{"id":14116,"date":"2025-09-03T20:03:48","date_gmt":"2025-09-04T03:03:48","guid":{"rendered":"https:\/\/news.dream.press\/news\/?post_type=announcement&p=14116"},"modified":"2025-09-03T20:03:50","modified_gmt":"2025-09-04T03:03:50","slug":"burning-breakers-cow-field-fixes-and-the-fbi-true-stories-from-dreamhosts-data-centers","status":"publish","type":"announcement","link":"https:\/\/news.dream.press\/news\/announcements\/burning-breakers-cow-field-fixes-and-the-fbi-true-stories-from-dreamhosts-data-centers\/","title":{"rendered":"Burning Breakers, Cow Field Fixes, and the FBI: True Stories From DreamHost\u2019s Data Centers"},"content":{"rendered":"\n

Haven’t heard much about DreamHost’s Infrastructure team (aside from Parts 1<\/a> and 2<\/a> of this series)? That means they’re doing their job! <\/p>\n\n\n\n

In their world, silence is golden, and chaos is always just a few milliseconds away.<\/p>\n\n\n\n

While the rest of us are streaming, sleeping, or building our online presence, this team is walking cold aisles, checking dashboards, replacing RAID cards, and preparing for emergencies that (hopefully) never happen.<\/p>\n\n\n\n

This is a peek into the war rooms, weird moments, and 24\/7 vigilance that keeps DreamHost, and your site, online. <\/p>\n\n\n\n

\"\"
Luke Odom (Director IT Operations) and Chris Lewis (Portland Data Center Manager) monitor aisles at PDX Data Center. <\/figcaption><\/figure>\n\n\n\n

The Day Portland Went Dark<\/strong><\/h2>\n\n\n\n

If there’s one story that sums up the team’s resilience, it’s the Portland outage of November 2023.<\/p>\n\n\n\n

\u201cPGE (Portland General Electric) was performing maintenance on one of the two power feeds to the data center,” explained Chris Lewis, DreamHost’s Manager of Data Center Operations, regarding an issue with the building’s power supply. <\/p>\n\n\n\n

Then the other<\/em> power feed failed, causing a complete power loss. <\/p>\n\n\n\n

Batteries kicked in. Generators roared to life and everything seemed OK. But when PGE turned the maintenance feed back on, it caused a switching malfunction, sending a dangerous power charge that sparked a fire and blew out a bank of breakers. With minutes of backup battery left, the team faced a worst-case scenario: complete data center failure.<\/p>\n\n\n\n

\u201cAlmost 75-80% of our racks went down, including core networking,\u201d Chris said. \u201cIt was madness.\u201d <\/p>\n\n\n\n

And yet, within 14 hours<\/strong> and a total team collaboration, everything was back up. Here’s how the team recovered service significantly faster than the estimated 2 days to a week (worst case):<\/p>\n\n\n\n

    \n
  1. Anticipation of Catastrophe<\/strong><\/li>\n<\/ol>\n\n\n\n
      \n
    • The team expected a catastrophic failure (not this one specifically) to take out the power and proactively migrated to a system where most machines could start without network dependency, saving significant time.<\/li>\n<\/ul>\n\n\n\n
        \n
      1. Preventative Maintenance and Resilient Systems<\/strong><\/li>\n<\/ol>\n\n\n\n
          \n
        • Consistent preventative maintenance and resilient systems (raids and zpools) protected data and allowed for sane replacement times rather than rushing to restore broken arrays.<\/li>\n<\/ul>\n\n\n\n
            \n
          1. Proximity and Preparedness of On-Call Staff<\/strong><\/li>\n<\/ol>\n\n\n\n
              \n
            • On-call admins live within an hour of the data center and were on-site quickly (this instance, the on-call admin was on-site twenty minutes after they were alerted).\u00a0<\/li>\n\n\n\n
            • With the power loss locking doors from the outside and disabling physical security systems, having someone on-site to manually verify identities was critical\u2014since letting anyone onto the floor without proper checks would be a major security violation (or worse, a Mission: Impossible breach).<\/li>\n<\/ul>\n\n\n\n
                \n
              1. Skilled and Global TechOps Team with Clear Leadership<\/strong><\/li>\n<\/ol>\n\n\n\n
                  \n
                • A vast, skilled, and global TechOps team provided fresh shifts and remote oversight.<\/li>\n\n\n\n
                • Clear leadership funneled information and prioritized tasks, enabled by a deep understanding of the layers and trust in the team to keep everyone updated on the status of all the moving pieces.<\/li>\n<\/ul>\n\n\n\n

                  \ud83d\udca1Did You Know?<\/strong>
                  <\/strong>DreamHost\u2019s diesel generators can power entire data centers for up to 24 hours<\/strong> as long as fuel deliveries keep coming. During the outage, they were the only reason lights (and sites) stayed on.<\/p>\n\n\n\n

                  Redundancy: The Art of Being Paranoid (In a Good Way)<\/strong><\/h2>\n\n\n\n

                  Disaster recovery isn\u2019t made up on the spot, it\u2019s engineered well in advance. <\/p>\n\n\n\n

                  \u201cWe assume something will break,\u201d said Luke Odom, DreamHost\u2019s Director of IT Operations. \u201cSo we design for failure.\u201d<\/p>\n\n\n\n

                    \n
                  • Power?<\/strong> Every rack has redundant PDUs (power distribution units) on separate feeds.
                    <\/li>\n\n\n\n
                  • Network?<\/strong> Multiple ISPs (internet service providers) via different entry points.
                    <\/li>\n\n\n\n
                  • Storage?<\/strong> RAID arrays, replication layers, and backups ready to go.<\/li>\n<\/ul>\n\n\n\n

                    Chris puts it plainly, applying Murphy’s Law, “Anything that can go wrong, will go wrong,\u201d it’s not a matter of if<\/em> failure occurs, but when<\/em>. The critical factor lies in how fast they can recover, or better yet, prevent such unexpected events from occurring. <\/p>\n\n\n\n

                    \"\"<\/figure>\n\n\n\n

                    Monitoring: The Silent Shield<\/strong><\/h2>\n\n\n\n

                    Keeping things online isn\u2019t glamorous, but it\u2019s constant. <\/p>\n\n\n\n

                    \u201cWe walk the floor, check lights, review dashboards every day,\u201d Chris said. \u201cSometimes the hardware doesn\u2019t report its own failure, so we go looking.\u201d<\/p>\n\n\n\n

                    That diligence means small issues get caught before they become full-blown outages. <\/p>\n\n\n\n

                    Current projects include upgrading firmware, code patching vulnerabilities, and rolling out better reporting so issues can be spotted (and solved) even faster.<\/p>\n\n\n\n

                    \u201cWe monitor everything we can,\u201d Chris said. \u201cAnd when something slips through, we figure out why and fix the system.\u201d <\/p>\n\n\n\n

                    When Automation Breaks (Because It Does)<\/strong><\/h2>\n\n\n\n

                    Automation is useful, until it isn\u2019t. <\/p>\n\n\n\n

                    \u201cWe break automation on the regular,\u201d Luke said. \u201cHalf of what we do isn\u2019t really automation, it\u2019s just tools that make our hands-on work faster and easier.\u201d<\/p>\n\n\n\n

                    A major security vulnerability, for example, forced the team to reboot nearly everything and upgrade the BIOS across their infrastructure. It was tedious and unautomated, until they built a tool to streamline repeatable steps. They still use it today. <\/p>\n\n\n\n

                    Server provisioning that once required hours now takes 20 minutes using OpenStack and Ansible. But if firmware, OS versions, or drivers change, scripts fail, and it\u2019s back to hands-on work. <\/p>\n\n\n\n

                    \"\"<\/figure>\n\n\n\n

                    War Story #2: Cow Field Edition<\/strong><\/h2>\n\n\n\n

                    Sometimes, infrastructure support happens in\u2026 unconventional places.<\/p>\n\n\n\n

                    While vacationing in southern Georgia, Luke got a call: RAID failure. Hardware down and servers offline. As a last resort, the team phoned Luke for help. <\/p>\n\n\n\n

                    \u201cI sat on a four-wheeler while herding cows,\u201d he said. \u201cI walked the tech through reassembling the array with one working drive. We duplicated it, rebuilt the RAID, and got the customers back online,\u201d he said. <\/p>\n\n\n\n

                    Yes, from the middle of a cow field.<\/p>\n\n\n\n

                    War Story #3: Gamer Jimmy, Fraud Slayer<\/strong><\/h2>\n\n\n\n

                    Nearly a decade ago, DreamHost was hit with a wave of fraudulent sign-ups. Hackers would compromise accounts, order dedicated servers or VPS offerings, and use them to launch attacks or mine crypto.<\/p>\n\n\n\n

                    Enter: Gamer Jimmy.<\/em>
                    <\/em>Not a team member, but a notorious hacker whose activity inspired an internal script.<\/p>\n\n\n\n

                    \u201cOne of the guys wrote a script named after \u201cGamer Jimmy\u201d Chris said. \u201cIt scanned for fraud indicators and auto-rejected suspicious requests.\u201d<\/p>\n\n\n\n

                    It worked.<\/p>\n\n\n\n

                    \ud83d\udca1Did You Know?<\/strong>
                    <\/strong>The FBI once installed a secret surveillance tap on a DreamHost customer: under the data center floor.<\/strong> It was later discovered during a center shutdown. The asset tag location was literally: underfloor.<\/em><\/p>\n\n\n\n

                    DreamHost Infrastructure: The Unsung Heroes <\/strong><\/h2>\n\n\n\n

                    At the end of the day, DreamHost\u2019s infrastructure team isn\u2019t just about blinking lights and airflow. They\u2019re the ones who drop everything\u2014meetings, sleep, even vacations\u2014to keep your site online.<\/p>\n\n\n\n

                    \u201cYou don\u2019t see us until something breaks,\u201d Chris said. \u201cBut that\u2019s kind of the point. If we\u2019re invisible, we\u2019re doing it right.\u201d<\/p>\n\n\n\n

                    And if you do<\/em> see them?<\/p>\n\n\n\n

                    There\u2019s probably a burning switchboard or a four-wheeler involved. And somehow, your server still made it. <\/p>\n","protected":false},"excerpt":{"rendered":"

                    Haven’t heard much about DreamHost’s Infrastructure team (aside from Parts 1 and 2 of this series)? That means they’re doing their job!  In their world, silence is golden, and chaos is always just a few milliseconds away. While the rest of us are streaming, sleeping, or building our online presence, this team is walking cold […]<\/p>\n","protected":false},"author":35,"featured_media":14118,"menu_order":0,"template":"","meta":{"_acf_changed":false,"_yoast_wpseo_metadesc":"","footnotes":""},"class_list":["post-14116","announcement","type-announcement","status-publish","has-post-thumbnail","hentry"],"acf":[],"yoast_head":"\nBurning Breakers, Cow Field Fixes, and the FBI: True Stories From DreamHost\u2019s Data Centers - DreamHost<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/news.dream.press\/news\/announcements\/burning-breakers-cow-field-fixes-and-the-fbi-true-stories-from-dreamhosts-data-centers\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Burning Breakers, Cow Field Fixes, and the FBI: True Stories From DreamHost\u2019s Data Centers - DreamHost\" \/>\n<meta property=\"og:description\" content=\"Haven’t heard much about DreamHost’s Infrastructure team (aside from Parts 1 and 2 of this series)? That means they’re doing their job!  In their world, silence is golden, and chaos is always just a few milliseconds away. While the rest of us are streaming, sleeping, or building our online presence, this team is walking cold […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.dreamhost.com\/news\/announcements\/burning-breakers-cow-field-fixes-and-the-fbi-true-stories-from-dreamhosts-data-centers\/\" \/>\n<meta property=\"og:site_name\" content=\"DreamHost\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/DreamHost\/\" \/>\n<meta property=\"article:modified_time\" content=\"2025-09-04T03:03:50+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.dreamhost.com\/news\/wp-content\/uploads\/2025\/09\/The-Chaos-and-Calm-of-a-Data-Center-1024x683.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1024\" \/>\n\t<meta property=\"og:image:height\" content=\"683\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:site\" content=\"@dreamhost\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"6 minutes\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Burning Breakers, Cow Field Fixes, and the FBI: True Stories From DreamHost\u2019s Data Centers - DreamHost","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/news.dream.press\/news\/announcements\/burning-breakers-cow-field-fixes-and-the-fbi-true-stories-from-dreamhosts-data-centers\/","og_locale":"en_US","og_type":"article","og_title":"Burning Breakers, Cow Field Fixes, and the FBI: True Stories From DreamHost\u2019s Data Centers - DreamHost","og_description":"Haven’t heard much about DreamHost’s Infrastructure team (aside from Parts 1 and 2 of this series)? That means they’re doing their job!  In their world, silence is golden, and chaos is always just a few milliseconds away. While the rest of us are streaming, sleeping, or building our online presence, this team is walking cold […]","og_url":"https:\/\/www.dreamhost.com\/news\/announcements\/burning-breakers-cow-field-fixes-and-the-fbi-true-stories-from-dreamhosts-data-centers\/","og_site_name":"DreamHost","article_publisher":"https:\/\/www.facebook.com\/DreamHost\/","article_modified_time":"2025-09-04T03:03:50+00:00","og_image":[{"width":1024,"height":683,"url":"https:\/\/www.dreamhost.com\/news\/wp-content\/uploads\/2025\/09\/The-Chaos-and-Calm-of-a-Data-Center-1024x683.png","type":"image\/png"}],"twitter_card":"summary_large_image","twitter_site":"@dreamhost","twitter_misc":{"Est. reading time":"6 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/news.dream.press\/news\/announcements\/burning-breakers-cow-field-fixes-and-the-fbi-true-stories-from-dreamhosts-data-centers\/#article","isPartOf":{"@id":"https:\/\/news.dream.press\/news\/announcements\/burning-breakers-cow-field-fixes-and-the-fbi-true-stories-from-dreamhosts-data-centers\/"},"author":{"name":"Annette Irwin","@id":"https:\/\/news.dream.press\/news\/#\/schema\/person\/6c421e9cde996bf3ed388cc1cf397ba4"},"headline":"Burning Breakers, Cow Field Fixes, and the FBI: True Stories From DreamHost\u2019s Data Centers","datePublished":"2025-09-04T03:03:48+00:00","dateModified":"2025-09-04T03:03:50+00:00","mainEntityOfPage":{"@id":"https:\/\/news.dream.press\/news\/announcements\/burning-breakers-cow-field-fixes-and-the-fbi-true-stories-from-dreamhosts-data-centers\/"},"wordCount":1154,"publisher":{"@id":"https:\/\/news.dream.press\/news\/#organization"},"image":{"@id":"https:\/\/news.dream.press\/news\/announcements\/burning-breakers-cow-field-fixes-and-the-fbi-true-stories-from-dreamhosts-data-centers\/#primaryimage"},"thumbnailUrl":"https:\/\/news.dream.press\/news\/wp-content\/uploads\/2025\/09\/The-Chaos-and-Calm-of-a-Data-Center.png","inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/news.dream.press\/news\/announcements\/burning-breakers-cow-field-fixes-and-the-fbi-true-stories-from-dreamhosts-data-centers\/","url":"https:\/\/news.dream.press\/news\/announcements\/burning-breakers-cow-field-fixes-and-the-fbi-true-stories-from-dreamhosts-data-centers\/","name":"Burning Breakers, Cow Field Fixes, and the FBI: True Stories From DreamHost\u2019s Data Centers - DreamHost","isPartOf":{"@id":"https:\/\/news.dream.press\/news\/#website"},"primaryImageOfPage":{"@id":"https:\/\/news.dream.press\/news\/announcements\/burning-breakers-cow-field-fixes-and-the-fbi-true-stories-from-dreamhosts-data-centers\/#primaryimage"},"image":{"@id":"https:\/\/news.dream.press\/news\/announcements\/burning-breakers-cow-field-fixes-and-the-fbi-true-stories-from-dreamhosts-data-centers\/#primaryimage"},"thumbnailUrl":"https:\/\/news.dream.press\/news\/wp-content\/uploads\/2025\/09\/The-Chaos-and-Calm-of-a-Data-Center.png","datePublished":"2025-09-04T03:03:48+00:00","dateModified":"2025-09-04T03:03:50+00:00","breadcrumb":{"@id":"https:\/\/news.dream.press\/news\/announcements\/burning-breakers-cow-field-fixes-and-the-fbi-true-stories-from-dreamhosts-data-centers\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/news.dream.press\/news\/announcements\/burning-breakers-cow-field-fixes-and-the-fbi-true-stories-from-dreamhosts-data-centers\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/news.dream.press\/news\/announcements\/burning-breakers-cow-field-fixes-and-the-fbi-true-stories-from-dreamhosts-data-centers\/#primaryimage","url":"https:\/\/news.dream.press\/news\/wp-content\/uploads\/2025\/09\/The-Chaos-and-Calm-of-a-Data-Center.png","contentUrl":"https:\/\/news.dream.press\/news\/wp-content\/uploads\/2025\/09\/The-Chaos-and-Calm-of-a-Data-Center.png","width":1536,"height":1024},{"@type":"BreadcrumbList","@id":"https:\/\/news.dream.press\/news\/announcements\/burning-breakers-cow-field-fixes-and-the-fbi-true-stories-from-dreamhosts-data-centers\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.dreamhost.com\/news\/"},{"@type":"ListItem","position":2,"name":"Announcements","item":"https:\/\/www.dreamhost.com\/news\/announcements\/"},{"@type":"ListItem","position":3,"name":"Burning Breakers, Cow Field Fixes, and the FBI: True Stories From DreamHost\u2019s Data Centers"}]},{"@type":"WebSite","@id":"https:\/\/news.dream.press\/news\/#website","url":"https:\/\/news.dream.press\/news\/","name":"DreamHost News","description":"Product announcements, events, and more.","publisher":{"@id":"https:\/\/news.dream.press\/news\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/news.dream.press\/news\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/news.dream.press\/news\/#organization","name":"DreamHost","url":"https:\/\/news.dream.press\/news\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/news.dream.press\/news\/#\/schema\/logo\/image\/","url":"https:\/\/www.dreamhost.com\/news\/wp-content\/uploads\/2023\/03\/dreamhost-events.png","contentUrl":"https:\/\/www.dreamhost.com\/news\/wp-content\/uploads\/2023\/03\/dreamhost-events.png","width":1598,"height":921,"caption":"DreamHost"},"image":{"@id":"https:\/\/news.dream.press\/news\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/DreamHost\/","https:\/\/x.com\/dreamhost"]},{"@type":"Person","@id":"https:\/\/news.dream.press\/news\/#\/schema\/person\/6c421e9cde996bf3ed388cc1cf397ba4","name":"Annette Irwin","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/bded6b0010ba90c5fa29aa5ece5b06f7f19eda9a6f4d37dd0dfd46afddf8391a?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/bded6b0010ba90c5fa29aa5ece5b06f7f19eda9a6f4d37dd0dfd46afddf8391a?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/bded6b0010ba90c5fa29aa5ece5b06f7f19eda9a6f4d37dd0dfd46afddf8391a?s=96&d=mm&r=g","caption":"Annette Irwin"},"description":"Annette Irwin is the Communications Lead at DreamHost, where she juggles storytelling, strategy, PR, and the occasional spreadsheet. A former sports producer and lifelong sports nerd, she now channels that competitive energy into crafting copy for DreamHost. She\u2019s passionate about writing with humor, clarity, and just the right amount of sarcasm\u2014because why be boring? When she\u2019s not in comms mode or policing grammar, she\u2019s mainlining coffee, devouring books, watching sports, or navigating the chaos of life with her favorite people\u2014and one extremely opinionated cat.","url":"https:\/\/news.dream.press\/news\/author\/annetteirwin\/"}]}},"lang":"en","translations":{"en":14116,"es":14123},"pll_sync_post":[],"_links":{"self":[{"href":"https:\/\/news.dream.press\/news\/wp-json\/wp\/v2\/announcements\/14116","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/news.dream.press\/news\/wp-json\/wp\/v2\/announcements"}],"about":[{"href":"https:\/\/news.dream.press\/news\/wp-json\/wp\/v2\/types\/announcement"}],"author":[{"embeddable":true,"href":"https:\/\/news.dream.press\/news\/wp-json\/wp\/v2\/users\/35"}],"version-history":[{"count":1,"href":"https:\/\/news.dream.press\/news\/wp-json\/wp\/v2\/announcements\/14116\/revisions"}],"predecessor-version":[{"id":14122,"href":"https:\/\/news.dream.press\/news\/wp-json\/wp\/v2\/announcements\/14116\/revisions\/14122"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/news.dream.press\/news\/wp-json\/wp\/v2\/media\/14118"}],"wp:attachment":[{"href":"https:\/\/news.dream.press\/news\/wp-json\/wp\/v2\/media?parent=14116"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}