Who Is Guarding the Grid?

A trio of highly trained hackers baited the employees of an electric utility north of Seattle last year with carefully crafted phishing emails. They had a bite in 22 minutes.

Neither alarm bells nor flashing lights drew attention to the mistake, even as a tiny package of malware slipped silently onto the computer of an unsuspecting employee. Had they been hackers with evil intent, they could have cut power to the utility's 325,000 customers.

Instead, the team was part of an unprecedented partnership between the cybersecurity unit of the Washington State National Guard and the Snohomish County Public Utility District.

The success of the attack should be no surprise, said Lt. Col. Thomas Muehleisen, one of the planners of the hack. These weekend warriors were veteran computer specialists from technology giants Microsoft, Google and Amazon.

"They are all world-class," said Muehleisen. "Would you be surprised that three Pro Bowl-ers can beat a high school football team?"

The national security implications of the exercise became clear months later when a similarly styled hack crippled three electric facilities in Ukraine, plunging some 225,000 homes and businesses into darkness for several hours. Ukraine has blamed Russia for the attack.

President Barack Obama has repeatedly called for enhanced protection of the nation's vast power grid — a network that has been pieced together since the Age of Edison into what engineers and scientists describe as "the largest machine on earth." But it is unclear in the waning months of the Obama administration whether the issue will remain a key priority. Neither of the major-party candidates has championed protecting the grid this campaign season.

In a two-month study, seven national security reporters with Northwestern University's Medill School of Journalism traveled to the Pacific Northwest to determine whether disruptive attacks are possible, likely or even inevitable and whether measures being taken are sufficient to secure the nation's power grid.

The Pacific Northwest is a logical focal point for studying efforts to protect the power grid not only from cyber but from physical and natural threats as well. The need for long power lines linking dams on the Columbia River to major population centers such as Portland and Seattle creates numerous choke points for terrorists to disrupt the power supply. And scientists predict the region could experience a 9.0-magnitude earthquake within the next 50 years, prompting local and federal authorities in Washington and Oregon to conduct drills last summer in preparation for such a disaster.

The investigation found government officials, industry experts, engineers and scientists are well aware of the challenges they face and are working to combat threats to the grid. But dire predictions of absolute collapse in the event of a concerted attack are overstated.

Scott Aaronson, executive director of security and business continuity for Edison Electric Institute, said the system's history points to its greatest strength.

"This is a grid that grew up over quite literally 100 years," Aaronson said. "There is any number of redundancies throughout the system, so taking out one or two or 10 nodes is not going to have the impact that you'd think it's going to have where the lights go out for 18 months."

Still, vulnerabilities persist and are not limited to one region. Covering the entire country and parts of Canada, the grid is a network of more than 7,000 power plants, hundreds of thousands of miles of high-voltage transmission lines and upwards of 55,000 substations.

"Everything is dependent on electricity. Everything," said David Holcomb, an infrastructure security expert for the Department of Homeland Security. "Without electricity, we're basically back in the 1850s."

Battling back and back and back

With the Washington National Guard hack team inside Snohomish's systems, the utility's cybersecurity team began working to reclaim control.

But once the guardsmen had their foothold, it was not easy to push the intruders out completely.

The two sides used a nearly identical test lab network to preserve service-delivery during the two-week test.

After eight hours of work, the guardsmen had full domain control, including of the utility's Supervisory Control and Data Acquisition, or SCADA, system. The commonly used network manager enables employees to monitor the grid, and it acts as a hub for information streaming among devices across the utility. With that access the team could have added computers, routers, new users and other cyber assets to the system remotely, Muehleisen said.

"And that was a degree of control they didn't think we would have ever," Muehleisen added.

The ease with which the hackers gained access doesn't mean the utility was unprepared. Quite the contrary, Muehleisen compared the networks to a well-guarded castle.

During the penetration test, while the hackers tried to move about the utility's systems undetected, leaders on both sides — Muehleisen and Benjamin Beberness, the utility's chief information officer — chatted daily, recapping what had occurred and planning for the next day.

"I could ask them to ratchet up their noise," Beberness said. "Step on that leaf, right? Be a little noisier, see if we detect you, and that helped us figure out where our tolerance was from a noise perspective."

Although the U.S. grid has never suffered a debilitating cyber attack, there is still cause for concern. The U.S. Department of Homeland Security has warned that components of the U.S. grid have been infected with BlackEnergy malware — the same type of code found on Ukrainian systems after last December's hack.

In the aftermath, DHS officials have said it is "imperative" that utilities shore up the their cyber defenses. Legislators also are calling for action.

"There's been more attention to this paid in Congress in recent years, but I still think we've got substantial vulnerability here," said U.S. Rep. Derek Kilmer, D-Wash., who has backed a variety of cybersecurity measures since he took office in 2013. "I think the power grid is a legit concern."

After revelations in July that the Democratic National Committee's network had been hacked, the White House called such cyber incidents a "fact of contemporary life." Then in August, the Department of Energy requested up to $34 million in appropriations for 12 projects in nine states, including Washington, to improve grid resiliency through cybersecurity research.

Rising hackability

As the grid becomes more dependent upon technology, it becomes increasingly vulnerable as well.

More and more, utilities across the country are investing in "smart grid" upgrades, often with federal grant money — several billion dollars since 2010, according to a recent Department of Energy report. The benefits include streamlined communication and remote control. But with increased technological integration come added risks.

"If you push the grid to meet the demands of the 21st and 22nd centuries, these are the kinds of things you have to do," said Clay Perry, spokesman for the Electric Power Research Institute, or EPRI, an industry nonprofit. "It's kind of the damned-if-you-do, damned-if-you-don't approach."

Besides doling out funds, federal authorities enforce mandatory critical infrastructure protection standards to keep personnel training, system management and information security, among other things, consistent across the grid. There are hefty fines for non-compliance, and new guidelines that took effect in July increased the number of facilities under federal oversight. Yet there are still some local utilities free from these top-level standards.

But power grid specialists, including EPRI, say federal standards should take into account the grid's diversity.

"It's kind of the damned-if-you-do, damned-if-you-don't approach." Clay Perry, EPRI spokesman "Are you going to mandate the same set of controls for every utility?" said Annabelle Lee, EPRI senior technical executive and cybersecurity expert. "Every utility is different."

Many utilities, furthermore, use programs and systems created decades ago, early in the development of computer-controlled systems, Lee added.

"Thirty, 40, 50 years ago, nobody worried about cybersecurity," Lee said.

As the Snohomish County hacking exercise shows, every employee connected to a company network is a potential access point. As companies continue integrating technology into day-to-day operations, employee awareness only increases in importance.

Since something as mundane as an email could open the door to a system shutdown, utilities are using employee-training programs to limit their cyber vulnerability. Beberness and his staff, for instance, send out fake phishing emails to Snohomish utility employees.

"When they take the bait, it immediately brings up a little 'Hey, here's why you shouldn't have clicked on this email,'" Beberness said. "So it's a fairly nice set of tools we can use."

Deterrence plans

Rep. Kilmer had been in office less than two weeks in 2013 when he met then-Secretary of Defense Leon Panetta at the Pentagon. Kilmer recalls asking what he thought would be a simple question: "So what keeps you up at night?"

Without hesitation, Panetta replied that the U.S. appears to be at risk and unprepared for a "Cyber 9/11," that the next big cyber attack could rival the devastation inflicted by terrorists on Sept. 11, 2011. Kilmer said he has focused on cybersecurity ever since.

During the Cold War, nations with nuclear weapons were said to be in a state of "Mutually Assured Destruction." Riffing on that term, some have described today's cyberspace standoff as a form of "Mutually Assured Disruption." Foreign governments may be disinclined to meddle in American computer networks for fear of firm-fisted reprisal.

"The greatest concentration of offensive cyber power on this planet is at the intersection of Baltimore-Washington Parkway and Maryland Route 32, no question about it," Gen. Michael Hayden, a former director of the National Security Agency, said in July, pointing to Fort Meade, where the NSA is headquartered.

But these techniques are unlikely to deter cyber-capable criminal and terrorist organizations as effectively.

Such groups control limited territory and infrastructure, muting the effectiveness of a forceful U.S. response.

Although an attack on Snohomish County's utility would likely affect local customers exclusively, an attack on a larger power production and distribution facility upstream — such as Bonneville Power Administration, from which Snohomish purchases the majority of its electricity — could have a prolonged regional impact, Beberness noted.

The prospect of such a widespread outage is increasingly unsettling. Financial transactions, communication, certain forms of transportation and even medical treatment could be interrupted.

In the event of a Ukraine-style takedown, Snohomish County utility workers could follow the example of their Ukrainian counterparts and restore power by physically reporting to substations across the county to operate breakers manually. But a manual override would be labor-intensive and difficult to sustain, Beberness said.

Some are calling the Snohomish penetration test a model that should be replicated nationwide, with local utilities soliciting aid from their local guardsmen. Late last year, the National Guard announced plans to establish 13 new cyber units by the end of fiscal year 2019. But, with thousands of utilities across the country, others question whether that's the best approach.

"Is this really creating a precedent that's going to bite us later?" said Michael Hamilton, a former chief information security officer for Seattle and current CEO of cybersecurity firm Critical Informatics.

"These organizations, go 'Well, we don't have to invest in controls because the government is going to come take care of it for us.' "

The question isn't unique to Snohomish County, and it isn't limited to the local level.

Jamil Jaffer, director of the Homeland and National Security Law Program at the George Mason University Antonin Scalia Law School, told

the House Small Business Committee in July that Americans need to have a national debate about who's in charge of providing for the common defense in cyberspace — not only for the power grid but for commerce and other critical sectors as well.

The legal questions at play are particularly troublesome when they pertain to private businesses, Jaffer said.

"We don't expect Target, for example, to have surface-to-air missiles on the top of their warehouses," Jaffer said. "To be sure, we expect them to have high fences, armed guards and perhaps guard dogs. But we don't expect them to defend against a Russian bomber coming and bombing their warehouses."

A Wake Up Call

The Grand Coulee Dam dominates the landscape along the Columbia River in Eastern Washington, a colossal hydroelectric plant made of 12 million cubic yards of concrete, enough to pave a highway stretching from Seattle's Puget Sound cross country to Miami Beach.

The dam is a product of President Franklin Delano Roosevelt's Works Progress Administration, a project that sent Americans back to work in the 1930s during the Great Depression. For some time after its completion in 1940, cars and cattle meandered along the mile-long roadway across the top of the dam. Not anymore.

An electric brotherhood: On the front lines of power grid protection Post 9/11, the Grand Coulee Dam and other crucial pieces of energy infrastructure throughout the United States are recognized as potential targets of terrorism. A phalanx of armed guards protect the dam's entryways, fences and gates are controlled by key cards, and barbed wire keeps intruders out of labyrinthine passageways where children once rode their bicycles and families strolled on weekend outings. Visitors can enter the entrails of the dam on guided tours, but hard-hats are required and closed circuit cameras follow their every move.

Coulee appears impervious to attack, but consider that security effort multiplied 3,000 times at each of the public and privately owned power utility plants across the country. Securing the nation's power grid also includes assessing the risk for hundreds of thousands of miles of transmission lines, thousands more substations and critical high-voltage transformers.

The potential for damage to the U.S. electrical grid became very real several years ago after a confounding attack in San Jose, California which has still not been solved.

On April 16, 2013 the PG&E Metcalf transmission substation in San Jose, California, was attacked. In 19 minutes, gunfire destroyed one large power transformer and damaged 16 more, jeopardizing the flow of power to Silicon Valley. Communication lines were cut, slowing the utility's ability to react.

The FBI is still investigating the incident, and no one has been charged. There is speculation among industry insiders that Metcalf was a practice run, meant to test the resilience of the electrical grid in the event of an attack.

The Metcalf attack pressured the federal government to act, sending Department of Homeland Security officials around the country to explain the attack, and the threat, to utilities. In addition to the DHS roadshow, utility regulators created the very first set of standards forcing power companies to identify their most critical substations and address any security shortcomings. For many, that meant putting in some combination of cameras, motion sensing equipment and fortified fencing.

"Metcalf was definitely a wake-up call for the industry itself," said Neil Arthurs, Lead Physical Security Specialist at Bonneville Power Administration (BPA), which owns three quarters of the transmission lines in the Pacific Northwest and is responsible for distributing the power produced by the Grand Coulee Dam.

"It's not like we're just going to go down to Bob's Transformer Moving for Less or something." Jim Phillips, Ross Dam Powerhouse operator Though the Metcalf incident has been widely discussed in energy, utility and national security circles, it was largely overshadowed by the Boston Marathon bombings, which happened just one day earlier.

Duane Highley, president & CEO of Arkansas Electric Cooperative Corporation, recognized early on how Metcalf would force utilities to think differently about energy security. "It wasn't until the last decade that utilities had become the front lines of a war," Highley said.

Power generating behemoths like the Grand Coulee Dam are critical, but well-defended. Still, there are less visible pieces of the grid that also need protection. Security and industry officials have identified high-voltage transformers as uniquely vulnerable, expensive and immensely difficult to replace.

In late July, one such transformer sat on a cargo barge, making its way across Diablo Lake, a stunningly blue reservoir in the North Cascade mountains of Washington State near the Canadian border. It would soon become scrap metal — and a lot of it.

The 60-ton high-voltage transformer was one of the original six from the Ross Dam powerhouse. New transformers had been installed the month before for the first time in the dam's 67-year history.

"These are a real weak point, at least if power was what they're after," said Ross Dam powerhouse operator Jim Phillips, pointing to the towering transformer as he watched it float away.

"It's a real vulnerable spot. If you put an AP [armor-piercing] round through them and they fail, they won't fail gracefully," he added.

The Metcalf attack is a prime example of the damage a bullet can do to a transformer.

While transformers vary widely in size and the amount of voltage they can handle, they all perform the same basic job: To transform electricity from one voltage to another, stepping it up to travel long distances across power lines or stepping it down for distribution at a local level. Large, high-voltage transformers play a critical role in the transmission of electricity, often referred to as the "backbone" or the "heart" of the system by people in the field.

The average transformer at a suburban substation is relatively small, but those used to get electricity on or off transmission lines can be as large as a small house. Less than 3 percent of all transformers in the U.S. are high-voltage, but 60 to 70 percent of the nation's electricity passes through them, according to a recent report by the Congressional Research Service.

The critical role that high-voltage transformers play in moving electricity across the grid makes them a unique — and volatile — target. Cooled by oil and powered by hundreds of thousands of volts, they often explode if damaged or as they near the end of their lifespan.

But the lifespan of a high-voltage transformer is decades, sometimes upwards of 40 years. While this makes their multi-million dollar price tag more cost-effective, it also made the large transformer manufacturing industry in the United States unstable. Sales skyrocketed during the '60s and '70s when the modern U.S. power grid was solidifying, but dropped dramatically as the nation's transformers hummed along healthily.

Without steady business, manufacturing plants closed. By the early 2000s, if the need for a new high-voltage transformer did arise, it often had to be shipped from far away places like South Korea, Israel, or Germany.

High-voltage transformers fall into the same category as much of the nation's decaying critical infrastructure: largely taken for granted until a dramatic failure occurs.

Moving a high-voltage transformer is a logistical nightmare, and shipping them from overseas only complicates the matter.

At the Ross Dam, moving out one of the old transformers took a crew of 15 men almost 12 hours. It involved a barge, two flatbed trucks, several forklifts, and a special machine called a Self-Propelled Mobile Transport. The level of the lake had to be lowered by a foot to load it, and then raised back up to unload it on the other side.

Preparing for that day took almost two years. It required state-issued driving permits, thousands of miles of rail lines, and two different international companies specialized in moving large equipment.

"It's not like we're just going to go down to Bob's Transformer Moving for Less or something," Phillips said.

Moves like this — with permits, railroad cars, crews and machinery — caused government and industry officials alike to start to wonder. What if there weren't months to spare? What if an area's power was depending on getting a high-voltage transformer quickly?

In 2008, long before Metcalf, the private and public sector joined forces to try to answer those questions. The Department of Homeland Security and the Electric Power Research Institute began RecX, a project developed with the challenge of transporting and installing a high-voltage transformer in less than a week, using only readily available equipment.

They paired up with ABB, one of the largest transformer manufacturers in the world — at the time, the only one capable of producing a high-voltage transformer in the U.S. — and CenterPoint Energy in Texas. The drill would take the transformer from ABB's plant in St. Louis south to Houston, carried solely on semi-trucks.

It worked. In 2014, the RecX transformer was successfully installed in five days, 10 hours, and 10 minutes.

"There's a good chunk that can be learned from planning for this type of disaster and understanding what you can do ahead of time. Just having that in your back pocket so that when things come about, it's like — okay, I know how to deal with this," said Sarah Mahmood, who led the DHS Science and Technology team through the six-year RecX planning process.

RecX is only one of several programs and projects developed in the last few years that focus on high-voltage transformer recovery in the event of a sudden failure. Three separate industry groups have devised spare transformer sharing programs.

Craig Stiegemeier, director of technology and transformer services for North America at ABB, said that having a spare transformer is a lot like having a spare tire in the trunk of your car.

"I never use my spare," he said, "but when I need it I'd really like to have it."

The Edison Electric Institute (EEI) has a transformer sharing program which would be used in the case of a presidentially-declared terrorist attack. Scott Aaronson at EEI stressed that the programs are non-competing, and that any program that adds to resiliency and recovery is helpful.

"I view these as lots of different tools in a toolbox," he said.

Moving large transformers isn't the only obstacle. Often custom designed and built, they are largely assembled by hand with components that can be difficult to source. All of this factors into lead times that can stretch over a year.

"It's not widgets being produced at a high volume," said Greg Bayman, a sales engineer at Mistubishi's new high-voltage transformer plant in Tennessee. "There's very little that can be easily automated."

In recent years, the number of transformer manufacturing plants in the country able to build large, high-voltage transformers has risen, fueled by an increase in demand as utilities need replacements and energy consumption grows. The federal government has also been discussing high-voltage transformer availability as a national security issue, encouraging manufacturing growth.

In the last year, power companies have been required to increase security at critical substations to contend with intentional attacks like the one at Metcalf, and other 'nuisance crimes' like copper theft.

Utilities have begun to install costly improvements like new cameras, reinforced fencing and motion sensing technology, in addition to devoting resources to countering cyber threats and, in some areas of the country, seismic protection.

David Holcomb, DHS protective services advisor for the State of Washington, finds it hard to justify putting ballistic walls around transformers when there are other threats to contend with. "If a big enough earthquake occurred it's going to shake them hard enough where it's going to kill them anyway." said Holcomb. "It would be nice if we had hundreds of millions of dollars to spend on every substation, but we don't."

Even when expensive security improvements are installed, it's not a guarantee against further intrusions.

Electric utility Arizona Public Service recently spent $20 million in security upgrades at their Trilby Wash substation, just west of Phoenix. Soon after, 27-year-old John Cooper squeezed into 7-inch gap and shimmied his way into the substation.

Cooper shut off several security cameras and crawled through another small opening to enter the substation control house. The utility shut off the high-voltage lines running to the substation until local law enforcement were able to arrest Cooper. He pled guilty to a reduced charge of criminal damage.

The vast majority of blackouts are triggered by relatively innocuous factors. Weather, trespassers and even squirrels who crawl into substations and get electrocuted have, so far, caused far more headaches for electrical utilities than malicious actors.

For those protecting massive pieces of infrastructure like the Coulee Dam, the fact that there hasn't been a major attack on the grid makes it difficult to justify a high level of security.

"It's hard to sell security," said Patrick Delfing, security captain for the Grand Coulee Dam. "Trying to make a rational business case for our existence sometimes takes up two-thirds of my day."

Considering the size and scope of the U.S. power grid, it would be impossible for government, industry regulators and utilities to perfectly protect the entire system.

That doesn't mean a major blackout is inevitable. The diversity and redundancy built into the grid makes it resilient. The Metcalf attack terrified the electricity industry, true, but even then power was rerouted. The lights in Silicon Valley didn't even blink.

Neighborhood substations like those operated by Snohomish County's public utility are important to rate-payers, but not necessarily to the well-being of the entire Western Interconnection. "If one substation gets shot out or disabled, electricity can come in through another way," said Doug Williams, security manager for the utility. "There isn't a single substation here, if we went offline, that would even be noticed."

A malicious physical attack on the grid is a legitimate concern, and one that is being studied from utility operations up to the Oval Office, but gunshots to a small neighborhood distribution substation aren't going to cause a regional blackout. Industry insiders say it would take a coordinated, multi-site attack by someone with intricate knowledge of the grid to do considerable damage.

Utilities cannot predict the likelihood of a terrorist attack or an act of war by a hostile nation-state, but they can prepare. Upping security around critical substations — especially those that feed power to major metropolitan areas like Seattle — and investing in transformer sharing programs are cost-effective ways to temper a constantly evolving threat.

Below the surface

In certain situations, efforts to protect the electrical grid intersect with emergency planning for possible catastrophic natural disasters such as earthquakes and resulting tsunamis, floods and extreme weather as a result of climate change and even power surges provoked by solar flares.

When Sue and Steven Andrews decided to retire in Washington state, they spent years seeking a quiet and beautiful community to suit their lifestyle. When they chose a home on Whidbey Island, a half-hour ferry ride from Everett, they didn't realize the hidden danger: a predicted magnitude-9 earthquake, followed by a tsunami.

The Andrews were no strangers to seismic threats. Steven Andrews said he remembers three or four major earthquakes from his time living in California, and knew living on the West Coast came with that risk. It wasn't until the couple became friendly with the local fire chief in their new neighborhood that they realized how unaware they were.

"We started hearing about this Subduction Zone," Sue Andrews said. "I never heard that term before. It is something here that other places don't have."

The Cascadia subduction zone stretches about 700 miles along the west coast, from Vancouver Island in British Columbia to northern California's Cape Mendocino. It is projected to release more than 300 years' worth of stored energy in an earthquake of unprecedented magnitude in recorded North American history.

According to FEMA's estimate, the total economic impact of such a quake could cost up to $70 billion. With nearly $50 billion in economic losses, Washington would be hardest hit among West Coast states.

"In America, we've got this horrible pattern where instead of someone being responsible for their own well-being, it's litigated." Chris Heimgartner, Snohomish County PUD The National Infrastructure Simulation and Analysis Center projects that the earthquake and ensuing tsunami would result in more than 3,000 deaths.

The Puget Sound region is also home to a network of smaller fault lines closer to the surface, including the Seattle Fault and the South Whidbey Fault, that can cause earthquakes up to magnitude 7.4, strong enough to devastate old buildings and threaten critical infrastructure such as substations and power lines.

The area's unique geology contributes to seismic threats.

When the glaciers began receding 10,000 years ago, soft sediments settled in a basin in the Puget Sound area. The main basin, between the southern tip of Whidbey Island and Tacoma, Washington, is about 600 feet deep. The area, which has been called "a water bowl," causes any earthquake to reverberate and amplify the motion by up to 20 times, said John Vidale, a seismologist and professor at the University of Washington.

"It is hard to see which place is most dangerous, we don't know. It is clear that the center of the basin got the most amplification, what's pretty close to downtown Seattle," said Vidale.

Other geologists in Washington state agree that the soft sediments within the basin cause the shaking to act in a complex way unique to the area.

"You can think about it like if you have a bowl of concrete or a bowl of Jello," said Tim Walsh, chief hazards geologist for the Washington Geological Survey at the Department of Natural Resources. "Hit it on the side, [the energy] will ring right through the concrete. Whereas it will slow down going through the Jello and oscillate."

An earthquake 1,000 years ago on South Whidbey Island raised one tip of the island 21 feet, sending a massive tsunami reverberating between the coast of Snohomish County and south to Seattle, Walsh said. A similar earthquake is possible again.

While geologists and emergency management departments agree a devastating seismic event is not probable in the near future, making plans for a worst-case-scenario Cascadia event would provide protections for less-severe seismic events as well.

Sue Andrews said that while she was not frightened at the prospect of a major earthquake, she decided the "healthy thing to do" was everything within their capacity to prepare their home and neighborhood.

"We live on an island, in a small community," Sue said. "It was made clear to us that the infrastructure that was on the island in terms of response would be very limited. It is necessary that local people be involved in preparing for such emergency."

The Andrews have food, water, cash and perhaps most importantly, an electrical generator that will kick in if their electricity goes out for any reason.

John Ufford, a preparedness unit manager for Washington State Emergency Management Division, said electricity, communications and roadways are central to a 21st Century society.

"Which of those are impacted by a massive earthquake?" Ufford said. "All of the above."

Ensuring a rapid recovery of the electrical grid is the state's top priority, helping to speed recovery in other sectors, Ufford said.

But funding for local emergency management has shrunk by 25 percent since the economic downturn in 2008, placing a heavier burden on electric utility companies to protect their own infrastructure and a greater need for private individuals to secure their home's electricity. This is a reality nationwide, Ufford said.

Together the Bonneville Power Administration and Snohomish County Public Utility District are the leading providers of electricity to the nearly 750,000 people who live in Snohomish County, and while the two companies have invested in securing their infrastructure, it does not eliminate the possibility that thousands will be left in the dark after an earthquake.

More than 170 transmission towers are predicted to fail if the Cascadia earthquake occurs in winter months, according to a BPA earthquake risk assessment. Crustal faults, such as the South Whidbey and Seattle faults can trigger the loss of a few towers, mostly due to landslides.

Leon Kempner Jr., Principal Structural Engineer and Seismic Program Manager at BPA, said the grid was built to a "very low seismic level" in the 1960s when the Northwest was thought to be seismically inactive.

For more than 20 years, BPA has implemented a long-term seismic mitigation program to address the vulnerability of their existing high-voltage electrical transmission line system.

More than 90 percent of BPA's transformers, the most critical and seismically vulnerable component, are now anchored to the ground to prevent them from sliding or overturning during an earthquake. However, BPA said anchoring the transformers to the foundations also has the effect of amplifying the acceleration, which could result in more damage to transformer components.

As a result, BPA has invested in new technology that can provide additional isolation between the ground motion and the transformer.

Kempner said BPA spends about $5 million each year, a small percentage of the annual budget, to retrofit grid components that are not yet up to standards and research for technology to reduce an earthquake's impact.

"Obviously we could spend a lot more, but we have a fixed income when it comes to what we can charge our customers," Kempner said. "We are making the right upgrades at the right time so we're not passing on huge rate increases to our customers."

BPA also participated in a four-day FEMA exercise called Cascadia Rising, that tested its emergency response to a Cascadia level event.

"The purpose of the Cascadia Rising exercise was to test our coordinated plans, uncover issues and learn how to better manage through a highly stressful situation," said John Hairston, chief administrative officer of BPA in written testimony before the DOE.

Through the exercise, Bonneville recognized that it cannot rely on its own capabilities to repair the damage caused by a significant earthquake. The exercise resulted in an agreement between BPA and 44 area utilities to share resources when a major seismic event occurs.

Because earthquakes are so unpredictable in both timing and magnitude, structural engineers have been researching how best to secure electrical grid components.

The Pacific Earthquake Engineering Research Center or PEER, at the University of California in Berkeley is home to the world's largest Shaking Table — a concrete table that simulates motions caused by seismic waves on structures weighing up to 75 tons.

A team of engineers has been testing electrical grid components including porcelain insulators and supporting structures for electrical switches in substations. On the Shaking Table, the engineers run scenarios that are unlikely to occur, but challenge the engineering standards of any given component.

"We have a very small probability but at the same time when we are testing to this level of agitation, we are telling our clients and ourselves that essentially the equipment is very robust and can withstand an earthquake," said Shakhzod Takhirov, Manager of Structures Laboratory at University of California.

The PEER research team is sponsored by utilities like Bonneville and companies that manufacture grid components. Takhirov and his team are testing equipment to withstand a worst-case-scenario — the Cascadia event.

"When we're testing our equipment, we're trying to make sure we meet that threshold," Takhirov said. "Our performance level means the equipment will perform and withstand a very strong earthquake. When manufacturers are trying to qualify their equipment to performance standards, they all have to do comprehensive analysis to ensure their equipment will not fail."

However, the Shaking Table test does have limitations. When structures and grid components are tested, they shake in isolation, meaning they are not connected to other components as they would be in an actual substation.

"Once you connect everything it will become a different system," Takhirov said. "You have to see what it's going to be or predict what it's going to be. In this case, we must predict."

Takhirov said that flexible components are the best way to ensure that substations and power lines do not fail. Slack between cable lines, for instance, could curb damage by allowing more space for a given structure to move.

"Failing can happen quite easily if you don't allow slack," Takhirov said. "Since two pieces are moving in different directions, they can destroy each other."

While these components would greatly increase survivability during an earthquake, funding to utilities for such upgrades is not readily available.

"It is the responsibility of each utility to maintain and upgrade its system using funding from its ratepayers,"said Tony Usibelli, special assistant to the director energy and climate policy at the Washington State Department of Commerce.

Public utilities, like Snohomish, are subject to safety standards and other requirements imposed by the state, but there are no regulations specific to energy infrastructure, Usibelli said.

Chris Heimgartner, assistant general manager of Distribution & Engineering Services at Snohomish County Public Utility District, said the utility has been retrofitting their substations and transmission lines to the current engineering standards which are designed to withstand a Cascadia level event — the really big one.

While the Snohomish public utility would most likely be able to quickly identify and repair the damaged portions of the local power grid, it would be up to the Bonneville Power Administration, which provides more than 80 percent of the local utility, to ensure power was actually getting to Snohomish, said Heimgartner.

Depending on the extent of damage to the BPA transmission system, service might not be restored for weeks or even months, he said, adding that individuals should understand the threat and the limited resources available and take preparedness into their own hands.

"In America, we've got this horrible pattern where instead of someone being responsible for their own well-being, it's litigated," Heimgartner said. "At the end of the day, we want people to take care of themselves."

On Whidby Island, the Andrews contributed to a neighborhood fund that bought equipment for disaster relief, but some of their neighbors were offended when the couple began to push for even more preparedness.

After a neighborhood meeting where Sue and Steve made recommendations to their neighbors, they received "nasty" emails for weeks, accusing them of pushing a government agenda.

"I think people are not ready to face the fact that you need to get some of these things in hand, and have it ready," she continued. "We're just gonna do the best we can until something happens."

Earthquakes, like cyber and physical attacks, are unpredictable, requiring utilities to invest in preventive and protective measures.

Research on earthquake resistant electrical equipment, investment in spare transformers and the Washington National Guard's cyber penetration test in Snohomish County are examples of how the Pacific Northwest is attempting to address threats to the power grid.

Cybersecurity expert Michael Hamilton notes that this region, with its large military bases and concentration of lucrative tech companies, cannot wait for solutions from Washington, D.C. "We try to solve our own problems, we think out of the box," Hamilton said. "We are much more willing to collaborate amongst ourselves, rather than look to the federal government for the solution to our problems."

While federal agencies certainly play a role in improving grid security, differences in geography, weather and local regulations make it almost impossible for the government to legislate one security solution for every possible threat.

After spending eight years at DHS, Sarah Mahmood understands the importance of guarding the grid better than most. "It's just so vital," Mahmood said. "To really understand the scope of how critical this is — to all of us — from national security all the way down to getting your groceries, it covers everything."