"The HPC is down!"
"But the competition just started!"
Our high-performance computing cluster (HPC) blinked red on the big screen. Minutes ticked by.
"Get it up! Get it up! We're losing points!"
"Working on it!"
Red team had been circling since the day before, hawks swooping and diving. They'd been scanning and probing all day Friday, but weren't allowed to attack until the checkered flag dropped Saturday morning at 8 a.m. We'd hoped to evade their talons, but they wasted no time, and now one of our critical assets blinked out--a meal for a hungry predator.
Our blue team was tasked with defending a mock oil refinery's industrial control system (ICS), the HPC and the integrated back-office IT system―all of it default insecure, some of it insecure by design―and the only real defense active monitoring and split-second eviction before red team could take us down.
"The HPC isn't coming up. What is going on?"
Four Raspberry Pis running the oil refinery and HPC sat on the table in front of us, water pumps clicking on and off. Click-click. Click-click. Click-click. Someone else was in control. Not us.
J.M. Porup
Meet Big Oil Logistics and Transportation Corporation (BOLT Corp), its ICS, and an HPC running on four Raspberry Pis.
CyberForce ― a competition to defend critical infrastructureAcross the country in seven different Department of Energy labs, university teams fought alongside us to defend identical infrastructure. The DoE runs its CyberForce competition every year to introduce college cybersecurity students to the challenges of securing critical energy infrastructure, and to recruit the best and brightest. We were the UC Berkeley team representing the Berkeley iSchool's new Masters of Information and Cybersecurity (MICS, pronounced "mikes") program, and I was not only reporting on the event, I was playing to win.
The six of us hunched over our laptops in the competition space at Berkeley Lab, perched cliffside just east of the UC Berkeley campus, too distracted to enjoy the view of the Golden Gate Bridge, San Francisco Bay and the iconic Berkeley Campanile, the picture postcard bell tower. For eight hours we fought to keep our systems up and red team out, until, by the end, the uncontrollable twitch in my right eye told me I had chewed on too much stress for one day.
Teammate Daren tugged at his grey goatee. "Let me check something."
Predator and preyI stood in the dark under the dripping canopy of the Strada Cafe in Berkeley waiting for my ride to the lab that morning. The piranha in my bowels gnawed away at my spasming entrails. How were we going to defend these completely defenseless industrial control systems? These things were running modbus, insecure by design. We weren't just sitting ducks, we were Peking ducks ready to serve with plum sauce.
This wasn't your normal capture the flag (CTF) , where teams competed to break into the systems and steal information. We weren't predators. We were the prey.
I sipped my black coffee and checked my phone: 6:12. Doors opened at 6:30, and the competition began at 8 a.m. Pacific―11 a.m. for the competitors at Brookhaven on Long Island, 10 a.m. at Argonne outside Chicago, and 9 a.m. at Sandia in New Mexico.
It was going to be a long day.
My teammate Josh rocked up with a coffee and we stood watching the rain. The darkness seemed unending. A car sloshed by. Was it―? No. Not our ride. We both pretended to be chill.
"We gonna win today, you think?"
"Gonna try."
Then "The Anvil of Crom" thumping in the car, Nathan our mentor driving, up the rainy black hillside to the unclassified cliffside lab that overlooks campus, to the guard post, documents glanced at, a flicker at us in the back seat, then up, up, up to the SOC we had laid out on two tables the day before.
Paul Mueller, photographer
Our ad hoc SOC: six laptops, two big screens, many coffees.
The music still rings in my ear, so wrong in retrospect. We were not warriors going into battle; we were lab mice in a maze with hawks circling, predators ready to crush us with their talons at any moment.
We sat down at our battle stations―two giant big screens propped up on our competition table―and prepared to defend. The piranha gnawed harder.
The dirty half-dozen"They're all just kids," we muttered, glancing sideways at each other. The other three teams at Berkeley Lab were undergraduates, most of them too young to enjoy a beer after. Our team's average age was 42, and together the six of us have a combined 107 years of experience working in IT, including a couple of folks from the defense sector, a software architect, a systems integrator, an early Facebook employee, and a coder turned wordslinger―me.
We were double their age and had ten times more experience than they did. Somehow that didn't make us feel any more confident about our chances of winning.
J.M. Porup
Predator humor
"Having fun?" my teammate Karel asked. I stared at the big screens, looking for rogue logins. The log info came at us in an unending stream. On the CyberForce Slack channel for competitors, threat intel overwhelmed with gossip and meme noise. And the memes! What was with the undergraduate obsession with memes, many of dubious hilarity? Out of place, old and weary, facing an impossible task, a goshawk's meal in waiting.
"Not really," I answered.
Fear and loathing washed over me like a shower of sewage. Were we fools or tools? I shuddered. Not the lesson the DoE wanted to impart when they launched CyberForce, I felt certain.
Red team goes vishing"Team 69 help desk, can I help you?" Terry clutched the cell phone to his ear in the noisy competition space. Of the 70 teams competing, UC Berkeley had been assigned number 69.
J.M. Porup
Team 69, at your service
CyberForce awarded huge points for usability and help-desk support, not just keeping red team out. To secure the ICS modbus service in the handful of weeks we had available, we'd implemented a crude two-step authentication process. Sniffing legitimate LDAP credentials sent in the clear wouldn't be enough to pop our mock oil plant. Red team would also need to know the correct two-step authentication code listed in the user guide.
Terry muted the phone. "Green team wants the two-step auth code."
"It's in the user guide. Tell them to read the user guide."
"Says they don't have access to it."
We had written the user guide from scratch to include an easy look-up table of all the two-step authentication codes. Hacking the green team to steal the user guide was out of scope. Within the constraints of the competition, our solution was as good as it gets.
Karel Baloun
Battlestations!
Daren rose to his full height, grey goatee jutting forward, and took the phone. "Hello?" He listened for a moment. "I'm going to have to assume you are red team trying to social engineer us. The auth code you need in the user guide." He hung up and passed the phone back to Terry.
"How did they get the number?"
"It's right there on the login page, call this number if you're having troubles."
Nice try, we all thought. Not gonna get that past this team. We weren't born yesterday. Try to socially engineer us ...
Until we read our scoring feedback the day after the competition. It included this nugget: “Tried calling help desk on HMI authentication problem, but another user was logged in and they thought I was from red team.”
Turned out the caller really was green team.
Lab time at the DoEThe competition kicked off with a red-tied Rick Perry beaming on the screen.
"Oh, look it's Trump's Secretary of Energy," I said.
"No time. Less than an hour before things get underway."
A week later, while writing this story, I hunted up Perry's opening remarks . Watching him struggle to read a teleprompter made my eye twitch again.
"Today the digital infrastructure that serves this country is literally under attack," Perry intones. "Protecting our energy infrastructure against those threats is my highest priority as Secretary."
"You are this nation's next generation of innovators, defenders, cyberwarriors"―a twinkle in his eye when he says the sexy cyber word―"We need you to bring your knowledge, passion, competitive spirit to, uh, the job at hand."
The worst part of it is, though, Perry's not wrong. Behind the puzzling upbeat muzak in the video and the "howdy, partner" political happy-clappy lurks a truth to wipe the smile off your face: America's critical infrastructure was never meant to be plugged into the internet. Next door to every spy and gangster on the planet, the energy systems on which our economy―and lives―depend are about as secure as a wet paper bag.
Worse, the massive skills shortage in cybertown means few qualified workers have any interest in building a career in OT/ICS/SCADA security. If Google and Facebook pay top dollar for security talent, a water treatment facility in southwestern Montana pays bottom dollar. The DoE wants to expose cybersecurity students to the problem in the hopes of attracting them to the ICS security space―or at least raising awareness of the issue more broadly among career beginners.
CyberForce ―cue flexing muscular men and women on the cover of a vintage Conan the Barbarian pot boiler, oil glistening on scantily-clad physiques, blades flashing, stentorian voice like Zeus announcing their presence―"SIGH BURR FORSS"―launched in 2016, and the December 2018 competition was the fourth so far, and saw double the number of participants as the April competition. The next competition will be held in November 2019.
"The competition is meant for collegiate students to defend and secure an energy-simulated environment," Amanda Joyce, CyberForce Competition Director and Strategic Cybersecurity Analysis and Research Group Lead at Argonne National Laboratory, says in the video. "So every year we change the scenario to be a different energy-based component, this year being oil and high-performance computing. Their job is to take a vulnerable system and to secure it to the best of their ability within realistic environment restrictions."
One surprise gotcha―CyberForce competitors were also lab rats being studied as part of an experiment. I growled at my laptop when I discovered this tidbit buried in a wordy research consent form. Recruiting talent? Cool. Stimulating innovation? Even better. Watching all our VPN traffic and studying us as research subjects? Creepy. Were we the researchers, or the lab mice? Maybe both....
Department of Energy
Prey to both red team and researchers.
Out of equal parts alarm and curiosity, I called the Institutional Review Board (IRB) manager at the DoE, a woman with a thick Southern accent by the name of Lindsay Motz. I told her the consent form was opt-out, not opt-in. She seemed genuinely surprised. "You don't have to opt-in in order to compete," she told me over the phone, and encouraged me to reach out to the academic researcher in charge of the project.
"Talk to the pee-AHH," she told me.
I blinked. "The, uh... the who?"
"The pee-AHH."
"I'm sorry, I mean, I, uhh―"
"The principal investigator. The pee-AHH."
"Right, the PI, of course, right... thanks."
The pee-AHH, Benjamin Blakely of Argonne National Lab, sent me a copy of the HRP-503 form, which crisply informs the reader that "gaining an understanding of how to better measure cybersecurity expertise....will help many stakeholders improve training programs, accreditation programs, and workforce frameworks."
Considering the US government's long history of unethical experiments on unwitting human subjects, not to mention totalitarian mass surveillance, giving the DoE the benefit of the doubt was beyond my poor power. The explanations tally, the words all seem correct, but it still felt like they were trying to pull a fast one.
"I don't mind," Karel said. "I've got nothing to hide."
The white-on-black console text blurred together. "I suppose since you've got nothing to say, you don't care about free speech either?"
A shrug. "I think maybe you're reading too much into it."
Then the sCOARboard went down, and a swarm of piranhas feasted on my spleen.
What are the rules again, exactly?"sCOARboard is down."
"What do you mean sCOARboard is down?"
"It's down. Everyone on Slack is complaining."
"How are we supposed to submit our incident reports?"
Want to check the real-time score updates? Only a refresh-refresh-refresh away―F5 the badly acronymized s COAR board, the competition's score-tracking system. Teams could also earn points by submitting intrusion reports, or bonus points for solving so-called "anomalies"―discrete security problems like analyzing a pcap file in Wireshark , extracting a message from steganography, etc.
