The hack revealed cracks in the largely hidden infrastructure that enables our computers to make decisions about which websites to load or software updates to run.
SuradechK/Thinkstock
On Saturday, Aug. 27, 2011, an Iranian man who went by the online alias alibo tried to check his email―only to find he couldn’t connect to Gmail. Yet the problem disappeared when he connected to a virtual private network that disguised his location. Whatever was going on, it seemed to only affect computer users in Iran.
His first hunch was that the problem might be somehow tied to the Iranian government―which was known for interfering with online activity―or a problem with his local internet service provider. So alibo posted a question about the issue on the Gmail Help Forum. Two days later, Google responded to this apparently small problem in a big way: It issued a public statement about the incident, attributing the problem to security issues at a Dutch company called DigiNotar. Within a month, DigiNotar had been taken over by the Dutch government. Not long after that, it declared bankruptcy and dissolved.
Advertisement
Cybersecurity breaches don’t usually spell the end of companies, much less spur national governments to seize control of private firms. But the DigiNotar compromise was unusual in many ways. Usually, the cybersecurity incidents we read about involve a company failing to protect the information entrusted to it by users. DigiNotar was different: Its whole reason for existence was to tell internet users who and what they could trust―and in 2011, it failed spectacularly in that mission. In the process, it revealed the cracks in the largely hidden infrastructure that enables our computers to make decisions about which websites to load or which software updates to run. Those decisions may seem mundane, but they are critical to the safety and security of the internet.
Five years later, the story of DigiNotar’s demise is all but forgotten, eclipsed by a series of more recent, more easily understandable, and more exciting breaches directed at organizations like Target, Sony, Ashley Madison, and the Democratic National Committee. But DigiNotar’s case has had long-lasting impacts, motivating some much needed improvements in the security of our online trust infrastructure, including a set of new minimum security requirements for companies like DigiNotar that were announced earlier this month by the Certificate Authority Security Council. But even as announcements like those suggest that we’re moving, gradually, in the right direction, the DigiNotar breach―even five years later―still serves as an important reminder of the risks inherent to our often-confusing online ecosystem.
* * *
Understanding what happened to DigiNotar requires some understanding of how your computer decides who and what to trust. There are a lot of people in the world building websites and coding software, and at any given moment, at least a few of them are up to no good―designing webpages built to look exactly like your bank’s in order to steal your login credentials, for instance, or writing programs that will encrypt your hard drive and hold it for ransom .
Advertisement
So every time you try to visit a webpage, your browser checks to make sure that the site you’re loading is really the one you’re trying to access, not a malicious page some wily attacker is trying to redirect you to. Similarly, when you download a new piece of software, your operating system will often check to make sure it’s coming from a trustworthy vendor.
But browser and operating system companies don’t want to be responsible for screening every single website and software developer in the world. Instead, they rely on third parties to vouch for those sites and developers. The third parties do this by issuing what are called certificates.
Bear with me, because this gets a little complicated―but it’s worth it. Those certificates are the bedrock of much of the security we enjoy online. They’re the reason we can do online banking, the reason we can download and install software updates without fearing malware. The organizations that screen people and companies and issue them these certificates are called certificate authorities, or CAs, and they make money by vetting and selling certificates to website operators and software manufacturers. There are hundreds of certificate authorities around the world, but the major browser and operating systems list only a small number as authorities whose certificates they will automatically trust. These elite certificate authorities are called root CAs. And the root CAs can, in turn, grant that same authority to any other intermediate CA they choose to endorse.
For instance, in 2015, a root CA operated by the China Internet Network Information Center issued an intermediate certificate to one of its customers, which then used the certificate to perform man-in-the-middle attacks and potentially intercept traffic between users and websites.
Advertisement
Any of those trusted CAs, whether they are root CAs or intermediate CAs that have been endorsed, can then issue certificates for any website they choose―even websites that have chosen to buy certificates from different CAs. This complex and often opaque hierarchy of relationships is one reason why things can go so wrong.
Most of this happens behind the scenes. If you’re a normal internet user, you probably only encounter certificates when you get a warning from your browser about trying to visit a website whose certificate was issued by an untrusted CA. But of course, that’s often not a clear―or alarming―enough message to stop users from trusting those sites. Admit it: You’ve probably clicked through such a warning.
There are different kinds of certificates: Some just allow for encrypted communication between you and the website you’re visiting, while other “Extended Validation” certificates involve a more thorough vetting of the website operator and certify that the organization running that site really is who it claims to be. When you see a little lock next to a site’s URL in your browser window, that usually means there’s an encrypted connection; a green bar next to the URL usually indicates that the site has an EV certificate.
Got all that?
Advertisement
* * *
Now that the background is out of the way: DigiNotar was a certificate authority―a well-established and reputable one. It was one of the root CAs for all of the major web browsers and issued many of the digital certificates used by the Dutch government for its online services. That made it a tempting target for criminals: If they could control one of these root CAs and issue trusted certificates themselves, they could potentially lure victims to a phishing site or infect computers with malware, bypassing many operating system and browser protections.
Because CAs are prime targets, they have to―and tend to―take security very seriously. DigiNotar was no exception. Among other things, it had segmented its computer networks into several different isolated partitions to constrain access attempts and used an intrusion prevention system to monitor incoming traffic. Every request for a new certificate had to be vetted and approved by two DigiNotar employees. Then, to issue the certificate, an employee had to insert a physical key card into a computer kept in a heavily guarded room. According to a postmortem report on DigiNotar’s compromise by security firm Fox-IT:
This room could be entered only if authorized personnel used a biometric hand recognition device and entered the correct PIN code. This inner room was protected by an outer room connected by a set of doors that opened dependent on each other creating a sluice. These sluice doors had to be separately opened with an electronic door card that was operated using a separate system than for any other door. To gain access to the outer room from a publicly accessible zone, another electronic door had to be opened with an electronic card.
Advertisement
This mix of physical and virtual safeguards demonstrates that DigiNotar was not a company that had failed to think about or invest in security. It understood that its security was vital for its own reputation―and for the wider world of internet users who relied, often without even knowing it, on DigiNotar’s certificates to tell them whom to trust online.
But DigiNotar also made some serious mistakes during the summer of 2011. For one, it was running some unpatched software one its web servers, which allowed an intruder to begin burrowing into its maze of partitioned networks in June 2011. On July 10, the intruder successfully issued his first rogue certificate. All told, by the end of the summer, he would go on to issue 531 rogue certificates for domains ranging from aol.com and microsoft.com to mossad.gov.il and cia.gov. (Once you’ve got access to a CA server, issuing rogue certificates for high-value targets like the CIA is no harder than issuing them for sites like AOL.)
It’s still unclear how exactly the intruder managed to bypass all the physical security in place to protect the inner sanctum where certificates were generated, but the investigators’ best guess was that the keycards for a few computers were left permanently in place. If true, it would have largely defeated the purpose of requiring the keycard insertion―not to mention all those sluiced doors and biometrics and PIN codes―in the first place.
On July 19, a routine check by DigiNotar revealed that some of the certificates it had ostensibly signed were not listed in the company’s logs―indeed, DigiNotar had no records of ever issuing these certificates. They were promptly revoked, and DigiNotar launched an internal investigation that uncovered still more rogue certificates. But by the end of July, the company believed the problem had been dealt with.
Advertisement
So it came as a shock when the report from alibo, the Iranian user, surfaced on the Gmail Help Forum a month later, and Google, in turn, blamed an unauthorized google.com certificate issued by DigiNotar. Some of the rogue certificates, it seemed, had slipped through the cracks of DigiNotar’s internal audit. And they were being used to certify impostor websites.
Thousands of Iranians who tried to visit Google websites in August 2011 were apparently redirected to sites that looked like Google webpages and were also certified as belonging to Google according to certificates issued by DigiNotar. Users from 298,140 unique internet protocal addresses trying to access Google websites were affected, and 95 percent of those IP addresses originated in Iran.
Why bother redirecting hundreds of thousands of Iranian Google users to fraudulent websites? Probably in order to read their email. Only one thing stood in the way: Google Chrome.
Part of what makes changing and securing the certificate ecosystem so difficult is that there are a lot of different stakeholders involved. Besides the hundreds of CAs, there are five major browsers (Firefox, Chrome, Internet Explorer, Safari, Opera), and these two groups have slightly different agendas. The CAs make money by selling certificates, so they want to sell as many as possible without alienating any of the major browsers. The browsers, in turn, are competing with one another for users. They don’t want to constantly block people from visiting websites due to certificate problems that those users are unlikely to understand and very likely to blame on the browser.
Back in 2011, Google Chrome included DigiNotar on its list of trusted CAs and would have happily accepted the company’s certificates for any other domain. But Chrome had a special extra check for Google’s own certificates. Google knew exactly which certificates it had purchased to validate its own domains―and which CAs it had purchased those certificates from. So to make sure that no other certificates were being issued for its domain, Google “pinned” its own valid certificates to its browser, Chrome, and didn’t accept any others, even if they had been issued by trusted vendors like DigiNotar. So on Aug. 27, when alibo tried to log into his Gmail account through Chrome and was redirected to the site signed by the rogue certificate, his browser knew something was wrong. It blocked the webpage, prompting alibo’s post on the Gmail Help Forum and the unraveling of the entire incident―and DigiNotar itself.
No one has ever been caught or charged with the compromise, though many have speculated that Iran’s government was likely involved. The only clue left by the intruder―a message left behind on a DigiNotar server―offers little insight into the perpetrator’s mission or identity other than a profound sense of self-importance. “I know you are shocked of my skills, how I got access to your network,” the message begins. “THERE IS NO [sic] ANY HARDWARE OR SOFTWARE IN THIS WORLD EXISTS WHICH COULD STOP MY HEAVY ATTACKS MY BRAIN OR MY SKILLS OR MY WILL OR MY EXPERTISE.”The discovery of the DigiNotar compromise left the browser and CA community―to say nothing of the Dutch government―reeling. Browser vendors rushed to revoke trust in DigiNotar certificates, but removing a root CA was not entirely straightforward. “We actually needed to push out an update to Firefox because the CA information was hard-coded to the browser,” Firefox security lead Richard Barnes said. Additionally, many legitimate websites (including some operated by the Dutch government) were still relying on DigiNotar certificates, so the browser vendors were forced to hold off on a blanket ban. Instead, Mozilla decided to block all DigiNotar certificates issued after July 1, 2011, but allowed users to decide whether they wanted to trust certificates issued by the company before that date. But giving users that autonomy over their online security only works if they understand what it is they’re choosing and the implications of that choice―a task that surely went beyond many Firefox users.
While the browsers scrambled to protect their users, the Dutch government took charge of DigiNotar and commissioned Fox-IT to investigate what had gone wrong. Hans Hoogstraaten, who led the investigation, said in an email, “What really shocked me was when I realized the impact it had for the people of Iran. In those days … people got killed for having a different opinion. The hackers (presumably the state) had access to over 300,000 Gmail accounts. The realization that the … security of a small company in Holland [may have] played a part in the killing or torture of people really shocked me.”Both CAs and browsers have made significant changes to their security practices since 2011. “DigiNotar was a real wake-up call for the entire industry,” said Rick Andrews, senior technical director for website security at Symantec, one of the largest CAs in the world.
Certificate pinning―what Google did with its certificates in Chrome, which resulted in the DigiNotar compromise first being detected―has become more common and has spread beyond companies that manufacture their own browsers. Another initiative, called Certificate Transparency, aims to make sure logs of all valid certificates issued by CAs are publicly accessible. This makes it easier for individual domain owners to monitor whether any certificates have been issued for their domains without their knowledge. To incentivize CAs to log their certificates publicly, Google announced in December 2014 that, beginning in 2015, it would no longer display the green address bar in Chrome for certificates that belong to verified companies unless those certificates had also been logged. In the back and forth between CAs and browsers, the browsers wield much of the power since they ultimately decide which websites users can load and which of the subtle security signals―green bars, lock icons―are conveyed to those users.
Perhaps the most significant change in the certificate landscape is simply that there are now many more certificates than there were five years ago. This is part of a larger push for widespread online encryption spearheaded by the CA Let’s Encrypt , launched earlier this year, which provides free certificates to anyone who wants them. Let’s Encrypt doesn’t provide the Extended Validation certificates that involve verifying a website owner’s identity (the kind that most high-value targets generally get and that warrant a green box in many browsers) because that process cannot be automated. “There are hundreds of millions of websites and devices out there, and in the future there will be many billions. For every one to have a certificate we’ll need issuance systems that can be fully automated,” said Josh Aas, founder of the Internet Security Research Group, which established Let’s Encrypt. Issuing more certificates helps spread encryption, but it also raises the stakes for the security of CAs and the risks posed by incidents like the DigiNotar compromise because it means that an increasing amount of our online communication relies on the protection provided by digital certificates.
And problems with CAs have not gone away. On March 20, 2015, years after the collapse of DigiNotar, Google discovered another set of rogue certificates for Google domains. These certificates had been issued by an Egyptian company, MCS Holdings, which had, in turn, received its certificates from CNNIC, the CA operated by the China Internet Network Information Center, an agency in the Chinese government’s Ministry of Information Industry. Soon afterward, Firefox and Chrome both removed CNNIC from their root CA lists. Just this summer, Chinese CA WoSign was accused of issuing fake certificates for Github and Alibaba, and in October Mozilla announced that it would no longer trust WoSign certificates.
Thanks in no small part to the legacy of DigiNotar, browsers and CAs alike are better able to deal with problems like these than they were five years ago―they can revoke compromised certificates faster, check certificates against public logs, and restrict the use of rogue certificates with pinning. But in other, more fundamental ways, the system of relying on CAs to tell us who we can trust online remains inherently vulnerable―and, perhaps more importantly, largely invisible to most internet users. The complexity of the certificate infrastructure can make it difficult for the wider public―beyond the community of browsers and CAs who have long been attuned to the importance of the DigiNotar compromise―to understand the risks they face online, as well as the signals and warnings that their browsers provide.
“The folks who operate the CAs are really a very tempting point of attack,” said Daniel Kahn Gillmor, a senior staff technologist with the American Civil Liberties Union’s Speech, Privacy and Technology Project. “If I wanted to attack someone else I would be looking for a lever of control that they might not even know existed.” The more we gloss over the crucial components of the trust infrastructure underlying our online communications―an infrastructure that is every bit as relevant today as it was five years ago―the harder it becomes to grasp how deeply and fundamentally all of our security is predicated on the security of digital certificates and the companies that issue them.
This article is part of Future Tense , a collaboration among Arizona State University , New America , and Slate . Future Tense explores the ways emerging technologies affect society, policy, and culture. To read more, follow us on Twitter and sign up for our weekly newsletter .
