Many security teams are plagued by the eternal question of whether to “ build” or “ buy” technical solutions for the complex challenges they face. At BuzzFeed, our security infrastructure team experienced this tension when our newly developed microservice ecosystem began to outgrow our existing auth patterns, which relied on Bitly’s tried and tested oauth2_proxy . This solution was reliable, but it lacked a centralized way for our users to sign on, which presented a handful of challenges for our growing platform. So we did what any good engineers would do: We looked for other existing, tested solutions in open source. However, while we heard from our peers that this was a common problem, we never found something that met our specifications. Thus, we created sso , a centralized authentication proxy of our own. It provided us with what we saw as a novel solution to a common industry problem, so we made a commitment to open source it ― to provide that tried and tested solution we too had been searching for in thepast .
Our problem: microservice authWhen BuzzFeed launched over 10 years ago, we started, as many companies do, with a glorious single application. Over time, when both the engineering team and its monolith application had doubled in size, we came to a familiar realization for companies of our size: We lacked the sophistication of infrastructure and tooling to support the organizational model of our teams and systems. Thus, rig wasborn .
Rig is our opinionated Platform as a Service (PaaS), which enables high-velocity development and productionizing of services. The transformation was wonderful ― we went from struggling to deploy several times a week to deploying 150 times a day! However, as anyone who has made the move to microservices can attest, there were pain points in migrating to a much more distributed set of services. In this article, we’ll explore one pain point in particular: authenticating and authorizing access to internalservices .
Today, BuzzFeed’s software ecosystem is comprised of hundreds of microservices that interact with each other in various ways. A large subset of these services is comprised of tools that support our content creation and business teams, which means we need to have a secure way for users to access these internal services. Early in the development process, when we were building out proofs of concept, we were inspired by Google’s BeyondCorp methodology , which addresses the challenge of allowing users to access internal services over untrusted networks without a traditional VPN. This approach requires these services to have a user interface open to the internet, and those interfaces need to both authorize and authenticate those users. Rather than pushing this responsibility onto each service, we opted to protect each of them with an instance of oauth2_proxy , which is a reverse proxy that uses third-party OAu th2 providers (Google, GitHub, and others) to authenticate and authorize requests. Using an auth proxy is generally an effective pattern for microservices because it allows developers to focus on their services’ primary functionality instead of re-implementing authentication every time a new service is developed. This proved to be the case for us, and we rapidly grew the number of deployed internalapplications .
While our platform made it easy to create new services, correctly configuring the accompanying auth proxy was overly difficult and confusing fordevelopers.However, problems soon emerged. As the number of individual boilerplate oauth2_proxy services grew beyond 70, we ran into a variety of fresh issues ― not just as maintainers and developers of the platform, but also as users of the internalapplications .
While our platform made it easy to create new services, correctly configuring the accompanying auth proxy was overly difficult and confusing for developers. This negatively impacted the productivity of our product teams and led to inconsistent and difficult-to-audit OAu th2 applications and permissions with Google, our third-party provider. Furthermore, as operators, managing the proliferation of auth proxy services proved difficult. Critical security fixes required over 70 patches and deploys, instead of just one. Auditing and controlling access across services was also an ongoing challenge. To remediate this, we needed to find a way to centralize access and administration. This would also create a more pleasant experience for the end user, who in the current state needed to separately log in to every service. In an ideal world, a user would perform a single sign-on and have access to all authorized services for a configured timespan .
Our solution: centralized authOur solution to these pain points is sso , our OAu th2-friendly adaptation of the Central Authentication Service ( CAS) protocol. sso allowed us to replace every individual oauth2_proxy service with a single, centralized system, providing a seamless and secure single sign-on experience, easy auditing, rich instrumentation, and a painless developerexperience .
Our implementation is comprised of two services, sso-auth and sso-proxy , that cooperate to perform both a nested authentication flow and proxyrequests .
sso-authsso-auth acts as a central authentication service, directing a user through an authentication flow with a third-party provider ( e.g. , Google). It uses the third-party provider’s groups API ( e.g. , Google Groups) to provide a simple administrative user experience forauthorization .
sso-proxysso-proxy goes through an OAu th flow similar to sso-auth , but with sso-auth as its authentication provider. After going through this flow, it proxies the request back to the upstream. Additionally, it signs the requests, providing a mechanism for upstreams to verify that the request originated from sso-proxy .
Both sso-auth and sso-proxy store user session information in long-lived, encrypted cookies, but sso-proxy transparently revalidates the user’s session with sso-auth on a short, configurable interval to ensure quick propagation of authentication and authorizationchanges .
User experience using ssoThese two services work together to create a single sign-on experience that, behind the scenes, consists of nested OAu thflows .
Authentication
Authorization
When end users visit an internal resource, like cms.example.com :
