by Will Bengtson
Previously we wrote about a method for detecting credential compromise in your AWS environment. The methodology focused on a continuous learning model and first use principle. This solution still is reactive in nature ― we only detect credential compromise after it has already happened.. Even with detection capabilities, there is a risk that exposed credentials can provide access to sensitive data and/or the ability to cause damage in our environment.
Today, we would like to share two additional layers of security: API enforcement and metadata protection. These layers can be used to help prevent credential compromise in your environment.
ScopeIn this post, we’ll discuss how to prevent or mitigate compromise of credentials due to certain classes of vulnerabilities such as Server Side Request Forgery (SSRF) and XML External Entity (XXE) injection. If an attacker has remote code execution (RCE) or local presence on the AWS server, these methods discussed will not prevent compromise. For more information on how the AWS services mentioned work, see the Background section at the end of this post.
Protecting Your CredentialsThere are many ways that you can protect your AWS temporary credentials. The two methods covered here are:
Enforcing where API calls are allowed to originate from. Protecting the EC2 Metadata service so that credentials cannot be retrieved via a vulnerability in an application such as Server Side Request Forgery ( SSRF ). Credential EnforcementCredential enforcement only allows API calls to succeed if they originate from a known environment. In AWS, this can be achieved by creating an IAM policy that checks the origin of the API call. To achieve this, it is important to understand where API calls come from (described below in Background). An example policy is shown below.
One way to deploy this is to create a managed policy that encompasses your entire account across all regions. To do this, describe each region and collect your NAT gateway IPs, VPC IDs, and VPC endpoint IDs to create the policy language for the managed policy (similar to the example above) that can be attached to the IAM Roles that you want to protect. The limitation of this method is that you can only protect IAM Roles that are used on EC2 instances deployed to the internal subnet. IAM Roles that are associated with EC2 instances in the external subnet should be excluded. Exposing your service publicly through a Load Balancer would allow you to deploy your EC2 instance into the internal subnet and allow you to attach this policy to your IAM Role.
Another limitation to this method is that AWS often makes calls on your behalf that are triggered by certain API calls. For example, when you restore an encrypted RDS instance, AWS will make KMS calls on your behalf to figure out which key should be used in the restore process. When these services make calls for you, the AWS credentials that are tied to the IAM Role that made the first call are used. The originating IP address will be one from AWS and not reflect what is in your policy. You can see this in CloudTrail by looking from events with sourceIPAddress resembling <service>.amazonaws.com . Even with this limitation, you will find that you can protect most IAM Roles and find workarounds to address this.
Metadata Service ProtectionAs described above, the EC2 Metadata service is the mechanism for providing credentials to your application running on an EC2 instance in AWS. It is available by making a request to the IP address of 169.254.169.254. The current AWS Metadata service does not require any HTTP headers to be present and allows any process to make HTTP requests.
Server Side Request Forgery (SSRF) is a vulnerability that allows an attacker to trick the application into making a HTTP/HTTPS requests on their behalf. One of the most common attacks against applications that are vulnerable to SSRF target the Metadata service credential path. When an attacker exploits a SSRF vulnerability, they cannot control which HTTP headers are sent in the request. The lack of header control by an attacker enables a required header on the metadata service to mitigate this class of vulnerability. If an attacker is able to set HTTP headers, such as having a shell on the server and controlling headers in a curl command, the header protection is useful in protecting against an attacker that does not realize there is a header required.
If the Metadata service required a HTTP Header when talking to it, the SSRF attack vector that aims to steal your AWS credentials can be mitigated. In the past it was not possible to create your own Metadata proxy to protect the Metadata service from attacks such as SSRF. The Metadata proxies you might find in open source are typically scoped to providing credentials for containers running on your hosts and not able to protect against these attacks. We have been working with AWS to enable the ability to protect against this attack by setting the User-Agent HTTP Header when making requests to the Metadata service from the AWS SDKs to something known. By knowing what User-Agents will be set when official AWS SDKs make requests to the Metadata service and combining this with the fact that in the SSRF vulnerability scenario you cannot control HTTP Headers, we are now able to proxy traffic to the Metadata service and reject requests without the appropriate User-Agent HTTP Header, thus mitigating the SSRF attack vector on AWS Credentials.
The current User-Agents that you will see when proxying traffic from the SDKs start with the following strings:
Summary
Default configuration in the cloud leaves your environment at increased risk in the event of a credential exposure/compromise. Coupling a Metadata proxy with API enforcement increases the security stance of your AWS environment, implementing defense in depth protections. Combining this approach with Detecting Credential Compromise in AWS paves a road for protecting IAM in your cloud environment.
Be sure to let us know if you implement this, or something better, in your own environment.
Will Bengtson, for Netflix Security Tools and Operations
Background What is a credential?“Credential” in this post is the Amazon Web Services (AWS) API key that is used to describe and make changes within an AWS account.
The main focus are the credentials that are used on an AWS Elastic Compute Cloud (EC2) instance, although the outlined approach is valid beyond EC2. AWS provides an ability to assign permissions to an instance through an