Lessons Learned from an AWS (Amazon Web Services) Intrusion
Recently, one of the more than 80 AWS accounts that Cornell uses was accessed by an intruder that had obtained the password to it.
A Cornell developer had accidentally included AWS credentials in source code committed to a Git software repository that was publicly accessible. Within minutes, the intruder had found them and began using the Cornell account to start up large AWS EC2 instance types across all AWS regions. The presumed goal was to steal computing cycles for cryptocurrency mining.
Once some irregular activities related to the intrusion were noticed by the Cornell developer, the credentials that were being used by the intruder were identified and deactivated. After that, the EC2 instances the intruder had launched were shut down to stop charges from mounting further. Even so, the cost of the affected account, usually $200 per day, grew to $6,700 the day of the intrusion.
Some factors contributed to keeping the impact of the break-in from being worse, including:
- Part of the standard audit configuration required for Cornell AWS accounts is to have monitoring services turned on that made it easier to see how the compromised credentials were being used.
- Amazon has soft limits on how many instance types can be running at any given time, preventing the intruder from creating more than they did.
Other mitigating factors:
- No Cornell confidential data was being held or processed by the affected account.
- Destruction of data did not appear to be part of the intruders' goal, so no data was lost.
- Actions taken by the intruder made Cornell aware there was a problem fairly quickly.
- The compromised credentials were limited to one specific service. If they were associated with other services, the fallout could have been far more extensive and the series of intrusions could have been harder to shut down.
Even so, the overall impact of the credential compromise could have been limited even more by better scoping of privileges, such as:
To a specific AWS region. The cost of this event could have been reduced by over 90% if privileges had been restricted to a specific region of interest.
To work just from a limited set of source IPs. Most Cornell AWS users connect from a limited set of IP addresses, such as Cornell campus IPs and possibly a home IP address. Scoping to that limited set of IPs won't hinder legitimate use of the credentials.
To home in on just the AWS operations that an IAM principal should have access to.
More tools and techniques that can help
People trying to break into AWS and other accounts watch public software repositories, like Git. Tools like git-secrets can be used to avoid accidentally committing things like AWS credentials to Git repositories.
Also, it's often possible to use temporary credentials instead of static, fixed ones. Tools like aws-vault make it possible to improve your AWS credential hygiene in a variety of scenarios. In fact, one team at Cornell has integrated aws-vault into their development processes.
You can avoid AWS credentials entirely if you are operating from an EC2 instance. In that case, you can use IAM instance profiles to give your EC2 instance privileges to perform AWS operations without having to configure AWS access key credentials.
There are also a variety of notification tools, some real-time, that can be used as an alert of unexpected activity.
Cloudification is Cornell's offering of public cloud services through enterprise agreements with Amazon Web Services (AWS) and Azure (Microsoft). The mission of the Cloudification team is to facilitate the campus transition to the cloud while encouraging modern devops practices. The Cloudification service is a partnership between CIT staff and IT staff in the colleges and administrative units to help facilitate consistent, secure, efficient processes for moving campus applications to the cloud. For more information, visit the Cloudification site or email firstname.lastname@example.org.