The Code Spaces Nightmare
By
Mark Nunnikhoven

To say that this continues to be a really bad week for the Code Spaces team is a massive understatement.

After gaining access to their administrative credentials, an attacker managed to quickly and easily wipe out their entire business. Both production and backup data was destroyed.

The question that’s been posed to me is simple, “Is this a result of a problem with AWS security?”

The answer is an unequivocal “No.”

Root Cause

While we don’t have all of the details of the attack, we can pull quite a bit of information from the Code Spaces statement on their site.

They stated that, “An unauthorised person…had gained access to our Amazon EC2 control panel…” From the actions that were taken (removing EBS snapshots, deleting AMIs, etc.), it’s safe to assume that the attacker had not just gained access but had gained access with a high-privilege account.

There is no mention of an exploit being used successfully against the AWS management tools.

A colleague pointed out that the statement, “our Amazon EC2 control panel” could actually refer to a set of internal tools instead of the AWS Management Console.

If that’s the case, the attacker may have compromised a system that hosted the tools (like a development laptop or administrator’s workstation) and might never have had direct access to any AWS credentials, just the tools that leverage the credentials.

The end result and advice is the same, but this alternate avenue highlights the importance of addressing the security of the systems that have access to your AWS credentials (in any form), in addition to the credentials themselves.

Based on what we know, this is an operational failure. One of dire consequences but also one that could have been prevented.

What You Should Do… Now

If you haven’t already, you need to take the following steps:

IAM is extremely powerful and free. There is no excuse for not using that power to protect your organization.

If you haven’t heard of IAM before, you should take a minute and watch the introduction video from AWS and then review this slide deck on the top 10 tips for IAM.

Multi-Factor Authentication

MFA is a simple concept. In addition to a username & password, a user must have a temporary access code generated by a token (which can be as simple as an app on their smartphone) in order to be granted access.

It’s a must-have for any account that can log into the AWS management console through the web.

It only takes a minute to setup MFA. Do it. Now.

Step two is to re-configure your IAM policies to require MFA for users with web access (yes, you can block web access if you want to) and any high-privilege accounts.

As typical, AWS has provided simple & straight forward documentation for this process.

Administration Strategy

Separation of duties is a long-held principle in information security. It’s a simple concept: one person shouldn’t be able to do everything. This way, you have multiple checks before anything catastrophic happens.

This was obviously not the case with Code Spaces. Accessing one account provided the attacker with sufficient privileges to destroy all of the company’s assets.

The way to prevent this is to ensure that you divide duties between IAM users (and, ideally, real-life users).

Again, IAM is free so there is no financial cost to do this. There is an operational impact, but given the alternative, it’s worth imposing that small burden on your ops team.

The first place you want to start is for your backups. If you’re automating backups (and you are, right?!?), ensure that the IAM role has the privileges to create new backups but not to delete existing ones.

In fact, prevent any IAM user/group/role from deleting backups with the exception of a dedicated account for only that action.

Make deleting a backup hard.

It’s not something you’ll be doing on a regular basis, and storage is cheap; take advantage of that. Keep the MFA token and credentials that can delete backups stored somewhere safe (physically and logically).

That step alone would have saved Code Spaces.

From there, continue to work through the administrative actions you take in your environment. The creation of new IAM users and granting of permissions should be next.

Rinse and repeat until you’ve hit a level of risk that you’re comfortable with.

IAM Roles

The best way to protect credentials is to not need them in the first place. That’s where IAM roles come into play.

A role provides you with the ability to provide an IAM policy to an asset in AWS, an EC2 instance being the simplest example.

You can assign an EC2 instance a role. In the background, the simple token service (STS) will automatically provide the instance with temporary credentials that are automatically rotated on a regular basis.

This means you can allow you applications access to various AWS resources without providing a set of credentials that could be compromised.

Does your app need to write to an S3 bucket? Use a role.

Using an SQS queue? Leverage a role to make that happen.

Sending out notifications? Use a role.

You get the idea. IAM roles reduce the risk of someone stealing a set of credentials by avoiding having to create those credentials in the first place.

Operational Security

What happened to Code Spaces is unfortunate, but it ultimately comes down to an operational failure.

AWS operates on a shared responsibility model. It’s up to you as the client to secure the operating system, your applications, your data, and access to your resources.

Administrative access is a risk in any environment, and it needs to be managed properly. IAM is a flexible, free tool to help you protect access to your AWS assets.

Please take the time to ensure that you don’t end up in this situation. Start working on the three areas I’ve outlined in this post:

The time you invest just might prevent this nightmare scenario from happening to you.

Mark Nunnikhoven is the Vice President of Cloud and Emerging Technologies for Trend Micro where he meets regularly with clients (and prospective clients) to understand their security challenges and to share the research and vision for cloud and data center security. He speaks regularly on cloud computing, usable security systems, and modernizing security practices at conferences and events.