Note that in order to avoid confusion with this separate issue, the publish date of this post was pushed back.

On July 29th, 2019, I discovered what I consider to be a major security risk with the use of the CircleCI CI/CD product that should be unacceptable to any corporation or open source project. This article details how commonly used CircleCI features interact with GitHub security mechanisms in a way that exposes GitHub secret keys to malicious users.

I have been in communication with the CircleCI security team since July 29th, 2019. To their credit, CircleCI quickly escalated this issue. However, as of August 28th, 2019, the stance of the CircleCI team is that these features do not represent a vulnerability. Since disclosure, CircleCI has updated their documentation to mitigate the risk of these build features. Details of these updates are below.

• CircleCI has updated their documentation in order to make the difference between a Deploy Key and a User SSH Key more clear, and push towards to the best practices of using a Machine User
• CircleCI has updated their documentation to explicitly state that these keys are exposed: https://circleci.com/docs/2.0/gh-bb-integration/#what-about-security
• CircleCI is conducting research into build processes which would eliminate the need to load private keys into build contexts

When a repository is configured to build on CircleCI, CircleCI will create one of the following to clone and build the repository:

1. A “Deploy Key” on the repository in Github. This key has access to clone a single repository.
2. An “User SSH Key” on the user in Github. This key has access to read/write all repositories in Github which the user has access too.

The default is to use a “Deploy Key”, however a “User SSH Key” is required if your build needs to:

• Clone private dependencies
• Clone private submodules
• Push updates, tags, or releases back to Github

Whichever of these key options is used, the RSA Private Key is injected in plain text into the build context of any job which runs on CircleCI. This means that anyone who has ‘Write’ access (in Github) can extract this key in one of two ways:

1. Modifying the build code to read and exfiltrate the private key (for instance, over the network)
2. Use the “Rerun job with SSH” option to SSH into a build container and directly read the key.

• This option does not leave any evidence in Github, or in the git log.

Other Sensitive Stuff

As a quick note, CircleCI has a couple other sensitive values that can be injected into your build context. These values are all masked within the CircleCI Application Dashboard, however are available within the build context, and exposed to exfiltration, in a similar fashion to the Private Keys. These values include:

• Environment Variables set on the Repository in the CircleCI Application
• Passphraseless private SSH Keys (to access arbitrary external hosts) set on the Repository in the CircleCI Application
• AWS Credentials set on the Repository in the CircleCI Application
• Potentially: Values in CircleCI Contexts across the entire Organization (not limited to a specific Repository).
  • All Contexts default to available, however additional security tool exist to lock these down a bit.

Any user with the ability to start a build can access these sensitive config values. It is important to remember when adding these values to follow best practices in order to prevent the risk of these values being exfiltrated.

Risk Vectors:

Exposed Deploy Key

The risk of an exfiltrated “Deploy Key” is fairly limited. In the event that an individual previously had access to a repository, but no longer does (for example, they were terminated from your organization), you cannot guarantee that this individual has not extracted CircleCI’s Private Key, and is continuing to use that key to clone the repository (maintaining Read access). This can be managed by rotating the key, however this is difficult (and discussed further below).

Exposed User Key

This is where things get dangerous. If a User SSH Key is exfiltrated, this is a classic privilege escalation attack. Anyone who can exfiltrate the key can obtain the same Github git permissions as the individual who initially set up and configured the repository. This is only git access, and does not include API or Application access.

Consider the following (non-exhaustive) cases that a typical user might follow in order to quickly get their CI/CD environment running. We have to remember that a large majority of users will take the easier route without a second thought, even if this goes against documented best practices. Some of these use cases now violate documented best practices, and readers should evaluate if they currently have exposed keys.

Due to the above attack vectors, in my opinion, the following keys should be considered compromised, and rotated as soon as possible:

• Any Deploy Key which has had a contributor lose access since the key was generated
• Any User SSH Key which belongs to a user which, and has ever been used in a build kicked off by any other user.
  • I understand that there are likely some situations where these exposed User SSH Keys would not lead to a potential privilege escalation, such as in an Organization where every account has the same Repository access, and everyone uses Employer-Specific Github accounts. In my experience, these cases are slim enough to ignore.

Key Rotation

Key rotation is a good defense to the risk of the exposed Deploy Keys. If these keys are rotated every time an organization member loses access to the repository, there is no risk of a privilege escalation through this channel. Unfortunately, CircleCI is not current capable of automatically rotating keys on these events. As well, there is no simple way to manually rotate either these types of keys.

In order to rotate the “Deploy Key”, it appears that a user needs to:

• Remove/“Stop Building” the project in CircleCI
• Manually remove the Deploy Key from the Repository in Github
• Wait ~10 minutes (this is how long it took during my tests) until the option to “Start Building” the repository is available again (this looked like an internal cache issue unrelated to security)
• Hit the “Start Building” button again to re-add the repository.

Note that during the waiting time, new PRs and code changes will not build.

In order to rotate the “User SSH Key”, you will need to:

• Delete the existing “User SSH Key” from the CircleCI Repository Settings page
• Find and Remove the corresponding Key from your Github Account Settings page
• Authorize your Github Account with CircleCI (the button will appear after removing the previous key)
• Click to add a new User Key

Note that during the time that the User Key is not on the account, the project will build with the Deploy Key instead, which may fail.

In either of these situations, CircleCI should:

• have a 1-click option to rotate any key without impacting the running system.
• automatically revoke Keys from Github when they are removed from Circle

Public Key/Secret Injection Toggle

CircleCI is a common platform for building and managing open source projects as well. They have a set of documentation specifically geared towards building an Open Source Project on CircleCI (https://circleci.com/docs/2.0/oss/). In today’s world many of these projects are pulled into an unbelievable number of places, and run on an unbelievable number of systems. Compromise of these projects can have wide reaching ramifications on systems all over the world.

A majority of open source projects hosted on Github accept Pull Requests publicly, and those that use Circle often enable CircleCI to build (typically run tests) for PRs from repository Forks. For security purposes, sensitive information (User SSH Keys Deployment Keys, (Application-specified) Environment Variables, and AWS credentials) are not injected into PR-from-Fork builds. The logic here is simple: it would be way to easy to:

• Fork either repository
• Modify the .circleci/config.yml to include malicious code
• Submit a PR back into the main repository
• That new code will automatically execute within the CircleCI context, allowing the private environment variables to be exfiltrated (either via logs, or via the network)

If this works, and the PR was immediately closed, many open source projects have enough volume, or are lacking in sources, that the malicious run could easily go unnoticed.

All CI/CD platforms that support open source projects (that I have seen) have a way to prevent this. CircleCI’s way is through the two options on the “Advanced Settings” page in Repository settings: Build forked pull requests, and Pass secrets to builds from forked pull requests. This is a perfectly acceptable solutions, however in CircleCI’s case, the risk is not entirely eliminated. This is due to numerous UX issues around these toggles. When UX issues cause security mistakes, I consider them a security issue. In fact, even when an organization or project has these properly enabled, the risk still exists. Below are 5 UX issues which contribute to an increased risk of key exposure on CircleCI.

1. What-Should-Be-Invalid States

With the combination of the two settings, and the public/private state of a repository, below are the following states you can have.

+---------+--------------+-----------+-------------------------------+
|         |     Build    |   Pass    |             Safe?             |
|         |  Forked PRs  |  Secrets  |                               |
+---------+--------------+-----------+-------------------------------+
| Private | False        | False     | Safe                          |
| Public  | False        | True      | Safe (but invalid)            |
| Private | True         | False     | Safe                          |
| Public  | True         | True      | **Extremely Unsafe**          |
+---------+--------------+-----------+-------------------------------+

The final state, where both settings are True, leaves the repository in a very dangerous state, where all of the sensitive information stored in CircleCI is, for all intents and purposes, public. While I recognize that CircleCI has seen more CD requirements than I have, I quite frankly cannot imagine a single situation where building forked PRs, and injected sensitive configuration into the builds, is a valid configuration. Both settings being True should not be a valid configuration. If a project needs that, it seems like the (effectively public) configuration should just live in the .circleci/config.yml file, rather than living in the CircleCI application.

Apologies, rant over, lets get back to UX issues.

2. Lack of Safe Defaults

The defaults around these options took a second to figure out, but from my testing it appears to be as follows for private/public repositories (flag names abbreviated):

+------------------+---------+--------+
|                  | Private | Public |
+------------------+---------+--------+
| Build Forked PRs | False   | False  |
| Pass Secrets     | True    | False  |
+------------------+---------+--------+

Note that for a Private Repository, the default for Pass secrets to builds from forked pull requests appears to be True. First, this makes no sense, as having this setting set True if you are not even building forked pull requests is unnecessary. Second, This makes it very easy to see a situation where:

• A new project is begun as a private repository
• The Open Source Project is “Launched” and flipped to public
• Build forked pull requests is activated (since it’s now an open source project and this makes sense)
• The dangerous Pass secrets to builds from forked pull requests is left as True

The default for this setting should always be false.

3. Lack of Visual Feedback

In the event that a repository is in the dangerous configuration (I will refer to this going forward as the True/True configuration), there is no visual feedback on the page of the dangers. There is a text note as part of the field description pointing out the dangers, it is not visually capturing, and is easily missed. Further more, this documentation explicitly calls out that in some situations this feature “isn’t for you”. CircleCI has all of the required information, and should be able to detect and block this feature in those cases. Here is a screenshot:

From just a quick glance, it is not clear that this configuration would allow any arbitrary person on the internet to publish new versions of your project, or deploy malicious code to your servers. For a setting that is explicitly called out in the documentation as being dangerous, it should be more clear when it is turned on.

4. Lack of Confirmation

For a setting as dangerous as this one, there should be a confirmation box with a clear warning to the potential side-effects of enabling the option. One could go as far as suggesting that the user re-authenticate before changing settings as sensitive as these ones (especially if the Repository is public, which CircleCI knows). Instead, this is implemented as a simple radio box with no confirmation, or feedback of a change. Accidentally pressing to hard on your track pad at the wrong time while looking through the repository settings pages can inadvertently expose sensitive credentials for your project to the world.

5. Lack of Notification

In the event that either of these security-impacting settings are changed, CircleCI should be sending notifications to the repository administrator, as well as the user that made the change. This would allow oversight to the possibility of the setting being changed either accidentally, maliciously, or through an account compromise. In the event that a CircleCI user has disabled build notifications, security-related notifications should still be sent.

6. Lack of Audit Log

In the event that either of these security-impacting settings are changed, there should be an audit log of who made the change, and the ability to view this history. This would allow an organization who finds themselves exposed to determine how it happened and appropriately evaluate their risk. By knowing a time range, it would be possible to inspect all builds from the time in question to determine if they were compromised. I have not seen any evidence of an audit log recording changes to these security settings, and would be curious if CircleCI has one internally.

What is the impact?

In my research I was not able to confirm any Open Source Projects in the wild which had this invalid configuration - but that was mostly because I didn’t look. The vulnerable projects would be public Github projects which openly accept PRs. In theory, determining a list of vulnerable projects would be as simple as:

1. Search github.com for any public repository containing a .circleci/config.yml file
2. Fork the repository, and modify the CircleCI config to exfiltrate the secret values (environment variables, user keys, deployment keys, and AWS configuration). These are the values that are documented to be excluded from “public” builds without this option enabled
3. Create a PR against the master project
4. Maybe quickly delete the PR so no one notices
5. Check the exfiltrated data for leaked values

Due to the fact that this can be run from a collection of new/anonymous Github accounts, any project with both of these options enabled is at risk.

Can this be worse?

Note: This section is mostly speculation, and has not been fully investigated. I hope I’m not getting to tin-foil-hat-y here

Yes. When chaining these issues with previously documented common security risks in dashboards (in particular, this great write-up investigating the risks of including analytics JS in your application dashboards), it would be possible for a malicious or compromised analytics library, JS dependency, or browser extension to maliciously modify CircleCI Repository security settings to expose sensitive information. Many of these “analytics” tools go as far as allowing arbitrary code injections from the platform (such as Google Tag Manager or Optimizely), which could allow a compromised CircleCI account at one of these vendors to conduct a similar attack.

A spearfishing-style approach could be used to target a single repository, hoping to go unnoticed. When considering UX issues #3 and #5 above, this malicious setting could stay in place for an extended time without anyone noticing. Further, without an audit log or notifications, malicious code could reset the settings after the sensitive information has ben exfiltrated and eliminate the possible of anyone noticing the historical change.

Does Github make this problem worse?

In my opinion, the Github “Social Network” model greatly increases the risk associated with these types of vulnerabilities. This model encourages an individual to have a single account, used as their “developer identity”, which is shared across work, pet projects, side hustles, and open source contributions.

At the same time, Github does not offer granular enough access control to allow secure boundaries to be enforced across this domain. In general, if your Github account is compromised, everything is compromised. While it is the specific implementation details of CircleCI’s build process that allows these keys to be exposed, at least some of the blame lies on Github for not giving CircleCI (or other providers) the ability to create User keys which access only specific repositories.

Note: I understand that Github Apps are a more appropriate solution to this access control problem at an Application level. Even with their existence, I should still, as an individual user, be able to create an SSH key on a machine which only has access to repositories in a specific organization, or a specific set of repositories.

What can be done?

Unfortunately, one of the main impacts of the issues is that you are at risk of having secret information spill out of CircleCI based on employee/contributor activity outside of your organization.

Enforce Boundaries

Enforce that your organization code can only be accessed by an Organization-specific Github Account. In order to effectively enforce this, you will likely have to use Github Enterprise.

This will only protect you from a compromise of an individual “Github Social Network” account. It will effectively quarantine you, preventing a chain of compromises from reaching your repositories. It will not protect you from privilege escalation within your organization.

Use Machine Users

Machine users are a pain, but can be used to effectively prevent privilege escalation within your organization. They will not protect you from having your code/secrets compromised by a compromised employee account.

By combining Machine Users with Enforced Boundaries, you should be fairly protected from a privilege escalation through the Exposed Keys channel.

If your organization manages both public and private repositories, you should consider separating the open source projects into a separate Organization, using only Machine Users for access. This gives a hard boundary to prevent a public configuration error or credential leak from impacting any of your private repositories.

In order to protect yourself from exposed secrets of other types (AWS credentials, environment variables, deployment keys, etc), the best way is to simply remember that anything you put in there should be considered public to your entire organization, including the case that they have any accounts get compromised. Anything you put in there should be limited in scope and access.

Contexts can be more tightly locked down (you can control who is allowed to use a given context), but unfortunately only a single context can be used in a build step at once (ie, you can’t combine or overlap Contexts).

Signed Commits

Begin requiring your team to sign their commits, and configure your CI system to only run builds on signed commits. This is especially important if your organization uses a GitOps workflow. Building only signed commits will prevent malicious code from being executed even in the event that a team members SSH Key is exposed. Unfortunately this is not possible is CircleCI, and has been a feature request for over 3 years.

This requirement (only running on signed commits) should, if possible, be configured outside of the in-repository configuration in order to prevent a malicious actor from modifying this security setting.

A Final Note

These are not new risks, and I am not the first to discover this type of risk. CI/CD systems are common access points and for many reasons, are generally a common risk vector for an organization.

Most of the patterns discussed are common for many CI/CD systems, and these risk vectors are not limited to CircleCI. Similar risks could be found on Travis CI, Codeship, Drone, or your self-hosted Jenkins machine. There was a good talk at Defcon25 (2017) which focussed on many of the underlying issues at hand here. I encourage you to explore these similar risks in your own set ups, and take the necessary actions to protect both yourself and your organizations.

It is unfortunately that in this situation the tight integration with the “Github Social Network”, and the use of User SSH Keys rather than application-managed tokens, increases the potential surface area that a compromise can impact. The User permissions of reachable tokens overlap in an often limitless and unpredictable fashion, which gives way to the potential for privilege escalation to cascade beyond the walls of your particular organization.