GCP – Holes in Your Bitbucket: Why Your CI/CD Pipeline Is Leaking Secrets

Written by: Mark Swindle

While investigating recent exposures of Amazon Web Services (AWS) secrets, Mandiant identified a scenario in which client-specific secrets have been leaked from Atlassian’s code repository tool, Bitbucket, and leveraged by threat actors to gain unauthorized access to AWS. This blog post illustrates how Bitbucket Secured Variables can be leaked in your pipeline and expose you to security breaches. 

Background

Bitbucket is a code hosting platform provided by Atlassian and is equipped with a built-in continuous integration and continuous delivery/deployment (CI/CD) service called Bitbucket Pipelines. Bitbucket Pipelines can be used to execute CI/CD use cases like deploying and maintaining resources in AWS. Bitbucket includes an administrative function called “Secured Variables” that allows administrators to store CI/CD secrets, such as AWS keys, directly in Bitbucket for easy reference by code libraries. 

CI/CD Secrets: CI/CD Secrets serve as the authentication and authorization backbone within CI/CD pipelines. They provide the credentials required for pipelines to interact with platforms like AWS, ensuring pipelines possess the appropriate permissions for their tasks. Secrets are often extremely powerful and are beloved by attackers because they present an opportunity for direct, unabated access to an environment. Maintaining confidentiality of secrets while balancing ease of use by developers is a constant struggle in securing CI/CD pipelines. 

Bitbucket Secured Variables: Bitbucket provides a way to store variables so developers can quickly reference them when writing code. Additionally, Bitbucket offers an option to declare a variable as a “secured variable” for any data that is sensitive. A secured variable is designed such that, once its value is set by an administrator, it can no longer be read in plain text. This structure allows developers to make quick calls to secret variables without exposing their values anywhere in Bitbucket. Unless…

Exporting Secrets from Bitbucket in Plain Text

CI/CD pipelines are designed just like the plumbing in your house. Pipes, valves, and regulators all work in unison to provide you with reliable, running water. CI/CD pipelines are a complicated orchestration of events to accomplish a specific task. In order to accomplish this, these pipelines are highly proficient at packaging and deploying large volumes of data completely autonomously. As a developer, this creates countless possibilities for automating work, but, as a security professional, it can be a cause for anxiety and heartburn. Perhaps it’s a line of code with a hardcoded secret sneaking into production. Maybe it’s a developer accidentally storing secrets locally on their machine. Or maybe, as we have seen in recent investigations,  it’s a Bitbucket artifact object containing secrets for an AWS environment being published to publicly available locations like S3 Buckets or company websites. 

Bitbucket secured variables are a convenient way to store secrets locally in Bitbucket for quick reference by developers; however, they come with one concerning characteristic—they can be exposed in plain text through artifact objects. If a Bitbucket variable—secured or not secured—is copied to an artifact object using the artifacts: command, the result will generate a .txt file with the value of that variable displayed in plain text. 

Mandiant has seen instances in which development teams used Bitbucket artifacts in web application source code for troubleshooting purposes, but, unbeknownst to the development teams, those artifacts contained plain text values of secret keys. This resulted in secret keys being exposed to the public internet where they were located and subsequently leveraged by attackers to gain unauthorized access.

Once a secured variable—such as an AWS Key—is copied to a .txt file in plain text, the secret has been leaked, and it’s up to the pipeline as to where that secret flows and how long until an attacker finds it.

Reproducing the Secret Leak

The following are steps to recreate the secret leak in a Bitbucket environment. One important note—the commands detailed in this guide illustrate only one possibility, but there are several other methods that export secured variables to artifacts in Bitbucket. Administrators and developers should closely review any references to artifact objects in their bitbucket-pipelines.yml file or any other files in the repository. 

Establish Secured Variables in Bitbucket

This can be done at the repository level or the workspace level as long as they are set to “secured variable.”

Update the bitbucket-pipelines.yml File to Create an Environment Artifact

The following lines of code execute the command printenv to copy all environment variables from Bitbucket to a .txt file called environment_variables.txt. This is a common practice in development when troubleshooting because developers need to review a wide range of variables for legitimate development purposes. Once the .txt file is created, the code passes it to a Bitbucket artifact object where it can be used by future stages in the pipeline, if necessary.

Navigate to the Pipeline Execution History and Download the Artifact

Open the Artifact and Search for Secured Variables

After exporting the .txt file, secrets can be read in plain text among all the variables in the Bitbucket environment. One note on this step—it is possible you will need to extract components of a .tar file as an additional step here. In this event, extract the .tar file using your data extraction tool of choice.

Secrets Flow Where the Pipeline Goes

Once the secrets are printed to the environment_variables.txt file, they are free to flow out of Bitbucket through the pipeline and become exposed. Any combination of development mistakes, malicious intent, or accidental disclosure can lead to secret exposure and misuse by a threat actor. 

Recommendations

Bitbucket Pipelines is a great platform for storing, collaborating, and deploying code. Bitbucket, however, is not a dedicated secrets manager, and storing secrets directly in Bitbucket introduces opportunities for secrets to be leaked. Safely protect your secrets when using Bitbucket Pipelines by:

Storing secrets in a dedicated secrets manager and then referencing those variables in the code stored in your Bitbucket repository
Closely reviewing Bitbucket artifact objects to ensure they are not exposing secrets as plain text files
Deploying code scanning throughout the full lifecycle of your pipeline to catch secrets stored in code before they are deployed to production

Conclusion

This is not an indictment against Bitbucket. Instead, it’s a case study in how seemingly innocuous actions can snowball into serious problems. We use the word “leak” for a specific reason. All it takes is one keystroke, one line of code, or one misconfiguration for a slow, seemingly untraceable drip of secrets to flow through your pipeline out into the world.