This case study shows how Bazel maintainers proactively invested in fortifying their GitHub Actions workflows against CI/CD supply chain attacks.
Industry: Technology
Runners: GitHub-Hosted
On Feb 1, 2024, security researchers detailed how a potential CI/CD supply chain vulnerability could have compromised the Bazel project. Bazel is a fast, scalable, and extensible build system from Google.
An attacker could have exploited this command injection vulnerability to exfiltrate the GITHUB_TOKEN from Bazel’s CI/CD environment. In this case, the GITHUB_TOKEN had read/write permissions to the project’s repository and could have been used to add a backdoor to Bazel releases/source code, leading to an industry-wide supply chain attack.
Bazel maintainers had already invested in fortifying their GitHub Actions workflows against such CI/CD supply chain attacks. The following sections show how the maintainers have been using StepSecurity’s Platform to secure their GitHub Actions workflows.
Bazel has been using StepSecurity Harden-Runner since September 2023. Harden-Runner enables outbound network control and runtime security for GitHub-hosted and self-hosted runners. It has been purpose-built to detect and prevent these kinds of CI/CD security attacks.
Because these workflows were using Harden-Runner, the outbound DNS and network calls for each and every workflow run since then were meticulously monitored and logged. This not only enabled real-time detection of malicious activities but also provided invaluable data for forensic analysis.
Here you can see the Bazel Project’s vulnerable workflow using the Harden-Runner GitHub Action.
The Harden-Runner GitHub Actions installs an agent on the runner that then monitors all outbound DNS and network calls. The link below shows the network events for a run of this workflow.
https://app.stepsecurity.io/github/bazelbuild/bazel/actions/runs/7741733879
You can see that the outbound DNS and network traffic is correlated with each step of the job. Since Bazel is a public repository, these insights are visible to everyone. When Harden-Runner is used on private repositories, these insights are only accessible to those who have access to the repository.
StepSecurity aggregates the outbound calls for each job and makes the list available in the Recommended Policy tab so a block policy can be applied. The screenshot below shows us that in the last 244 runs of this job, outbound calls were made to expected destinations. This confirms that the workflow hasn’t communicated with any malicious endpoints and this vulnerability hasn’t been exploited. This information is critical to investigate if the vulnerability was exploited to exfiltrate the CI/CD credentials to an attacker-controlled destination.
In addition, StepSecurity creates a baseline of network events for each job based on the first few runs of the job. If there was a new destination that was not in the baseline, it would have shown up as an anomalous request on the dashboard.
While this Bazel project was using Harden-Runner in the audit mode, it is also possible to set a block policy which would have actively blocked outbound calls that are not in the allowed list both at the DNS and the network layers.
StepSecurity also enables you to view a list of all unique destinations to which outbound calls have been made across all workflow runs in the Bazel project. This makes it easy to periodically review the list of destinations to spot any suspicious calls across all workflow runs.
https://app.stepsecurity.io/github/bazelbuild/actions/all-endpoints
One of the reasons the potential impact of this command injection vulnerability was high was because the GITHUB_TOKEN in this workflow had elevated permissions.
StepSecurity’s platform also suggests and applies minimum permissions for the GITHUB_TOKEN across your workflows using automated pull requests.
The Bazel project had in the past used StepSecurity to apply these best practices as can be seen from the following pull request from April 2023. This pull request set the minimum GITHUB_TOKEN permissions for 4 GitHubActions workflow files in the Bazel project. The affected workflow was unfortunately not present in the Bazel project at that time, else it would also have had the least privileged GITHUB_TOKEN.
https://github.com/bazelbuild/bazel/pull/18264
Kudos to Bazel maintainers for their foresight in securing their CI/CD pipelines to protect a widely used open-source software.
This case study shows how StepSecurity’s GitHub Actions Security platform can defend CI/CD pipelines against known as well as unknown security vulnerabilities. This is why several enterprises and over 2,300 open-source projects including Bazel trust StepSecurity to harden their GitHub Actions runners and apply GitHub Actions security best practices using automation.
CISA Enforces Network Egress Control and CI/CD Infrastructure Security to Harden their GitHub-hosted Runners
CISA’s case study talks about how it leverages StepSecurity Harden-Runner 's network egress control and runtime security in over 175 GitHub repositories to prevent Codecov and SolarWinds-style attacks.
