How Harden Runner Detected the Sha1-Hulud Supply Chain Attack in CNCF's Backstage Repository

Introduction

On November 23-24, 2025, the npm ecosystem experienced one of its largest coordinated supply chain attacks. Dubbed "Sha1-Hulud: The Second Coming" this attack compromised over 780 npm packages including packages from Zapier, ENS Domains, PostHog, Postman, and AsyncAPI. The malicious code targeted CI/CD environments, harvesting credentials and attempting to establish persistent backdoors.

Among the thousands of CI/CD pipelines that ran these compromised packages were those of Backstage, the Cloud Native Computing Foundation's open-source developer portal platform created by Spotify, with over 32,000 GitHub stars and widespread enterprise adoption.

StepSecurity Harden Runner detected this attack across all repositories where it was monitoring workflow runs. Because Backstage is a prominent open source project using Harden Runner's free community tier (which makes insights publicly accessible), we're using it as a case study to explain how baseline-driven anomaly detection identified the attack and what this means for securing CI/CD pipelines.

What Happened

During the attack window, nine GitHub Actions workflow runs in the `backstage/backstage` repository executed compromised npm packages. The affected jobs were part of Backstage's E2E test workflow, specifically during the step that runs:

- name: Run E2E test
  run: |
    sudo sysctl fs.inotify.max_user_watches=524288
    yarn e2e-test run

When `yarn e2e-test run` resolved and installed dependencies, it fetched the latest versions of certain packages since the package does not have a lock file. During the attack window, those latest versions happened to be compromised. The malicious `preinstall` scripts executed immediately, triggering the Sha1-Hulud payload.

How Harden Runner Detected the Attack

Backstage uses Harden Runner's community tier, which is free for open source projects. The Backstage maintainers have been using Harden Runner since October 2023, demonstrating their proactive approach to implementing security best practices for their CI/CD pipelines. Harden Runner operates as an EDR (Endpoint Detection and Response) agent for GitHub Actions runners, monitoring runtime behavior including all outbound network connections.

Baseline-Driven Anomaly Detection

Harden Runner builds a baseline of expected network destinations for each workflow. When a workflow run makes connections to endpoints outside this baseline, it triggers an anomaly alert.

During the Sha1-Hulud attack, the compromised packages initiated connections to endpoints that had never appeared in Backstage's E2E workflow history:

These connections immediately stood out in Harden Runner's network telemetry. They were not part of the established baseline for Backstage's E2E tests, which typically connect to npm registries, GitHub APIs, and test infrastructure—not to executable download endpoints.

With Harden Runner's block mode enabled, these connections would have been blocked entirely. The malware would have been unable to download Bun or TruffleHog, preventing the attack from progressing beyond the initial compromise.

Verifiable Evidence

Because Backstage uses the community tier of Harden Runner, the insights for these workflow runs are publicly accessible. Anyone can verify the detection:

- Run 19625457065 - E2E Linux Jobs (https://app.stepsecurity.io/github/backstage/backstage/actions/runs/19625457065)

- Run 19631408557 - E2E Linux Jobs (https://app.stepsecurity.io/github/backstage/backstage/actions/runs/19631408557)

- Run 19632292852 - E2E Linux Jobs (https://app.stepsecurity.io/github/backstage/backstage/actions/runs/19632292852)

‍

Explore this interactive demo to see how StepSecurity Harden-Runner detected the attack:

When you visit these insights pages, you'll see the malicious connections clearly highlighted. Destinations that were new in these runs—such as `bun.sh`, `oss.trufflehog.org`, and `keychecker.trufflesecurity.com`—are displayed in orange with "Anomalous" tags, immediately distinguishing them from expected network activity.

You can also view the baseline that Harden Runner built for this workflow, including how many previous runs the baseline is based on. This makes it easy to understand why certain connections triggered alerts: they simply never appeared before in the workflow's history.

Deeper Visibility with Harden Runner Enterprise Tier

While the community tier detected this attack through network anomaly detection, Harden Runner's enterprise tier provides additional capabilities that offer even deeper visibility into supply chain attacks.

HTTPS Monitoring and Process Telemetry

The enterprise tier includes: ‍

- HTTPS monitoring: Full visibility into HTTPS requests, not just connection endpoints

- Process events: Complete process execution telemetry including process tree and command-line arguments

- Real-time notifications: Immediate alerts via email, Slack, S3, and webhook integrations when anomalies are detected

Detecting the Persistence Mechanism

The Sha1-Hulud malware attempted to establish persistence by registering a self-hosted GitHub Actions runner named "SHA1HULUD" on affected repositories. This would have given attackers the ability to execute code remotely via repository discussion events.

With the enterprise tier's HTTPS monitoring, this persistence attempt was directly observable—the malware made HTTPS calls to the GitHub API to register the runner. This provides additional evidence of compromise beyond the initial payload download, and reveals the full scope of what the malware was attempting to do.

You can see this in action in our enterprise demo, where `Runner.Listener` made a POST call to `/actions/runner-registration` on `api.github.com`. This triggered a detection even without a baseline—workflows typically do not register runners, making this behavior inherently suspicious.

Explore this interactive demo to see how StepSecurity would have detected this attack for enterprise customers:

Real-Time Alerting

For enterprise customers, these detections trigger immediate notifications through configured integrations. Rather than discovering a compromise during a post-incident review, security teams are alerted in real-time as anomalous behavior occurs—enabling rapid response and containment.

Responsible Disclosure and Impact Assessment

We reported our findings to the Backstage maintainers, who reviewed the detection data and assessed the impact. Their analysis concluded that the affected E2E workflow only had access to a read-only GitHub token with no additional secrets configured. As a result, there was no impact from the compromise on the Backstage repository.

We want to acknowledge the Backstage maintainers for their quick response and support in investigating this incident. Their collaboration enabled us to rapidly confirm the detection findings and assess the impact.

From Detection to Prevention

While detecting the Sha1-Hulud attack provided valuable visibility, Harden Runner's block mode can actively prevent such attacks from succeeding.

In this case, Backstage was running Harden Runner in audit mode, which monitors and alerts on anomalous connections but does not block them. If block mode had been enabled, the connections to `bun.sh` and `oss.trufflehog.org` would have been blocked at the network level. The malware's `preinstall` script would have executed, but it would have been unable to:

- Download the Bun runtime needed to execute its obfuscated payload

- Download TruffleHog to scan for secrets

- Exfiltrate any discovered credentials without the payload

The attack would have been stopped before any damage could occur. For workflows with access to sensitive secrets, this is the difference between a security alert and a security incident.

Why Runtime Monitoring Matters

This case study validates a fundamental principle: you cannot rely solely on pre-deployment security checks to catch supply chain attacks.

Traditional security approaches focus on:

- Scanning dependencies for known vulnerabilities (CVEs)

- Reviewing lock files for unexpected changes

- Using allowlists for npm packages

These approaches fail against zero-day supply chain attacks like Sha1-Hulud because:

1. The compromised packages had no CVEs at the time of attack

2. Many projects do not use lock files, and in many cases the latest versions of packages are installed

3. The compromised packages were legitimate, trusted packages that had been hijacked

Runtime monitoring provides the missing layer. By establishing a baseline of expected behavior and alerting on anomalies, Harden Runner detected the attack based on what the code actually did—not what it claimed to be.

This isn't the first time Harden Runner has detected supply chain attacks. Previous detections include:

- tj-actions/changed-files compromise: A popular GitHub Action with 23,000+ dependent repositories was compromised

- @aspect-build/rules_js compromise: A popular Nx build system package was compromised with data-stealing malware

- ctrl-tinycolor and 40+ npm packages: Mass compromise of npm packages detected through runtime analysis

In each case, runtime monitoring identified the attack through anomalous behavior that traditional security tools missed.

‍

Recommendations

Based on this incident, we recommend:

For Open Source Projects

1. Enable Harden Runner on your GitHub Actions workflows. The [community tier](https://github.com/step-security/harden-runner) is free for public repositories.

2. Review your workflow permissions. The principle of least privilege limited Backstage's exposure.

3. Consider enabling egress policy enforcement. Harden Runner can block connections to endpoints outside your baseline, stopping attacks in real-time.

For Enterprises

1. Implement runtime monitoring on your CI/CD workflows to gain visibility into supply chain attacks.

2. Treat CI/CD runners as high-value targets. They often have access to production credentials and deployment capabilities.

3. Monitor for anomalous behavior, not just known threats. Zero-day attacks require behavioral detection.

4. Assume that your dependencies can be compromised. Build your security posture around this assumption.

‍

Conclusion

The Sha1-Hulud attack compromised over 780 npm packages and affected thousands of repositories. In the Backstage repository, Harden Runner detected the attack through baseline anomaly detection—identifying network connections to `bun.sh` and `oss.trufflehog.org` that had never appeared in the workflow's history.

This detection wasn't based on signatures or CVEs. It was based on observing that the workflow suddenly started doing something it had never done before.

For Backstage, the limited workflow permissions meant no impact. But the attack serves as a warning: supply chain attacks are increasingly sophisticated, and without runtime monitoring, you may never know your CI/CD pipelines were compromised.

Runtime monitoring provides the visibility needed to detect these attacks. If you're not monitoring what your CI/CD runners are actually doing, you're flying blind.

To learn more about Harden Runner and how it protects CI/CD pipelines from supply chain attacks, visit stepsecurity.io or explore the GitHub Action.

For technical details on the Sha1-Hulud malware and the full list of compromised packages, see our technical analysis‍