Lesson 6 of 8 · 11 min · Pipelines in practice
Pipeline secrets and securing the delivery chain
What you will learn
- Why pipelines are attractive attack targets
- Where to keep secrets and how to limit their reach
- Short-lived credentials and OIDC instead of long-lived keys
- Detecting secrets in Git history, and what to do when one leaks
- Hardening the pipeline: permissions, untrusted code, dependencies
In simple words
A pipeline holds the keys to production: cloud passwords, access tokens, signing keys. That makes it a target for attackers. This lesson shows where to keep those keys, how to use keys that expire within minutes instead of lasting for years, and what to do the moment one leaks.
Words to know
- Secret: a password, token or key that the pipeline needs.
- Secret store: a protected place for secrets, kept apart from the code.
- Short-lived credential: access that stops working after a short time, so a leaked copy is soon useless.
- OIDC: a way for the pipeline to prove who it is and receive a short-lived credential, with no stored password.
Why the pipeline is a target
A CI/CD system can read your source code, hold credentials for registries and cloud accounts, and push code to production. Compromise the pipeline and an attacker can ship malware to every user without touching production servers. Several famous supply-chain incidents were exactly this: a stolen CI token, a poisoned dependency, a modified build script. So the pipeline needs the same care as production.
Where secrets live (and where they must not)
Secrets are credentials: API keys, passwords, tokens, private keys, certificates. They must never be in:
- the Git repository (including
.env, config files, test fixtures, notebooks), - Dockerfiles (
ENV,ARG), images, or build logs, - CI YAML files, or arguments on the command line (visible in process lists),
- chat, tickets, or screenshots.
Proper homes:
| Place | Use |
|---|---|
| CI secret stores (GitHub Actions secrets, GitLab CI variables, Jenkins credentials) | credentials the pipeline itself needs, scoped per repository/environment |
| Cloud secrets managers (AWS Secrets Manager / SSM Parameter Store, Azure Key Vault, Google Secret Manager) and HashiCorp Vault | secrets the application needs at runtime; central rotation and audit |
| Kubernetes Secrets (ideally from an external store via CSI driver or External Secrets) | runtime secrets in clusters |
| Encrypted files in Git (SOPS, sealed-secrets) | GitOps workflows: only the ciphertext is committed |
CI stores mask secret values in logs, but masking is a courtesy: a script that prints a transformed value (base64, reversed, split) or sends it to an attacker's server defeats it. Treat a secret given to a job as readable by whoever controls that job's code.
Scope it down
- Per environment: production secrets only in the
productionenvironment, usable only by jobs that target it, only from protected branches, with required reviewers. - Least privilege: the deploy credential may deploy to one service, not administer the account. The registry token pushes to one repository.
- Separate credentials per pipeline and per environment so one leak is contained and revocable.
- Do not give secrets to pull requests from forks or to jobs that run untrusted code. Platforms withhold them by default; do not override that.
Short-lived credentials: OIDC
The best secret is the one that does not exist. Instead of storing a long-lived cloud key in CI, use OpenID Connect (OIDC) federation: the CI job asks its platform (GitHub, GitLab) for a signed identity token ("I am job X of repository R on branch B in environment E"); the cloud provider is configured to trust that issuer and to allow that specific identity to assume a role; it returns temporary credentials valid for about an hour.
Benefits:
- No access key sits in the CI secret store, so there is nothing to steal, rotate or accidentally print.
- Trust is bound to repository, branch and environment through conditions in the role's trust policy (for example
repo:myorg/shop:environment:production). - Every use is logged against a role session in CloudTrail.
We validated the GitHub workflow fragment in the previous lesson (permissions: id-token: write and aws-actions/configure-aws-credentials with role-to-assume). The same pattern exists for Azure, Google Cloud, HashiCorp Vault and many registries. For anything that must be a stored secret, prefer short expiry and automated rotation.
Finding secrets already in Git
Developers sometimes commit a key. Deleting the file in the next commit does not remove it: it stays in history for anyone who clones. We reproduced it in a scratch repository:
printf 'AWS_ACCESS_KEY_ID=AKIA...\nAWS_SECRET_ACCESS_KEY=...\n' > .env
git add .env && git commit -m "add config"
git rm --cached .env && echo ".env" > .gitignore && git add .gitignore && git commit -m "remove env"
git show HEAD~1:.env # the key is still there
A secret scanner reads the whole history. gitleaks found it:
docker run --rm -v "$PWD:/repo" zricethezav/gitleaks:latest git /repo --no-banner --redact
INF 2 commits scanned.
INF scanned ~107 bytes (107 bytes) in 40.8ms
WRN leaks found: 1
(The scanner exits with a non-zero code when it finds leaks, which fails a CI job.) Note: AWS's documented example key AKIAIOSFODNN7EXAMPLE was not flagged, because scanners deliberately ignore well-known placeholder values; our test used a realistic fake key. Where to run scanners:
- Pre-commit hook on the developer's machine (stops it before it exists in history).
- CI check on every pull request and a scheduled scan of full history.
- Platform features: GitHub secret scanning and push protection block known token formats and notify the provider.
Also scan images and artifacts (Trivy has a secret scanner) and logs.
When a secret leaks: the response
- Revoke / rotate immediately. Assume it is already compromised. This is the only step that actually fixes the problem.
- Check the provider's logs (CloudTrail, audit logs) for use since the leak; look for new resources, users or keys created by the attacker.
- Remove it from the code and, if the repository is public or widely shared, rewrite history (
git filter-repo) and force clones to update, understanding this does not undo the exposure. - Find the cause (why was it committed?) and add a control (hook, scanner, template).
- Write a short incident note.
Hardening the pipeline itself
- Least-privilege tokens: explicit
permissions:in GitHub Actions, read-only by default; deploy roles limited to what is needed. - Protect the default branch and the pipeline definition: required reviews on changes to workflows and Jenkinsfiles (
CODEOWNERS), because whoever edits the pipeline controls the secrets. - Untrusted input: pull request titles, branch names and issue text are attacker-controlled. Never splice them into shell commands (the actionlint finding in the GitHub Actions lesson). Be very careful with triggers that give fork code access to secrets (
pull_request_target). - Dependencies and actions: pin third-party actions to a commit SHA, lock dependencies, use a private artifact proxy, run Dependabot or Renovate, review new dependencies. A malicious or hijacked dependency runs in your build.
- Isolated, ephemeral runners: a fresh container or VM per job so nothing leaks between jobs; do not share self-hosted runners between trust levels; do not run builds on the CI controller.
- Separate build and deploy credentials and separate who may approve what.
- Sign and verify artifacts so production runs only what your pipeline built.
- Audit logs and alerts on credential use, new workflow files, changed repository settings.
- Network egress control from build runners where practical, so a compromised build cannot send data anywhere.
Common mistakes
- Long-lived cloud keys in CI secrets shared by every repository.
- One all-powerful credential for every environment.
- Assuming masking protects secrets.
- Echoing environment variables for debugging (
env,set -x). - Deleting a committed secret without rotating it.
- Unpinned third-party actions and unreviewed pipeline changes.
- Secrets for pull requests from forks.
Try it yourself
- Create a throwaway repository, commit a fake realistic key and remove it. Show it is still in history, then find it with gitleaks.
- Set up a pre-commit hook that runs gitleaks on staged changes.
- Write down which of your pipeline's credentials could be replaced by OIDC or shorter-lived tokens.
- Draw which jobs of a pipeline need production credentials, and make sure only those get them.
Check yourself
Why isn't it enough to delete a committed secret in a later commit?
Git history preserves the earlier commit, so anyone with a clone (or the remote) can still read it. The secret must be revoked and rotated; rewriting history is cleanup, not a fix.
Why is OIDC better than storing an access key in CI?
The job receives short-lived credentials for a role after proving its identity, so there is no long-lived key to leak, and trust can be limited to a specific repository, branch and environment.
Does masking in CI logs make secrets safe to use in any script?
No. Masking only hides exact matches in logs. Code in the job can transform or exfiltrate the secret, so secrets must only be exposed to trusted code.
What belongs in a secrets manager rather than in CI variables?
Secrets the running application needs: database passwords, API keys, certificates. They need central rotation and audit, and should be fetched at runtime using the workload's identity.
Interview questions
How do you handle secrets in a CI/CD pipeline end to end?
Keep them out of Git and images, store pipeline credentials in the CI secret store scoped per environment, prefer OIDC and short-lived credentials, fetch runtime secrets from a secrets manager at start-up, scan for leaks in pre-commit and CI, rotate regularly, and have a revoke-and-rotate response ready.
A pull request modifies the CI configuration to print environment variables. How do you prevent damage?
Do not expose secrets to pull-request builds from forks or unreviewed branches, require review (CODEOWNERS) for pipeline changes, scope secrets to protected environments and branches, and use short-lived credentials.
What is a software supply-chain attack and how do you reduce the risk in CI?
An attacker compromises something you trust: a dependency, build tool, action, or the build system itself. Reduce it with pinned and reviewed dependencies, an artifact proxy, isolated ephemeral runners, least-privilege credentials, signed artifacts and verification, and monitoring.
Summary
- The pipeline holds the keys to production and must be hardened as such.
- Keep secrets out of code, images and logs; scope them per environment; prefer OIDC and short-lived credentials.
- Scan history and artifacts for secrets; if one leaks, revoke and rotate first.
- Restrict pipeline permissions, protect pipeline definitions, treat PR input as hostile, pin dependencies, isolate runners, sign artifacts.
Going deeper
- GitHub Actions: security hardening: secrets, permissions, injection and pinning.
- GitHub secret scanning and push protection: blocking known tokens.
- OWASP CI/CD Security Cheat Sheet: a thorough checklist.
- gitleaks: the scanner used above.
- AWS: using IAM roles with OIDC identity providers: the cloud side of keyless access.
See how much of this you can explain
A free mock interview asks you new questions on CI/CD pipelines and releases for DevOps Engineer and scores every answer, with what to fix.