Lesson 5 of 8 · 11 min · Pipelines in practice
Artifacts, caching and versioning
What you will learn
- What an artifact is and what a cache is, and why they are different
- Caching dependencies and Docker layers, with correct cache keys
- Passing files between jobs, and retention
- Artifact repositories and image registries; immutable, traceable versions
- Reproducible builds, SBOMs and provenance
In simple words
A pipeline produces things worth keeping, such as the built application: that is an artifact. It also downloads things it will need again next time, such as libraries: keeping those is a cache. An artifact is the result. A cache is only a shortcut. This lesson shows how to handle both.
Words to know
- Artifact: a file the pipeline produces and keeps, such as a built application or an image.
- Cache: saved downloads that are reused to make the next run faster. Safe to delete.
- Version: a label that identifies exactly one build.
- Immutable: never changed after it is made. A given version always means the same file.
Two different needs
CI runners are usually fresh and empty. Two problems follow:
- Every run would download all dependencies again: slow and wasteful. The remedy is a cache.
- The result of one job (a built package, a test report) must reach the next job or be stored for later. The remedy is an artifact.
| Cache | Artifact | |
|---|---|---|
| Purpose | speed up builds | carry or store build output |
| Correctness | must never matter: a build must work (more slowly) with an empty cache | is the product: later stages and deployments depend on it |
| Contents | downloaded dependencies, compiled objects, Docker layers | binaries, packages, images, test reports, coverage |
| Keyed by | a hash of the inputs (lock file) | the build / commit id |
| Lifetime | evicted when old or unused | kept deliberately (days to forever) |
Rule of thumb: if losing it breaks the pipeline it is not a cache.
Caching dependencies
The cache key decides when a cache is reused. A good key changes exactly when the dependencies change:
- uses: actions/cache@v4
with:
path: ~/.cache/pip
key: pip-${{ runner.os }}-${{ hashFiles('requirements.txt') }}
restore-keys: |
pip-${{ runner.os }}-
keyis an exact match. Including the hash of the lock file (requirements.txt,package-lock.json,pom.xml,go.sum) means a changed dependency list creates a new cache.restore-keysare prefixes used as a fallback: start from the most recent similar cache and add what changed, instead of from nothing.- Setup actions often bundle caching (
actions/setup-pythonwithcache: pip,setup-nodewithcache: npm,setup-javawithcache: maven), so you need no manual step. - Cache the package manager's download cache (
~/.cache/pip,~/.npm,~/.m2,~/go/pkg/mod), not the installed environment, which can be OS- or version-specific. - Use lock files and commands like
npm ci, so builds are deterministic, not "whatever is newest".
Dangers: a cache that includes build outputs may hide a broken clean build ("works only with cache"); a poisoned cache (written by an untrusted pull request) could inject malicious files into later trusted builds, so platforms isolate caches by branch and do not let fork pull requests write to the main branch's cache. Run a clean build without cache periodically (nightly).
Docker layer caching
Docker reuses layers whose instruction and inputs are unchanged. We measured it in the Docker course: the second build of an unchanged project, and a build after editing only app.py, showed CACHED on the earlier steps (including the slow pip install). That only works if the cache exists on the machine. On an empty CI runner, every build starts cold, so export and import the cache:
cache-from: type=gha
cache-to: type=gha,mode=max
(or type=registry,ref=registry.example.com/team/shop-api:buildcache, or a local/S3 backend). Also order your Dockerfile with dependencies first and code last, so the expensive layers stay valid across commits.
Persistent self-hosted runners keep a local Docker cache automatically but accumulate disk usage and state; prune regularly or use ephemeral runners with a remote cache.
Artifacts between jobs
Within a workflow:
- uses: actions/upload-artifact@v4
with:
name: test-report
path: report.xml
retention-days: 14
# in a later job
- uses: actions/download-artifact@v4
with:
name: test-report
Use them to hand a built package to a deploy job, publish test and coverage reports, keep logs of failed runs, or store release files. Set retention (storage costs accumulate) and never upload secrets. In Jenkins the equivalent is archiveArtifacts and stash/unstash; in GitLab it is artifacts:.
For a container-based workflow the "artifact" between build and deploy is the image in a registry, identified by tag and digest. Rebuilding in the deploy job breaks "build once".
Where artifacts live: repositories and registries
| Artifact type | Typical home |
|---|---|
| Container images | Docker Hub, GitHub Container Registry, Amazon ECR, Google Artifact Registry, Harbor |
| Java (jar/war) | Maven repository: Nexus, Artifactory, GitHub Packages, AWS CodeArtifact |
| JavaScript | npm registry (private: Verdaccio, GitHub Packages, CodeArtifact) |
| Python | PyPI, private index (devpi, Artifactory, CodeArtifact) |
| Generic files, installers | S3 buckets, GitHub Releases |
| Infrastructure modules | Terraform registry, private module repositories, OCI registries |
An artifact repository gives you a single, controlled source of dependencies (a proxy/cache for public registries, protecting you from outages and from malicious or deleted packages) and a place to publish your own builds with retention and access policies.
Versioning and traceability
Every artifact must answer: which source commit and which build produced this, and what is inside?
- Tag images with the commit SHA (
shop-api:sha-1a2b3c4) and/or a semantic version (1.4.2from a Git tag). Add moving tags (1.4,latest) only as conveniences. Deploy by the immutable tag or digest. - Make tags immutable in the registry (ECR has "tag immutability"), so a tag can never silently change.
- Embed metadata: OCI labels (
org.opencontainers.image.revision,.source,.version), and expose the version in the app (/versionendpoint, a build-info log line) so you can see what runs. - Record what was deployed (a deployment log or Git-based GitOps repository) so you can answer "what version was live at 14:05?".
Retention and cost
Registries grow without bound. Set lifecycle policies: keep the last N tagged releases, delete untagged images older than 14 days, delete branch/PR images after merge. Artifacts, caches and logs all cost storage.
Reproducible, verifiable builds
- Pin everything: lock files, base images by version or digest, tool versions, action versions.
- Hermetic where possible: no network access during the build except to the artifact proxy.
- SBOM (software bill of materials): a list of every package in the artifact, generated at build time (
docker buildx build --sbom=true, Syft, Trivy), stored with the image. When a new vulnerability appears you can answer "which of our images contain this library?" immediately. - Provenance and signing: attest how the artifact was built (SLSA provenance) and sign it (cosign / Sigstore). Deployment can then verify that an image was produced by your pipeline and not pushed by hand. Kubernetes admission policies can enforce this.
- Scan the artifact (the Docker course covers Trivy) before promotion.
These steps turn "we think this is what we deployed" into evidence.
Common mistakes
- Treating a cache as an artifact, or depending on a cache for correctness.
- A cache key that never changes (stale dependencies) or changes every run (never hits).
- Rebuilding for each environment instead of promoting one artifact.
- Mutable
latesttags in production. - Unbounded retention and surprise storage bills.
- Uploading build outputs that contain secrets (for example
.envcopied into a package). - No way to map a running version back to a commit.
Try it yourself
- Add dependency caching to a workflow and compare the run time on a cold and a warm cache.
- Build the same Docker project twice in CI with and without a cache backend; compare the times.
- Make your pipeline upload a test report as an artifact on failure (
if: failure()), and download it. - Tag a pushed image with the SHA and a semantic version, and add OCI labels. Show how to find the commit from a running container.
Check yourself
What is the difference between a cache and an artifact?
A cache only speeds things up and can be discarded without breaking the build; an artifact is a deliberate output (package, image, report) that later stages or people rely on.
Why include the hash of the lock file in a cache key?
So the cache is reused while dependencies are unchanged and a new cache is created exactly when they change, avoiding stale or constantly missed caches.
Why use immutable tags and the commit SHA for images?
They make every deployed image traceable to its source and guarantee that a tag always means the same bytes, which makes rollbacks and audits reliable.
What is an SBOM useful for?
It lists the components in an artifact, so when a vulnerability is announced you can quickly find which artifacts are affected.
Interview questions
Your CI takes 25 minutes, mostly installing dependencies and building Docker layers. What do you do?
Cache the package manager's download directory keyed on the lock file, use a Docker build cache backend (registry or GitHub cache), reorder the Dockerfile so dependencies precede source, run jobs in parallel, and keep a nightly clean build to catch cache-dependent breakage.
How do you ensure that what is deployed to production is exactly what was tested?
Build once, publish an immutable artifact tagged with the commit, promote that same artifact across environments with configuration changes only, and verify its digest or signature at deploy time.
What goes wrong with a build cache shared across all branches?
A pull request could write a poisoned or inconsistent cache that a trusted branch later restores. Isolate caches by branch or trust level, and do not let untrusted code write to caches used for releases.
Summary
- Caches speed builds and must never be required for correctness; artifacts are outputs that later stages and deployments depend on.
- Key caches on lock-file hashes; cache download directories and Docker layers (with a remote backend in CI).
- Store artifacts in the right repository or registry with retention rules; tag with commit SHA and semantic versions, make tags immutable, and embed metadata.
- Pin inputs, generate SBOMs, scan, sign and verify to make builds reproducible and trustworthy.
Going deeper
- GitHub Actions: dependency caching: keys, restore-keys and limits.
- Docker build cache backends: registry, GitHub Actions, S3 and local caches.
- SLSA (Supply-chain Levels for Software Artifacts): provenance and build integrity levels.
- Sigstore documentation: signing and verifying artifacts with cosign.
- Amazon ECR: image tag immutability: preventing tag overwrites.
See how much of this you can explain
A free mock interview asks you new questions on CI/CD pipelines and releases for DevOps Engineer and scores every answer, with what to fix.