Lesson 5 of 8 · 11 min · Pipelines in practice

Artifacts, caching and versioning

Covers: Docker BuildKit cache behaviour; GitHub Actions examples. Updated 13 Nov 2026. Part of the DevOps Engineer roadmap.

What you will learn

  • What an artifact is and what a cache is, and why they are different
  • Caching dependencies and Docker layers, with correct cache keys
  • Passing files between jobs, and retention
  • Artifact repositories and image registries; immutable, traceable versions
  • Reproducible builds, SBOMs and provenance

In simple words

A pipeline produces things worth keeping, such as the built application: that is an artifact. It also downloads things it will need again next time, such as libraries: keeping those is a cache. An artifact is the result. A cache is only a shortcut. This lesson shows how to handle both.

Words to know

  • Artifact: a file the pipeline produces and keeps, such as a built application or an image.
  • Cache: saved downloads that are reused to make the next run faster. Safe to delete.
  • Version: a label that identifies exactly one build.
  • Immutable: never changed after it is made. A given version always means the same file.

Two different needs

CI runners are usually fresh and empty. Two problems follow:

  1. Every run would download all dependencies again: slow and wasteful. The remedy is a cache.
  2. The result of one job (a built package, a test report) must reach the next job or be stored for later. The remedy is an artifact.
Cache Artifact
Purpose speed up builds carry or store build output
Correctness must never matter: a build must work (more slowly) with an empty cache is the product: later stages and deployments depend on it
Contents downloaded dependencies, compiled objects, Docker layers binaries, packages, images, test reports, coverage
Keyed by a hash of the inputs (lock file) the build / commit id
Lifetime evicted when old or unused kept deliberately (days to forever)

Rule of thumb: if losing it breaks the pipeline it is not a cache.

Caching dependencies

The cache key decides when a cache is reused. A good key changes exactly when the dependencies change:

- uses: actions/cache@v4
  with:
    path: ~/.cache/pip
    key: pip-${{ runner.os }}-${{ hashFiles('requirements.txt') }}
    restore-keys: |
      pip-${{ runner.os }}-
  • key is an exact match. Including the hash of the lock file (requirements.txt, package-lock.json, pom.xml, go.sum) means a changed dependency list creates a new cache.
  • restore-keys are prefixes used as a fallback: start from the most recent similar cache and add what changed, instead of from nothing.
  • Setup actions often bundle caching (actions/setup-python with cache: pip, setup-node with cache: npm, setup-java with cache: maven), so you need no manual step.
  • Cache the package manager's download cache (~/.cache/pip, ~/.npm, ~/.m2, ~/go/pkg/mod), not the installed environment, which can be OS- or version-specific.
  • Use lock files and commands like npm ci, so builds are deterministic, not "whatever is newest".

Dangers: a cache that includes build outputs may hide a broken clean build ("works only with cache"); a poisoned cache (written by an untrusted pull request) could inject malicious files into later trusted builds, so platforms isolate caches by branch and do not let fork pull requests write to the main branch's cache. Run a clean build without cache periodically (nightly).

Docker layer caching

Docker reuses layers whose instruction and inputs are unchanged. We measured it in the Docker course: the second build of an unchanged project, and a build after editing only app.py, showed CACHED on the earlier steps (including the slow pip install). That only works if the cache exists on the machine. On an empty CI runner, every build starts cold, so export and import the cache:

cache-from: type=gha
cache-to: type=gha,mode=max

(or type=registry,ref=registry.example.com/team/shop-api:buildcache, or a local/S3 backend). Also order your Dockerfile with dependencies first and code last, so the expensive layers stay valid across commits.

Persistent self-hosted runners keep a local Docker cache automatically but accumulate disk usage and state; prune regularly or use ephemeral runners with a remote cache.

Artifacts between jobs

Within a workflow:

- uses: actions/upload-artifact@v4
  with:
    name: test-report
    path: report.xml
    retention-days: 14

# in a later job
- uses: actions/download-artifact@v4
  with:
    name: test-report

Use them to hand a built package to a deploy job, publish test and coverage reports, keep logs of failed runs, or store release files. Set retention (storage costs accumulate) and never upload secrets. In Jenkins the equivalent is archiveArtifacts and stash/unstash; in GitLab it is artifacts:.

For a container-based workflow the "artifact" between build and deploy is the image in a registry, identified by tag and digest. Rebuilding in the deploy job breaks "build once".

Where artifacts live: repositories and registries

Artifact type Typical home
Container images Docker Hub, GitHub Container Registry, Amazon ECR, Google Artifact Registry, Harbor
Java (jar/war) Maven repository: Nexus, Artifactory, GitHub Packages, AWS CodeArtifact
JavaScript npm registry (private: Verdaccio, GitHub Packages, CodeArtifact)
Python PyPI, private index (devpi, Artifactory, CodeArtifact)
Generic files, installers S3 buckets, GitHub Releases
Infrastructure modules Terraform registry, private module repositories, OCI registries

An artifact repository gives you a single, controlled source of dependencies (a proxy/cache for public registries, protecting you from outages and from malicious or deleted packages) and a place to publish your own builds with retention and access policies.

Versioning and traceability

Every artifact must answer: which source commit and which build produced this, and what is inside?

  • Tag images with the commit SHA (shop-api:sha-1a2b3c4) and/or a semantic version (1.4.2 from a Git tag). Add moving tags (1.4, latest) only as conveniences. Deploy by the immutable tag or digest.
  • Make tags immutable in the registry (ECR has "tag immutability"), so a tag can never silently change.
  • Embed metadata: OCI labels (org.opencontainers.image.revision, .source, .version), and expose the version in the app (/version endpoint, a build-info log line) so you can see what runs.
  • Record what was deployed (a deployment log or Git-based GitOps repository) so you can answer "what version was live at 14:05?".

Retention and cost

Registries grow without bound. Set lifecycle policies: keep the last N tagged releases, delete untagged images older than 14 days, delete branch/PR images after merge. Artifacts, caches and logs all cost storage.

Reproducible, verifiable builds

  • Pin everything: lock files, base images by version or digest, tool versions, action versions.
  • Hermetic where possible: no network access during the build except to the artifact proxy.
  • SBOM (software bill of materials): a list of every package in the artifact, generated at build time (docker buildx build --sbom=true, Syft, Trivy), stored with the image. When a new vulnerability appears you can answer "which of our images contain this library?" immediately.
  • Provenance and signing: attest how the artifact was built (SLSA provenance) and sign it (cosign / Sigstore). Deployment can then verify that an image was produced by your pipeline and not pushed by hand. Kubernetes admission policies can enforce this.
  • Scan the artifact (the Docker course covers Trivy) before promotion.

These steps turn "we think this is what we deployed" into evidence.

Common mistakes

  • Treating a cache as an artifact, or depending on a cache for correctness.
  • A cache key that never changes (stale dependencies) or changes every run (never hits).
  • Rebuilding for each environment instead of promoting one artifact.
  • Mutable latest tags in production.
  • Unbounded retention and surprise storage bills.
  • Uploading build outputs that contain secrets (for example .env copied into a package).
  • No way to map a running version back to a commit.

Try it yourself

  1. Add dependency caching to a workflow and compare the run time on a cold and a warm cache.
  2. Build the same Docker project twice in CI with and without a cache backend; compare the times.
  3. Make your pipeline upload a test report as an artifact on failure (if: failure()), and download it.
  4. Tag a pushed image with the SHA and a semantic version, and add OCI labels. Show how to find the commit from a running container.

Check yourself

What is the difference between a cache and an artifact?

A cache only speeds things up and can be discarded without breaking the build; an artifact is a deliberate output (package, image, report) that later stages or people rely on.

Why include the hash of the lock file in a cache key?

So the cache is reused while dependencies are unchanged and a new cache is created exactly when they change, avoiding stale or constantly missed caches.

Why use immutable tags and the commit SHA for images?

They make every deployed image traceable to its source and guarantee that a tag always means the same bytes, which makes rollbacks and audits reliable.

What is an SBOM useful for?

It lists the components in an artifact, so when a vulnerability is announced you can quickly find which artifacts are affected.

Interview questions

Your CI takes 25 minutes, mostly installing dependencies and building Docker layers. What do you do?

Cache the package manager's download directory keyed on the lock file, use a Docker build cache backend (registry or GitHub cache), reorder the Dockerfile so dependencies precede source, run jobs in parallel, and keep a nightly clean build to catch cache-dependent breakage.

How do you ensure that what is deployed to production is exactly what was tested?

Build once, publish an immutable artifact tagged with the commit, promote that same artifact across environments with configuration changes only, and verify its digest or signature at deploy time.

What goes wrong with a build cache shared across all branches?

A pull request could write a poisoned or inconsistent cache that a trusted branch later restores. Isolate caches by branch or trust level, and do not let untrusted code write to caches used for releases.

Summary

  • Caches speed builds and must never be required for correctness; artifacts are outputs that later stages and deployments depend on.
  • Key caches on lock-file hashes; cache download directories and Docker layers (with a remote backend in CI).
  • Store artifacts in the right repository or registry with retention rules; tag with commit SHA and semantic versions, make tags immutable, and embed metadata.
  • Pin inputs, generate SBOMs, scan, sign and verify to make builds reproducible and trustworthy.

Going deeper

See how much of this you can explain

A free mock interview asks you new questions on CI/CD pipelines and releases for DevOps Engineer and scores every answer, with what to fix.