Lesson 4 of 8 · 11 min · Pipelines in practice

Jenkins pipelines

Covers: Declarative Pipeline syntax for current Jenkins LTS. Updated 13 Nov 2026. Part of the DevOps Engineer roadmap.

What you will learn

  • Jenkins architecture: controller, agents, plugins, jobs
  • The declarative Jenkinsfile: agent, environment, stages, steps, post
  • Credentials, parallel stages, parameters and approvals
  • Running builds in Docker agents
  • Operating Jenkins: security, backups, upgrades, and when to choose something else

In simple words

Jenkins is an older and very widely used automation server, and many companies still run their builds on it. Unlike GitHub Actions, your team installs and looks after it. The idea is the same: a file in your project lists the stages, and Jenkins runs them.

Words to know

  • Jenkins: an open-source server that runs build and release pipelines.
  • Jenkinsfile: the file in your project that describes the pipeline's stages.
  • Agent: a machine that Jenkins sends work to.
  • Plugin: an add-on that gives Jenkins a new ability.

What Jenkins is

Jenkins is the long-established open-source automation server. You host it yourself, which gives total control (any language, any network, any plugin) and total responsibility (patching, scaling, security, backups). Many enterprises still run large Jenkins estates, so you will meet it in jobs even if you start new projects on GitHub Actions or GitLab CI.

Architecture:

  • The controller (formerly "master") holds configuration, schedules builds, serves the UI and stores history. Do not run builds on it.
  • Agents (nodes) are the machines or containers where build steps actually run. They connect to the controller; they can be permanent servers, cloud VMs started on demand, or Kubernetes pods created per build.
  • Plugins add almost everything: Git, Docker, credentials, Kubernetes agents, notifications. A big plugin set is Jenkins' strength and its upgrade headache.
  • Jobs can be freestyle (click-configured, discouraged) or Pipeline jobs defined by a Jenkinsfile in your repository. Use the Jenkinsfile: it is versioned and reviewed like code. A multibranch pipeline discovers branches and pull requests in a repository and creates a job for each.

Declarative Jenkinsfile

Two syntaxes exist: declarative (structured, validated, recommended) and scripted (full Groovy, flexible, easier to abuse). A complete declarative example, placed in the repository root as Jenkinsfile:

pipeline {
    agent { docker { image 'python:3.12-slim' } }

    options {
        timeout(time: 20, unit: 'MINUTES')
        disableConcurrentBuilds()
        buildDiscarder(logRotator(numToKeepStr: '30'))
    }

    environment {
        IMAGE = "registry.example.com/shop-api"
        TAG   = "${env.GIT_COMMIT.take(7)}"
    }

    stages {
        stage('Install') {
            steps {
                sh 'pip install -r requirements.txt'
            }
        }
        stage('Quality') {
            parallel {
                stage('Unit tests') {
                    steps { sh 'pytest -q --junitxml=report.xml' }
                    post { always { junit 'report.xml' } }
                }
                stage('Lint') {
                    steps { sh 'ruff check .' }
                }
            }
        }
        stage('Build image') {
            when { branch 'main' }
            agent any
            steps {
                withCredentials([usernamePassword(credentialsId: 'registry-creds',
                                                  usernameVariable: 'REG_USER',
                                                  passwordVariable: 'REG_PASS')]) {
                    sh '''
                        echo "$REG_PASS" | docker login registry.example.com -u "$REG_USER" --password-stdin
                        docker build -t "$IMAGE:$TAG" .
                        docker push "$IMAGE:$TAG"
                    '''
                }
            }
        }
        stage('Deploy to production') {
            when { branch 'main' }
            input { message 'Deploy to production?'; ok 'Deploy' }
            steps {
                sh './deploy.sh "$TAG"'
            }
        }
    }

    post {
        failure { echo 'Build failed' }
        cleanup { cleanWs() }
    }
}

Reading it:

  • agent says where to run: a Docker container from the given image (a clean, reproducible toolchain), any, a labelled node (agent { label 'linux-large' }), or a Kubernetes pod template. A stage can have its own agent, as the image-build stage does here, since building images needs Docker access.
  • options: a timeout, no overlapping builds, and log retention. Always set these.
  • environment: variables for all stages. env.GIT_COMMIT.take(7) is the short commit id used as the image tag, the same traceability rule as in the previous lessons.
  • stages / stage / steps: the structure. Steps are things like sh '...' (run a shell command), junit, input, echo, archiveArtifacts, withCredentials.
  • parallel: the unit tests and linting run at the same time.
  • when: run a stage only for a condition (branch 'main', tag 'v*', expression { ... }, changeset '**/*.py').
  • post: actions after a stage or the pipeline: always, success, failure, unstable, cleanup. Publishing test reports with junit in post { always { ... } } means failures still produce reports.
  • input: a manual approval gate (an in-pipeline version of continuous delivery). The stage-level input directive pauses before that stage's own agent is allocated, so a stage-level agent is not held while people decide. Be aware that a pipeline-level agent (as in this short example) stays allocated for the whole run; in real pipelines that wait for humans use agent none at the top and give each stage its own agent, so no executor sits idle for days.

Credentials

Never put passwords in a Jenkinsfile. Store them in Jenkins' Credentials store (username/password, secret text, SSH key, files) and bind them in a step with withCredentials, or credentials('id') in environment. Jenkins masks the values in the console log, but a script that echoes a transformed value, or sends it elsewhere, can still leak it, as in every CI system. Use secrets managers (Vault plugin, AWS) for production credentials and scope credentials by folder. Notice that the example logs in with --password-stdin so the password is not in the process list.

Parameters, triggers and shared libraries

  • parameters { string(name: 'TAG') choice(...) } make a build parameterised ("Build with Parameters").
  • Triggers: webhooks from Git (best), pollSCM (inefficient fallback), cron, upstream jobs.
  • Shared libraries keep reusable pipeline code (functions, steps) in a separate repository so dozens of Jenkinsfiles stay short.

Docker and Kubernetes agents

Running each build in a fresh container makes builds reproducible and avoids "this agent has Java 8 and that one Java 17" problems. With the Kubernetes plugin, Jenkins starts a pod per build with the containers you declare and deletes it afterwards, so you pay for capacity only while building. Mounting the host's Docker socket into build containers gives them root on the host; prefer rootless builders such as Kaniko or BuildKit-based tools in untrusted environments.

Running Jenkins in production

  • Security: enable authentication and authorisation (matrix or role-based), keep Jenkins and plugins updated, restrict who can configure jobs and credentials, keep the controller off the public internet or behind an authenticated proxy and VPN. Jenkins has a history of plugin vulnerabilities; treat it as a high-value target because it holds deployment credentials.
  • Executors: set the controller's executors to 0.
  • Configuration as code: the Jenkins Configuration as Code (JCasC) plugin defines the whole Jenkins configuration in YAML, so a rebuilt controller is identical.
  • Backups: the JENKINS_HOME directory (jobs, credentials, config). Test restoring it.
  • Upgrades: pin plugin versions, test updates in a staging Jenkins, use the LTS line.
  • Observability: monitor queue length, agent availability, build durations, disk usage (artifact and log retention!).

Jenkins versus hosted CI

Jenkins GitHub Actions / GitLab CI
Hosting you run it managed (or self-hosted runners)
Flexibility extreme, any plugin large marketplace, simpler model
Maintenance high low
Cost your servers and time per-minute pricing
Pipeline format Groovy Jenkinsfile YAML
Best for complex, regulated or on-premises environments most new projects

The concepts transfer completely: stages, agents/runners, secrets, artifacts, approvals, parallelism.

Common mistakes

  • Building on the controller.
  • Freestyle jobs configured by clicking, which nobody can review or reproduce.
  • Credentials in the repository or echoed in logs.
  • No timeouts or retention, filling disks and hanging executors.
  • Polling Git every minute instead of using webhooks.
  • Outdated plugins on an internet-facing controller.
  • Waiting for approval while holding an executor (a pipeline-level agent plus a long input).

Try it yourself

  1. Start Jenkins in Docker (jenkins/jenkins:lts-jdk17, port 8080, a volume for /var/jenkins_home), unlock it with the initial admin password and install the suggested plugins.
  2. Create a Pipeline job from SCM pointing at a repository with the Jenkinsfile above (adapted), and run it.
  3. Add a credentials entry and use it in withCredentials. Check that the console masks it.
  4. Convert the GitHub Actions workflow from the previous lesson to a Jenkinsfile and compare the two.

Check yourself

What is the difference between the Jenkins controller and an agent?

The controller manages configuration, scheduling and the UI. Agents execute the build steps. Builds should not run on the controller.

Why use a Jenkinsfile in the repository instead of configuring jobs in the UI?

It is versioned, reviewed, reproducible and travels with the code, so every branch can have its own pipeline and a rebuilt Jenkins can recreate jobs automatically.

How should a Jenkins pipeline use a password?

Store it in Jenkins credentials and bind it with withCredentials (or a secrets manager plugin). Never hard-code it in the Jenkinsfile or print it.

What does the post section do?

Defines actions that run after the pipeline or a stage depending on the outcome (always, success, failure, cleanup), for reports, notifications and cleanup.

Interview questions

How would you scale Jenkins for a large team?

Keep the controller small and build on agents, ideally ephemeral ones (Kubernetes pods or cloud VMs started on demand), use shared libraries and JCasC, label agents by capability, and monitor queue time to scale capacity.

What security risks does a CI server have and how do you reduce them?

It stores credentials and can deploy to production, so it is a prime target. Restrict access, keep it patched, isolate builds, scope credentials narrowly, avoid running untrusted code with secrets, and audit changes.

Declarative or scripted pipelines?

Declarative for most work: it is structured, validated up front and readable. Use scripted blocks (script { }) or shared libraries only where logic needs it.

Summary

  • Jenkins is a self-hosted automation server: controller, agents, plugins, and Pipeline jobs defined by a Jenkinsfile.
  • Declarative pipelines express agent, options, environment, stages (with parallel and when), and post actions.
  • Keep secrets in the credentials store; run builds in containers or ephemeral agents; never build on the controller.
  • Hosted CI is simpler to maintain; the concepts are the same everywhere.

Going deeper

See how much of this you can explain

A free mock interview asks you new questions on CI/CD pipelines and releases for DevOps Engineer and scores every answer, with what to fix.