Lesson 4 of 8 · 11 min · Pipelines in practice
Jenkins pipelines
What you will learn
- Jenkins architecture: controller, agents, plugins, jobs
- The declarative
Jenkinsfile: agent, environment, stages, steps, post - Credentials, parallel stages, parameters and approvals
- Running builds in Docker agents
- Operating Jenkins: security, backups, upgrades, and when to choose something else
In simple words
Jenkins is an older and very widely used automation server, and many companies still run their builds on it. Unlike GitHub Actions, your team installs and looks after it. The idea is the same: a file in your project lists the stages, and Jenkins runs them.
Words to know
- Jenkins: an open-source server that runs build and release pipelines.
- Jenkinsfile: the file in your project that describes the pipeline's stages.
- Agent: a machine that Jenkins sends work to.
- Plugin: an add-on that gives Jenkins a new ability.
What Jenkins is
Jenkins is the long-established open-source automation server. You host it yourself, which gives total control (any language, any network, any plugin) and total responsibility (patching, scaling, security, backups). Many enterprises still run large Jenkins estates, so you will meet it in jobs even if you start new projects on GitHub Actions or GitLab CI.
Architecture:
- The controller (formerly "master") holds configuration, schedules builds, serves the UI and stores history. Do not run builds on it.
- Agents (nodes) are the machines or containers where build steps actually run. They connect to the controller; they can be permanent servers, cloud VMs started on demand, or Kubernetes pods created per build.
- Plugins add almost everything: Git, Docker, credentials, Kubernetes agents, notifications. A big plugin set is Jenkins' strength and its upgrade headache.
- Jobs can be freestyle (click-configured, discouraged) or Pipeline jobs defined by a
Jenkinsfilein your repository. Use the Jenkinsfile: it is versioned and reviewed like code. A multibranch pipeline discovers branches and pull requests in a repository and creates a job for each.
Declarative Jenkinsfile
Two syntaxes exist: declarative (structured, validated, recommended) and scripted (full Groovy, flexible, easier to abuse). A complete declarative example, placed in the repository root as Jenkinsfile:
pipeline {
agent { docker { image 'python:3.12-slim' } }
options {
timeout(time: 20, unit: 'MINUTES')
disableConcurrentBuilds()
buildDiscarder(logRotator(numToKeepStr: '30'))
}
environment {
IMAGE = "registry.example.com/shop-api"
TAG = "${env.GIT_COMMIT.take(7)}"
}
stages {
stage('Install') {
steps {
sh 'pip install -r requirements.txt'
}
}
stage('Quality') {
parallel {
stage('Unit tests') {
steps { sh 'pytest -q --junitxml=report.xml' }
post { always { junit 'report.xml' } }
}
stage('Lint') {
steps { sh 'ruff check .' }
}
}
}
stage('Build image') {
when { branch 'main' }
agent any
steps {
withCredentials([usernamePassword(credentialsId: 'registry-creds',
usernameVariable: 'REG_USER',
passwordVariable: 'REG_PASS')]) {
sh '''
echo "$REG_PASS" | docker login registry.example.com -u "$REG_USER" --password-stdin
docker build -t "$IMAGE:$TAG" .
docker push "$IMAGE:$TAG"
'''
}
}
}
stage('Deploy to production') {
when { branch 'main' }
input { message 'Deploy to production?'; ok 'Deploy' }
steps {
sh './deploy.sh "$TAG"'
}
}
}
post {
failure { echo 'Build failed' }
cleanup { cleanWs() }
}
}
Reading it:
agentsays where to run: a Docker container from the given image (a clean, reproducible toolchain),any, a labelled node (agent { label 'linux-large' }), or a Kubernetes pod template. A stage can have its ownagent, as the image-build stage does here, since building images needs Docker access.options: a timeout, no overlapping builds, and log retention. Always set these.environment: variables for all stages.env.GIT_COMMIT.take(7)is the short commit id used as the image tag, the same traceability rule as in the previous lessons.stages/stage/steps: the structure. Steps are things likesh '...'(run a shell command),junit,input,echo,archiveArtifacts,withCredentials.parallel: the unit tests and linting run at the same time.when: run a stage only for a condition (branch 'main',tag 'v*',expression { ... },changeset '**/*.py').post: actions after a stage or the pipeline:always,success,failure,unstable,cleanup. Publishing test reports withjunitinpost { always { ... } }means failures still produce reports.input: a manual approval gate (an in-pipeline version of continuous delivery). The stage-levelinputdirective pauses before that stage's ownagentis allocated, so a stage-level agent is not held while people decide. Be aware that a pipeline-levelagent(as in this short example) stays allocated for the whole run; in real pipelines that wait for humans useagent noneat the top and give each stage its own agent, so no executor sits idle for days.
Credentials
Never put passwords in a Jenkinsfile. Store them in Jenkins' Credentials store (username/password, secret text, SSH key, files) and bind them in a step with withCredentials, or credentials('id') in environment. Jenkins masks the values in the console log, but a script that echoes a transformed value, or sends it elsewhere, can still leak it, as in every CI system. Use secrets managers (Vault plugin, AWS) for production credentials and scope credentials by folder. Notice that the example logs in with --password-stdin so the password is not in the process list.
Parameters, triggers and shared libraries
parameters { string(name: 'TAG') choice(...) }make a build parameterised ("Build with Parameters").- Triggers: webhooks from Git (best),
pollSCM(inefficient fallback),cron, upstream jobs. - Shared libraries keep reusable pipeline code (functions, steps) in a separate repository so dozens of Jenkinsfiles stay short.
Docker and Kubernetes agents
Running each build in a fresh container makes builds reproducible and avoids "this agent has Java 8 and that one Java 17" problems. With the Kubernetes plugin, Jenkins starts a pod per build with the containers you declare and deletes it afterwards, so you pay for capacity only while building. Mounting the host's Docker socket into build containers gives them root on the host; prefer rootless builders such as Kaniko or BuildKit-based tools in untrusted environments.
Running Jenkins in production
- Security: enable authentication and authorisation (matrix or role-based), keep Jenkins and plugins updated, restrict who can configure jobs and credentials, keep the controller off the public internet or behind an authenticated proxy and VPN. Jenkins has a history of plugin vulnerabilities; treat it as a high-value target because it holds deployment credentials.
- Executors: set the controller's executors to 0.
- Configuration as code: the Jenkins Configuration as Code (JCasC) plugin defines the whole Jenkins configuration in YAML, so a rebuilt controller is identical.
- Backups: the
JENKINS_HOMEdirectory (jobs, credentials, config). Test restoring it. - Upgrades: pin plugin versions, test updates in a staging Jenkins, use the LTS line.
- Observability: monitor queue length, agent availability, build durations, disk usage (artifact and log retention!).
Jenkins versus hosted CI
| Jenkins | GitHub Actions / GitLab CI | |
|---|---|---|
| Hosting | you run it | managed (or self-hosted runners) |
| Flexibility | extreme, any plugin | large marketplace, simpler model |
| Maintenance | high | low |
| Cost | your servers and time | per-minute pricing |
| Pipeline format | Groovy Jenkinsfile |
YAML |
| Best for | complex, regulated or on-premises environments | most new projects |
The concepts transfer completely: stages, agents/runners, secrets, artifacts, approvals, parallelism.
Common mistakes
- Building on the controller.
- Freestyle jobs configured by clicking, which nobody can review or reproduce.
- Credentials in the repository or echoed in logs.
- No timeouts or retention, filling disks and hanging executors.
- Polling Git every minute instead of using webhooks.
- Outdated plugins on an internet-facing controller.
- Waiting for approval while holding an executor (a pipeline-level agent plus a long
input).
Try it yourself
- Start Jenkins in Docker (
jenkins/jenkins:lts-jdk17, port 8080, a volume for/var/jenkins_home), unlock it with the initial admin password and install the suggested plugins. - Create a Pipeline job from SCM pointing at a repository with the Jenkinsfile above (adapted), and run it.
- Add a credentials entry and use it in
withCredentials. Check that the console masks it. - Convert the GitHub Actions workflow from the previous lesson to a Jenkinsfile and compare the two.
Check yourself
What is the difference between the Jenkins controller and an agent?
The controller manages configuration, scheduling and the UI. Agents execute the build steps. Builds should not run on the controller.
Why use a Jenkinsfile in the repository instead of configuring jobs in the UI?
It is versioned, reviewed, reproducible and travels with the code, so every branch can have its own pipeline and a rebuilt Jenkins can recreate jobs automatically.
How should a Jenkins pipeline use a password?
Store it in Jenkins credentials and bind it with withCredentials (or a secrets manager plugin). Never hard-code it in the Jenkinsfile or print it.
What does the post section do?
Defines actions that run after the pipeline or a stage depending on the outcome (always, success, failure, cleanup), for reports, notifications and cleanup.
Interview questions
How would you scale Jenkins for a large team?
Keep the controller small and build on agents, ideally ephemeral ones (Kubernetes pods or cloud VMs started on demand), use shared libraries and JCasC, label agents by capability, and monitor queue time to scale capacity.
What security risks does a CI server have and how do you reduce them?
It stores credentials and can deploy to production, so it is a prime target. Restrict access, keep it patched, isolate builds, scope credentials narrowly, avoid running untrusted code with secrets, and audit changes.
Declarative or scripted pipelines?
Declarative for most work: it is structured, validated up front and readable. Use scripted blocks (script { }) or shared libraries only where logic needs it.
Summary
- Jenkins is a self-hosted automation server: controller, agents, plugins, and Pipeline jobs defined by a
Jenkinsfile. - Declarative pipelines express agent, options, environment, stages (with
parallelandwhen), andpostactions. - Keep secrets in the credentials store; run builds in containers or ephemeral agents; never build on the controller.
- Hosted CI is simpler to maintain; the concepts are the same everywhere.
Going deeper
- Jenkins documentation: user handbook, tutorials and administration.
- Pipeline syntax reference: declarative and scripted syntax.
- Using credentials in Jenkins: binding and masking.
- Jenkins security: hardening a controller.
- Jenkins Configuration as Code: configuration in YAML.
See how much of this you can explain
A free mock interview asks you new questions on CI/CD pipelines and releases for DevOps Engineer and scores every answer, with what to fix.