A Jenkins that does not rot: configuration as code, ephemeral agents, shared libraries

A Jenkins controller that has been alive for five years contains a large amount of state that exists nowhere else: credentials someone pasted in 2021, job definitions edited in the UI, plugin versions that happen to work together. The recovery plan for that machine is "hope".

Three changes fix it. They are independent, so you can do them in any order, but this is the order we use.

1. Configuration as code (JCasC)

The Configuration as Code plugin takes a YAML file and applies it to the controller at boot: security realm, authorisation, clouds, agent templates, tool installations, global settings. Anything not in the YAML is either defaulted or drifted.

jenkins:
  systemMessage: "Managed by JCasC — do not edit in the UI"
  numExecutors: 0
  authorizationStrategy:
    roleBased:
      roles:
        global:
          - name: admin
            permissions: [Overall/Administer]
            entries: [{ group: "platform-team" }]
  clouds:
    - kubernetes:
        name: k8s
        namespace: jenkins-agents
        jenkinsUrl: "http://jenkins.jenkins.svc.cluster.local:8080"
        templates:
          - name: build
            label: build
            containers:
              - name: jnlp
                image: ghcr.io/acme/ci-agent:2026.09
                resourceRequestCpu: "1"
                resourceLimitMemory: "4Gi"
unclassified:
  location:
    url: https://ci.acme.example/

numExecutors: 0 on the controller is not cosmetic. Builds running on the controller have access to the Jenkins home directory, which means access to every credential. Do not run builds there.

Adoption is incremental: export your current config with the plugin's "view configuration" page, commit it, and start deleting the parts you do not understand until boot still works. Budget a week. The finish line is that you can docker run the controller image with the YAML mounted and get your Jenkins back.

Credentials stay out of the YAML. Reference them from the secrets store you already have with the Vault or cloud-native credentials providers, or better, drop static credentials entirely.

2. Ephemeral agents

Static agents are a standing fleet of machines, each one a snowflake accumulating build tools, and each one billed 24 hours a day to do 90 minutes of work.

The Kubernetes plugin (configured above) starts a pod per build and deletes it after. The economics change completely: you pay for build-seconds, and a machine that never survives a build cannot drift. In the Jenkinsfile:

pipeline {
  agent {
    kubernetes {
      yaml '''
        spec:
          containers:
            - name: build
              image: ghcr.io/acme/ci-agent:2026.09
              command: ["cat"]
              tty: true
              resources:
                requests: { cpu: "2", memory: "4Gi" }
      '''
    }
  }
  stages {
    stage('test') {
      steps { container('build') { sh 'make test' } }
    }
  }
}

Two practical notes. First, put the agent image in your registry and version it; an agent image pinned to latest reintroduces the drift you just removed. Second, ephemeral agents make caching explicit — nothing survives the pod, so you need a remote cache or your builds get slower before they get faster. Plan that in the same change.

If you are not on Kubernetes, the EC2 and Azure VM agent clouds do the same thing with a slower start time (60 to 120 seconds instead of 5 to 15). Still worth it.

3. Shared libraries with a real interface

Most Jenkinsfiles are copy-pasted and then diverge. A shared library makes the common shape one artefact with a version:

// vars/serviceBuild.groovy
def call(Map cfg) {
  pipeline {
    agent { kubernetes { yaml libraryResource('agents/build.yaml') } }
    stages {
      stage('test')  { steps { container('build') { sh cfg.testCmd ?: 'make test' } } }
      stage('image') { steps { container('build') { sh "make image TAG=${env.GIT_COMMIT}" } } }
      stage('deploy') {
        when { branch 'main' }
        steps { deployTo(env: 'staging', service: cfg.name) }
      }
    }
  }
}

and each repository's Jenkinsfile becomes:

@Library('acme-ci@v4') _
serviceBuild(name: 'orders', testCmd: 'make test-integration')

Pin the library to a tag, not to main. A library on main means every repository in the company gets your refactor at the same instant, which is how a one-line change becomes an outage. Tag it, roll it out repo by repo, and keep the previous tag working for a release or two.

The library is also where policy lives — image signing, SBOM generation, the approval step before production — because it is the one place that changing it changes every pipeline.

What this buys you

After all three: the controller is disposable, agents are cattle, and pipeline logic is versioned and reviewable. Rebuilding from nothing is a helm install plus a YAML file, and takes about twenty minutes.

That is also the point at which the keep-or-migrate question becomes easy to answer — a Jenkins in this shape is cheap to keep, and if you do migrate later, pipelines expressed as library calls port far more easily than 600-line Jenkinsfiles.