When working with our customers, we frequently get questions around integrating Atomist into an existing Jenkins-based CI/CD environment: How does Atomist relate to Jenkins? How can we use Jenkins to do builds and use Atomist for the broad, holistic orchestration of our delivery flows? How can we bring Jenkins pipelines into an actionable and collaborative ChatOps model?

Ever since we introduced the Software Delivery Machine (SDM) framework it was possible to integrate a Jenkins job or pipeline as a goal into an SDM goal set.

Once build events are flowing into Atomist, the following SDM goal set would schedule two goals: a Build goal that waits for a Jenkins job or pipeline to finish and a deploy goal that will start a deployment to a staging environment once the Jenkins job has finished.

export const configuration = configure(async sdm => {

    const deploy; // ... ommitted code to set up deploy goal

    return {
        "ci/cd": {
            goals: [
                new Build().with({ externalTool: "jenkins" }),
                deploy,
            ],
        },
    };
});

Recently we introduced a Jenkins-specific SDM extension pack to deepen the integration with Jenkins. The remainder of this post introduces the features of sdm-pack-jenkins.

Trigger Jenkins jobs

Traditionally Jenkins polls your Git SCM to determine when to start new jobs. There are also ways to set up job triggers without polling by using Git commit hooks or specific plugins like the Jenkins GitHub plugin. This means that every push to your Git repository triggers a job. Great!

What's not so great in practice is that this approach creates jobs for immaterial changes as well. Immaterial change detection is an Atomist concept—or rather a PushTest—to determine what change is considered material enough to require a full CI/CD process to start.

When combining a powerful SDM PushTest with the capability of sdm-pack-jenkins to trigger Jenkins jobs, a lot of unnessecary builds can be prevented:

export const configuration = configure(async () => {

    const deploy; // ... ommitted code to set up deploy goal

    const options: Pick<JenkinsRegistration, "server"> = {
        server: {
            url: process.env.JENKINS_URL || "http://127.0.0.1:8080",
            user: process.env.JENKINS_USER || "admin",
            password: process.env.JENKINS_PASSWORD || "123456",
        },
    };

    const build = jenkins("build", {
        ...options,
        job: async gi => `${gi.goalEvent.repo.name}-build`,
    });

    return {
        "ci/cd": {
            test: [
                isMaterialChange({
                    extensions: ["java", "properties", "yaml"],
                    files: ["pom.xml"],
                })],
            goals: [
                build,
                deploy,
            ],
        },
    };
});

In the example above we first configure the Jenkins connection details. Here we are reading from environment variables to overrride defaults. Because an SDM is just TypeScript code, the credentials can be obtained from different places like configuration, Kubernetes secrets or HashiCorp Vault.

With those connection details we can create the build goal using the jenkins function provided by sdm-pack-jenkins. In this example, the build goal will trigger a Jenkins job called <name of repo>-build. It is assumed that those jobs already exist in your Jenkins instance.

Lastly we add a isMaterialChange push test that checks if the push contains changes to files that are considered material for Maven-based Java projects.

In summary, this SDM will trigger a spring-service-build Jenkins job for every material change to a spring-service repository.

Banish pipeline proliferation

The previous section's example had one assumption: the Jenkins pipeline needed to exist already! This means that somebody must have created the pipeline in Jenkins and/or created the Jenkinsfile in the repository.

What if those manual steps wouldn't be needed? What if somehow each new repository your team creates automatically gets a Jenkins pipeline and all pipeline definitions are kept in sync across your many repositories?

We can extend the previous sample to have the SDM automatically create Jenkins pipelines and keep build definitions in sync on every push.

export const configuration = configure(async () => {

    const deploy; // ... ommitted code to set up deploy goal

    const options; // ... ommited code to set up Jenkins options

    const build = jenkins("build", {
        ...options,
        job: async gi => `${gi.goalEvent.repo.name}-build`,
        definition: async gi => mavenPipeline(gi),
    });

    return {
        "ci/cd": {
            test: [
                hasFile("pom.xml"),
                isMaterialChange({
                    extensions: ["java", "properties", "yaml"],
                    files: ["pom.xml"],
                })],
            goals: [
                build,
                deploy,
            ],
        },
    };
});

The jenkins function now takes an additional definition property. This property defines a callback to be executed for every push. The callback function returns a Jenkins build definition as XML which will be diffed against the existing build definition in Jenkins. If the build definition differs, it will be updated. If no build definition exists, it will get created.

With this SDM every push to a repository that has a pom.xml file, a Jenkins job called <name of repo>-build will be updated or created. This requires no Jenkinsfile in the repository or manual set up of the job!

The previous example manages the Jenkins pipeline definition for every push and also triggers the pipeline. If you are only interested in converging your pipelines, you can set the convergeOnly property to true like:

    const build = jenkins("build", {
        ...options,
        job: async gi => `${gi.goalEvent.repo.name}-build`,
        definition: async gi => mavenPipeline(gi),
        convergeOnly: true,
    });

Orchestrate Jenkins Jobs

Using an SDM it is possible to orchestrate different Jenkins jobs or pipelines. This is useful to plug together a build pipeline with staging and subsequent production deploy pipelines.

The following example uses a second Jenkins goal to trigger an existing pipeline called deploy that takes two parameters: TARGET_ENV to determine where to deploy to and IMAGE_NAME to identify what Docker image to deploy.

The deploy goal is configured to require preApproval which means the goal has to get started explicitly. This can happen from chat, API, or our web interface.

export const configuration = configure(async () => {

    const options; // ... ommited code to set up Jenkins options

    const build = jenkins("build", {
        ...options,
        job: async gi => `${gi.goalEvent.repo.name}-build`,
        definition: async gi => mavenPipeline(gi),
    });

    const deploy = jenkins(
        {
            displayName: "deploy",
            preApproval: true,
        },
        {
            ...options,
            job: "deploy",
            parameters: async gi => {
                const repo = gi.goalEvent.repo;
                return {
                    TARGET_ENV: "staging",
                    IMAGE_NAME: `${repo.owner}/${repo.name}:${gi.goalEvent.sha}`,
                };
            },
        });

    return {
        "ci/cd": {
            test: [
                hasFile("pom.xml"),
                isMaterialChange({
                    extensions: ["java", "properties", "yaml"],
                    files: ["pom.xml"],
                })],
            goals: [
                build,
                deploy,
            ],
        },
    };
});

ChatOps-enable Jenkins

At Atomist we believe in the power of ChatOps. We surface all the build and delivery-related information in your team's Slack or Microsoft Teams channels and allow you to take action right there in the context of your conversation. The Atomist bot also knows how Jenkins jobs relate to different channels; channel members see only the information they care about.

With sdm-pack-jenkins this applies to information about your Jenkins jobs as well. We update the messages in chat while Jenkins jobs move through the build stages and let you click through to the logs. Additionally, you can start Jenkins jobs from chat as part of an SDM goal set, giving you full control over who is allowed to start certain jobs.

Jenkins goal updates and approval in Slack

Try it out

To try this out for yourself, follow the quick start guide and create an Atomist workspace and install the Atomist CLI.

Install Jenkins

If you don't have an existing Jenkins installation, use the following command to run a Jenkins instance in a Docker container:

$ docker run -p 8080:8080 \
    -v jenkins_home:/var/jenkins_home \
    -v /var/run/docker.sock:/var/run/docker.sock \ 
    jenkinsci/blueocean

Once Jenkins has started, visit the interface to finish the installation process and create your first admin user. You'll need the admin credentials in the next step.

Run the Jenkins Sample SDM

Start the Jenkins SDM from our atomist/samples repository like so:

$ JENKINS_URL=http://<your jenkins ip>:8080/ \
    JENKINS_USER=<jenkins admin user> \
    JENKINS_PASSWORD=<jenkins admin password> \
    atomist start \
    --repository-url=https://github.com/atomist/samples.git \
    --index=lib/sdm/jenkinsJob.ts

With the SDM running and connected to your Atomist workspace, you can now make a commit to any Maven repository. This will plan the Jenkins build goal and trigger a newly created job in your Jenkins instance.

If you don't have such project ready, you can create a new Spring-based project from our web interface; select the + in the left menu bar and choose Spring REST Project as your seed.

The Jenkins sample SDM creates pipelines suitable for Maven-based projects. If you prefer to trigger some of your existing jobs, you can pass the name of the job to trigger into the SDM via the JENKINS_JOB environment variable.

If something is not behaving as it should while you try this out, please reach out to us in our Slack community workspace at https://join.atomist.com.

Summary

The above examples should demonstrate how you can integrate Jenkins into an Atomist goal set. The support in sdm-pack-jenkin should allow you to quickly evaluate Atomist without changing your existing Jenkins pipelines or configuration.

With this integration, it is possible to combine Atomist's capabilities such as Autofixes, Fingerprints, or Container goals with existing Jenkins pipelines to build powerful and resuable goal sets across your entire organization.