CI Tools: Dagger, The New World (Part 3/3)

Pipeline Migration

Let’s take a closer look at how to migrate a GitLab pipeline into a Dagger pipeline.

In this example, there are two jobs. First is the build:rails_test_image job in the dependecies stage. It builds a test image based on ruby-slim:latest that contains all Ruby source code.

build:rails_test_image:
  stage: dependencies
  tags:
  - image_builder
  script:
  - docker build -t $RAILS_TEST_IMAGE --build-arg COMMIT_SHA=ab45dsaa --build-arg RELEASE_DATE=11111 --build-arg BUNDLER_VERSION="2.3.16" . 
  - docker push $RAILS_TEST_IMAGE

Second is the test:erd job in the lint stage, which runs the ERD creation job. The second job depends explicitly on the first one.

test:erd:
  image: "$RAILS_TEST_IMAGE"
  cache:
    policy: pull
    key:
      files:
      - core/Gemfile.lock
    paths:
    - core/vendor/bundle
  before_script:
  - cd core
  - bundle install
  tags:
  - openshift-runner
  stage: lint
  needs:
  - job: build:rails_test_image
  allow_failure: true
  script:
  - bundle exec rake db:setup erd

Below you can see the Gitlab CI pipeline, written with the Dagger Go SDK. I show only the content of the Dagger function.

ctx := context.Background()

// Create new dagger engine or connect to existing one
client, err := dagger.Connect(ctx)
if err != nil {
    panic(err)
}

// Load core directory into dagger build context
root := client.Host().Directory("./core")

// Build core container from dockerfile with arguments
coreContainer := client.Container().Build(root, dagger.ContainerBuildOpts{
    Dockerfile: "Dockerfile",
    Target:     "test",
    BuildArgs: []dagger.BuildArg{
        {Name: "COMMIT_SHA", Value: "ab45dsaa"},
        {Name: "RELEASE_DATE", Value: "11111"},
        {Name: "BUNDLER_VERSION", Value: "2.3.16"},
    },
})

// execute "install bundle" on core container
// return new container called bundled 
bundled := coreContainer.WithWorkdir("/opt/app-root/src/core").WithExec([]string{"bundle", "install"})

// execute bundle command on bundled container 
contents, err := bundled.
    WithExec([]string{"bundle", "exec", "rake", "db:setup", "erd"}).
    Stdout(ctx)
if err != nil {
    logWarning(err)
    // Allowed to fail
    // return
}

Let’s break down the steps involved in using Dagger to run an ERD (Entity Relationship Diagram) creation task. First, we use the Connect(ctx Context) function to connect to an existing Dagger engine, and if no engine exists, Dagger will create a new one for us. This is achieved with the following code:

client, err := dagger.Connect(ctx)

The next step is to load the local ./core directory from the host, which contains all the backend code we need. The Host() function is used to return the host’s execution environment. From there, we can access files on our host system. Here’s the code for this step:

root := client.Host().Directory("./core")

After that, we can call the client.Container().Build function to build a container that contains all the necessary components to run our tests. This function takes three parameters: the Dockerfile path, the Docker context and the Docker build options. These options can include arguments, target stage, Dockerfile name, etc. The function returns a container that we can use in the next stages. Here’s the code for those actions:

coreContainer := client.Container().Build(root, dagger.ContainerBuildOpts{
    Dockerfile: "Dockerfile",
    Target:     "test",
    BuildArgs: []dagger.BuildArg{
        {Name: "COMMIT_SHA", Value: "ab45dsaa"},
        {Name: "RELEASE_DATE", Value: "11111"},
        {Name: "BUNDLER_VERSION", Value: "2.3.16"},
    },
})

Now that we have our base image, which contains all the necessary things to run our tests, we can set the current working directory and execute the bundle install command to install the Rails dependencies. This is achieved with the following code:

bundled := coreContainer.
    WithWorkdir("/opt/app-root/src/core").
    WithExec([]string{"bundle", "install"})

After we have installed all the dependencies, we can finally run the ERD creation step. Again, we use the newly created container and execute the bundle command to run the ERD creation task:

bundled.
    WithExec([]string{"bundle", "exec", "rake", "db:setup", "erd"}).
    Stdout(ctx)

This is a small example to show how parts of a GitLab CI pipeline can be migrated into a Dagger pipeline. It is not just the same pipeline logic in a new technology. The test can be executed locally on your computer. This for developing and testing the pipeline itself or for the application developer to test their Ruby code.

The Dagger pipeline can then be called from GitLab CI again or any other build server.
This example shows additonally that single jobs can be migrated to Dagger. Using the Strangler pattern a whole pipeline can be migrated step by step (job by job).

Goff example

If you are interested to see an production example in the wild. Checkout the GitOps Diff Tool (called GOFF) on GitHub. The pipeline is enterly written with Dagger. It builds Docker images, binary artifacts and even the documentation. And the best thing about it, you can run the whole pipeline local on your machine with the same command as you do in the CI system. Just execute dagger run go run ci/main.go and let the magic happen.

Small remark on this one, this pipeline was written ways before Dagger releaesed the Project Zenith. So it is maybe not the most up-to-date example. But it is a prove that Dagger is already working for productive systems.

Infrastructure

Running pipelines needs an infrastructure, at least a build server. We show you an as-a-service and a self hostet option.

Dagger Cloud

End of September 2023 Dagger released their commercial cloud offering. The Dagger cloud. It offers a centralized UI for your Dagger runs. It collects Dagger pipelines data, execution times, layer caching, logs and DAG metrics. Everything you need to observe and monitor your pipelines at one place. No matter if you run your pipelines locally, on-prem or within Dagger Cloud. The Dagger Engine and Dagger Cloud form the Dagger Platform, with Dagger Cloud providing a production-grade control plane.

In the image below you can see a pipeline run on the Dagger cloud. Every step in the Pipeline is listed with the execution time, cache operation and logs.

Dagger cloud offers also the DAG view (still marked as experimental) This view allows you to identify critical bottlenecks and fans-in/out as well in your pipeline.

Self Hosted

At Puzzle ITC we created a specific infrastructure to run Dagger pipelines with GitLab CI. It contains the Dagger Engine and is optimized for the Dagger caching.

Runners

The special GitLab Runners for Dagger establish the connection to the Dagger Engine. The graphic shows how the pods created by the runners connect to the engine via socket.

 

Dagger Engine

We contrtibutet the infrastructrure setup to the official Dagger documentation. Find there the details on how to set up your Dagger GitLab CI infrastructure on OpenShift:

Run Dagger on OpenShift with GitLab Runners

Call Dagger Pipeline

The code below shows an example implementation for Dagger in Gitlab CI. It is the GitLab CI part to run a Dagger Pipeline on our Dagger infrastructure.

The most important part ist the environment variable _EXPERIMENTAL_DAGGER_RUNNER_HOST which is set to the local Unix socket unix:///var/run/dagger/buildkitd.sock. This allows us to connect the Gitlab Builder Pod with the Dagger Engine which runs on the same node.

stages:
  - build

variables:
  KUBERNETES_POD_LABELS_1: "project=${CI_PROJECT_NAME}"
  KUBERNETES_POD_LABELS_2: "project-id=${CI_PROJECT_ID}"
  KUBERNETES_POD_LABELS_3: "job-id=${CI_JOB_ID}"
  OTEL_ENDPOINT: tempo.pitc-buildkit-test.svc.cluster.local:4317
  _EXPERIMENTAL_DAGGER_JOURNAL: "out/journal.log"

.go-cache:
  variables:
    GOPATH: $CI_PROJECT_DIR/.go
    GOCACHE: $CI_PROJECT_DIR/.cache/go-build
    GOMODCACHE: $CI_PROJECT_DIR/go/pkg/mod
  before_script:
    - mkdir -p .go/pkg/mod/
    - mkdir -p .cache/go-build
    - mkdir -p go/pkg/mod
  cache:
    paths:
      - .go/pkg/mod/
      - .cache/go-build/
      - go/pkg/mod

.dagger:
  timeout: 2h
  extends: .go-cache
  image: registry.puzzle.ch/cicd/dagger-golang:latest
  tags:
    - buildkit-test
  variables:
    "_EXPERIMENTAL_DAGGER_RUNNER_HOST": "unix:///var/run/dagger/buildkitd.sock"
    "OTEL_RESOURCE_ATTRIBUTES": "commit=${CI_COMMIT_SHA},group=${CI_PROJECT_NAMESPACE},project=${CI_PROJECT_NAME},job=${CI_JOB_ID},pipeline=${CI_PIPELINE_ID}"
    "OTEL_EXPORTER_JAEGER_ENDPOINT": http://tempo.pitc-buildkit-test.svc.cluster.local:14268/api/traces
  after_script:
    - otel-collector --name ${CI_PROJECT_NAME}-${CI_JOB_ID} ${_EXPERIMENTAL_DAGGER_JOURNAL}
    - dagger-graph ${_EXPERIMENTAL_DAGGER_JOURNAL} out/dag.svg
  artifacts:
    expire_in: "30 days"
    paths:
      - "out/"

build:
  stage: build
  extends: .dagger
  script:
  - mkdir -p out
  - dagger version
  - echo "Run dagger"
  - dagger call -m ./ci/ build

The last line of the GitLab CI build job calls the Dagger pipeline.

Observability

While Dagger provides a compelling cloud solution with built-in observability features, our quest for deeper insights led us to explore the possibilities of observability within a self-hosted Dagger engine. To achieve this, we’ve seamlessly integrated Dagger into our existing monitoring and tracing stack, which comprises:

• Grafana Tempo, serving as a tracing endpoint.
• Grafana Loki, facilitating efficient log management.
• Prometheus, responsible for scraping Kubernetes metrics.
• Grafana, enabling the visualization of metrics.

This combination empowers us to unlock a comprehensive understanding of our CI/CD processes, enhancing our ability to monitor and optimize the performance of our self-hosted Dagger engine. The result is a more holistic and data-driven approach to observability, regardless of whether you’re utilizing Dagger in the cloud or self-hosting it.

Logs

Managing logs proved to be a relatively straightforward aspect of our transition. Leveraging the presence of Grafana Tempo within our cluster, all we needed to do was annotate the Dagger engine DaemonSet to seamlessly route logs into Loki.

Metrics

Metrics, on the other hand, posed a more intricate challenge. Regrettably, neither Gitlab nor Buildkit offered a Prometheus/OpenMetric endpoint that could be scraped for metrics. As a result, we resorted to extracting metrics from existing OpenShift workload metrics. Initially, we focused on monitoring CPU and memory consumption.

In addition, we fine-tuned our Gitlab Runner to enhance the metadata associated with our metrics. We achieved this by introducing a set of custom labels to each pod. These labels contain information about the Gitlab project and the unique pipeline job ID. These labels simplifies the process of correlating data from multiple sources.

  KUBERNETES_POD_LABELS_1: "project=${CI_PROJECT_NAME}"
  KUBERNETES_POD_LABELS_2: "project-id=${CI_PROJECT_ID}"
  KUBERNETES_POD_LABELS_3: "job-id=${CI_JOB_ID}"

Traces

Handling traces posed the most significant challenge in our journey. Initially, we attempted to send metrics directly from Buildkit to Tempo. Unfortunately, the traces generated by Buildkit lacked a common trace ID, causing them to be split and fail to correlate effectively. After thorough investigation and research, I stumbled upon the dagger.io/dagger/cmd/otel-collector Go package. While it lacked documentation, the code showed promise in producing clean execution traces. After making some modifications, we successfully generated and transmitted traces based on the “dagger journal” – yet another undocumented feature.

Our exploration didn’t stop there. We delved into the dagger.io/dagger/cmd/dagger-graph module, which, as you might have guessed, was another undocumented gem. Essentially, it creates a mermaid diagram based on the dagger journal. With slight code adjustments, we made it output the graph in a format compatible with the Grafana Graph Panel. This meant that we not only had the ability to capture traces but also generated a graphical representation of the DAG in Grafana.

Why new Pipelines?

One of the most challenging aspects of my job is convincing people to transition to a different CI system. The mere idea of switching to a new CI tool often triggers reluctance and a stigma among individuals. Vlad, from Earthly (another CI system built on Buildkit), has written about this problem in the Earthly Blog (Shutting down Earthly CI).

I’ve encountered this challenge many times. Many companies initially adopted CI out of necessity, not choice. They might have set up a Jenkins instance a decade ago, and, over time, someone started writing scripts, installing plugins, and continuously patching things together. This often led to a convoluted evolution of their CI systems, but not in a positive way. These systems morphed into disfigured creatures, laden with unsightly hacks and workarounds.

From an operational perspective, these systems frequently fail to align with the principles of a cloud-native environment.

In the realm of pipeline development, a fundamental shift is essential, and Dagger emerges as a game-changer for this. Its advantages offer a new Developer experience, reducing complexity by allowing even complex builds to be expressed as a few simple functions. The “push and pray” practice becomes obsolete as Dagger empowers developers to mirror CI capabilities within their local environments. With the ability to use the same language for app and pipeline development…, fostering a more efficient and collaborative environment. Achieving parity between development and CI environments fosters consistency, while the ease of cross-team collaboration enables the reuse of workflows without the need to learn different stacks.

And for platform teams driving the infrastructure, the adoption of Dagger as a CI system unlocks a multitude of benefits as well. Dagger’s versatility offers a significant advantage by eliminating CI lock-in, as its functions seamlessly operate across major CI platforms, negating the need for proprietary DSL. By empowering app teams to craft their own functions, Dagger catalyzes standardization through a library of reusable components, simultaneously liberating platform teams from being a bottleneck. Embracing Dagger not only fosters faster CI runs but also harmonizes the platform teams’ need for control with the app teams’ call for flexibility, heralding a new era in CI efficiency and collaboration.

New Release: Dagger v0.10.0

The release of Dagger v0.10.0 introduces a new feature for CI/CD pipelines: Dagger Functions. This new feature aims to change how developers manage their CI scripts, transforming cluttered and complex scripts into clear, efficient, and dependable code. It not only simplifies and speeds up CI pipelines but also improves their portability and makes the coding process more enjoyable.

For those interested in learning more about how Dagger Functions can redefine the CI/CD structure, continue reading for further details.

---

The past blogposts of this series:
CI Tools: The Legacy (Part 1/3)
CI Tools: Dagger, The Future (Part 2/3)