CI Tools: Dagger, The Future (Part 2/3)

Creating a Dagger pipeline

The figure below showcases a pipeline encompassing multiple steps. At its core, it executes a classic multi-stage build, featuring a build container (Container B) and a runtime container (Container R). This runtime container is an Alpine image embedded with the application binary. Additionally, we’ll be establishing a service binding to facilitate the execution of unit tests against a Redis database.

package main

import (
	"context"
	"os"

	"dagger.io/dagger"
)

func main() {
	ctx := context.Background()

	// create dagger client
	client, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stderr))
	if err != nil {
		panic(err)
	}
	defer client.Close()

	// get host directory
	project := client.Host().Directory(".")

	// build app
	builderContainer := client.Container().
		From("golang:1.20").
		WithDirectory("/src", project).
		WithWorkdir("/src").
		WithEnvVariable("CGO_ENABLED", "0").
		WithExec([]string{"go", "build", "-o", "myapp"})

	//Create redis container
	redis := client.Container().From("redis").WithExposedPort(6379)

	//Create service binding and run tests against redis cache
	_, err = builderContainer.WithServiceBinding("redis", redis).
		WithExec([]string{"go", "test", "./..."}).Sync(ctx)
	if err != nil {
		panic(err)
	}

	// publish binary on alpine base
	runtimeImage := client.Container().
		From("alpine:3.18").
		WithFile("/bin/myapp", builderContainer.File("/src/myapp")).
		WithEntrypoint([]string{"/bin/myapp"})

	_, err = runtimeImage.Publish(ctx, "ttl.sh/myapp:latest")
	if err != nil {
		panic(err)
	}

}

Let’s dive deeper into the process of harnessing Dagger to orchestrate an ERD (Entity Relationship Diagram) task. Initially, we use the Connect(ctx Context) function, aiming to establish a connection with an existing Dagger engine. If an engine is unavailable, Dagger steps in to spawn a new one on our behalf. The corresponding code to achieve this is as follows:

client, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stderr))

Following the initial connection, our next move is to access the local directory from the host. This directory holds all the requisite code. By invoking the Host() function, we obtain a representation of the host’s execution environment, enabling us to seamlessly access the files located on our host system. The corresponding code snippet for this procedure is:

project := client.Host().Directory(".")

Subsequent to accessing our local directory, our next objective is to fetch the golang build image. This is achieved using the client.Container().From function. This function gives us a container, which we can easily set up and run commands against. The code snippet illustrating this step is as follows:

builderContainer := client.Container().
    From("golang:1.20").
    WithDirectory("/src", project). //Copy project files into container
    WithWorkdir("/src"). //Set the workind directory within the container
    WithEnvVariable("CGO_ENABLED", "0"). //Set env var
    WithExec([]string{"go", "build", "-o", "myapp"}) //Execute command on container

The functions we invoke within Dagger are very similar to the commands we typically use in a Dockerfile.

• WithDirectory -> COPY
• WithEnvVariable -> ENV
• WithWorkdir-> WORKDIR
• WithExec -> RUN

In the next section we’re creating a new Container from the latest Redis image and expose the default Redis port. After that we can create a service binding to our build container. This allows us to connect from our build container to the Redis container.
As soon we created our service binding, we can run the go test command in our builder container

//Create redis container
redis := client.Container().From("redis").WithExposedPort(6379)

//Create service binding and run tests against redis cache
_, err = builderContainer.WithServiceBinding("redis", redis).
    WithExec([]string{"go", "test", "./..."}).Sync(ctx)
if err != nil {
    panic(err)
}

Once our application has been built and tested, our next step is to create a runtime container, optimized to only contain our application binary. To facilitate this, we create a new container using the alpine:3.18 base image. Dagger provides the WithFile() function, a utility allowing us to transfer individual files either from our host system or from another container. In our current context, we’ll be utilizing this function to transfer our application binary from the build container to our new runtime container.

// publish binary on alpine base
runtimeImage := client.Container().
    From("alpine:3.18").
    WithFile("/bin/myapp", builderContainer.File("/src/myapp")).
    WithEntrypoint([]string{"/bin/myapp"})

In the final step, we aim to publish our runtime image to a container registry. To accomplish this, we simply invoke the Publish() function, specifying the target registry URL to which our image will be pushed.

_, err = runtimeImage.Publish(ctx, "ttl.sh/myapp:latest")
if err != nil {
    panic(err)
}

Problems Dagger solves

Fast Feedback and Reduced Overhead

Dagger fundamentally revolutionizes the feedback process by allowing pipelines to run directly on your local machine. All that’s required is an OCI-compatible runtime, such as Docker. This means you no longer have to commit changes to Git and then anxiously wait for a cloud runner to become available. Instead, simply modify your pipeline code, save it, execute, and instantly verify the results.

Additionally, Dagger’s inherent design with containers simplifies caching. Your pipeline code and application code remain distinct, ensuring that only jobs directly impacted by pipeline changes are executed. All other jobs benefit from default caching, which not only speeds up the feedback loop but also prevents unnecessary executions of unchanged jobs.

Monitoring and Observability with Dagger

Integrated OpenTelemetry Support: Dagger seamlessly integrates with OpenTelemetry, making pipeline insights more accessible than ever. In our implementation, we’ve linked Dagger to our InfluxDB, allowing us to store and analyze pipeline traces. Every instruction relayed to Buildkit by Dagger is meticulously traced, facilitating the identification and rectification of any prolonged actions within the pipeline.

Consistent Metrics Collection: With Dagger, the location of your pipeline execution becomes irrelevant. Whether it’s run on your local machine or a remote server, Dagger can consistently send metrics to a centralized collector. This includes popular platforms like Grafana, Loki, Tempo, Dagger Cloud, and Elasticsearch, ensuring uniformity and ease in data analysis.

Visual Insights with dagger-graph: Dagger also introduces dagger-graph, a valuable tool for visualizing your Directed Acyclic Graph (DAG). It helps in pinpointing potential bottlenecks, critical fan-in/fan-out scenarios and other aspects of your pipeline that might demand attention.

This visual representation even distinguishes between cached and uncached steps in your pipeline, offering a holistic view of your system’s efficiency.

Debugging

While Dagger has emerged as a revolutionary tool in many aspects of CI/CD, there’s room for enhancement when it comes to debugging. As of now:

Limited Debugging: Dagger currently does not extend debugging capabilities to your containers. The debugging process is constrained to the pipeline code, which might not be comprehensive enough for all debugging scenarios. This limitation could impact the user experience, especially when intricate issues arise within containers.

The Promise of the Future: On a positive note, there’s active acknowledgment of this limitation. An open issue on Github is addressing this feature, indicating the developers’ commitment to enhancing Dagger’s functionality.

Potential for Seamless Integration: Given that Dagger’s architecture revolves around containers, there doesn’t seem to be a technical impediment to integrating debugging capabilities. Once this feature rolls out, it promises to transform the debugging experience, making Dagger even more versatile and user-friendly.

In essence, while Dagger might have a temporary shortcoming in the debugging department, the roadmap suggests promising advancements. With debugging capabilities, Dagger will further solidify its position as a comprehensive CI/CD tool.

Templating/Reusability

Dagger’s language-independent API enables you to compose pipelines in your preferred programming language. This capability significantly simplifies the process of crafting custom templates suited to your unique requirements. You can define functions in the language you’re most comfortable with, be it Go, Python or JavaScript. Such versatility reduces time and effort by removing the necessity of mastering a distinct language solely for pipeline creation.

By making custom templates with Dagger, you can encapsulate and repurpose current patterns across your pipelines. For example, if there’s a recurring sequence of steps across various pipelines, you can design a template for that sequence. These templates are versioned and can be shared with your team, enhancing both collaboration and the manageability of pipeline maintenance.

Opting to create templates in Go offers the advantage of the language’s robust typing system. Consequently, your templates become less liable to errors, reinforcing their reliability and ease of maintenance in the long run. Additionally, the Godoc tool can be used to proficiently document your templates. Such comprehensive documentation helps in establishing proper abstractions and more reusable, extensible pipeline modules. It also streamlines the process of maintaining and updating your pipelines as your project evolves. In summary, Dagger empowers you to harness the capabilities of your chosen programming language for bespoke template creation, optimizing both time and effort and enhancing the sustainability and dependability of your pipelines.

Vendor Lock-in

Despite the adoption of Dagger, the specter of vendor lock-in still exists, given that developers remain bounded to the Dagger API and the SDK of their choice. Nonetheless, Dagger does offer a key advantage: it facilitates the migration of pipelines from one CI tool, say GitHub, to another like GitLab CI, eliminating the need for a complete pipeline rewrite. Such a feature introduces a increased level of flexibility and portability among different CI tools, thereby decreasing the risks associated with commitment to a singular vendor or platform.

To illustrate, our pipeline only mandates a few prerequisites: an OCI-compatible runtime such as Docker, and the Go SDK, specifically version 1.20. That’s the extent of it.

Conclusion

We seem to be close to a new era in CI Tooling. With its foundation built on cloud-native standards such as OCI, GraphQL, and OpenTelemetry, Dagger integrates seamlessly into our existing setups. While I hold GitLab in high regard — it brilliantly addresses many nuances of contemporary software development and product management — I’ve found pipeline development with Dagger to be a cut above the rest. Its consolidation of current software standards with a language-agnostic API creates a unique experience. The rapid feedback and potential for local development with Dagger significantly enhance the efficiency of pipeline development and execution.

Though Dagger is still a newcomer in its field, with good room for growth — especially in the realms of documentation, examples, and best practices — I’m optimistic about its potential to revolutionize the approach engineers take towards pipeline development.

The game changer isn’t merely about the speed of pipelines; it’s about crafting and maintaining them with superior efficacy.

Yes, Tony, after experiencing my personal CI hell, I can only assume this is heaven.

Stay tuned for our forthcoming blog post, where we’ll delve deeper, guiding you through Dagger’s installation within a Kubernetes setup and sharing valuable insights on creating or transitioning to Dagger pipelines.

_________________________________________________________________________________________________________________

The past blogposts of this series:
The state of CI Tools in 2023 – The Legacy (part 1/3)