How to use the Pulumi Operator to build Docker Images

How to use the Pulumi Operator to build Docker Images

I recently faced a problem when trying to build a Docker image using the @pulumi/docker provider.

In this Blog Post I explain my journey on how I use the Pulumi Operator to build Docker Images and what are the problems I encountered.

If you simply want the solution, click here to scroll down


import * as docker from '@pulumi/docker'
import * as kubernetes from '@pulumi/kubernetes'

const image = new docker.Image("app", {
    build: {
        context: "../",
        args: {
            BUILDKIT_INLINE_CACHE: "1",
        },
        cacheFrom: {
            images: ["ghcr.io/kerwanp/pulumi-preview-test:latest"]
        }
    },
    imageName: "ghcr.io/kerwanp/pulumi-preview-test:latest",
})

const appLabels = { app: "preview-test" }

const deployment = new kubernetes.apps.v1.Deployment("app", {
    spec: {
        replicas: 1,
        selector: { matchLabels: appLabels },
        template: {
            metadata: { labels: appLabels },
            spec: {
                imagePullSecrets: [
                    {
                        name: 'regcred'
                    }
                ],
                containers: [
                    {
                        name: "app",
                        image: image.imageName,
                        imagePullPolicy: 'Always',
                    }
                ]
            }
        }
    }
})

And then deploy it by creating a stack using the Pulumi Operator:

---
apiVersion: pulumi.com/v1
kind: Stack
metadata:
  name: app-stack
  namespace: pulumi
spec:
  envRefs:
    PULUMI_ACCESS_TOKEN:
      type: Secret
      secret:
        name: pulumi-secret
        key: pulumiAccessToken
  gitAuth:
    basicAuth:
      password:
        type: Secret
        secret:
          name: pulumi-secret
          key: githubToken
      userName:
        type: Literal
        literal:
          value: kerwanp
  stack: kerwanp/preview-test/test
  projectRepo: https://github.com/kerwanp/pulumi-preview-test
  repoDir: deploy
  branch: "refs/heads/main"
  destroyOnFinalize: true

The problem

When the Pulumi Operator tries to deploy the stack it throws the following error:

failed to run update: exit status 255code: 255
stdout: Updating (test)

View Live: https://app.pulumi.com/kerwanp/preview-test/test/updates/1


 +  pulumi:pulumi:Stack preview-test-test creating (0s) 
@ Updating......
 +  pulumi:pulumi:Stack preview-test-test creating (2s) panic: runtime error: invalid memory address or nil pointer dereference
 +  pulumi:pulumi:Stack preview-test-test creating (2s) [signal SIGSEGV: segmentation violation code=0x1 addr=0x29 pc=0x124befc]
 +  pulumi:pulumi:Stack preview-test-test creating (2s) goroutine 27 [running]:
 +  pulumi:pulumi:Stack preview-test-test creating (2s) github.com/pulumi/pulumi-docker/provider/v4.dockerHybridProvider.Configure({{}, {0x21b7ca0, 0x26157, 0x26157}, {0x17dd3fc, 0x6}, {0x17fe470, 0xc0000d1400}, {0x17fe3c0, 0xc0000b0690}}, ...)
 +  pulumi:pulumi:Stack preview-test-test creating (2s)     /home/runner/work/pulumi-docker/pulumi-docker/provider/hybrid.go:93 +0x17c
 +  pulumi:pulumi:Stack preview-test-test creating (2s) github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Configure_Handler.func1({0x17f1f08, 0xc00042e6f0}, {0x14c4e00?, 0xc0007d7900})
 +  pulumi:pulumi:Stack preview-test-test creating (2s)     /home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/v3@v3.64.0/proto/go/provider_grpc.pb.go:462 +0x78
 +  pulumi:pulumi:Stack preview-test-test creating (2s) github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1({0x17f1f08, 0xc0006ffe90}, {0x14c4e00, 0xc0007d7900}, 0xc000136ac0, 0xc00043eca8)
 +  pulumi:pulumi:Stack preview-test-test creating (2s)     /home/runner/go/pkg/mod/github.com/grpc-ecosystem/grpc-opentracing@v0.0.0-20180507213350-8e809c8a8645/go/otgrpc/server.go:57 +0x3e8
 +  pulumi:pulumi:Stack preview-test-test creating (2s) github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Configure_Handler({0x153bb40?, 0xc0007644b0}, {0x17f1f08, 0xc0006ffe90}, 0xc0005da8c0, 0xc000653960)
 +  pulumi:pulumi:Stack preview-test-test creating (2s)     /home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/v3@v3.64.0/proto/go/provider_grpc.pb.go:464 +0x138
 +  pulumi:pulumi:Stack preview-test-test creating (2s) google.golang.org/grpc.(*Server).processUnaryRPC(0xc000234000, {0x17fa100, 0xc000702b60}, 0xc0003c5e60, 0xc00075cba0, 0x21f98e8, 0x0)
 +  pulumi:pulumi:Stack preview-test-test creating (2s)     /home/runner/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1345 +0xdf3
 +  pulumi:pulumi:Stack preview-test-test creating (2s) google.golang.org/grpc.(*Server).handleStream(0xc000234000, {0x17fa100, 0xc000702b60}, 0xc0003c5e60, 0x0)
 +  pulumi:pulumi:Stack preview-test-test creating (2s)     /home/runner/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1722 +0xa36
 +  pulumi:pulumi:Stack preview-test-test creating (2s) google.golang.org/grpc.(*Server).serveStreams.func1.2()
 +  pulumi:pulumi:Stack preview-test-test creating (2s)     /home/runner/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:966 +0x98
 +  pulumi:pulumi:Stack preview-test-test creating (2s) created by google.golang.org/grpc.(*Server).serveStreams.func1
 +  pulumi:pulumi:Stack preview-test-test creating (2s)     /home/runner/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:964 +0x28a
    docker:index:Image app  error: error reading from server: EOF
    docker:index:Image app **failed** 1 error
 +  pulumi:pulumi:Stack preview-test-test created (2s) 19 messages

Diagnostics:
  docker:index:Image (app):
    error: error reading from server: EOF

  pulumi:pulumi:Stack (preview-test-test):
    panic: runtime error: invalid memory address or nil pointer dereference
    [signal SIGSEGV: segmentation violation code=0x1 addr=0x29 pc=0x124befc]
    goroutine 27 [running]:
    github.com/pulumi/pulumi-docker/provider/v4.dockerHybridProvider.Configure({{}, {0x21b7ca0, 0x26157, 0x26157}, {0x17dd3fc, 0x6}, {0x17fe470, 0xc0000d1400}, {0x17fe3c0, 0xc0000b0690}}, ...)
        /home/runner/work/pulumi-docker/pulumi-docker/provider/hybrid.go:93 +0x17c
    github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvide

r_Configure_Handler.func1({0x17f1f08, 0xc00042e6f0}, {0x14c4e00?, 0xc0007d7900})
        /home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/v3@v3.64.0/proto/go/provider_grpc.pb.go:462 +0x78
    github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1({0x17f1f08, 0xc0006ffe90}, {0x14c4e00, 0xc0007d7900}, 0xc000136ac0, 0xc00043eca8)
        /home/runner/go/pkg/mod/github.com/grpc-ecosystem/grpc-opentracing@v0.0.0-20180507213350-8e809c8a8645/go/otgrpc/server.go:57 +0x3e8
    github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Configure_Handler({0x153bb40?, 0xc0007644b0}, {0x17f1f08, 0xc0006ffe90}, 0xc0005da8c0, 0xc000653960)
        /home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/v3@v3.64.0/proto/go/provider_grpc.pb.go:464 +0x138
    google.golang.org/grpc.(*Server).processUnaryRPC(0xc000234000, {0x17fa100, 0xc000702b60}, 0xc0003c5e60, 0xc00075cba0, 0x21f98e8, 0x0)
        /home/runner/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1345 +0xdf3
    google.golang.org/grpc.(*Server).handleStream(0xc000234000, {0x17fa100, 0xc000702b60}, 0xc0003c5e60, 0x0)
        /home/runner/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1722 +0xa36
    google.golang.org/grpc.(*Server).serveStreams.func1.2()
        /home/runner/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:966 +0x98
    created by google.golang.org/grpc.(*Server).serveStreams.func1
        /home/runner/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:964 +0x28a

Resources:
    + 1 created

Duration: 4s


stderr: warning: A new version of Pulumi is available. To upgrade from version '3.66.0' to '3.68.0', visit https://pulumi.com/docs/reference/install/ for manual instructions and release notes.

The Debuging Process

Replicating the issue

I opened a shell on the Pulumi Operator Pod to see if I can replicate the pulumi up inside the container.

I found that the Operator cloned my repository inside /tmp/pulumi-working/<stack-name> I jumped into it and simply ran pupumi up.

Problem...

getcwd() failed: No such file or directory

This error usually appears when the folder you are in does not exist anymore, so I decided to duplicate it and re-run pulumi up.

And voilà, error replicated!

Docker is simply not there..

If I try to build the Docker image by myself directly from the container I receive the following error:

Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

It makes completely sense! If you do not have a Docker Daemon running, you cannot build a Docker image.

The solution

The idea is to add a docker:dind container inside the Pulumi Operator pod to be used as a Docker Daemon.

Starting in 18.09+ the Docker Dind automatically generate TLS certificates. We have to mount a directory accross the containers to share thoses.

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pulumi-operator-fc266171
  namespace: pulumi
  spec:
    replicas: 1
    selector:
      matchLabels:
        name: pulumi-kubernetes-operator
    template:
      spec:
        volumes:
          - name: dind-storage
            emptyDir: {}
          - name: docker-certs-client
            emptyDir: {}
        containers:
          - name: dind
            image: docker:24-dind
            resources: {}
            volumeMounts:
              - name: dind-storage
                mountPath: /var/lib/docker
              - name: docker-certs-client
                mountPath: /certs/client
              # ^^^^^^
              # Mount volume to share certificates to the operator
            securityContext:
              privileged: true
            # ^^^^
            # Must be defined to run Docker
          - name: pulumi-kubernetes-operator
            image: pulumi/pulumi-kubernetes-operator:v1.12.0
            args:
              - "--zap-level=error"
              - "--zap-time-encoding=iso8601"
            env:
              - name: DOCKER_HOST
                value: 'tcp://localhost:2376'
                     # ^^^^
                     # Define address for remote Docker Daemon 
              - name: DOCKER_CERT_PATH
                value: '/certs'
                     # ^^^^
                     # Define custom certificate directory (mounted volume) 
              - name: DOCKER_TLS_VERIFY
                value: 1
                     # ^^^^
                     # Force Docker client to use TLS 
              - name: WATCH_NAMESPACE
                valueFrom:
                  fieldRef:
                    apiVersion: v1
                    fieldPath: metadata.namespace
              - name: POD_NAME
                valueFrom:
                  fieldRef:
                    apiVersion: v1
                    fieldPath: metadata.name
              - name: OPERATOR_NAME
                value: pulumi-kubernetes-operator
              - name: GRACEFUL_SHUTDOWN_TIMEOUT_DURATION
                value: 5m
              - name: MAX_CONCURRENT_RECONCILES
                value: "10"
            resources: {}
            volumeMounts:
              - name: docker-certs-client
                mountPath: /certs
              # ^^^^^^
              # Mount volume to get Docker certificates

            terminationMessagePath: /dev/termination-log
            terminationMessagePolicy: File
            imagePullPolicy: Always

You can actually check that it works by running docker info inside the Operator Pod.

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.10.4
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.17.3
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 24.0.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 1677a17964311325ed1c31e2c0a3589ce6d5c30d
 runc version: v1.1.7-0-g860f061
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
 Kernel Version: 5.15.90.1-microsoft-standard-WSL2
 Operating System: Alpine Linux v3.18 (containerized)
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 31.32GiB
 Name: pulumi-operator-anaxago-fc266171-6f896556dc-lxkvp
 ID: 32511430-499e-4fa7-bbce-9b801ffb77ec
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

Tada! 🎉 You can now build images using Pulumi and the Pulumi Operator!


I hope that this Blog Post helped you! If you have any questions, feel free to use the comment section! 💬

Oh and if you want more content like this, follow me: