Using AWS ECR with Spinnaker and Kubernetes

Published in

The Spinnaker Community Blog

4 min readApr 17, 2017

As of January 27, 2017 AWS ECR started supporting the Docker Image Manifest V2, Schema 2 which means that Spinnaker users can now use ECR as an image repository where it was previously unsupported. This is great news for those who have already invested in using ECR with Kubernetes!

ECR, however, requires users tackle some other challenges that you don’t necessarily encounter using registries like Quay.io or Dockerhub. This post will hopefully help you use ECR while deploying images to Kubernetes with Spinnaker.

The Problem

With registries like Quay.io or Dockerhub, individual user accounts can be used to access repositories. Quay.io even has robot accounts that can be provisioned for use cases such as this. Unfortunately, things aren’t so easy with ECR. Logging into ECR with docker login requires an IAM Role that has access to your ECR Registry. If you have the correct permissions, you can then run aws ecr get-login to get your docker login command. This will output a command with as username and password, issued by AWS. The catch? The password rotates every so often, causing them to expire. The question then becomes: How can we work around this limitation and allow Spinnaker to interact with ECR?

This post assumes that you already have the appropriate IAM permissions in place to allow Spinnaker to access the registry as well as allowing Kubernetes to pull your images.

Sidecars to the Rescue

One of my favorite patterns to emerge from the Kubernetes community is sidecar containers. Sidecars are containers that run alongside your primary container to perform functions that aren’t directly built into the application itself. One example of sidecar use case if renewing certificates and updating them while an application is still running. Following this example, we will be doing something similar to ensure that our ECR authentication tokens stay up to date.

ecr-token-refresh

While trying to solve this problem, I ended up writing a application that refreshed authentication tokens on an interval. I’ve also dropped it in a container that can be used as a sidecar. You can find it on Github here. The work ecr-token-refresh is responsible for is dead simple. It takes a list of ECR registry IDs (AWS Account IDs) and regions, gets an authentication token (i.e. password) and writes that password to a file. The use case here is that you use a Volume that’s shared between Clouddriver and ecr-token-refresh for reading and writing these passwords. The best part? You don’t even have to bootstrap Clouddriver with these credentials. Just tell Clouddriver where the passwordFile is located and start the Pod.

Configuring ecr-token-refresh

Configuration of ecr-token-refresh is done via a YAML file. The schema is simple as there are only two required options. The snippet below is a basic configuration file, which is located at /opt/config/ecr-token-refresh/config.yaml by default.

interval: 30m #refresh every 30 minutes
registries:
  — registryId: “123456789”
    region: “us-west-2”
    passwordFile: “/opt/passwords/my-registry.pass”

That’s it! We’ve now told ecr-token-refresh to refresh our authentication tokens every 30 minutes for our registry in us-west-2. Additional registry entries can be added to the list as well!

Configuring Clouddriver

As I stated earlier, the idea of a sidecar is that it shares a volume between the main application, Clouddriver, and itself. Our instance of ecr-token-refresh will write files to /opt/passwords/, so we’ll need to make sure Clouddriver is reading passwords from the shared directory.

The first thing we need to do is configure our Docker registries in clouddriver-local.yml. We’ll configure our registry with ID “123456789” as my-registry.

# ...above omitted for brevitykubernetes:
  enabled: true
  accounts:
    - name: my-kubernetes-account
      dockerRegistries:
        - accountName: my-registrydockerRegistry:
  enabled: true
  accounts:
    - name: my-registry
      address: https://123456789.dkr.ecr.us-west-2.amazonaws.com
      username: AWS
      passwordFile: /opt/passwords/my-registry.pass
      repositories:
        - testrepo/testimage

You’ll notice that we’ve configured Clouddriver to find the password file for our registry at /opt/passwords/my-registry.pass which is the same name in the configuration we used for our sidecar.

The username AWS may vary across accounts. To figure out what yours is, run aws ecr get-login and see what it uses for the username.

Tying It All Together

The last step in setting up Spinnaker to use ECR is to add our sidecar to the Clouddriver ReplicaSet. We’ll assume that we’ve already added our ecr-token-refresh configuration as a ConfigMap called tokenRefresh.config.

What you’ll see in the manifest is that the Clouddriver container shares a mounted volume with our sidecar, mounted at /opt/passwords. This will enable our sidecar to write passwords to what is essentially the same directory that Clouddriver is reading them from. (I’ve omitted some configuration for simplicity).

apiVersion: extensions/v1beta1
kind: ReplicaSet
spec:
  template:
    spec:
      containers:
        - image: quay.io/spinnaker/clouddriver:master
          imagePullPolicy: Always
          name: clouddriver
          ports:
            - containerPort: 7002
              name: clouddriver
          volumeMounts:
            - mountPath: /opt/passwords
              name: token-refresh-shared
        - image: quay.io/skuid/ecr-token-refresh:latest
          name: ecr-token-refresh
          imagePullPolicy: Always
          volumeMounts:
           - name: token-refresh-config
             mountPath: /opt/config/ecr-token-refresh
           - name: token-refresh-shared
             mountPath: /opt/passwords
      volumes:
        - name: token-refresh-shared # shared volume
          emptyDir: {}
        - name: token-refresh-config
          configMap:
            name: tokenRefresh.config

After you add those entries to the manifest, you’re just a kubectl apply -f rs/spin-clouddriver.yml away from using ECR with Spinnaker!

Conclusion

Using ECR with Spinnaker may prove to be a bit more work than other services, but for users who are sticking with ECR, a sidecar is the best way to handle refreshing your credentials. Following this pattern, you can create a type of “set it and forget it” scenario which will feel as if you are using a simple username and password to access your registries. Hopefully teams who were previously blocked from using Spinnaker are no longer blocked and can start deploying their ECR images right alongside other registries!

If you have any questions or ideas for improvement, feel free to submit a PR or file an issue!