Spinnaker/Echo + Google Cloud Functions + Stackdriver Logging == Spinnaker Audit Log

(Co-authored with Daniel Peach)

A good audit log allows you to easily determine who did what, where and when. In this article we demonstrate how to create an audit log for Spinnaker by integrating Spinnaker’s Echo microservice with Google Cloud Functions and Stackdriver Logging.

Spinnaker’s workflow engine, Orca, pushes all events to Echo, an eventing and notification service. Echo can be configured to propagate these events to an external url via rest calls.

We will use this webhook capability to push events to a new HTTP Cloud Function. Then our function will consume the event stream and publish human-friendly messages to Stackdriver Logging.

Google Cloud Functions is a serverless platform for building event-based microservices. Cloud Functions are written in Javascript and execute in a standard Node.js runtime environment.

You’ll first need to enable the Cloud Functions APIs:

Next, you’ll need to update your gcloud cli with the beta component, if it’s not already installed:

$ gcloud components install beta

It’s usually a good idea to update the other gcloud components as well:

$ gcloud components update

Now we’re ready to create our Cloud Function.

Create a Cloud Storage bucket to stage your Cloud Functions artifacts, where YOUR_STAGING_BUCKET_NAME is a globally-unique bucket name:

$ gsutil mb gs://YOUR_STAGING_BUCKET_NAME

Create a directory on your local system for the application code:

$ mkdir gcf_spinnaker && cd gcf_spinnaker

Create a config.json file in the gcf_spinnaker directory with the following contents:

{
  "USERNAME": "SOME_USERNAME",
  "PASSWORD": "SOME_PASSWORD",
  "TIMEZONE": "America/New_York",
  "PROJECT_ID": "YOUR_PROJECT_ID",
  "CREDENTIALS_PATH": "YOUR_CREDENTIALS_PATH",
  "AUDIT_LOG_NAME": "SOME_AUDIT_LOG_NAME"
}

Make sure to note whatever values you select for the username and password as we’ll need them again to configure Echo.

You can specify any timezone from this list.

Note that it is your GCP project id, not project name that is required.

For the credentials, you’ll need to retrieve a service account key. Ensure that the associated service account has permission to write logs. Save the .json file in the gcf_spinnaker directory and set the value of the credentials path to ./MY-SVC-ACCT-KEY.json.

Select a name for your audit log; e.g. mySpinnakerAuditLog. You’ll need this name when you want to view the logs.

Download the index.js file from the Spinnaker Cloud Functions sample project on GitHub and save it to the gcf_spinnaker directory.

Download the package.json file and save it to the gcf_spinnaker directory.

If you open the index.js file in your local editor you’ll see verifyWebhook and spinnakerAuditLog functions. The verifyWebhook function uses basic authentication to verify the username and password of incoming webhooks. Once the request is authorized, the spinnakerAuditLog function processes the incoming event and publishes human-friendly messages to Stackdriver Logging.

Deploy the spinnakerAuditLog function with an http trigger:

$ gcloud beta functions deploy spinnakerAuditLog --source . --stage-bucket YOUR_STAGING_BUCKET_NAME --trigger-http --memory 2048MB

Once the deploy operation completes, note the httpsTrigger entrypoint:

$ gcloud beta functions deploy spinnakerAuditLog --local-path . --stage-bucket YOUR_STAGING_BUCKET_NAME --trigger-http --memory 2048MB
Copying file:///var/folders/00/1vqr8000h01000cxqpysvccm007fz1/T/tmpDIdlvT/fun.zip [Content-Type=application/zip]…
/ [1 files][ 4.4 KiB/ 4.4 KiB]
Operation completed over 1 objects/4.4 KiB.
Waiting for operation to finish…done.
availableMemoryMb: 256
entryPoint: spinnakerAuditLog
httpsTrigger:
  url: https://us-central1-YOUR-PROJECT-ID.cloudfunctions.net/spinnakerAuditLog

Let’s use this entrypoint to test the new function:

$ curl https://us-central1-YOUR-PROJECT-ID.cloudfunctions.net/spinnakerAuditLog

The request should be rejected with a 401 status code since it was not authorized.

Now with the proper basic authorization headers:

$ curl -u SOME_USERNAME:SOME_PASSWORD https://us-central1-YOUR-PROJECT-ID.cloudfunctions.net/spinnakerAuditLog

Since we haven’t sent any request body, the request should return “Spinnaker audit log request body is malformed.” But we will have verified that the function is reachable and is attempting to authorize incoming requests.

Navigate to the Stackdriver Logging page and view the logs for your new function:

There should be log entries from our testing of the entrypoint.

Let’s next configure Echo to send events to our new webhook. In your echo-local.yml file, add the following configuration:

rest:
  enabled: true
  endpoints:
    - wrap: true
      flatten: false
      url: https://us-central1-YOUR-PROJECT-ID.cloudfunctions.net/spinnakerAuditLog
      username: SOME_USERNAME
      password: SOME_PASSWORD
      eventName: spinnaker_events

After updating Echo’s configuration, make sure to cycle Echo to pick up the configuration changes.

We have created our function, verified its reachability and security, located its logs, and configured Echo to push events to its entrypoint. We’re all set to interact with Spinnaker and monitor our new audit log.

Most log entries are written at the info level. But cancellations are written at the warning level and failures are written at the error level. The lowest-level details, which you probably don’t need to regularly examine, are written at the debug level.

The audit log will include the user, if provided, whether each activity was ad-hoc or initiated via a pipeline, if a reason was provided for the activity, the timestamp of the activity, and so on.

If your Stackdriver Logging console doesn’t become populated with the new logging messages, you should check your local echo log file to see if there are any rest errors (in case something was misconfigured).

You now have an audit log for Spinnaker that is available any time you can reach your cloud console, and without having to manage any new infrastructure. You can even populate an audit log in a different GCP project from the one in which your Spinnaker installation runs.