I recently ran into a little issue trying to execute a startup script on
Google’s Container Optimized OS (COS). Running a startup script
using startup-script worked fine. For some reason, no matter the contents, anything specified in startup-script-url
failed to execute; and it seemed to be only COS that is affected.
Trawling the log data
Thankfully, all the startup information is logged in the OS’ journal. Trawling through the logs (using the command journalctl),
I came across the following few lines:
startup-script: INFO Starting startup scripts.
startup-script: INFO Found startup-script-url in metadata.
startup-script: WARNING gsutil is not installed, cannot download items from Google Storage.
startup-script: INFO No startup scripts found in metadata.
startup-script: INFO Finished running startup scripts.
Notice the line containing WARNING. I had naively thought that the gcloud and gsutil binaries were already bundled
inside COS (as they are with other images). It turns out this isn’t the case.
Attempting to fix it
Considering gsutil isn’t available, my first thought was to reference the GCS URL for the startup script: something
like https://storage.cloud.google.com/test-bucket-name/startup.sh. However, it appears this still requires the use
of gsutil to download the contents.
Making use of the public URL (such as https://storage.googleapis.com/test-bucket-name/startup.sh) seems to circumvent
the need to use gsutil. The log entries below show how GCE is able to fetch a publicly accessible startup script (Simply
pulling the latest version of the Alpine Docker image as a test):
startup-script: INFO Starting startup scripts.
startup-script: INFO Found startup-script-url in metadata.
startup-script: INFO Downloading url from https://storage.googleapis.com/test-bucket-name/startup.sh to /var/lib/google/startup-aXNBqI/tmpyUFrNE.
startup-script: INFO startup-script-url: + docker pull alpine:3.6
startup-script: INFO startup-script-url: 3.6: Pulling from library/alpine
startup-script: INFO startup-script-url: 88286f41530e: Pulling fs layer
startup-script: INFO startup-script-url: 88286f41530e: Verifying Checksum
startup-script: INFO startup-script-url: 88286f41530e: Download complete
startup-script: INFO startup-script-url: 88286f41530e: Pull complete
startup-script: INFO startup-script-url: Digest: sha256:f006ecbb824d87947d0b51ab8488634bf69fe4094959d935c0c103f4820a417d
startup-script: INFO startup-script-url: Status: Downloaded newer image for alpine:3.6
startup-script: INFO startup-script-url: Return code 0.
startup-script: INFO Finished running startup scripts.
If you have no sensitive information in your startup scripts and are happy for your startup scripts to be publicly accessible, then this is your solution.
If you don’t want to leave your startup scripts as publicly available, then continue reading for some ways of securing them.
Securing the startup script
There are a few ways of securing your startup scripts. Ideally, the best way would be to not include any sensitive data in your startup scripts (the metadata server is available, which could be used to keep sensitive data out of your scripts) - thus removing the requirement for them to be secured.
However, this is not always possible. So, here are three methods that I know of that possibly be used to secure your startup scripts:
1. Host the startup scripts on an external server
If you spin up a separate server, you could store the startup scripts on this server. Using something like NGiNX’s secure link module, you are able to ensure that a specific token is provided before accessing your startup scripts.
The downside to this is that you need a separate and standalone server to host these scripts.
2. Host the startup scripts on an internal server
Pretty much the same as the previous suggestion, with one difference. Ensure that the server hosting the startup scripts has no external IP address assigned (assuming this is a GCE instance in the same project), and reference the download URL using the internal project hostname.
Side note: I haven’t tested this method, but I don’t see why it wouldn’t work.
3. Use gsutil to generate a signed URL
The gsutil command has the ability to generate a signed URL. This is basically a URL that is pubcliy valid for a defined
period of time, but only if you know the full URL.
The following example shows how a GCS object is signed, and publicly available via GET requests for the next two minutes:
gsutil signurl -d 2m -m GET path/to/serviceAccount.json gs://test-bucket-name/startup.sh
This generates a signed URL like the following:
https://storage.googleapis.com/test-bucket-name/startup.sh?GoogleAccessId=xxx@project-id.iam.gserviceaccount.com&Expires=1508428954&Signature=xxx
Using this method, you’re able to ensure your startup script is publicly available only for those with the direct link and those who access it within the specified time period.
As you can see, this is a very manual process. However, there’s nothing stopping this from being automated.
I hope this helps in ensuring your COS images are able to make use of the startup-script-url metadata keys in Google
Compute Engine. If you have any additional ideas on how to secure your startup script URLs, please drop a comment below.