Handling Tricky Dependencies For Serverless With Lambda Layers

I’ve fallen behind on my daily blog post initiative, but oh well. It turns out that I don’t really have time to write a good blog post every day. But at least this time I have a good excuse, because I was busy swimming in the ocean (and it was awesome).

“Serverless” computing is a new-ish paradigm with many use cases. I tend to use it for anything that I can lately, since I never have to worry about uptime and I also don’t have to pay for it (my projects stay small enough to live within the free tier—especially with the growing number of providers these days).

Basically, serverless just means “some unknown server that we don’t have to worry about”. So if we’re running some server code that isn’t too coupled with a persistent filesystem, we can probably just give the code to AWS or another cloud provider and tell them to run it on our behalf. There are no servers for us to maintain, so there may as well be no servers at all. I’ll probably talk more about serverless computing in another post, but for now I’m going to focus on a more specific topic.

AWS Lambda offers a bunch of runtimes for different programming languages. But suppose we wanted to build a Jekyll-powered static site as part of our serverless application, but that application was written in Python instead of Ruby (Jekyll’s language of implementation). If we owned the server running the application, we could just install Jekyll on it and our static sites would be one subprocess.run away. But serverless computing doesn’t work that way; any non-standard environment setup that would normally be a one-time job would have to be repeated on every Lambda function invocation, since we don’t control the lifetime of the servers that run our code. It should be fairly obvious that such an overhead would slow our application down to a crawl.

Alternatively, we could split whatever code we needed to call that Ruby program into a new function that used the Ruby runtime. However, having to write code in an unfamiliar language just to run an external program that should be a black box is not ideal. And of course, not all dependencies will be conveniently written in languages that have corresponding serverless runtimes.

Our last resort would be to bundle any necessary resources with our deployment package, but having to upload a massive archive every time we updated our code would become very tedious.

Thankfully, AWS provides a solution to this conundrum: Lambda Layers. In a nutshell, they let us define a filesystem overlay with whatever files we want, which can include libraries and executables. So to solve our example problem, we could create a layer with a Ruby installation and the Jekyll gem installed, then include that layer in our Jekyll-using function.

I’ll be demonstrating the process with the Serverless Framework, since that’s what I use to develop serverless applications, but the steps for Lambda on its own should be similar. For other providers, I don’t think a similar mechanism exists yet. Here’s the necessary configuration/code (see the Serverless Layers documentation for more information):

# serverless.yml

service: jekyll-on-python
provider:
  name: aws
  runtime: python3.7
layers:
  jekyll:
    path: ./layer
    package:
      exclude:
        - ./**
      include:
        - ./bin/**
        - ./lib/**
functions:
  build:
    handler: jekyll.handler
    layers:
      - !Ref JekyllLambdaLayer

# jekyll.py

import os
import subprocess

def handler(_evt=None, _ctx=None):
    site = path_to_site_to_be_built()
    os.chdir(site)
    subprocess.run(["jekyll", "build"], check=True)
    # TODO: Do something with the output.

The next step is to put the required files at ./layer so that they’ll be added to the Lambda environment.

There are a few approaches that we could take, the most obvious being “copy our existing Ruby/Jekyll installation to the destination”. This might work for some dependencies, like static binaries, if we’re running Linux on our local machine, but it’s far from a generic solution. Ruby, for example, won’t work when transplanted to a different location on disk, since it relies on some absolute paths.

A more thorough method, if we’re running Linux, would be to compile Ruby at our destination, which should solve our path issues. We can build Ruby easily, thanks to ruby-build. This probably works, but it’s not optimal.

Lambda runs its functions in Amazon Linux, so if we want to ensure compatibility, we should build our dependencies in the exact version of Amazon Linux that Lambda is running. The easiest way to do this, by far, is with Docker. We can use the Amazon Linux Docker image and Docker bind mounts to build our dependencies and leave the results in a local directory outside of the container. Here’s a handy script that will do the job:

#!/usr/bin/env bash

set -e

GEMS="jekyll"
IMAGE="amazonlinux:2018.03"
RUBY_VER="2.6.3"

if [ "$1" = build ]; then
  yum -y update
  yum -y install bzip2 findutils gcc-c++ openssl-devel readline-devel zlib-devel
  curl -L https://github.com/rbenv/ruby-build/archive/v20190615.tar.gz | tar zxf -
  RUBY_CONFIGURE_OPTS="--enable-shared" ./ruby-build-20190615/bin/ruby-build --verbose "$RUBY_VER" /opt
  /opt/bin/gem install $GEMS
  rm -rf /opt/{include,share}
else
  cd $(dirname "$0")
  script=$(basename "$0")
  docker run -t --rm --mount "type=bind,source=$(pwd),destination=/opt" "$IMAGE" "/opt/$script" build
fi

The key here is the Docker --mount argument which causes the source to be mounted inside the container at destination, thus making any changes to that directory inside the container persist outside of it as well.

Jekyll is a pretty good example gem, since it includes not just Ruby code, but dependencies with native extensions as well. The layer gets built properly, but there’s a packaging problem:

$ sls deploy
# ...
  ENOENT: no such file or directory, open '/path/to/service/layer/lib/ruby/gems/2.6.0/gems/ffi-1.11.1/ext/ffi_c/libffi-x86_64-linux/include/ffitarget.h'
# ...

This might seem weird, because if we go looking for that file, it definitely exists. However, it’s a symlink, and it points to the absolute path /opt/lib/ruby/..., which existed in our Docker container, but doesn’t on our local machine. I’m not sure if this would be an issue without the Serverless Framework, but either way, we can work around it by replacing the link with the actual file:

$ cd layer/lib/ruby/gems/2.6.0/gems/ffi-1.11.1/ext/ffi_c
$ sudo cp --remove-destination libffi/src/x86/ffitarget.h libffi-x86_64-linux/include/

In case you encounter this error more than once, this StackExchange question might be useful.

Now that we’ve dealt with our issues, sls deploy should work correctly and our application should be deployed. It’ll take some time to package up and upload the layer the first time. But after the first upload, we get the major benefit of the technique: we can update our application code without having to re-upload the layer. This is the difference between uploading 5KB and uploading 60MB on every code update.

I’ve used this strategy this most notably for running GitHub Changelog Generator from a Python function, and the benefits have been significant. Code deploys are orders of magnitude faster, testing and CI is simpler, and I don’t have to deal with maintaining any components that are isolated from the rest of my application.