LocalStack’s guide to run AWS serverless environment locally : Discover the power of Lambda + Docker + SQS

Because local testing is a must have for every developer, Localstack provides an AWS-like environment without breaking the bank.

[UPDATE 15–03–2021] — A work around for the last docker engine is to specify change ports: — “53:53” to ports: — “127.0.0.1:53:53” . https://github.com/localstack/localstack/issues/3700

[UPDATE 10–03–2021] — It seems like the new docker engine version do not work with the latest localstack image (problem with the port 53 forwarding). Keep the 20.10.2 version of docker-engine. If you are on MacOs download the 3.1.0 docker desktop version which is composed of the good version.

• Docker engine release note : https://docs.docker.com/engine/release-notes/
• Docker desktop release note for Mac : https://docs.docker.com/docker-for-mac/release-notes/

As AWS stands for one of the greatest cloud providers, one day you will probably have to deploy something on it : lambda functions, S3 storage, Kinesis Streams, SQS queues, DynamoDB… Even if devops is even more accessible with the AWS CDK (infrastructure as code using typescript, python and other languages), multi-environments management could be time consuming and quite expensive.

That’s what localstack is for, it provides a local environment with all the AWS resources available in your machine with the same AWS API.

Moreover, serverless event-based production infrastructure can’t be reproduced at all without Localstack. How can you trigger a lambda function when your SQS queue just received a message which has been pushed by one of your workers ? We are currently evolving in a world where serverless renews the way we used to build our infrastructure.

What’s Localstack ?

Localstack is an open-source project launched by Atlassian which mocks each AWS resources on your local machine. A big part is free such as Cloudformation, Dynamo, EC2, Kinesis, S3 but a great UI and some services need the «pro» version of localstack like EMR, docker lambda, Athena (everything is described here).

Anyway, you still will have 14 days free trial after registering. Such enough to complete this tutorial. The product is maintained and is evolving. By the way, you will be automatically added to their slack community channel as soon as your subscribe !

Localstack UI

Lambda news

While lambda was already an insane tool to manage performance and scalability in your infrastructure, it became even more powerful in the autumn 2020 during the AWS re:invent by supporting dockerized function. What an upgrade !

NO MORE ZIP PACKAGE, WELCOME DOCKER IMAGES

• Up to a 10GB docker image is allowed, which should be sufficient for all your dependencies for any programming languages.
• And because a great news never comes alone, lambda functions are now billed per milliseconds of time processing. (If optimized, it will save a lots of your money)

One of the common use of lambda is to bind them to S3 storage or SQS queue to perform specific actions. However, it can be a mess to setup from scratch without any third parties.

I will walk you through the complete road of how I have approached this. (with valuable tips at each step! )

What are we doing and why ?

In a serverless architecture, a powerful combo is to link an SQS queue and a lambda function to deal with asynchronous workflow. Something is pushing messages to any kind of queue and when you rise the number of events specified in the configuration a lambda is automatically deployed to handle those messages.

Serverless architecture

This is a pay on-demand serverless infrastructure which minimizes costs and allows scalability.

The biggest pain of this beautiful structure is to reproduce this workflow on your machine.

That’s what localstack is for! Use the exact same configuration than your staging/production environment without any additional $$$ expenses.

Since docker image has been supported by lambda, nothing explains how to deal with it!

Requirements

• Have Docker installed (https://docs.docker.com/get-docker/)
• Have subscribed to the localstack Pro version to use docker lambda images (https://app.localstack.cloud/dashboard). You have 14 days free trials after setup your account and add subscription
• Have installed AWS CLI V2 (https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html)
• Have installed awscli-local which make your config even more readable. No magic, the awslocal command replaces the standard aws --endpoint-url=http://localhost:4566. (https://github.com/localstack/awscli-local)
• pip/pip3 is required
• Have configured your AWS credentials. Run aws configure (no need to provide real access key and secret key but value must be setup either by this way or by the environment variable of the localstack container)

Setup

Localstack could be installed either using pip pip install localstack and launched with localstack start but we will use the consistent method regardless of your current working environment : Docker.

If you choose the first method, I can’t guaranty you the full guide will work as well as described. Don’t worry, just docker basis are required.

We are going through the complete developer workflow to ensure you that no secret remains lock. First of all, create a test folder where you will setup all of this project. (A GitHub repository will be linked at the end, but follow this tutorial to understand everything).

Localstack config

The docker-compose.yml is our configuration entry point where your will be able to make everything works perfectly. Create a .env with the LOCALSTACK_API_KEY value.

docker-compose.yml file

What the hell does that mean ?

We just created a docker service named localstack which is based on the localstack/localstack:0.12.6 image provided by the localstack team. We use the 0.12.6 version because everything seems to work on it right now.

As docker creates it’s own environment, we need to precise which local ports are bound to which container’s ports. For instance, we bind the ports range from 4566 to 4620 of your machine to the same of the container.

Some environment variables are needed to make this works :

• DEBUG=1 is used to provide more logs inside the container
• SERVICES=s3,sqs,lambda,sns lists services that you want to deploy
• DEFAULT_REGION=us-east-1 specifies the targeted region of your resources
• LAMBDA_EXECUTOR=docker tells localstack to use dedicated docker container to run your lambda functions (seems to be the best way to reproduce an AWS real infrastructure)
• LAMBDA_REMOTE_DOCKER=true and LAMBDA_REMOVE_CONTAINERS=true are additional config for lambda docker execution
• DATA_DIR=/tmp/localstack/data is the dedicated folder path used by localstack to save its own data
• DOCKER_HOST=unix:///var/run/docker.sock

Volumes are needed as docker does not store any state. Using them will allow persistent data and avoid build everything each time you launch your stack.

The .env file is specified to use the LOCALSTACK_API_KEY value. Please keep your credentials secrets

This config should be enough to make it work. Now, you need to open a terminal at the same path of your docker-compose file.

Test it!

For the first run, do not specify the -d option to run it in background : docker-compose up localstack

Localstack start logs

You should now have those logs with :

• INFO:botocore.credentials: Found credentials in environment variables
• INFO:localstack_ext.bootstrap.licensing: Successfully activated API key
• Waiting for all LocalStack services to be ready. Ready.

Check health and go to this url : http://localhost:4566/health

{"services": {"lambda": "running", "logs": "running", "s3": "running", "sns": "running", "sqs": "running", "cloudwatch": "running"}}

Localstack dashboard status

!! Tips !!

• Be careful with the latest tag. It is recommended to use it but the team often pushes updates on it without creating a new tag. So if you delete your localstack docker image even if you’ve pinned the latest one, you might have created a non-consistent environment. (You can see the last push date from the docker hub repository on the tag section)
• As explained before, we need to have the localstack pro version to make docker lambda works. After sign in, you will have to fill the billing information and create a subscription to be able to copy your own API_KEY from the dashboard tab.
• It is really recommend to use a .env to add your LOCALSTACK_API_KEY=XXXXX and use a .dockerignore and a .gitignore with .env. Avoid adding your personal API_KEY anywhere.
• When your docker runs, you will have to wait a bit (might be day) to see the status on the cloud UI to be on «running» state and have infos on your resources. But no worries, everything is still working well (the logs table on the dashboard will be up to date)
• I had to add sns service to the list even if I do not use it to make everything work (logs are straight forward)
• If you want to use a local dashboard, you can use the localstack-full image and the PORT_WEB env variable. It will be accessible from the http://localhost:8080 url.
• If you are using python lambda function, do not use the EDGE_PORT=4566 variable, it was not working for me and led to many hours of debugging with the localstack team — on slack community channel — to find the problem (might be solved right now but don’t be a fool, the default value is 4566 as well, no need to explicitly add it).
• Note that on MacOS you may have to run TMPDIR=/private$TMPDIR docker-compose up if $TMPDIR contains a symbolic link that cannot be mounted by Docker.

Now your Localstack env is setup, you can keep it running in background

docker-compose up --build -d localstack

Build your lambda image

Create a dedicated folder mkdir ~./lambda/python-lambda && cd ~./lambda/python-lambda and create a Dockerfile as followed :

Python lambda dockerfile

Our based image comes from AWS and imports everything we want to trigger our lambda function. After what we copy our requirements file and install it.

The CMD is composed of the <file_name>.<function_name>

And a handler.py file in a src/ folder which contains the lambda function code. Nothing impressive, just some logs to make sure everything works fine.

Lambda function

We can now build it!

Rundocker-compose build test-lambda

Test

We are now able to test our image to be sure everything works before deploying it with localstack. We have to run it with port mapping to execute a curl on it.

• docker run -p 9000:8080 test-lambda:latest
• curl -XPOST “http://localhost:9000/2015-03-31/functions/function/invocations" -d ‘{}’

Docker lambda executed with curl

Congrats! Your image is well formatted, we can now make it work all together

Make it work all together

Update our docker-compose.yml to create a dedicated bridge network to allow your containers to see each other. And add it either to the localstack and the test-lambda image. Rebuild everything now to avoid many creepy errors..

Full docker-compose.yml file

Be sure your localstack is up and running as described before and you have a well formatted lambda function image. Here all the commands we will use in this tutorial.

Create and invoke lambda

• Create the lambda function with the good image tag of your function. We need to specify a dummy role even if it is not created.

awslocal lambda create-function --function-name test-lambda --code ImageUri=test-lambda:latest --role arn:aws:iam::000000000:role/lambda-ex

Create-function output

Don’t be afraid, even if the PackageType is set to Zip, the lambda function is still based on a docker container.

Test it

awslocal lambda invoke --function-name arn:aws:lambda:us-east-1:000000000000:function:test-lambda response.json

Invoke output

Result should appear in a response.json file with true word inside.

Update lambda

You can update your lambda if you have made some changes on your image by running this command : awslocal lambda update-function-code --function-name test-lambda --image-uri test-lambda:latest

Create SQS queue and bind it

Create a dedicated sqs queue :

• awslocal sqs create-queue --queue-name test-lambda-queue

{
“QueueUrl”: “http://localhost:4566/000000000000/test-lambda-queue”
}

You can now bind it! Specify the — batch-size 1 argument to trigger a lambda function on each message received in your queue.

• awslocal lambda create-event-source-mapping --function-name test-lambda --batch-size 1 --event-source-arn arn:aws:sqs:us-east-1:000000000000:test-lambda-queue

{
“UUID”: “ebff1d61-d0d0–4513–89cd-26dd7aa78f13”,
“StartingPosition”: “LATEST”,
“BatchSize”: 1,
“EventSourceArn”: “arn:aws:sqs:us-east-1:000000000000:test-lambda-queue”,
“FunctionArn”: “arn:aws:lambda:us-east-1:000000000000:function:test-lambda”,
“LastModified”: “2021–02–24T23:32:51+01:00”,
“LastProcessingResult”: “OK”,
“State”: “Enabled”,
“StateTransitionReason”: “User action”
}

In the localstack dashboard you can now see your linked details resources you’ve just created.

Localstack Dashboard

Send message and check you docker dashboard!

• awslocal sqs send-message --queue-url http://localhost:4566/000000000000/test-lambda-queue --message-body “This is a test”

{
“MD5OfMessageBody”: “ce114e4501d2f4e2dcea3e17b546f339”,
“MessageId”: “0ddc5ea2-feef-95e1–3861–7dc1fd213295”
}

Check your dashboard, a new container will be created!

Docker dashboard

Check the logs : Everything works fine!

Lambda container logs

!! Tips !!

• You need to define a bridge network to allow containers to communicate
• If you want to use awslocal config in a script, we highly recommend to add cli_pager= to your ~/.aws/config file to disable pager program (especially if you are using iterm), awslocal cli should prevent those problems. Otherwise, your script will prompt the result of some commands. (https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html)

Conclusion

You can download this example (with a node.js lambda image as well) in the following Github repository : https://github.com/theodorebourgeon/template-localstack-lambda-docker-sqs

Everything is moving fast, AWS and localstack, but I have tried to provide a consistent tutorial to make your docker lambda function works with localstack.

I choose SQS as an event-broker, but you can setup Kinesis as well to make your local architecture match your staging environment.

I hope you enjoyed it! Feel free to share, comment and give claps!

Made with ❤

Resources

AWS

• https://docs.aws.amazon.com/lambda/latest/dg/images-create.html
• https://aws.amazon.com/fr/blogs/aws/new-for-aws-lambda-container-image-support/

Localstack

• https://localstack.cloud/
• https://github.com/localstack/localstack
• https://roadmap.localstack.cloud/
• https://github.com/localstack/localstack#troubleshooting
• https://hub.docker.com/r/localstack/localstack

Others

• https://github.com/aws/aws-cli/tree/v2
• https://docs.aws.amazon.com/cli/latest/index.html

Articles

• https://betterprogramming.pub/dont-be-intimidated-learn-how-to-run-aws-on-your-local-machine-with-localstack-2f3448462254
• https://medium.com/expedia-group-tech/development-and-testing-of-aws-without-aws-localstack-ab02f9425c40
• https://dev.to/goodidea/how-to-fake-aws-locally-with-localstack-27me