Theodore Bourgeon

Feb 25, 2021

11 min read

LocalStack’s guide to run AWS serverless environment locally : Discover the power of Lambda + Docker + SQS

Because local testing is a must have for every developer, Localstack provides an AWS-like environment without breaking the bank.

[UPDATE 15–03–2021] — A work around for the last docker engine is to specify change ports: — “53:53” to ports: — “127.0.0.1:53:53” . https://github.com/localstack/localstack/issues/3700

[UPDATE 10–03–2021] — It seems like the new docker engine version do not work with the latest localstack image (problem with the port 53 forwarding). Keep the 20.10.2 version of docker-engine. If you are on MacOs download the 3.1.0 docker desktop version which is composed of the good version.

As AWS stands for one of the greatest cloud providers, one day you will probably have to deploy something on it : lambda functions, S3 storage, Kinesis Streams, SQS queues, DynamoDB… Even if devops is even more accessible with the AWS CDK (infrastructure as code using typescript, python and other languages), multi-environments management could be time consuming and quite expensive.

That’s what localstack is for, it provides a local environment with all the AWS resources available in your machine with the same AWS API.

Moreover, serverless event-based production infrastructure can’t be reproduced at all without Localstack. How can you trigger a lambda function when your SQS queue just received a message which has been pushed by one of your workers ? We are currently evolving in a world where serverless renews the way we used to build our infrastructure.

What’s Localstack ?

Localstack is an open-source project launched by Atlassian which mocks each AWS resources on your local machine. A big part is free such as Cloudformation, Dynamo, EC2, Kinesis, S3 but a great UI and some services need the «pro» version of localstack like EMR, docker lambda, Athena (everything is described here).

Anyway, you still will have 14 days free trial after registering. Such enough to complete this tutorial. The product is maintained and is evolving. By the way, you will be automatically added to their slack community channel as soon as your subscribe !

Localstack UI

Lambda news

While lambda was already an insane tool to manage performance and scalability in your infrastructure, it became even more powerful in the autumn 2020 during the AWS re:invent by supporting dockerized function. What an upgrade !

NO MORE ZIP PACKAGE, WELCOME DOCKER IMAGES

One of the common use of lambda is to bind them to S3 storage or SQS queue to perform specific actions. However, it can be a mess to setup from scratch without any third parties.

I will walk you through the complete road of how I have approached this. (with valuable tips at each step! )

What are we doing and why ?

In a serverless architecture, a powerful combo is to link an SQS queue and a lambda function to deal with asynchronous workflow. Something is pushing messages to any kind of queue and when you rise the number of events specified in the configuration a lambda is automatically deployed to handle those messages.

Serverless architecture

This is a pay on-demand serverless infrastructure which minimizes costs and allows scalability.

The biggest pain of this beautiful structure is to reproduce this workflow on your machine.

That’s what localstack is for! Use the exact same configuration than your staging/production environment without any additional $$$ expenses.

Since docker image has been supported by lambda, nothing explains how to deal with it!

Requirements

Setup

Localstack could be installed either using pip pip install localstack and launched with localstack start but we will use the consistent method regardless of your current working environment : Docker.

If you choose the first method, I can’t guaranty you the full guide will work as well as described. Don’t worry, just docker basis are required.

We are going through the complete developer workflow to ensure you that no secret remains lock. First of all, create a test folder where you will setup all of this project. (A GitHub repository will be linked at the end, but follow this tutorial to understand everything).

Localstack config

The docker-compose.yml is our configuration entry point where your will be able to make everything works perfectly. Create a .env with the LOCALSTACK_API_KEY value.

docker-compose.yml file

What the hell does that mean ?

We just created a docker service named localstack which is based on the localstack/localstack:0.12.6 image provided by the localstack team. We use the 0.12.6 version because everything seems to work on it right now.

As docker creates it’s own environment, we need to precise which local ports are bound to which container’s ports. For instance, we bind the ports range from 4566 to 4620 of your machine to the same of the container.

Some environment variables are needed to make this works :

Volumes are needed as docker does not store any state. Using them will allow persistent data and avoid build everything each time you launch your stack.

The .env file is specified to use the LOCALSTACK_API_KEY value. Please keep your credentials secrets

This config should be enough to make it work. Now, you need to open a terminal at the same path of your docker-compose file.

Test it!

For the first run, do not specify the -d option to run it in background : docker-compose up localstack

Localstack start logs

You should now have those logs with :

Check health and go to this url : http://localhost:4566/health

{"services": {"lambda": "running", "logs": "running", "s3": "running", "sns": "running", "sqs": "running", "cloudwatch": "running"}}
Localstack dashboard status

!! Tips !!

Now your Localstack env is setup, you can keep it running in background

docker-compose up --build -d localstack

Build your lambda image

Create a dedicated folder mkdir ~./lambda/python-lambda && cd ~./lambda/python-lambda and create a Dockerfile as followed :

Python lambda dockerfile

Our based image comes from AWS and imports everything we want to trigger our lambda function. After what we copy our requirements file and install it.

The CMD is composed of the <file_name>.<function_name>

And a handler.py file in a src/ folder which contains the lambda function code. Nothing impressive, just some logs to make sure everything works fine.

Lambda function

We can now build it!

Rundocker-compose build test-lambda

Test

We are now able to test our image to be sure everything works before deploying it with localstack. We have to run it with port mapping to execute a curl on it.

Docker lambda executed with curl

Congrats! Your image is well formatted, we can now make it work all together

Make it work all together

Update our docker-compose.yml to create a dedicated bridge network to allow your containers to see each other. And add it either to the localstack and the test-lambda image. Rebuild everything now to avoid many creepy errors..

Full docker-compose.yml file

Be sure your localstack is up and running as described before and you have a well formatted lambda function image. Here all the commands we will use in this tutorial.

Create and invoke lambda

awslocal lambda create-function --function-name test-lambda --code ImageUri=test-lambda:latest --role arn:aws:iam::000000000:role/lambda-ex

Create-function output

Don’t be afraid, even if the PackageType is set to Zip, the lambda function is still based on a docker container.

Test it

awslocal lambda invoke --function-name arn:aws:lambda:us-east-1:000000000000:function:test-lambda response.json

Invoke output

Result should appear in a response.json file with true word inside.

Update lambda

You can update your lambda if you have made some changes on your image by running this command : awslocal lambda update-function-code --function-name test-lambda --image-uri test-lambda:latest

Create SQS queue and bind it

Create a dedicated sqs queue :

{
“QueueUrl”: “http://localhost:4566/000000000000/test-lambda-queue
}

You can now bind it! Specify the — batch-size 1 argument to trigger a lambda function on each message received in your queue.

{
“UUID”: “ebff1d61-d0d0–4513–89cd-26dd7aa78f13”,
“StartingPosition”: “LATEST”,
“BatchSize”: 1,
“EventSourceArn”: “arn:aws:sqs:us-east-1:000000000000:test-lambda-queue”,
“FunctionArn”: “arn:aws:lambda:us-east-1:000000000000:function:test-lambda”,
“LastModified”: “2021–02–24T23:32:51+01:00”,
“LastProcessingResult”: “OK”,
“State”: “Enabled”,
“StateTransitionReason”: “User action”
}

In the localstack dashboard you can now see your linked details resources you’ve just created.

Localstack Dashboard

Send message and check you docker dashboard!

{
“MD5OfMessageBody”: “ce114e4501d2f4e2dcea3e17b546f339”,
“MessageId”: “0ddc5ea2-feef-95e1–3861–7dc1fd213295”
}

Check your dashboard, a new container will be created!

Docker dashboard

Check the logs : Everything works fine!

Lambda container logs

!! Tips !!

Conclusion

You can download this example (with a node.js lambda image as well) in the following Github repository : https://github.com/theodorebourgeon/template-localstack-lambda-docker-sqs

Everything is moving fast, AWS and localstack, but I have tried to provide a consistent tutorial to make your docker lambda function works with localstack.

I choose SQS as an event-broker, but you can setup Kinesis as well to make your local architecture match your staging environment.

I hope you enjoyed it! Feel free to share, comment and give claps!

Made with ❤

Resources

AWS

Localstack

Others

Articles