LocalStack’s guide to run AWS serverless environment locally : Discover the power of Lambda + Docker + SQS

What’s Localstack ?

Localstack UI

Lambda news

NO MORE ZIP PACKAGE, WELCOME DOCKER IMAGES

  • Up to a 10GB docker image is allowed, which should be sufficient for all your dependencies for any programming languages.
  • And because a great news never comes alone, lambda functions are now billed per milliseconds of time processing. (If optimized, it will save a lots of your money)

I will walk you through the complete road of how I have approached this. (with valuable tips at each step! )

What are we doing and why ?

Serverless architecture

The biggest pain of this beautiful structure is to reproduce this workflow on your machine.

Since docker image has been supported by lambda, nothing explains how to deal with it!

Requirements

Setup

Localstack config

docker-compose.yml file
  • DEBUG=1 is used to provide more logs inside the container
  • SERVICES=s3,sqs,lambda,sns lists services that you want to deploy
  • DEFAULT_REGION=us-east-1 specifies the targeted region of your resources
  • LAMBDA_EXECUTOR=docker tells localstack to use dedicated docker container to run your lambda functions (seems to be the best way to reproduce an AWS real infrastructure)
  • LAMBDA_REMOTE_DOCKER=true and LAMBDA_REMOVE_CONTAINERS=true are additional config for lambda docker execution
  • DATA_DIR=/tmp/localstack/data is the dedicated folder path used by localstack to save its own data
  • DOCKER_HOST=unix:///var/run/docker.sock

This config should be enough to make it work. Now, you need to open a terminal at the same path of your docker-compose file.

Test it!

Localstack start logs
  • INFO:botocore.credentials: Found credentials in environment variables
  • INFO:localstack_ext.bootstrap.licensing: Successfully activated API key
  • Waiting for all LocalStack services to be ready. Ready.
{"services": {"lambda": "running", "logs": "running", "s3": "running", "sns": "running", "sqs": "running", "cloudwatch": "running"}}
Localstack dashboard status

!! Tips !!

  • Be careful with the latest tag. It is recommended to use it but the team often pushes updates on it without creating a new tag. So if you delete your localstack docker image even if you’ve pinned the latest one, you might have created a non-consistent environment. (You can see the last push date from the docker hub repository on the tag section)
  • As explained before, we need to have the localstack pro version to make docker lambda works. After sign in, you will have to fill the billing information and create a subscription to be able to copy your own API_KEY from the dashboard tab.
  • It is really recommend to use a .env to add your LOCALSTACK_API_KEY=XXXXX and use a .dockerignore and a .gitignore with .env. Avoid adding your personal API_KEY anywhere.
  • When your docker runs, you will have to wait a bit (might be day) to see the status on the cloud UI to be on «running» state and have infos on your resources. But no worries, everything is still working well (the logs table on the dashboard will be up to date)
  • I had to add sns service to the list even if I do not use it to make everything work (logs are straight forward)
  • If you want to use a local dashboard, you can use the localstack-full image and the PORT_WEB env variable. It will be accessible from the http://localhost:8080 url.
  • If you are using python lambda function, do not use the EDGE_PORT=4566 variable, it was not working for me and led to many hours of debugging with the localstack team — on slack community channel — to find the problem (might be solved right now but don’t be a fool, the default value is 4566 as well, no need to explicitly add it).
  • Note that on MacOS you may have to run TMPDIR=/private$TMPDIR docker-compose up if $TMPDIR contains a symbolic link that cannot be mounted by Docker.

Now your Localstack env is setup, you can keep it running in background

Build your lambda image

Python lambda dockerfile
Lambda function

We can now build it!

Test

Docker lambda executed with curl

Congrats! Your image is well formatted, we can now make it work all together

Make it work all together

Full docker-compose.yml file

Create and invoke lambda

  • Create the lambda function with the good image tag of your function. We need to specify a dummy role even if it is not created.
Create-function output

Test it

Invoke output

Update lambda

Create SQS queue and bind it

  • awslocal sqs create-queue --queue-name test-lambda-queue
  • awslocal lambda create-event-source-mapping --function-name test-lambda --batch-size 1 --event-source-arn arn:aws:sqs:us-east-1:000000000000:test-lambda-queue
Localstack Dashboard

Send message and check you docker dashboard!

Docker dashboard
Lambda container logs

!! Tips !!

  • You need to define a bridge network to allow containers to communicate
  • If you want to use awslocal config in a script, we highly recommend to add cli_pager= to your ~/.aws/config file to disable pager program (especially if you are using iterm), awslocal cli should prevent those problems. Otherwise, your script will prompt the result of some commands. (https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html)

Conclusion

--

--

--

Lead Data Engineer at @LoneScale

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Math in Unity : Grid and Bitwise operation (Part V)

FaunaDB: An Introduction

Why Agile and How

Kubernetes Use Case in Industries

Flutter: DraggableScrollableSheet

UUID primary keys with Postgres and ROM

Haskell CheatSheet-(Modules & List intersection/List product)

How to Get Rid of Pop-ups

Theodore Bourgeon

Theodore Bourgeon

Lead Data Engineer at @LoneScale

More from Medium

AWS AppSync: Implementing the Simple Web Service - Serverless Pattern 🏗

Implementation of a Lambda function and creation of an SQS with Serverless

AWS HTTP API directly to SQS with User Agent and IP Address

Running a Containerized Application on Serverless Infrastructure