This article is for anyone struggling with testing cloud services locally, and specifically for people wanting to locally test an API that uses API Gateway and Lambda, with the Serverless framework, on AWS. I was once in desperate need of this knowledge, and my co-worker Joseph Jaffe helped me put this solution into place.
A very popular and quick API solution is using Serverless along with API Gateway and Lambda. If you have never done this before, you can read more about it here. If you already have experience, and are looking for creative ways to locally test, then keep reading. Bonus if you like 🌮🌮!
The Problem
When setting up an API using the Serverless framework, there are important choices to make. Some are extremely important: they can make your life much easier as you build your API, or cause you huge headaches in the future. For instance, when I first started my API, I was setting up each endpoint in its own Serverless project folder. Thus each endpoint had unique URLs. This is bad practice for an API solution that needs one base URL with multiple service points. We changed the solution to have all endpoints in the same Serverless project, and that allowed us to use one extra handy aspect of Lambda — layers. Lambda layers are a way to share code across API endpoints, reducing the amount of repeated code across the API project. You can read more about Lambda Layers here, but I digress. Let’s get back to the problem: local testing!
When you create a new API endpoint, you must deploy the entire API to include the new resource in API Gateway and get the URL for it. Once you have done that, you can deploy individual resources.
Deploying for Serverless takes time. Lots of time. For instance, see these average deploy times for one of our last projects:
Deploying a single endpoint: ~7 seconds 🙂
Deploying full API (~12 resources): ~24 seconds 😔
Deploying Layers (2 layers): ~32 seconds 💀
While these times don’t appear too bad at first glance, imagine trying to quickly and iteratively test out changes in your API. Having each deploy over 1 minute long is a huge time-hog and will kill your momentum. Stretch this over weeks of development and you will see why we all need a solution to test Lambda functions locally.
Our Solution
In order to quickly flush out the obvious errors, we require local testing. In order to test locally, we found that serverless invoke local allows us to do this in the simplest manner. This not only allows us to run Lambda scripts locally, but we can also add breakpoints using Debug Mode in Visual Studio Code. If you have never played with it, Debug Mode in VSC is really helpful.
The most important aspect of local testing is instant feedback. No more waiting around for layers, functions or entire APIs to deploy! Tons of time saved during the initial build and your (patience as well as your) Program Director will love you for it!
The last Serverless API project we worked on stored all the data in JSON files hosted on AWS S3 buckets. You may be reading the data from a database, so you need to make sure you can still access the data while running locally. One way would be to set up the database locally on your machine. Ultimately, every project is unique and requires you to think creatively for a solution that meets your needs.
In order for us to know if we are running locally or not, we created an environment variable to pass in through our local invocation. The variable is named LOCAL_DEV and is used to check if we should be loading the data from S3 or from a local file system folder, like so:
const data =
process.env.LOCAL_DEV === “true”
? require(`./data/tacos.json`)
: //handle loading/setting the data as you regularly would
Note above that the boolean value of true is in quotes. Environment variables always come through as strings, so be ready to handle this fact of life.
We have a snapshot of the data stored on S3 on our local computers, so when we are in local development mode, we use that local data in order to run and test the code locally.
Additionally, if you are using layers in Lambda, you will need to point directly to the code as opposed to referring to it by name, at the top of your Lambda file:
const apiCommon = process.env.LOCAL_DEV === “true”
? require(“../layers/apicommon/nodejs/node_modules/apicommon/index”)
: require(“apicommon”);
Once you have all code in place to allow the Lambda function to run successfully locally, then you can try invoking the function. Here is an example invocation of an endpoint called tacos (🌮🌮) that gets all tacos from a food API. Because I ❤️ 🌮🌮. Code for this example can found on Github.
This is copied and pasted from a command shortcut I defined in my package.json file. That command requires you to put literal markers in front of all quotes. Here is that command from package.json in its entirety:
“scripts”: {
“local-tacos”: “serverless invoke local –function tacos –data ‘{ “queryStringParameters”: {“type”: “breakfast”, “filling1”: “egg”, “filling2”: “bacon”, “filling3”: “cheese”, “tortilla”: “flour”, “salsa”: “Salsa Doña”}}’ -e LOCAL_DEV=true > output.json”
}
Ok, now let’s look at this command and what each part does. I am removing all of the literal markers for easier readability.
serverless invoke local –function tacos –data ‘{ “queryStringParameters”: {“type”: “breakfast”, “filling1”: “egg”, “filling2”: “bacon”, “filling3”: “cheese”, “tortilla”: “flour”, “salsa”: “Salsa Doña”}}’ -e LOCAL_DEV=true > output.json
First, the base part:
serverless invoke local –function tacos
The item above says to locally invoke the API endpoint “tacos” (local 🌮🌮 are the best, right?) which gets a set of tacos filtered by whatever query string parameters you send it. Next, let’s look at the second part.
–data ‘{ “queryStringParameters”: {“type”: “breakfast”, “filling1”: “egg”, “filling2”: “bacon”, “filling3”: “cheese”, “tortilla”: “flour”, “salsa”: “Salsa Doña”}}’
Here is where we can define whatever query string parameters we are passing into the API endpoint. For this example, we pass into the API endpoint all the attributes that describe the taco(s) we are looking for. In our case, we are looking for egg, bacon and cheese tacos on a flour tortilla with Salsa Doña.
Note: Any guess as to where the taco described above (with Salsa Doña) can be found in Austin, Texas, USA? If you know, please respond in the comments!
If you need to test a bunch of parameters, you could save them out in a testing/query.json file(s) so you could do something like:
yarn local-taco query-success yarn local-taco query-fail
This could be a good start for an API testing environment!
The third part of the call is for defining any environment variables.
-e LOCAL_DEV=true
This tells our Lambda code very clearly we are running this function locally and to make sure and prepare for that by pulling all resources in locally as well.
Last, I pipe the results into a JSON file.
> output.json
From here I can easily verify if the results are correct or if an error was thrown.
Conclusion
That sums it up! If you didn’t see the link earlier, I have a sample project written up that you can try on your own, using the Serverless framework, the AWS services API Gateway, and Lambda.
Also, if you have ideas on how to make this solution better, or other alternative solutions, I would love to hear about your feedback/tips/experiences in the comments below.
Orchestrating Complexity With Web Animations API, Kirill Myshkin
Choosing A New Serverless Database Technology At An Agency (Case Study), Michael Rispoli
Gatsby Serverless Functions And The International Space Station, Paul Scanlon
Flaky Tests: Getting Rid Of A Living Nightmare In Testing, Ramona Schwering
🎧 Bonus: Smashing Podcast Episode 22 With Chris Coyier: What Is Serverless? (moderated by Drew McLellan)