In this article I'll explain what a Smoke Test is and why you should implement one to catch bugs early in your website, API, or application.

What is a smoke test?

A Smoke Test is a suite of end-to-end/integration tests that runs against your production application on a scheduled basis to detect potentially serious bugs or "fires".

If a signs of smoke is found, the Smoke Test should alert someone who can investigate. Smoke Tests are especially useful for catching bugs or regressions that occur after a new deployment of your application.

The most typical response to a Smoke Test failure is for a human to roll back and redeploy an earlier, more stable version of your app.

So to summarize, a Smoke Test has three main properties:

  • It has a suite of end-to-end tests that test critical paths in your app
  • It is scheduled to continuously run against production
  • It alerts a human to failures that might indicate a serious problem in production

Why everyone should implement a Smoke Test

Ever received an email from a user reporting a production exception while you were out for dinner? It's embarrassing and sometimes panic inducing. It's much better to catch bugs early before anyone as a chance to experience them.

Ideally this would be done with extensive unit and integration testing in your application. But for complex systems, like MailSlurp, these tests can't replicate end user actions completely. This is because some of those actions rely on propriety cloud infrastructure components outside of your codebase. I.e.: how does one unit test Kubernetes ingress rules?

An example situation

Let's make these features more concrete. We'll use MailSlurp as an example. It's an API for sending and receiving emails from randomly generated email addresses. It was actually built to allow Smoke-testing of user sign-up, email verification, and support interaction - but we shouldn't use MailSlurp to test itself!

So, we'll need to write a suite of Smoke Tests from scratch that test the key paths of our application. As MailSlurp is an API, that means testing the following:

  • Can a signed-up user make API requests that create and destroy resources?
  • Are my selling-point features actually operational?
  • Are all components of my application healthy and accessible?
  • Can a new user sign-up for my application (for this we need MailSlurp itself, more later)?

Building the test suite

Let's tackle the easiest part first: are all the components of my application healthy and accessible. This means the main landing page, the dashboard, the documentation site, the support portal etc. So we want to make HTTP requests to each site and verify the response. For this we could really use any language or framework we like but I chose NodeJS and Jest as they are popular.

So let's create a new repository an install some dependencies:

Ok so the bash commands above created a new folder, created an empty and installed and using . You could use npm too; I just prefer yarn.

Testing production component health

Next, let's write a simple test to see if the landing page at https://www.mailslurp.com is up and responding. The landing page is deployed on AWS S3 and CloudFront and so has some redirects defined for non-https and non-www requests. We'll need to take those into account too.

Testing user actions

That was pretty straight forward. Now let's test something more informative: can users take actions on our platform that create and destroy resources. As this example is about MailSlurp we'll use the MailSlurp Javascript SDK to perform some API requests and use Jest to verify the outcomes.

First, let's install MailSlurp:

Next, let's write a test that creates a new randomized email address and then delete it on MailSlurp using a known test user (to create a new user each time we'd need to sign up for the app which requires email verification - this is what MailSlurp can help with).

Easy! Writing tests with jest is really smooth. Here we used a known apiKey (redacted) to create a new email address and then delete it. If this test in our Smoke Test passes it indicates that:

  • Users can create and destroy resources in our production application
  • Our API connects successfully with databases
  • Our authentication layers are functioning correctly
  • The official SDK client works with the current production API version

If these tests started to fail it would indicate that recent changes have introduce a bug affecting one of these components. We could then find out what changed and roll-black those changes until we have time to fix them properly.

Testing defining selling-point features for your application

A defining feature of MailSlurp is the ability to send and receive emails via a REST API or SDK client. Without that feature our application is not healthy and users will be upset. So we need one more test to cover that. Let's write that now:

Voila. The code above creates a new email address on the fly using the MailSlurp SDK, sends a email from it to its own address and then verifies that the email was received (think how useful this feature could be for testing your own application).

Scheduling our Smoke Tests with CircleCI

Now that we have our tests we need to deploy them and schedule their execution using CircleCI. First thing you want to do is sign up for CircleCI.

Then you want to connect the Github/Bitbucket repository containing your Smoke Tests to CircleCI using the CircleCI dashboard.

Next, we want to define a CircleCI config file in the repository that tells CircleCI what to do. Configs live inside a file at . So your project should look something like this:

Okay now let's define a job that executes our test inside a Docker container and a worfklow that schedules this job to execute every ten minutes. We do this inside .

In summary

So that's how you can set up a scheduled Smoke Test with NodeJS and CircleCI. Each time you deploy changes your tests should detect any exceptions within ten minutes. And these tests will run continuous, even during the night, to catch any regressions that might take longer to show up. You can enable Slack or email notifications within the CircleCI project settings.

You can also expand the CircleCI config so that tests are run immediately after deployment, but I'll leave that up to you.

One thing we didn't cover was testing with a new user each time (testing user sign-up). That is exactly what MailSlurp was made for! With MailSlurp you can create a new random email address during a test and use that to sign up and verify an account for your application. You can then use this user to test critical paths - and what's more critical that signing up new users?

For more information see https://www.mailslurp.com