Writing a Smoke Test Tool: PASS, FAIL, SKIP and a Tiny Harness

After a deployment, one question matters more than any other: does the thing work at all? Not "is the coverage high" and not "do the unit tests pass". Those ran before the deploy, against mocks. The smoke test runs after, against the real stack, with real Redis, a real database, real connectors and real configuration.

This post is about a smoke test tool we wrote for a safety-critical backend: alarms come in from an upstream system, pass through a few services, and end up as a call to an external emergency-notification system. A silent failure in that path is not a bug report, it's an incident. The tool is about 1,000 lines of plain Python and one dependency (requests). Here is how it's built and what we learned.

What You'll Learn

✅ Why a smoke test is a separate tool, not a pytest suite in disguise
✅ A tiny harness: Config, Api, Skip and a check runner
✅ One package per feature, ordered so cheap blockers run first
✅ Why a check should SKIP instead of failing when it can't run
✅ Simulators for the external systems you must not call
✅ How to handle environments that differ (local, dev, prod)
✅ A real regression it caught

1. Why a Separate Tool?

Unit tests and smoke tests answer different questions:

The last row shapes the design. A smoke test is a go/no-go signal, so it must be fast, it must be trustworthy (no flaky checks that teach people to ignore it), and its output must be readable by someone who is stressed and holding a deployment.

We kept it in its own repository. It has no import of the backend. It only knows URLs, and that's the point: it tests the system the way a client does.

2. The Whole Harness Is One File

The core idea is that a check is just a function. It passes if it returns, fails if it raises AssertionError, and skips if it raises Skip:

class Skip(Exception):
    """Raised by a check that cannot run on this profile."""
 
 
def expect(resp, code, what=""):
    assert resp.status_code == code, \
        f"{what or resp.request.url}: expected {code}, got {resp.status_code}"

A feature package is then a list of such functions:

def requires_auth(api):
    expect(api.post("/api/alarms", key=False, bearer=False), 401)
 
 
def invalid_value_rejected(api):
    body = _fire(api, _alarm(value="TRUE"))
    assert body.get("status") == "rejected", body
 
 
CHECKS = [requires_auth, invalid_value_rejected]

No decorators, no fixtures, no plugin system. The function name is the label in the output, and it's also the --only filter. The runner is about 30 lines:

def run_package(name, checks, argv=None, package_ini=None) -> bool:
    ap = argparse.ArgumentParser(prog=f"python -m {name}")
    ap.add_argument("--profile", default="local")
    ap.add_argument("--only", help="run a single check by function name")
    args = ap.parse_args(argv)
    cfg = Config.load(args.profile, package_ini)
    api = Api(cfg)
    results = []
    for fn in checks:
        if args.only and fn.__name__ != args.only:
            continue
        try:
            fn(api)
            results.append((fn.__name__, "PASS", ""))
        except Skip as e:
            results.append((fn.__name__, "SKIP", str(e)))
        except AssertionError as e:
            results.append((fn.__name__, "FAIL", str(e)))
        except Exception as e:  # network error etc.
            results.append((fn.__name__, "FAIL", f"{type(e).__name__}: {e}"))
        print(f"  {results[-1][1]:4} {fn.__name__} {results[-1][2]}")
    return not any(r[1] == "FAIL" for r in results)

Three details are worth copying:

• Any unexpected exception is a FAIL, including connection errors. If the stack is down, that is exactly what the smoke test exists to say.
• Each check is independent. One failure doesn't stop the rest of the package, so you see the full picture in one run.
• The failure message is part of the check. expected 401, got 200 is what the person on call reads.

Config: profiles, ini files and env vars

The same checks must run against several environments, so the target is a profile, a section in an ini file:

[DEFAULT]
base_url = http://localhost:8000
timeout = 20
 
[local]
base_url = http://127.0.0.1:8002
otp_in_response = true
simulators = true
 
[dev]
base_url = http://dev-host:8000
check_static = false
external_simulators = true

Precedence is simple: real environment variables win over .env.<profile>, which wins over .env, which wins over the ini files. Secrets only ever come from environment variables (API key, admin password, database passwords) and .env files that are gitignored. The loader even accepts only keys starting with SMOKE_, so a stray line in a .env can't leak into the process environment.

3. One Package Per Feature, Blockers First

Each feature of the backend gets a package with the same shape:

smoke_test/
  harness.py
  main.py              # SUITE: ordered (package, blocker?) list
  server.ini
  send_alarm/
    __main__.py        # python -m send_alarm
    common.py          # CHECKS, PACKAGE_INI, main()
    test.ini           # per-profile overrides for this package
  system_status/
  admin_auth/
  ...

Every package is runnable on its own (python -m send_alarm --only requires_auth), which is what you want when one check is red and you're iterating on a fix. main.py runs them all in a fixed order:

SUITE = [
    ("containers", True),
    ("api_access", True),
    ("system_status", True),
    ("inbound_connector", True),
    ("outbound_connector", True),
    ("admin_auth", True),
    ("send_alarm", True),
    ("rule_admin", True),
    ("webhook_alarm", True),
    ("config_monitoring", False),
    ("session_log", False),
    # ...
]

The boolean says whether the package is a blocker. The rules:

1. Cheap checks first. "Are the containers up", "does the API answer", "is Redis healthy" take milliseconds. There's no point running the full alarm flow if the API is down.
2. A failed blocker stops the run. The exit code is 1 and the output says so: BLOCKER package system_status failed -- stop and roll back.
3. Non-blockers run to the end. A broken reporting page should be fixed, but it shouldn't trigger a rollback of an alarm system.

Deciding what is a blocker is a product decision, not a technical one. Ask: "if this is broken, do we roll back?" Be strict. If everything is a blocker, nothing is.

4. SKIP Is Not PASS, and It Is Not FAIL

This is the single most useful rule we adopted:

A check must SKIP, never fail, when the profile lacks something it needs. And SKIPs are always printed.

A check might need admin credentials, a fixture rule, a simulator, a database password, or a human step (a one-time password that arrives by e-mail on production). If the tool failed on all of those, nobody could run it on production and it would be red by default, and red-by-default gets ignored. If the tool silently passed, you'd believe things were verified that were not.

So there are three outcomes and the summary shows all three:

== admin_auth (dev: http://dev-host:8000) ==
  PASS bad_credentials_rejected
  PASS protected_endpoint_needs_token
  SKIP login_otp_refresh_logout OTP arrives by email on this profile; log in manually
  PASS wrong_otp_rejected
  PASS pre_otp_token_cannot_access_admin_api
admin_auth: PASS (4 pass, 1 skip, 0 fail)

The reason is part of the line, so a SKIP reads as an instruction: "here is what you have to check by hand". Even the one check that deliberately always skips is useful:

def real_alarm_dispatch(api):
    raise Skip("starts a real dispatch: run manually against the "
               "external TEST system")

It's a reminder in every run that this path is covered by a manual procedure, not by the tool.

A related rule: checks are side-effect free by default. The default send-alarm checks only send messages that must be rejected (wrong sensor type, invalid value, unknown rule). The checks that create data use a context manager that creates a throw-away record and deletes it in __exit__, and it first purges leftovers from an earlier aborted run:

class smoke_rule:
    def __enter__(self):
        self.api.login_admin()
        self._purge_stale(str(body["rule_key"]))      # leftovers from a crashed run
        r = self.api.post("/api/rules", key=False, json=body)
        self.id = r.json()["id"]
        return self.id
 
    def __exit__(self, *exc):
        if self.id is not None:
            self.api.request("DELETE", f"/api/rules/{self.id}", key=False)

The purge matters more than it looks. A leftover duplicate made the system answer "conditions do not match" to every later run, and the failure message pointed at the wrong thing.

5. Simulators: Test the Whole Path Without Calling the Outside World

The most valuable checks follow an alarm all the way through: it enters the first service, passes the connectors and the API, and ends as a SOAP call (or a webhook) to an external system. We must not call the real one from a smoke test.

So the repository contains two tiny simulators, written with only the standard library: a fake SOAP server and a webhook receiver. Both record every call they receive and expose it:

class Simulators:
    """In-process SOAP stub + webhook receiver on free local ports."""
 
    def __enter__(self):
        self.soap = soap_stub.make_server(self.host, 0)   # port 0 = free port
        self.hook = webhook_receiver.make_server(self.host, 0)
        for srv in (self.soap, self.hook):
            threading.Thread(target=srv.serve_forever, daemon=True).start()
        ...
 
    def soap_calls(self):    return list(self._mods[0].STATE.calls)
    def webhook_calls(self): return list(self._mods[1].CALLS)

A check creates a rule pointing at the simulator's URL, fires an alarm, waits, and then asserts on what the simulator received: the session was opened, the dispatch started, with the right identifiers. That proves every hop in the chain, which no mock inside a unit test can.

There's a catch: the in-process simulators only work if the system under test can reach your machine. That's true locally, and false for a shared dev server. For that case there's a second mode: the same simulators run as containers next to the target, with pre-provisioned rules, and the harness talks to their /_calls and /_reset endpoints.

Both modes sit behind one function, so the checks don't care which they get:

@contextlib.contextmanager
def simulated_rule(api, kind):
    if api.cfg.extra.get("simulators", "false").lower() == "true":
        with Simulators(api.cfg) as sim, \
             smoke_rule(api, **_rule_fields(kind, sim.wsdl_url, sim.webhook_url)) as ds:
            yield sim, ds
    else:
        with ExternalSimulators(api.cfg) as sim, fixed_rule(api, kind) as ds:
            yield sim, ds

If the simulator containers aren't deployed yet, the check SKIPs with the reason. And before running, the flow checks refuse to run if the target is in a mode that would call the real external system.

6. Local, Dev and Prod Are Not the Same

Most of the work after the first version was not new checks. It was making the same checks honest on more environments. A few examples, all real:

Login with a one-time password. Locally the app runs with debug on and returns the OTP in the response, so the tool can log in. On dev and prod the OTP is e-mailed, so the login checks SKIP. We keep the checks that don't need a full login (wrong password, wrong OTP, no token) running everywhere.

Static files. One check requests an admin static file. The dev deployment doesn't run collectstatic, so the profile has check_static = false. The knob lives in the ini file, not in the code.

A list endpoint that grew with the table. One check fetched an unfiltered list of messages. On dev it started failing after 30 seconds: the table had 168,000 old rows, and the endpoint serialized every one with per-row queries. The smoke test found a real performance bug. The follow-up fixed the serializer, added pagination and a nightly cleanup job. The check now filters by status, so it can't grow with the table again.

One worker. Another surprise: dev ran the API with a single sync worker. An unauthenticated POST with a body was answered with a connection reset about every other time (the server replied 401 and closed the socket while the client was still sending the body). One slow request also blocked everything else. The fix was to run four workers, as production does, and to make the check send no body, since authentication runs before the body is read.

Neither of those was a test bug. Both were differences between environments, and the smoke test was the only thing that could see them.

7. The Regression It Caught

The best argument for the tool is one check that failed on its first authenticated run:

def pre_otp_token_cannot_access_admin_api(api):
    """The token returned by /api/auth (password only, OTP not yet entered)
    must not work on protected endpoints."""
    r = api.post("/api/auth", key=False, bearer=False,
                 json={"email": c.admin_user, "password": c.admin_password})
    api.token = r.json()["token"]   # deliberately never verified with the OTP
    leaked = [p for p in ("/api/users", "/api/rules", "/api/config-monitoring", "/api/session-log")
              if api.get(p, key=False).status_code == 200]
    assert not leaked, f"pre-OTP token accepted on: {leaked}"

The password-only token was accepted by four admin endpoints. The second factor was effectively optional. The OTP verification compared a user with itself, so it was always true, and most views only checked "is authenticated". No unit test had caught it, because each view was tested with a valid, fully logged-in user.

After the fix, the check stayed in the suite as a regression test, and it also confirmed the fix on the dev server. A smoke test is a good home for "security properties of the deployed system", since it exercises them from the outside, the way an attacker would.

8. Running It in CI

The suite is a plain command with an exit code, so a pipeline stage is small:

withCredentials([string(credentialsId: "smoke-api-key-${params.PROFILE}",
                        variable: 'SMOKE_API_KEY')]) {
    dir('smoke_test') {
        sh ". ../venv/bin/activate && python -u main.py --profile ${params.PROFILE}"
    }
}

Points worth noting:

• The API key is a CI credential that exists only inside that step's environment.
• A user-supplied base URL ends up in a shell command, so the pipeline validates it against a plain http(s):// pattern first.
• On CI, the checks that need a human, a database or a simulator SKIP and are listed in the log. That's the SKIP rule paying off again: the same tool runs on a laptop with everything and on CI with nothing but an API key.
• -u (unbuffered output) so the log shows progress while a slow check is still running.

9. Lessons

Keep the harness tiny. The whole framework is one file you can read in ten minutes. Anyone can add a check by writing a function and adding it to a list. We never needed pytest's machinery, and we never had to explain it.

Make the output honest. Three outcomes, reasons on every SKIP, a summary that says which blocker stopped the run, and an exit code a pipeline can trust.

Write down what you verified. We keep a progress file next to the tool: per package, what passes on which environment, what SKIPs and why, and facts learned about the backend (for example, "validation failures answer HTTP 200 with an error code in the body, not 4xx"). It's the memory of the tool, and it stops people re-discovering the same quirks.

Test the properties, not just the endpoints. "Returns 200" is weak. "A password-only token must not open admin endpoints" and "secrets are masked in the config monitoring response" are the checks that matter.

Never let it mutate what it shouldn't. Default to side-effect-free checks, clean up in __exit__, and make anything that touches real external systems manual.

Summary

✅ A smoke test is a go/no-go signal that runs after the deploy, against the real stack
✅ A check is a function: return = PASS, AssertionError = FAIL, Skip = SKIP
✅ One package per feature, cheap blockers first, stop and roll back on a blocker failure
✅ SKIP with a printed reason when a profile can't run a check, so the tool works everywhere
✅ Use simulators that record calls, and assert on what they received
✅ Expect most of the work to be making checks honest across environments
✅ Keep a progress file, and keep regression checks for security properties

Related: Upgrading Django 5.2 to 6.0: Tests First, Then Delete Dependencies ends with a manual smoke test. This tool is what we automated it into.