The deploy health check #

View this page as Markdown

On this page

Before switching traffic to a new running copy, Turn Zero Cloud checks the manifest's health path. It must return HTTP 200 three times in a row within 180 seconds. A failed check leaves the serving version in place.

Deploys, promotes, and platform redeploys run this check. A deploy or promote may also check the manifest's smoke_paths (Checking your other routes).

What the check expects of your server #

Return 200 when the application is ready and 503 while it is starting. Register the health route before accepting requests. The health route must not return a 4xx; repeated 404 responses can end the check early. A server that is not yet listening counts as starting. For an application with a database, finish migrations before starting the server.

The Database package's startup walk runs migrations before listening, so a migration failure stops startup. Listening first and returning 503 during migration also works, but the application must stop itself if migration fails.

Where the manifest declares the database kind, the skill:deploy skill asks for a health route that performs one read over the connection and returns 503 while that read fails. The Database package's integration guide describes what the read proves: a pass shows the connection setting reaches the database, not only that the process responds. Build a database-backed service shows such a route. The deploy does not check which queries the health route makes: the check sees only the route's answers, so a route that skips the database still passes.

A fixed body suits that 503, such as {"ok":false}, with no error text, credential, or host. Where the check fails, the deploy's record keeps the first 512 bytes of the last non-200 body your own process returned, and list_versions returns it too.

How the check probes #

The check sends GET to the manifest's health path on the new copy every three seconds, for up to 180 seconds. The path must return exactly status 200 on three probes in a row to pass. This takes at least six seconds after the first 200. A 503 or a connection error between them, or any other response, starts the count of 200s again.

The check follows no redirect, so a 3xx counts as a failed probe. A redirect alone never ends the check early.

Each probe waits at most ten seconds for the response to begin. Near the end of the check it waits only for the time left, but never less than one second. So a health handler must begin its response within ten seconds, and a probe waiting longer is recorded as timed_out.

When the check ends early #

A stable client error ends the check early. When twenty probes in a row return the same 4xx status, other than 408, 425, or 429, the platform checks whether your own process is responding. Immediately after the twentieth such response, the platform sends one control probe: a request without the router mark. Where your process responds to it, the check ends early, about a minute in, and the record's outcome.gate contains ended_early: true.

Any other response during the twenty probes, a 200 among them, starts the count of twenty again. A 5xx response or a transport error, such as a failed or timed-out connection, never counts toward the twenty. A process that is still starting returns them. A 4xx from something in front of your process does not trigger this client-error stop.

The check also reads the new copy's newest instance beside each probe, on container apps and pods alike. It ends early after two successive reads show at least two restarts, or an exit code while the instance is not running. A 200 response or a read showing neither condition clears the count. One restart followed by a running instance does not end the check. Failed or unfinished reads, or reads with no instance, are ignored. The outcome is health_gate_failed; its gate records ended_early: true and the evidence below.

On a pod, the check also ends early where the platform cannot start the new copy at all, so nothing will ever serve the health path. While it probes, the check reads whether the pod was refused. If two reads in a row show that the pod was refused, the check stops immediately. The record ends failed with outcome pod_create_refused and no outcome.gate, because your code never ran. A refusal that clears between reads, or a 200 from your process, lets the check go on. Nothing in your application causes this outcome: deploy again later, as Deploys describes.

The evidence a failed check keeps #

A failed deploy ends its record in the state failed, with the failure's name and detail in the record's outcome. A failed deploy or promote also gives the step it stopped at in step, such as image_build, compute_apply, or health_gate. Where the health check failed (health_gate_failed), the outcome also includes gate, the check's evidence, collected before the platform removes the new copy. list_versions returns the same outcome.

Member What it contains
probe_sequence The probe responses as runs of an HTTP status or a connection word, the last sixteen runs.
last The last non-200 response's status and content type, with the first 512 bytes of its body where your own process responded. Null where no probe got an HTTP response.
control_probe The control probe's own result: outcome answered with its status and any refusal, transport with a connection word such as connection_refused, unavailable with a cause, or skipped where the platform sent none.
answered_by What the control probe reached: application where your process responded 403 router_mark_required, intermediary where something in front of your process responded, nothing where nothing responded, and unknown otherwise.
compute The new copy's state and restart count, or unavailable with a cause word where the platform could not read it.
console The last forty lines of the new copy's console, at most 4,096 bytes, with truncated where the limit cut them.
ended_early True where the check ended early on a stable client error from your own process, or on a new copy that keeps stopping. False where it ran to its bound.
polls_200 How many probes returned 200 in all, counted whether or not they came together.

Where the smoke check failed instead, the outcome includes smoke in place of gate, as Checking your other routes describes.

The record's detail sums up the evidence and gives no address. Where the evidence shows a likely cause, such as no route at the health path or a process that stops, the detail opens with it and its remedy. Where some probes returned 200 but never three in a row, the detail says so, and it names a likely cause only where the new copy restarted or exited. Where the last response was a redirect, the detail says so and that the check does not follow it.

Each control probe may leave one harness_refusal line in your console, which is expected.

For an environment running as its own container, console lines reach the log store after a delay, so the platform waits up to 60 seconds for a first line before recording the tail. Where none arrived, console is empty and says so. Where a version is running, the lines are then readable through read_logs. After a failed first deploy they are not: read_logs returns only the lines the check kept.

Checking your other routes #

A passing health check does not prove that other routes work. Add up to five read-only GET routes to smoke_paths to check them before traffic switches (Listing smoke paths).

Once the health check has passed, the platform runs the smoke check. It sends GET to each listed path on the new copy, in the manifest's order, and the switch waits until every path has passed. A smoke request carries no body.

A smoke path passes on any status below 500 except 404. So a 3xx passes, and a route that needs a session or a body passes on its 401, 403, 400, or 422. A 404, a 5xx response, or a transport error, such as a refused connection, fails the request.

The check requests a failing path again every three seconds. One 30-second window, counted from the first smoke request, covers all the paths together. Each request waits at most ten seconds for the response to begin, or the window's time left where that is less, and never under one second.

List routes that serve GET. A route with no GET handler often returns 404, which fails the check. List only routes that change nothing on GET, because on a promote they run against production's data and secrets.

Like the health check, the smoke requests reach the new copy directly and never pass through the audience gate. So a route behind sign-in returns your application's own status. The paths under /__account/ and /__router/ belong to the platform, and a manifest that lists one as a smoke path is refused when you submit it.

A deploy runs the smoke check, and so does a promote of a version production does not serve now. A rollback, restart, or platform redeploy sends no smoke requests. Neither does a promote of the version production already serves. So a settings change those acts apply, such as a rotated credential, is not smoke-checked, and a recovery never waits on a route.

A path that has not passed within the 30 seconds ends the deploy or promote failed, with the outcome smoke_path_failed at the step smoke_check. The platform deletes the new copy, as after a failed health check, and a version already serving keeps serving.

The record's outcome.smoke keeps the evidence. Its path is the path that failed, answer its last status or connection word, and last that response's status and content type. A request that times out before any response records answer: "timed_out" and last: null, with no HTTP status. It also holds compute and console, as a failed health check's gate does. The console starts at the first smoke request, so its lines are ones the smoke check could have caused.

To fix a failed smoke check, read the console in outcome.smoke for the crash. Fix the route or what it calls, such as a missing upstream key, and deploy again. Or remove the path from smoke_paths and submit the manifest again.

A manifest without smoke_paths runs no smoke check, and the deploy requests the health path alone.

Testing a route that needs a body #

Smoke requests have no body. Test routes that need one after deployment. An AI tool with no shell calls write_route with environment set to the environment the deploy went to. Pass the route's method, such as POST or PUT, a body, and in headers the content-type and any signature header the route reads.

The body is a string, sent as UTF-8 text, of at most 64 KiB, so it cannot carry binary data such as an image. Check the release gives the call's other limits.

For a binary body, such as a file upload, send the request from your own machine to https://<hostname>/<path>, the hostname being the one read_status returns, for example with curl --data-binary @photo.jpg.

Unlike the smoke requests, both pass through the audience gate, as an end user's request does. So under an invited or workforce audience, send an end user's session with them, unless the path is one your manifest lists in session_free_paths.