Status and logs #

View this page as Markdown

On this page

Each entry starts from what read_status, read_logs, or a request to your application returns. The read_status reference describes every member of the status. Read the status describes how to wait for a deploy to end. Read logs and counters describes every log source.

The deploy succeeded but a route responds with a 500 #

Cause. A server application's deploy succeeds when the application starts, its health path returns 200 three times in a row, and any smoke paths the manifest lists pass. The health check calls that one path, and the smoke check calls only the listed paths, so a route neither calls can still fail. The platform records each request your application responds to with a 5xx status, a server error from 500 to 599.

Check. Call read_status for the environment. If any request failed since the current version started, the summary says how many failed, and on how many paths. The application.server_errors member lists each failing path with its count, its last status, and the time of the last failure. If its read member is skipped, a deploy is running, so read again when it ends. Use application.server_errors.counted_through to judge recent failures: a request after that time may not be counted yet. Read again before treating a missing recent failure as zero.

For a server deploy, the success response and the command's checked: line describe the health and smoke paths requested; no other route was checked. When no health or smoke check ran, the command's line says so; request the pages or routes yourself.

Remedy. Read the application's own output for the failing path. Call read_logs with source: "container" for what it printed, or with source: "app" for what it wrote through the Logging package. Fix the route and deploy again. The server_errors member counts from the new version's start, so a fixed route leaves the list.

To have each deploy request the route before the new version starts serving, list it in the manifest's smoke_paths. List each such route you add later there too, as Checking your other routes describes. The failing paths in the status describes every member.

When a log read returns the record of a 5xx response, its detail gives the container read that finds the error, and the time to read from. From a 5xx record to the error your application wrote follows it to the stack trace.

A log read with source app comes back empty though your code prints lines #

Cause. The app source holds only what your application writes through the Logging package. Lines your code prints with console.log or console.error go to the console, which the container source reads.

Check. The empty response's detail says that the app source holds only the Logging package's records. A read_logs call with source: "container" and the same window returns the printed lines.

Remedy. Read printed lines with source: "container". For records the platform keeps and can filter, write them through the Logging package and read them with source: "app". If the container read is empty too, its detail says why. For example, the running copy scaled to zero, or the lines have not arrived yet. Why a read comes back empty describes each case.

A request receives a 401 that your code never sends #

Cause. Under an invited or workforce audience, the serving router checks each request before your application sees it. The serving router is the platform process that receives every request to your hostnames. It refuses a request with no live session in the application's realm, such as one whose bearer token is missing or no longer valid. It does not check the paths the audience lists in session_free_paths. A refused request never reaches your code, so your own 401 branch never runs on the platform.

Check. The response is 401 with error: "authentication_required", and its detail opens with "The serving router refused this request before the application". A browser that opens a page is sent to the sign-in page instead.

Remedy. Sign in again, or refresh the session at the realm's token endpoint, then send the new bearer token. To test your application's own 401 and 503 branches, run them where the serving router does not run. Public or workforce for a backend that verifies the bearer gives the two places: offline tests on the Account package's test double, and a local run with the run line, turnzero run.

read_status shows the production state while development deploys #

Cause. A read_status call with no environment argument describes production in its top-level members, whichever environment is deploying. The top-level state, version, and hostname are then production's. Each environment's own record, including the development deploy, is in the environments object.

Check. The top-level environment member is production. The summary gives each environment its own sentence, then says that the top-level members describe production. environments.development.deploy.state is deploying, and its step shows where the deploy is.

Remedy. Call read_status with environment: "development" and wait_seconds until the top-level state leaves deploying. The top-level members then describe development, and the summary says so. The next call a deploy returns already names the deploy's environment, so make that call as given.

read_status says the version runs on an older base image #

Cause. Each deploy builds an image of your application on the platform's base image. The platform updates that base image from time to time, often for security fixes. A running version keeps the base it was built on. The platform never rebuilds it for you.

Check. In read_status, each environment's base member gives built_on, the base the serving version was built on, and current, the base the platform builds on now. stale is true where they differ, false where they match, and null where no version serves or its base was not recorded. fixes is critical or high where the newer base carries fixes of that severity. Where the base is stale, a base_apply sentence beside it says so. In list_versions, the serving version's row gives the same fixes.

Remedy. Deploy again, even with the same zip. A deploy builds a new image on the current base. On an application with two environments, production gets the new base when you deploy to development and then promote that version. A promote carries the image of the version it promotes. A restart keeps the old image, so it does not help.

The logs show SIGTERM lines after a promote or while the application is idle #

Cause. These shutdown lines are normal. During a promote, or a deploy that replaces a running version, the platform stops the old version's container with SIGTERM. A container app that scales to zero while idle is stopped with SIGTERM too. In either case, npm can print npm error lines as it ends.

Check. Call read_logs with source: "container". Each line from the container being replaced has [retiring] after its stream. Each line an idle-stopped container wrote from the stop onward has [idle-stop] there instead. A line from the new version is never marked [retiring]. If the environment is still idle when you read, the detail says the last lines are an idle stop.

Remedy. None is needed. The serving version keeps responding, and the next request after an idle stop starts the application again. A read within about a minute of the stop may leave its lines unmarked, so read again. A pod never marks an idle stop. Once a pod has ended, the lines kept from it are marked [pod-ended] instead.

read_status shows CrashLooping #

Cause. The application's copy keeps exiting soon after it starts, so the hosting platform keeps restarting it. compute_state reads CrashLooping once the copy has restarted five times or more and is not running. It must also have last stopped within ten minutes, or be waiting out a delay before its next restart. compute_restarts gives the restart count. The platform never halts the application for it. The restarts go on until the code starts cleanly.

Check. Call read_logs with source: "container". The last lines before each restart usually name the error, such as a missing setting, a failed database connection, or an exception at startup. Where the environment's not_stored in read_status lists a setting, that setting has no stored value, and the code may be exiting without it.

read_status shows the state on both a pod and a container app. The platform also writes a warn event, crash loop, to your application's log stream, counted as warn.crash_loop. The event names the copy, its restart count, and its last exit code where one is known.

On a pod, the event is written when the copy begins crash-looping. If the same copy keeps crash-looping, it can be reported again after seven days. On a container app, it is written once a day while the copy keeps crash-looping, and the count is the copy's stops in the last hour.

Remedy. Fix the cause in the code or store the missing setting, then deploy again. compute_state returns to its usual word once a copy starts and stays up.

The console says no migration was applied at a start #

Cause. The migration function applies each file once and records it in the schema_migrations table. At a start where every file is already recorded, it applies none, and the start code prints migrations: none pending at this start. This happens at every restart, and when a copy wakes from its idle stop. It also happens at a deploy after a local run applied the file to the same database, including a first deploy to development.

Where the environment runs as a pod, the live console contains only the lines of pods that still exist. A pod that scaled to zero, or one a deploy replaced, leaves only each process's first and last lines, marked [pod-ended], for a read's window. So the start that applied the file may be missing, even on a first deploy. It is missing where the migration ran as its own step of start, or where that start is older than the read's window.

Check. Request a route that exercises the change made by the migration. To confirm a particular migration ran, check its entry in schema_migrations, the history table used by the Database package and both runners in Keep the schema as migrations, and check the expected schema change. If your app uses another migration tool, check that tool's history instead. read_status with tables: true lists table names; a table's presence alone does not prove that every column, index or data migration ran.

Or run SELECT count(*) FROM <table> with query_database. It returns a count where the table exists, and the refusal query_failed with sqlstate 42P01 where it does not (Read your data). If the console still has the lines of the start that applied the file, that start printed the file's name.

Remedy. If the recorded migrations and expected schema match, no change is needed. If a migration is missing, check that its file was deployed and that startup reads the correct folder before the server listens. For a later schema change, add a new migration file. After production has started with a migration, never edit or rename that file. For a migration recorded only in development, follow the linked reset procedure before changing it. Keep the schema as migrations explains the function.

A read_logs or read_status call reports that the tool call failed #

Cause. "Tool call failed" is your AI tool's own message for a call that got no response. It carries no response from the platform, no reference, and no refusal name, so the platform refused nothing.

Check. The message holds no refusal name and no reference.

Remedy. A read such as read_logs or read_status changes nothing, so where one draws no answer or your tool reports the call failed, call it again. A call such as deploy can change the application. A call that gets a response from something other than the platform says how to tell whether it started.

A call is refused with cell_timeout #

Cause. The hosting cell that runs your application did not respond in time, so the call could not finish. Nothing in your request was wrong. Any call can meet it, read_status and read_logs most often.

Check. The response is 503 with error: "cell_timeout". Its detail says the hosting cell did not respond in time, and ends with a reference.

Remedy. Call again after a short wait. Before you repeat a deploy, a promote, or another call that changes something, call read_status first, since the change may already be made. If the refusal persists, report it and quote the reference.