How log records arrive

A log record is one entry of your application's log stream: an entry the logging service stores, or a line of your process's console. This page explains when records arrive in a read_logs read and in what order, how a read waits for a record, and why a read can come back empty. Read logs and counters shows how to make each read.

The store and the console

Every source except container reads the logging service's store. container reads the raw console of the named environment's compute, production where none is named. Where no compute runs after a failed first deploy or promote, it returns the lines that version's health check kept. A container read that names local is refused invalid_request, because a local run has no compute on the platform and so no console.

The response's shape depends on the source. A container read returns lines and no entries. Every store source, app, platform, router, egress, harness, and the whole stream, returns lines and adds entries, the same records parsed, each with its at, level, source, message, and any fields. The read_logs reference describes every member.

Without since, a container read covers the 24 hours before the read where the environment runs as a container. Where it runs as a pod, the read covers the newest pod's log, or the previous container's log where the current one has restarted. Name an earlier since to read older lines.

Where the environment runs as a pod, a container read is limited in bytes as well as in lines. The platform reads the newest 262,144 bytes of the pod's log and returns the whole lines in them. So a pod that writes very long lines can return fewer lines than limit. A single line longer than 262,144 bytes is not returned, and neither are the lines before it.

Where the newest 262,144 bytes begin inside such a line, the response's detail says that a console line longer than the read's byte window was not returned. The detail never leaves that sentence out.

When the environment has no version or no compute

read_logs and read_counters refuse never_deployed when the named environment has no deploy or promote in its history. A halted environment's records stay readable, because a halt keeps the environment's data.

A container read is refused never_deployed when the environment has no compute, except after a failed first deploy or promote, described below. Its detail opens by saying no compute runs there, which is true whether or not your code ran before. Where the environment has no version yet, the detail names the call that puts one there: deploy, or promote for production on an application with two environments.

During a first deploy or promote, the refusal's detail instead says no version serves there yet, and names the version in flight and its kind. It also names the step it is in with the step's start, or the version's start before its first step is recorded. The platform, harness, and app sources read at once meanwhile. To read the console, wait with read_status and wait_seconds until the version is deployed.

After a failed deploy or promote, with nothing in flight, the refusal's detail gives the failed version and, where recorded, its step and error. The platform, harness, and app sources read now. read_status keeps the failure in that version's outcome, with the health check's last console lines where the check failed. Where the check's record shows the container ran (console lines, a response from the application, a running or ready instance, or a restart or exit), the detail says it was removed.

A failed first deploy or promote is not refused where its health check kept a record and no earlier version served. The container read then returns the console lines that check kept, at most 40. They are the same lines read_status keeps at deploy.outcome.gate.console. The detail says whether the container started, and failed_check names the version, its kind, and when the check read the lines. Lines written after the check are not kept.

The record stays in the version history until the application is deleted, and purge_logs does not remove it. The read stops returning it once a later version runs or the development environment is deleted. since and wait_seconds neither narrow nor hold this response. Where production has no version and development's first deploy failed this way, production's refusal names the development read.

A read of the local stream needs neither a history nor compute.

How long console lines last

The console has limits the store does not:

  • Where the development environment runs as a pod, container reads the pod's console, which lasts only as long as the pod. A pod that has scaled to zero returns no line. The store's sources return records either way.
  • For a failed health check, the platform copies the console's last forty lines into the failed row's outcome.gate.console before it removes the new copy. Those lines can include a harness_refusal line from a control probe, which The deploy health check explains.
  • Where no line had reached the store by the deadline, that copy is empty. A container read a little later returns the lines while a running version exists. For a pod, the console ends with the pod, so the row's copy is the only record.

A scale to zero ends only a pod's console. Where the environment runs as a container, its console lines stay in the provider's log store, and a container read after the scale to zero still returns them. That store keeps each console line for 30 days. purge_logs does not reach it, since the purge erases the logging service's entries and counter totals alone.

When records arrive, and in what order

Records reach a read with a delay:

  • A store source's entries arrive within a few seconds, because the Logging client sends its buffer on a short interval.
  • The per-minute records under router, egress, and harness arrive once their minute has ended.
  • Where the environment runs as a container and the platform has the direct read enabled, a container read also reads the console of each running replica directly, through the hosting provider. A replica is one running copy of your container. Where that direct read succeeds, the newest lines arrive at once. The provider's log collection still delivers each line to the log store up to about a minute after your process writes it. A read with wait_seconds follows that console instead, as How a wait ends describes.
  • Where the development environment runs as a pod, container lines read at once.

So a line missing from a read made seconds after it was written may appear in the next one. The response tells you when that can happen.

A read's window ends at until, or at the read where you give none. Where the window ends within its source's delay, the response includes a detail saying the newest records may not have arrived yet. It says so whether the response contains entries or not. A container read on a container app always includes a detail, because its window ends at the read. Where the direct read is enabled, the list below says what it tells you. A pod's console has no delay, so its response carries no detail about a delay.

On a container app where the direct read is enabled, the detail says how it went:

  • Where the direct read returned lines, the detail gives the time from which the newest lines came from the replica directly. A later read can return those lines again, this time from the log store.
  • The detail says lines "may still be in the ingestion" where the direct read reached no replica or did not complete. It also says so where a replica is not running, has stopped, or has just restarted, or where the response may lack lines.
  • Where the direct read did not complete, the detail gives the reason. On a response with lines, or an empty one the filter emptied, it also says the log store's lines reach to about a minute before the read. A read a minute later returns the rest.
  • Where none of the items above applies and the direct read added no line of its own, the detail says the replicas' newest lines are in the response.

Where the direct read is not enabled, the detail says the newest lines may still be in the ingestion. A line reaches the log store up to about a minute after it is written.

Where the detail says lines "may still be in the ingestion", read again about a minute later. Those lines reach the log store by then.

A container read with wait_seconds can add two sentences to the detail: one saying more instances run than the wait followed, and one saying how the wait ended where it did not hold. How a wait ends describes both.

A container read's detail holds at most 600 characters. Where all it could say would run longer, it leaves sentences out, in this order, until it fits:

  1. The sentence saying more instances run than the wait followed.
  2. The sentence saying the response may lack lines.
  3. The sentence about the delay before lines reach the log store.
  4. The sentence saying the environment is idle.
  5. The sentence about a replica that is not running, has stopped, or has just restarted.
  6. The sentence about [retiring].
  7. The sentence saying how a wait ended.

The detail always keeps the time from which lines were read directly, the reason a direct read did not complete, and what a contains search found. An empty response follows the same order. It always keeps its first sentence and its sentence about the filter. On an empty response with no filter, the step for the delay shortens the first sentence to its opening words. It then says only that the compute may have scaled to zero and written nothing.

The first and the last items are the two sentences a wait adds. So the sentence on more instances is the first to go, and the sentence on how the wait ended is the last.

The reason is one word, such as timed_out, throttled, http_503, or a network error code. Where the word is longer than 29 characters, the detail shows its first 28 characters followed by …. The response's reason member contains the same word whole, however long it is. The response has a reason member only where a direct read did not complete.

A failed direct read is the platform's own read of your replicas through the hosting provider, and its cause may not be known. It does not say whether your application is healthy, which read_status reports. A replica's stream that returns a 5xx status, 501 and 505 aside, is asked once more where the read's time allows. Where the time is up, the first status is the reason.

A container read returns its lines in the order your process wrote them. Each line has the form <time> <stream> <text>, where the time is when the process wrote the line, and the stream is stdout or stderr. On a pod, the stream is -, because a pod's log does not name one.

During a promote or a replacing deploy, the old version's container keeps writing until it stops. Each line from the old version's container has [retiring] after its stream, including the lines it writes as it stops, also when you read them after it has stopped. So a SIGTERM or npm error line marked [retiring] is the old version's shutdown, not a fault of the new version. A line from the new version is never marked, and on a pod no line is marked.

When the response holds a line marked [retiring], its detail also says what the mark means, unless the detail is already near its length limit. The detail says so only for lines the platform marked. A line your own code writes that begins with [retiring] is not counted as marked.

An environment that runs as a container can scale to zero while idle, and each line the stopped container writes from the stop has [idle-stop] after its stream instead. Such lines are the idle stop's shutdown, not a crash, and the next request starts the application again.

Where the environment is still idle when you read, the detail says the last lines are an idle stop. A read whose window starts after the stop leaves the stop's lines unmarked, and so can a read within about a minute of it. The logs show SIGTERM lines after a promote or while the application is idle gives what each mark means and when a read leaves a line unmarked.

How a wait ends

A read with wait_seconds, a whole number from 1 to 45, waits for a record. On a store source, the response waits until an entry matching the read's filters exists, checking every two seconds, or until the seconds pass. It includes waited_ms, the time it waited in milliseconds.

An entry already in the window ends the wait at once, so a wait with since waits only for entries written after that moment. Where none arrives, the response is the empty read with its detail.

Only one wait per application runs at a time. That count includes a read_status wait, an action that waits, such as a deploy with wait_seconds, and a container wait. A store wait that finds another wait running returns at once, and its empty response's detail says the wait did not take place.

On a container read, the wait follows the console instead of checking the store. It follows the console of each running instance of your container, three at most, or the newest pod where the environment runs as one. The response comes once a line the read's filters allow is written, stamped at or after since, or at the stop. The stop is the earlier of wait_seconds and 40 seconds, so a wait_seconds above 40 is treated as 40. The response includes every line the follow read, with waited_ms.

Without since, the read covers its usual window, and a line already in that window ends the wait at once. The wait needs the direct read, a setting Turn Zero turns on or off for the whole platform, so a wait that met it off is not a passing state. Where the setting is off, every container wait answers at once as an ordinary container read, and its detail says the direct read is off.

A container wait that cannot run returns at once as an ordinary container read, and its detail names why. The reasons are five. Another wait for the application was already running. The platform's waits or follows are full. The hosting provider's management API is near its limit. The direct read is off. On a pod, no pod was running when the wait listed the instances a second time. Where the management API nears its limit during the wait, the response includes the lines read so far, not an ordinary read.

Each of those detail sentences says to read later or without the wait, because a repeat at once meets the same reason.

Where nothing runs when the wait starts, or every followed console closes, the wait lists the instances once more ten seconds later and follows a newcomer. It does not list a third time. On a container app, it then waits until the stop, and the response includes the store's lines read at the stop. On a pod, it returns at once with the lines read so far, and where no pod ran, its detail says so. Where more than three instances run, the detail says the wait followed three.

The wait follows the current container alone. On a pod whose process keeps crashing, the follow ends when the container exits. A crash loop's earlier lines come from a read without the wait: the ordinary container read returns the previous container's log where the current one has restarted.

Why a read comes back empty

An empty response includes a detail that says why:

  • Under app: the source contains only what your application writes through the Logging package. Its console.log and console.error lines are under container.
  • Under container: the compute may have scaled to zero and written nothing, or its recent lines may still be in the ingestion. Where the direct read reached a replica and found no line, with no failure, no stopped or restarted replica, and no missing lines, the detail mentions only the scale to zero. A pod's console ends with the pod. Where the read includes contains, the detail first says how many lines were searched and matched, and how many the filter kept where it is given.
  • Under any other source: no entry exists in the window, or none matches the read's filters.
  • Across all sources: none of the stored sources contains an entry in the window. Console output, console.log and console.error among it, is read with the container source alone.

A read across all sources that does return entries says the same about console output in its detail. A read of the local stream says neither, because a local run has no console on the platform. Where its entries include an answered_5xx record, the detail names the container read to make instead, as From a 5xx record to the error your application wrote describes.