Test in development, then promote #

View this page as Markdown

On this page

Prompt:

Add a development environment, so I can test before production.

Also works:

  • "Let me try each version somewhere before my users see it."
  • "The new version works. Put it live."

One call, tzcloud_create_environment, turns development on (Turn the development environment on). After that, each deploy goes to development first. Once you say yes, tzcloud_promote puts the tested version on production (Promote to production). Without development, an application has production alone: its deploy puts the version live, and that deploy is the whole of taking it to production, with no promote.

How the hostnames are formed explains the application's label, the unique suffix added at creation, and both hostname forms.

What your AI tool does #

  • Reads the develop-then-promote and release skills (read_context, ids skill:develop-then-promote and skill:release) and follows them.
  • Calls tzcloud_create_environment once, when you ask for a development environment.
  • Before promoting, requests each route the change touched on the development hostname and checks tzcloud_read_logs for errors.
  • Once development is deployed, promotes if you already asked for the promote; otherwise, asks you and waits for your yes. With no prior approval and nobody to ask, stops and reports.
  • Asks for no browser approval: tzcloud_create_environment and tzcloud_promote run without one.

What you need #

  • An application created as Deploy from your computer describes.
  • Every application can have a development environment. Its server is charged at the Shared 0.5 GB rate and always sleeps when idle. Size and usage gives the limits and explains charges; the pricing page gives the current rates.

Before your AI starts #

For the AI tool: Check the following before starting.

  • Your AI tool connected and signed in (Connect your AI assistant).
  • The application identifier, which tzcloud_list_applications returns.

Steps #

You turn development on once. After that, you deploy each version to development and promote it to production.

Turn the development environment on #

A new application has one environment, production. Its deploys go straight to production, and tzcloud_roll_back puts an earlier version back. A second environment, development, is a hosted copy of your app. There you test a version before your users see it.

Your AI tool calls tzcloud_create_environment with the application identifier and environment: "development". The work is done when the call returns, and it needs no browser approval.

{ "application": "<application id>", "environment": "development" }

The call returns created: true and the development hostname. If the application already has development, the call returns created: false. It then changes nothing, except to finish any step an earlier call left undone. The hosted development copy of your server needs its first deploy before it can serve application routes.

The answer also lists what development does not hold yet: each sign-in or push credential, and each secret a manifest setting refers to. Store each one with tzcloud_store_secret, naming development. A missing secret bound to a setting refuses the deploy. Missing sign-in or push credentials do not: the deploy can succeed while those providers remain unavailable. Submit the manifest again after storing a sign-in credential so the realm receives it. Check the deploy to see both kinds of missing credential before deploying.

You can call it before or after tzcloud_submit_manifest: either order ends in the same state. tzcloud_submit_manifest creates development's credentials on every application. Where the manifest declares a database, it also creates development's database. This holds whichever call comes first. tzcloud_create_environment adds the hosted development copy and creates no database. Each call reports the application's state as it read it, so two calls run at once can describe two different moments, and both are true.

From then on:

  • a deploy goes to development, at <label>-dev.ai.host, and tzcloud_promote moves a version to production, at <label>.ai.host, where <label> is the application's current label (The development hostname);
  • where the manifest declares the accounts service, development has its own realm, which is its own list of end users, and local runs sign testers in there;
  • the manifest's realm member and push providers apply to development as well as production;
  • development's schedules run from its first deploy;
  • test end users can be created on development too. Turning development on removes none from production, and your AI tool's session still creates them there (Test sign-in with test end users).

Local data that names production's users matches no one on development, because development's realm has its own users.

Deploy to development #

With development on, a deploy goes to development. Your AI tool calls tzcloud_deploy_command and runs the command line it returns, as steps 3 to 5 of Deploy from your computer describe. If production already serves a version, it keeps serving it while development deploys.

Check development before promoting #

For the AI tool: Check the changed behavior before promoting. A successful deploy proves the application starts and passes its health checks; it does not prove the changed routes work.

If you can run commands, write or update a verification script in the project. Give it the base URL so the same checks can run locally and against development. Check the expected responses and saved data, and make a failed check end the script with a nonzero status. Build a public upload gallery includes a tested upload service with such a script. Local tests with test doubles check application logic without calling the platform; also check the deployed service.

Without a shell, check the changed routes with tzcloud_request_route, or tzcloud_write_route for a route that writes, naming environment: "development". Check saved data with tzcloud_query_database where needed. For routes that require sign-in, use test_end_user as Test the signed-in paths describes. Read development's logs with tzcloud_read_logs and investigate errors before promoting. A failed tzcloud_request_route answer's detail gives the reads for that request: one tzcloud_read_logs read for its log entries, then a console search.

To check other routes before a deploy or promote switches traffic, list read-only GET routes in the manifest's smoke_paths (Listing smoke paths). A promote requests them from the new production copy, using production's data and secrets. Promoting the version production already serves skips these checks.

Promote to production #

Once development's deploy is deployed and you agree, or already asked for the promote, your AI coding tool calls tzcloud_promote with the application identifier and wait_seconds. Without a version, it takes the version development is serving.

{ "application": "<application id>", "wait_seconds": 45 }

The promote takes the same wait_seconds as a deploy and responds the same way. Without it, the promote returns at once with status 202 and state: "deploying". With it, the response waits for the promote to end, up to that many seconds. The response's environment is always production. Where something other than the platform responds, call tzcloud_read_status first, and promote again only where it shows no promote started after your call.

Besides wait_seconds, tzcloud_promote takes two optional members:

  • version, the number of a deployed row in the version history. If the call gives no version and development runs none, the call is refused 409 nothing_to_promote. A number that is not a deployed row of the application is refused 404 no_such_version.
  • rotate_database_credential, which also sets a new password on the production database role, as What happens during a promote describes.

A promote builds nothing. It reuses the image the deploy built, so production runs the same code development ran. The image also holds the runtime harness as it was at that deploy. So a promote keeps that harness, and only a new deploy takes the current one. On an application with one environment, tzcloud_promote is refused 409 environment_not_created, since the deploy already reaches production.

What a promote does to production's database #

Where the manifest declares the database kind and no production database exists yet, the first promote creates it on a production server. The platform creates no tables and runs no migrations. Your application's own migration walk does that. It applies your migration files to production's database when the new production container starts, before its server listens. It does the same in development. A promote takes no snapshot of the data first.

So a new migration changes production's data as soon as the new container starts, while the old version is still serving. A failed promote does not undo the change. A later rollback moves the code back but leaves the data as the migration left it. Add a database says how to write a migration that the running version keeps working through.

What happens during a promote #

On Shared 0.5 GB, production usually runs as a pod on shared machines. The steps below are the same there. So where tzcloud_read_status shows grain as pod, you see the same new copy, health check, and switch.

The promote works in this order:

  1. It creates a new production platform credential and gives it to the new container. The current value keeps working until the switch.
  2. With rotate_database_credential: true, it sets a new password on the production database role before the new container starts.
  3. It starts the new version's production container next to the one serving now. The new container gets production's settings and credentials, including TURNZERO_CLOUD_REALM_KEYS where production has a realm.
  4. It polls the health path on the new container alone, until it returns 200 three times in a row, three seconds apart. A 503 or a connection error between them starts the count again, inside the check's 180 seconds. Then any smoke paths are requested, unless production already serves this version. Once both pass, it switches the production hostname to the new container. A first promote records deployed at once. Where production already serves a version, the switch waits until the promote is 10 seconds old, and deployed comes 3 seconds after it, once every router has the switch.
  5. It removes the previous container about 15 seconds after the switch. That is the routers' lookup interval for the serving container, 10 seconds by default, plus a 5-second grace period.

After a first promote records deployed, a request may still be answered not_deployed for up to about 2 seconds while the routers pick up the new container.

A failed promote ends failed and names the outcome. Its new container is removed. The serving container is untouched, and the current platform credential stays in place. The previously serving container keeps its platform credential until the platform removes that container, so a request that still reaches it is served.

A database password rotation is the exception. From the rotation until the switch, the previous container cannot open new database connections, though its open connections keep working. That is why the rotation runs only when you ask for it, not at every promote.

At the switch, production's schedules restart from the promote. A scheduled run still in progress on the previous container can fail, and the schedule runs again at its next due time.

With wait_seconds, the promote's response includes the outcome where the promote ends within the wait. Where it ended deployed, the response also carries server_errors_since: production's server errors (5xx answers) counted since that version began serving (Check the release). The count covers only the routes requested since then, so request the others you rely on. Routes listed as smoke_paths in the manifest are checked before each deploy and promote switches. Otherwise make the next call the response gives, tzcloud_read_status with wait_seconds, or check the version's row in tzcloud_list_versions.

A promote interrupted by a platform restart before its switch ends failed with the outcome interrupted within about fifteen minutes. One interrupted after its switch ends deployed, because the new version is already running.

You can promote the version production already runs. The promote creates a new production platform credential and applies the settings again. With rotate_database_credential: true, it also sets a new database password.

Expected result #

With development turned on, the deploy is served at <label>-dev.ai.host, and the promote adds a "promote" row for production to the version history.

  • After tzcloud_create_environment, tzcloud_read_status lists development beside production, development as never_deployed until its first deploy.
  • After the promote, tzcloud_read_status shows production's deploy.state as deployed and its version as the version development served.

Refusals #

See the refusals reference for other platform errors.

A promote can also meet the refusals it shares with a deploy, such as deploy_in_flight and setting_secret_missing. The Refusals section of Deploy from your computer lists them.

After development is turned on, tzcloud_create_environment can also be refused name_bound_to_setting, custody_entry_missing, custody_entry_unmovable, or realm_not_declared. This happens while it copies the manifest's sign-in or push settings to development. Development stays on. The refusal's detail says what happened to the setting, and whether to call tzcloud_create_environment again or submit the manifest. Author the manifest explains each.

Refusal Status Cause Remedy
nothing_to_promote 409 tzcloud_promote gave no version, and the development environment runs none. Deploy to development first, or pass a deployed version.
no_such_version 404 tzcloud_promote: the version is not a deployed row of the application, or its environment was deleted. Name a row tzcloud_list_versions returns with state: "deployed".
version_image_pruned 409 tzcloud_promote: the platform has deleted the named version's image. Choose a version tzcloud_list_versions marks promotable. Or deploy that version's artifact again as a new version, then promote it.
environment_not_created 409 tzcloud_promote on an application with one environment, whose deploy already reaches production. Deploy instead, or turn development on with tzcloud_create_environment first.
target_environment_halted 409 tzcloud_promote while production is halted. Call tzcloud_halt_environment on production with resume: true, then promote again.
deletion_in_progress 409 tzcloud_promote while a deletion of the application or of either environment runs. Wait for the deletion to finish; read_approval_status shows whether it completed. If it failed, request the same deletion again to complete it.
invalid_request 400 tzcloud_create_environment named production, which always exists. Name development, or leave environment out.