Store files

Prompt:

Let users upload images, and keep each user's images private to that user.

Also works:

• "Save each generated report as a file the user can download later."
• "Let people attach a photo to a listing."

What your tool does

• Reads the manifest. If its services array has no {"kind": "object_storage"} entry, adds one and resubmits with submit_manifest, following the manifest-author skill.
• Calls list_storage_areas for the account's storage areas. Then calls declare_storage_area for each area the feature needs, with a name, account_keyed, version_keeping: false, and the application's identifier.
• Takes the Storage package from the library (Use the library) and writes the backend routes. Each route checks the end user's access, chooses a file name for that user, and calls the package's client under the application's platform credential with an acting identity.
• Where the browser uploads or downloads directly, writes a backend route that mints a transfer grant at POST /storage/v0/grants after the same access check. It also writes the browser code that uses the grant on its one file.
• Uploads a file of its own under an upload grant that mint_upload_grant returns, so no long-lived token reaches its shell.
• Deploys the change with deploy and follows it to its end with read_status (Deploy an application). The deploy's own zip is stored in the platform's deploy area, as step 6 describes.
• Asks you nothing before these actions, because submit_manifest, declare_storage_area, and deploy are reversible-tier actions.
• Asks first for one action: delete_environment erases the development files of every area that belongs to the application. On an application with one environment it does so where the call is accepted, as Applications and environments describes. The tool prints an approval link, and a person approves it in the browser.

What you need

• A decision on which files each end user may read and write. Your backend enforces this; the platform doesn't separate files by user automatically in this version.

Before your AI starts

This section is for your AI tool: what it checks and gathers before it begins. You don't need to do these steps yourself.

• A connected, signed-in tool (Connect your tool) and an application with a submitted manifest (Deploy an application).
• The application's identifier, to tie each area to the application. list_applications returns it. The deploy passes it to the container as TURNZERO_CLOUD_APPLICATION, and the provision line writes it into the local environment file under the same name. The backend reads it from that setting instead of a copy in its code.

Steps

1. Declare the service and the areas

Add {"kind": "object_storage"} to the manifest's services array. The entry records that the application uses storage, and the platform provisions nothing for it. The storage routes and declare_storage_area work for a declared area with or without the entry.

Before using an area, call declare_storage_area with:

• name: a letter or digit, then letters, digits, hyphens, or underscores, up to 64 characters;
• account_keyed: true or false;
• version_keeping: false;
• application: the application the area belongs to. list_applications returns the identifier, and a call without it is refused 400 application_required.

You can also declare an area with PUT /storage/v0/areas/{area} and a JSON body containing the same members. On that route, a call under the application's platform credential or a token for that application can leave out application, and the area goes to that application. A call under a session or an account-scoped token must give it, or is refused the same way. The Storage package's mintArea sends the declarations again on every call, including the application as application, so have your backend read the identifier from TURNZERO_CLOUD_APPLICATION.

Declaring an area again with the same declarations changes nothing. Different declarations, including a different application, are refused.

version_keeping: true is refused in this version. The platform records account_keyed, but it does not separate files by end user. Your backend must check each end user's access and choose file names to match; an end user's session cannot call the storage routes.

Each area has two parts, development and production, on every application. A file written from development is not visible from production, and the same name can refer to a different file in each. A local run under the development platform credential writes to the development part, also on an application with one environment, so its test files never reach production. delete_environment erases every area's development files and leaves production's as they were.

list_storage_areas is a management action, not a storage route, and returns the account's areas. The storage routes have no route that lists areas. The declaration route PUT /storage/v0/areas/{area} and list_storage_areas take no environment header.

To remove an area that contains no file in either environment, call undeclare_storage_area. It removes the area and frees the name, so a later declaration under that name creates a new area. Undeclaring an area that still contains a file is refused area_not_empty, so delete its files in both environments first. Only the account can undeclare an area, under a session or an account-scoped token. An application-scoped token is refused by name.

Every area belongs to one application, and none belongs to two or to none. Deleting an application deletes its areas and every file in them.

2. Put, fetch, list, delete

The storage routes are HTTPS on the platform's origin:

• PUT, GET, and DELETE at /storage/v0/areas/{area}/files/{name};
• a paged list at /storage/v0/areas/{area}/files.

The declare_storage_area response gives an area's whole file address in its detail: https://turnzero.ai/storage/v0/areas/<area>/files/<name>, with the file's name in place of <name>. An upload under a minted token shows one upload each with curl and Node.js.

The file name is taken exactly from the path, slashes included. A name is refused 400 invalid_request when one of its segments, the parts a slash or a backslash separates, is . or ...

A write is also refused 400 invalid_request when the name holds a backslash, or when any segment ends with a dot, the last one included. This is because the storage service may keep such a name under another name. Separate segments with a forward slash, and end no segment with a dot. A file already stored under such a name can still be read, listed, and deleted.

Every call needs the account's credential: a session, a minted token, or the application's platform credential. Every call reaches one environment's part of the area.

The credential sets the environment, or a header names it:

• An application's platform credential belongs to one environment and reaches that environment's files. A header naming the other environment is refused 403 environment_mismatch.
• A session or a minted token belongs to no environment. A call under either names the environment in the header x-turnzero-cloud-environment, development or production.

A file route, the file list, or a grant mint without that header is refused 400 environment_required. The platform never assumes an environment, so test files cannot reach production by mistake.

Every write gives the acting identity in the header X-Acting-Identity: a string your application chooses, separate from the credential. A write without it is refused 400 identity_required.

For a conditional write, send If-Match with the version you expect, or If-None-Match: * to require an unused name. A write whose condition fails is refused 412 version_mismatch, with current_version beside it. A put that gives a version for a missing file is refused 404 no_such_file.

3. Use the Storage package client

The Storage package provides StorageWireClient. Create it with new StorageWireClient(transport), giving it a function that adds your credential to each request, as A minimal upload service shows, then call its methods:

• put(area, name, bytes, { identity, contentType, ifVersion, mustNotExist }) returns the new version. mustNotExist requires an unused name, and ifVersion requires the stored version to match. When mustNotExist is true, ifVersion is ignored.
• get(area, name, { ifNoneMatch }) returns the bytes and metadata, or a not-modified result.
• getMetadata, list with a prefix and a cursor, and delete_. Deleting a missing name succeeds.

The client takes and returns file bytes as Uint8Array. It throws refusals as errors with a code from the package: no_area, no_file, exists, version_moved, or declarations_differ. From version 0.13.0, it refuses a file name the platform refuses, one with a . or .. segment, itself with invalid_name, before it calls the platform. From version 0.14.0, put refuses a name holding a backslash or a segment that ends with a dot the same way, and the reads and the delete still send it.

A minimal upload service

These five files make a service that stores uploaded files in one area, lists them, and returns them. A test runs them. Put them in your application's folder. Then take the package into lib/storage/ as Use the library shows, and run npm install.

manifest.json declares storage and records the package:

{
  "manifest_version": 1,
  "services": [{ "kind": "object_storage" }],
  "health": "/health",
  "region": "usa",
  "egress": [],
  "audience": { "kind": "public" },
  "packages": [{ "name": "storage", "version": "0.14.1" }]
}

package.json installs the package from its copy and starts src/main.mjs:

{
  "name": "files-service",
  "private": true,
  "type": "module",
  "scripts": {
    "start": "node src/main.mjs"
  },
  "dependencies": {
    "@turnzero/storage": "file:lib/storage"
  }
}

src/storage.mjs builds the client over a transport. The transport sends each call to TURNZERO_CLOUD_GATEWAY_URL, or to TURNZERO_CLOUD_API where that is not set. It adds the credential from TURNZERO_CLOUD_TOKEN. A deploy sets these settings, and the provision line writes them for a local run, so the same files run in both places.

// src/storage.mjs
import { StorageWireClient } from '@turnzero/storage';

export const AREA = 'uploads';

// Each call goes to the platform with the application's credential.
function transport(path, init = {}) {
  const origin = process.env.TURNZERO_CLOUD_GATEWAY_URL || process.env.TURNZERO_CLOUD_API;
  return fetch(`${origin}${path}`, {
    ...init,
    headers: { ...init.headers, authorization: `Bearer ${process.env.TURNZERO_CLOUD_TOKEN}` },
  });
}

export const storage = new StorageWireClient(transport);

src/server.mjs listens on PORT at once, then declares the area with mintArea, naming the application from TURNZERO_CLOUD_APPLICATION. Until the declaration succeeds, every route answers 503, the health path included. Then it serves four routes: GET /health, PUT /files/<name> to upload, GET /files to list, and GET /files/<name> to download. The declarations here must equal any declaration your tool made for the area with declare_storage_area. Otherwise every start is refused: the platform answers differing_redeclaration, which the client throws as declarations_differ.

The download route sends each file as an attachment, with x-content-type-options: nosniff. A browser then saves an uploaded page or image rather than running it on your application's hostname. Each response ends when the request's abort signal fires, as the deploy's rules ask.

// src/server.mjs
import { createServer } from 'node:http';
import { AREA, storage } from './storage.mjs';

// Every write records this acting identity.
const IDENTITY = 'files-service';

async function readBytes(request) {
  const chunks = [];
  for await (const chunk of request) chunks.push(chunk);
  return new Uint8Array(Buffer.concat(chunks));
}

async function listNames() {
  const names = [];
  let cursor = null;
  do {
    const page = await storage.list(AREA, { cursor });
    names.push(...page.names.map((file) => file.name));
    cursor = page.next;
  } while (cursor);
  return names;
}

let ready = false;

async function handle(request, response) {
  // The deployed runtime's abort signal: end the response when it fires.
  request.signal?.addEventListener('abort', () => response.destroy(), { once: true });
  const send = (status, body) => {
    if (response.destroyed) return;
    response.writeHead(status, { 'content-type': 'application/json' });
    response.end(JSON.stringify(body));
  };
  if (!ready) return send(503, { error: 'starting' });
  const { pathname } = new URL(request.url, 'http://localhost');
  if (request.method === 'GET' && pathname === '/health') return send(200, { ok: true });
  if (request.method === 'GET' && pathname === '/files') return send(200, { files: await listNames() });
  if (!pathname.startsWith('/files/')) return send(404, { error: 'not_found' });
  const name = decodeURIComponent(pathname.slice('/files/'.length));
  if (request.method === 'PUT') {
    const contentType = request.headers['content-type'] ?? 'application/octet-stream';
    const { version } = await storage.put(AREA, name, await readBytes(request), { identity: IDENTITY, contentType });
    return send(201, { name, version });
  }
  if (request.method === 'GET') {
    try {
      const { bytes, meta } = await storage.get(AREA, name);
      // An attachment, never shown in place, so an uploaded page cannot run on this hostname.
      response.writeHead(200, {
        'content-type': meta.contentType,
        'content-disposition': 'attachment',
        'x-content-type-options': 'nosniff',
      });
      return response.end(bytes);
    } catch (error) {
      if (error.code === 'no_file') return send(404, { error: 'no_file' });
      throw error;
    }
  }
  send(405, { error: 'method_not_allowed' });
}

export async function startServer(port) {
  const server = createServer((request, response) => {
    handle(request, response).catch((error) => {
      const refused = error instanceof URIError || error.code === 'invalid_name';
      if (!refused) console.error(error);
      if (response.destroyed) return;
      if (!response.headersSent) response.writeHead(refused ? 400 : 500, { 'content-type': 'application/json' });
      response.end(JSON.stringify({ error: refused ? 'invalid_name' : 'internal' }));
    });
  });
  await new Promise((resolve) => server.listen(port, resolve));
  // Declare the area before any file route runs. Every start restates the same declarations.
  await storage.mintArea(AREA, { accountKeyed: false, versionKeeping: false, application: process.env.TURNZERO_CLOUD_APPLICATION });
  ready = true;
  return server;
}

The routes write under one fixed acting identity and check no end user's access. Before real users reach them, give each route the access check step 4 describes.

src/main.mjs starts the server on PORT:

// src/main.mjs
import { startServer } from './server.mjs';

await startServer(Number(process.env.PORT ?? 8080));

4. Authorize access and issue transfer grants

The platform's routes do not support shared access levels or separate files by end user. Your backend must check the end user's access before it reads or writes a file or issues a transfer grant. The acting identity records who acted; it grants no access.

For a direct upload or download from the browser, call POST /storage/v0/grants under the application's platform credential or a token for that application. Name the environment as in step 2; the grant then reaches only that environment's files. An account session or an account-scoped token cannot issue a grant on this route. They use mint_upload_grant, below, instead. Send JSON with:

• area and name;
• direction: upload or download;
• identity: a nonempty acting identity, fixed for the grant;
• max_bytes, for an upload only: a positive integer up to the upload limit, which is the default when you leave it out.

The response returns grant once, with id, area, name, direction, identity, max_bytes, and expires_at. The browser sends the grant as the bearer credential on PUT to upload that one file, or GET to download it. Other files, directions, and routes are refused.

An upload grant is refused for a name that a write refuses, as step 2 describes. A download grant for a file already stored under such a name is issued as usual.

An upload grant allows one successful write; after a refused write, it stays usable until it expires. A download grant allows repeated reads until it expires. Upload conditions still apply.

A grant lasts five minutes by default; platform staff can change the default. A grant ends when its application or account is deleted. A gateway that checked the grant in the last 30 seconds can still answer it for up to 30 seconds after it ends. While the account is suspended, a grant is refused 403 account_suspended on its file route, and the backend cannot mint a new one. But suspension ends no grant: an unexpired grant works again once platform staff reinstate the account.

A platform credential and an application-scoped token each reach only their application's areas. Neither can issue a grant for another application's area.

An upload grant from your tool

To upload one file from a shell, your tool calls the management action mint_upload_grant instead of keeping a token there. It takes:

• application, and an area bound to that application;
• name, the file's name in the area;
• environment, optional: where you leave it out, the environment a deploy goes to, production on an application with one environment and development on one with two;
• identity, optional: the acting identity deploy where you leave it out;
• local_path, optional: the file on your machine, as an absolute path or one relative to the folder the line runs in. Where you leave it out, the file is the one named by the last segment of name, in that folder.

On an application with one environment, leaving environment out puts the file beside your live application's own files. Name development for a test or seed file, so it stays with the data your local runs use.

The platform never reads local_path. A local_path that a shell would change in the line is refused 400 invalid_request, and nothing is minted. That is a path with a control character, a double quote, $, a backtick, %, !, &, |, <, >, ^, a typographic double quote, a doubled backslash, or a trailing backslash. PowerShell passes a path that has no space to npx.cmd without its quotes, and cmd then reads &, |, <, >, and ^ as command syntax. So those five are refused on every operating system.

The line does not expand ~: name your home folder in full.

The response contains grant once, expires_at, max_bytes, the file's whole address, and three ready ways to upload the file:

• command, one line for macOS and Linux that runs the put subcommand of the turnzero-cloud command;
• command_windows, the same line for every Windows shell, with npx.cmd in place of npx;
• commands, two lines that need no turnzero-cloud command: commands.curl, run in a shell such as bash with curl 7.76 or later, and commands.powershell, run in Windows PowerShell.

Your tool runs one of them once, as given, before expires_at. The put line pipes the grant to the command on its input, never as an argument. Each of the two commands sends the grant as the bearer and nothing else, because the grant fixes the environment and the identity. All of them read the same file on your machine.

This is command for a call that names the area artifacts and the name app.zip, with <grant> where the call's own grant goes. A test holds it to the form the platform composes:

echo <grant> | npx -y https://turnzero.ai/packages/turnzero-cloud-0.8.0.tgz put --area artifacts --name app.zip --path "app.zip"

Where your tool is connected to an origin other than https://turnzero.ai, the line includes --origin with that origin.

The command prints one line that opens with put: and gives the file as the area stores it, as JSON: its name, area, size, version, content_type, and last_written_at. It ends with one of three statuses:

• 0: the file was written;
• 3: nothing was written, as on a refusal such as transfer_grant_spent for a second run of the line;
• 2: the outcome is unknown, so the file may have been written. Call mint_upload_grant again and upload under the new grant.

The response has no command and no command_windows where the name contains a character that one line cannot pass unchanged through every shell. A space is the common case. The line takes a name of up to 1,024 letters, digits, ., _, -, and / that begins with neither - nor /. The response's detail then says so, and your tool runs commands.curl or commands.powershell, which read the same file.

In Windows PowerShell 5.1, Invoke-WebRequest needs -UseBasicParsing, and commands.powershell includes it.

The grant is an upload grant like any other: one file, spent by the one successful write, and refused on every other route. A second write under it is refused transfer_grant_spent, even with the same bytes. The session, an account-scoped token, and a token for that application can call the action. The application's platform credential cannot, and the backend mints on POST /storage/v0/grants instead.

5. Limits and metering

A request body can be up to 8 MiB by default, and a list page up to 100 files; platform staff can change both. An upload grant can set a smaller max_bytes, never a larger one. A list request takes prefix, an opaque cursor, and limit, and a larger limit is reduced to the page maximum. Follow the returned cursor for the next page.

The platform records the bytes, outcome, and time of each file operation against the application. When the application stores more data than its plan allows, puts are refused usage_over_quota, and reads, lists, and deletes keep working. Deleting the account removes its areas and files; the metering history is kept.

6. The deploy area, and deploying from an area of your own

To deploy, your tool calls deploy naming neither artifact nor upload, and runs the line the response contains, once. The line runs the turnzero-cloud command, which zips the application's folder, uploads the zip, and starts its deploy. The command stores the zip in your application's deploy area, deploy-<application id>, which the platform creates and keeps.

The deploy area is the platform's, not yours:

• Only a deploy's short-lived upload grant reaches its files. The command alone receives it, from the call that prepares the upload. Your session, your tokens, and the application's platform credential are refused 403 area_scope_refused, whether they list, read, write, or delete there.
• That grant's upload can be sent again with the same zip, for example after a response was lost. Once the first upload has landed, and until the grant expires, the response has status 200 and the file's name, area, size, and sha256, with no version and no ETag, and nothing is written. Other bytes are refused transfer_grant_spent.
• It keeps one pending upload per environment. The next upload prepared for that environment replaces an upload no deploy has read, and the replaced upload's command is refused transfer_grant_spent. For up to 30 seconds after the replacement, a gateway that checked the replaced grant answers its upload as a repeated write, and nothing is written. A later deploy call from your tool also ends the last line's upload where it has not started.
• A file is deleted when the deploy that read it ends, and any file older than a day is deleted at the next daily pass.
• Its bytes count in your application's stored data, and a put over the plan's quantity is refused usage_over_quota.
• An export leaves it out.

The prefix deploy- is reserved: a new area whose name begins with it is refused 400 area_name_reserved. An area you declared under the prefix before it was reserved keeps its files and your access.

The artifact form is the route that needs no turnzero-cloud command. A script of your own deploys from an area of its own. It uploads with one of the two commands that mint_upload_grant returns, which need no turnzero-cloud command, or with the put line, as An upload grant from your tool describes. Or it uploads under a minted token, with the header naming the environment the deploy goes to and an acting identity, as An upload under a minted token shows.

That script then calls deploy with artifact: the area, the file name, and the SHA-256 hash. The reference's optional environment member gives the part of the area that contains the file, and defaults to the environment being deployed. The area must belong to the application you deploy, and must not be its deploy area; any other area is refused 403 artifact_area_mismatch before it is read. The platform checks the hash before it builds the image. A promote needs no upload, because it reuses that image. Deploy an application covers the whole deploy.

An upload under a minted token

The samples below are the form of an upload without the turnzero-cloud command. They upload app.zip under a minted token into an area of your own named artifacts, for a script of your own that deploys with artifact. The token is in the setting TURNZERO_CLOUD_MINTED_TOKEN, and deploy-tool is the acting identity.

With curl 7.76 or later, from a shell. The bearer header arrives on standard input, so the token never appears in curl's arguments. No test checks this sample.

printf 'Authorization: Bearer %s' "$TURNZERO_CLOUD_MINTED_TOKEN" | curl --fail-with-body --upload-file app.zip \
  -H @- \
  -H "x-turnzero-cloud-environment: production" \
  -H "X-Acting-Identity: deploy-tool" \
  -H "Content-Type: application/zip" \
  "https://turnzero.ai/storage/v0/areas/artifacts/files/app.zip"

With Node.js, saved as upload.mjs and run with node upload.mjs. No test checks this sample.

import { readFile } from 'node:fs/promises';

const response = await fetch('https://turnzero.ai/storage/v0/areas/artifacts/files/app.zip', {
  method: 'PUT',
  headers: {
    authorization: `Bearer ${process.env.TURNZERO_CLOUD_MINTED_TOKEN}`,
    'x-turnzero-cloud-environment': 'production',
    'x-acting-identity': 'deploy-tool',
    'content-type': 'application/zip',
  },
  body: await readFile('app.zip'),
});
console.log(response.status, await response.text());
if (!response.ok) process.exit(1);

Expected result

declare_storage_area returns the area and an outcome:

• created for a new area;
• unchanged when the same declarations were repeated.

list_storage_areas then lists the area with its declarations and its application.

A PUT on a file route returns 200, with the file's version in the ETag header and its metadata as JSON; the package's put returns that version. A grant mint returns the grant once with its expires_at, and the browser's transfer under it reaches the environment the grant names. mint_upload_grant returns the same once, with the file's address, the put line as command and command_windows, and the two commands. The put line, run as given, prints a line that opens with put: and ends with status 0. A file written from development is listed from development and absent from production.

undeclare_storage_area returns outcome: removed with the area as it was, or outcome: unchanged with area: null when no area has that name.

Refusals

A refusal on a storage route returns the platform's error shape with the refusal's name, and a refused write stores nothing. The rows cover declare_storage_area, undeclare_storage_area, mint_upload_grant, and the artifact checks of deploy, then the storage routes. A zip the deploy cannot read or run is refused 400 artifact_unreadable or artifact_layout_invalid, which the deploy page's Refusals describe. Deploy an application lists the deploy's other refusals.

Related

• Deploy an application covers the artifact upload and the deploy that reads it.
• Storage describes the package's client and what the platform's routes do not do.
• Use the library copies the package into the project.
• Mint a token mints the token a script uses on the storage routes.
• Applications and environments explains the two environments whose files each area keeps apart.
• declare_storage_area, undeclare_storage_area, list_storage_areas, and mint_upload_grant in the generated reference give each action's arguments and result.
• Glossary defines the terms used on this page.