Self-hosting Rakazo

The signed-in product is a long-running API, a Graphile Worker, Postgres, and a computer provider (Docker supervisor, E2B, Daytona, CreateOS, or Boat). It is not a static site. The marketing site in apps/www can be hosted separately.

Local (source checkout)

Same as the README quick start: .env from .env.example, pnpm sandbox:build, pnpm dev, then the URL it prints (http://127.0.0.1:5173 by default, or http://localhost:5173 — both loopback hosts are trusted). Electron: pnpm --filter @rakazo/desktop dev while that stack is up, choosing Existing instance with that address. The desktop app’s This computer option instead installs and runs the published images itself with Docker Compose (see Published images), using port 45173 by default so it can run alongside pnpm dev. If that port is occupied, the app selects and remembers another loopback port. The managed API gets a Docker-assigned loopback port; all desktop traffic uses the web origin.

For source development in WSL, keep the checkout and data directory in the Linux filesystem (for example, ~/rakazo), and run pnpm dev as your normal user. The host-run supervisor matches bot container UID/GID to that user. If Docker Desktop container IPs are unreachable, set SANDBOX_CONTROL_VIA_LOOPBACK=true in .env; this publishes the token-protected control service on a random loopback port. Leave this unset for the Compose-hosted supervisor.

Compose bot homes mount only their own subdirectory of the application volume using Docker volume semantics. Docker’s internal volume paths are never used as host bind mounts.

Published images (no checkout)

Pull Postgres and ghcr.io/elie222/rakazo/app into any empty folder. No clone or image build. Requires Docker Engine 26+ (API 1.45+ for bot home volume subpaths), the Compose plugin, curl, and OpenSSL.

mkdir -p rakazo && cd rakazo &&
curl -fsSLO https://raw.githubusercontent.com/elie222/rakazo/main/infra/compose/install-images.sh &&
bash install-images.sh

The installer downloads docker-compose.images.yml and .env.images.example, creates .env with random secrets, then pulls and starts the images. It preserves an existing .env when rerun. For the installer secret list, non-reuse rules, and recovery, see Self-host secrets checklist. To customize the public URL, image tag, or optional providers before startup, run bash install-images.sh --prepare-only, edit .env, then run bash install-images.sh. Flags may be combined in either order: --prepare-only, --local.

SANDBOX_PROVIDER defaults to docker. The images Compose file runs a sandbox supervisor (from the app image, on the internal network only) and pulls ghcr.io/elie222/rakazo/computer. Signup and local Docker computers work without an E2B account. Optional remote providers: set SANDBOX_PROVIDER to e2b, novita, daytona, createos, or boat and add the matching API key. The published-images Compose stack requires SANDBOX_SUPERVISOR_TOKEN for every provider; leave it empty and compose up fails closed.

Optional: set OPENROUTER_API_KEY or connect a model in the UI after signup. A provider that authenticates from the host needs no key: for Amazon Bedrock with an ECS task, IRSA or EC2 instance role, opt in with PI_DEFAULT_CREDENTIALS=host and set PI_DEFAULT_PROVIDER=amazon-bedrock, PI_DEFAULT_MODEL and AWS_REGION. With an instance role, keep restricted computer egress on so bot computers cannot read the role’s credentials from the metadata endpoint. Approvals have three modes in Settings → Approvals: Never ask runs every action, Auto lets an approval model pass actions that fit the request, and Ask asks before any change. A space starts in Auto, which asks like Ask until an approval model is available. Each person can allow or block single tools and sites in every mode, and a space can customize its approvals apart from the account.

In Settings → Models → Approval model, each user can select a model from an existing connection and add ordered backups, independently of their main model. Personal connections use their own credentials; hosted calls follow the same Rakazo credit rules as chat. Backups are tried only when a checker fails. An explicit ask, including an uncertain decision, still requires a person.

The deployment checker remains the default when no approval model is selected. For OpenRouter Jev, set RAKAZO_AUTO_REVIEW_PROVIDER=openrouter and RAKAZO_AUTO_REVIEW_MODEL=typesafe/jev-1.14; it uses OPENROUTER_API_KEY and the Decisions API. TypeSafe direct remains available through RAKAZO_AUTO_REVIEW_PROVIDER=jev and TYPESAFE_API_KEY. Core runs without either hosted service.

The example defaults to edge (main builds). Every publish is multi-arch (amd64 + arm64), so arm64 hosts need no special tag. Do not assume latest is present until a stable release exists.

Open http://127.0.0.1:5173. Signup never grants server-administration permissions. Put TLS in front of :5173 for a public host and set the three public origins to that HTTPS URL.

Images Compose binds web to loopback (127.0.0.1:5173). Terminate TLS on the host and proxy there. Vite preview same-origin-proxies /api and /rpc, so do not expose :3100. Set BETTER_AUTH_URL, WEB_ORIGIN, and API_URL to that same HTTPS origin, and set RAKAZO_HOST to its hostname (for example, app.example.com).

app.example.com {
	reverse_proxy 127.0.0.1:5173
}

Open Agent computer on a bot, or send a message that uses the desktop, to see the local Docker computer. For in-stack Caddy plus remote E2B computers, use the production Compose path and infra/compose/Caddyfile.prod instead of this host proxy.

Restricted networks / mirror downloads

If the installer, Compose downloads, or image pulls are blocked, use the restricted-network guide for mirror settings and local files.

Bot computer resource ceilings

Each Docker computer runs Xvfb, a window manager and a full Chromium driven by an agent that decides for itself what to open, so it is capped. These defaults provide a starting point for the Docker computer topology:

Set any of them to 0, none or unlimited to remove that ceiling. A malformed value fails the supervisor at startup naming the variable, rather than surfacing later as a failed bot.

API and worker memory

Chat runs execute in the API container that receives the message, and routines and other background runs execute in the worker, so both hold live agent turns. In the production Compose file, RAKAZO_API_MEMORY_LIMIT and RAKAZO_WORKER_MEMORY_LIMIT cap them (Compose sizes such as 2g or 3072m, default 2g each). Raise them on a larger host. Deploys briefly run the old and new container of each side by side, so leave room for both.

Restricted computer egress

Restricted egress gives Docker computers the public internet (browsing, DNS, apt, git over SSH) but not the Docker host, the LAN, or link-local cloud metadata such as 169.254.169.254, which on a cloud VM would hand instance credentials to anything a bot runs.

It takes two parts:

1. SANDBOX_COMPUTER_EGRESS=restricted gives each computer network a deterministic host bridge name (rakazo-c…) instead of a generic br-*.

2. The host ruleset, applied once per Docker host:

   sudo bash infra/compose/restrict-computer-egress.sh

   install-images.sh downloads and runs it (asking for sudo) and records the result in .env. Where the rules cannot be installed (macOS, Windows, Docker Desktop, rootless Docker, no iptables, or a failed run) it writes SANDBOX_COMPUTER_EGRESS=open and says why. The script drops forwarded traffic from rakazo-c* bridges to all non-public IPv4/IPv6 destinations via the DOCKER-USER chain, adds an INPUT drop so computers cannot open connections to the host (established replies to host-initiated control and screen connections still flow), and installs a systemd oneshot so the rules return after a reboot. --print shows the exact rules without changing anything; --remove uninstalls.

Before the first restricted computer starts, the supervisor checks that the host drops connections from a rakazo-c* bridge. Without the ruleset, computers fail to start with an error naming the script instead of running unrestricted; run the script (no restart needed) or set SANDBOX_COMPUTER_EGRESS=open and recreate the supervisor.

When SANDBOX_COMPUTER_EGRESS is unset, the Compose files use auto: the supervisor runs the same check and uses restricted egress if the ruleset is in place, otherwise open egress with a warning in its log. It keeps the first completed answer; a check that could not finish (a Docker error or timeout) fails that computer’s start and runs again on the next one. An install from before the ruleset existed therefore keeps working when it upgrades. After installing the ruleset, recreate the supervisor, or set restricted to refuse computers whenever the rules are missing. A host-run supervisor (pnpm dev) and the desktop app’s This computer stack default to open: Docker Desktop keeps its bridges in a VM the host rules cannot reach.

Screen streaming is unaffected: the supervisor and web proxy join each computer’s bridge, and the ruleset exempts traffic whose in- and out-interface are both rakazo-c* before any drop (that exemption is load-bearing on hosts where br_netfilter feeds bridged frames through FORWARD). Rules match interface names rather than subnets, so computer create/delete cycles need no firewall maintenance; deleting a computer removes its traffic from the match and nothing else. A computer provisioned while egress was open is replaced on its next provision — resuming it on an unnamed bridge would bypass the restriction.

Computers resolve names through Docker’s embedded DNS, which forwards each lookup to the host’s resolvers from inside the computer’s network. When the host’s resolver has a private address, as a cloud VPC resolver or a LAN router does, the ruleset drops that forward too and every lookup times out while connections by IP still work. Set SANDBOX_COMPUTER_DNS to public resolvers (comma-separated IP addresses, such as 1.1.1.1,1.0.0.1) and recreate the supervisor: Docker then forwards computers’ lookups there, and each computer picks the servers up on its next provision.

If computers must reach LAN services, an internal proxy, or endpoints bound to the host, set SANDBOX_COMPUTER_EGRESS=open, recreate the supervisor, and run sudo bash infra/compose/restrict-computer-egress.sh --remove. The flag alone does not uninstall the host rules, which keep matching the still-named rakazo-c* bridges until removed.

The ruleset requires Linux Docker Engine with the iptables backend; Docker Desktop, rootless Docker, and firewall-backend: nftables are unsupported. The SANDBOX_SCREEN_NETWORK=internal topology shares one network instead of per-bot bridges, so the mode does not apply there.

Where Docker Engine runs in a Linux VM, as with OrbStack on macOS, the rules belong in the VM, which sudo on the host cannot reach. Apply them from a container on the VM’s host network instead, and keep it running: the VM does not persist the rules, so a loop puts them back after a restart. The installer does not download the script on macOS. From the directory that holds the stack’s .env:

curl -fsSLO https://raw.githubusercontent.com/elie222/rakazo/main/infra/compose/restrict-computer-egress.sh

Save the following as egress.yml beside that script and start it with docker compose -f egress.yml up -d. The name keeps it in its own Compose project:

name: rakazo-computer-egress
services:
  computer-egress:
    image: alpine:3
    network_mode: host
    cap_add: [NET_ADMIN, NET_RAW]
    restart: unless-stopped
    volumes:
      - ./restrict-computer-egress.sh:/restrict-computer-egress.sh:ro
    command: >-
      sh -c "apk add --no-cache bash iptables ip6tables &&
      while true; do bash /restrict-computer-egress.sh --apply; sleep 60; done"

Then change SANDBOX_COMPUTER_EGRESS from open to restricted in .env and recreate the supervisor. Docker Desktop has not been tested this way.

Docker Compose (single machine)

1. Copy .env.example to .env and set POSTGRES_PASSWORD and POSTGRES_APP_PASSWORD (openssl rand -hex 16 each), plus BETTER_AUTH_SECRET, ENCRYPTION_KEY, and SCREEN_PROXY_SECRET to independent long random strings (32+ characters; 64 hex for ENCRYPTION_KEY). Docker sandboxes also need a dedicated SANDBOX_SUPERVISOR_TOKEN. Keep existing ENCRYPTION_KEY values so stored credentials stay decryptable.
2. Set OPENROUTER_API_KEY (and COMPOSIO_API_KEY if you want Plugins).
3. Build the computer image: pnpm sandbox:build (Compose also builds it via the computer service).
4. docker compose --env-file .env -f infra/compose/docker-compose.yml up --build
5. Open the web origin (http://127.0.0.1:5173 by default). Signup never grants server-administration permissions.

On Windows, if an older clone with core.autocrlf=true leaves the computer pane hung on boot (bash\r in sandbox logs): from a clean worktree, set git config core.autocrlf false, run git add --renormalize . && git checkout -- ., then rebuild with pnpm sandbox:build.

Compose runs Postgres, the sandbox supervisor (Docker socket), API, worker, and a Vite preview of the web app. Bot computers are sibling containers (rakazo/computer:local) on separate per-bot networks; only the supervisor and screen proxy join each one. The API process does not get an unrestricted Docker socket; the supervisor owns the lifecycle.

Postgres stays on the Compose network only (not published on the host), matching the images compose. Credentials come from .env (POSTGRES_PASSWORD is required). Prefer a URI-safe value (openssl rand -hex 16); characters such as @ : / ? # % break the interpolated DATABASE_URL inside Compose. Official Postgres images set user, password, and database only on first volume init, so an existing database volume keeps its original identity: keep those values in .env, or change them in place with ALTER ROLE / rename. Recreate the volume only after a backup (or when the data is disposable); docker compose down -v deletes all Postgres state. For host-side clients (pnpm db:migrate, GUI tools), add infra/compose/docker-compose.postgres-host.yml so Postgres is published on loopback 127.0.0.1:5433, or use docker compose --env-file .env -f infra/compose/docker-compose.yml exec postgres sh -c 'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB"'. Do not publish Postgres on a public interface.

Database roles

Rakazo uses two Postgres logins:

• The owner (POSTGRES_USER, or DATABASE_MIGRATION_URL outside Compose) runs migrations and owns the tables.
• The app login (rakazo_app with POSTGRES_APP_PASSWORD, or DATABASE_URL) is not a superuser, cannot bypass row-level security, cannot create objects, and can only read and write the app tables.

pnpm db:migrate (in containers, its prebuilt form pnpm --filter @rakazo/api db:deploy, which the one-shot migrate container runs before the API starts) applies the migrations and the job queue schema as the owner. Only that container gets the owner login, so code running in the API or worker cannot find one that bypasses row-level security. It then creates the app login named in DATABASE_URL, or updates its password, and grants it the rakazo_app role. When migrations run as a role without CREATEROLE, as on some managed Postgres, create the login yourself and GRANT rakazo_app TO <login>.

Messages, events, threads, memory, artifacts, secrets, model and voice credentials, skills, bots, runs, tasks, routines, connections, MCP servers and sign-ins, computers, boards and tickets, messaging identities and outbox, external conversations, steering messages, usage, scratchpad items and approval rules have row-level security. Each signed-in request sees only its Space’s rows, plus the user’s own account-level credentials. A query that leaves out its Space filter returns nothing from other Spaces. A background job that acts for one run, routine, computer, skill, thread or cloud agent runs as that row’s Space and user, so agent tools get the same backstop. Security events are written and read only in system scope. Reconcilers, queue sweeps, the messaging outbox drain and webhooks run in a named system scope (runAsSystem in packages/db). The job queue’s tables (graphile_worker) hold job metadata, not Space content: graphile-worker turns on row-level security for them, and pnpm db:migrate adds a policy that gives the app login every row.

With only DATABASE_URL set, migrations and the app share one login. Everything keeps working, but pnpm db:migrate, the API and the worker warn that row-level security is not enforced. To fix this, set DATABASE_MIGRATION_URL to the owner and DATABASE_URL to a new login.

Run pg_dump as the owner. The app login would only dump rows it is allowed to see.

The Docker supervisor is not published as its own image and is not exposed on the host. It runs from the app image, stays on the internal Compose network, and holds the Docker socket because access to it is equivalent to control of the Docker host. Docker sandboxes require SANDBOX_SUPERVISOR_TOKEN (API, worker, supervisor). SCREEN_PROXY_SECRET signs browser-screen capabilities (API and web proxy). Keep both distinct from BETTER_AUTH_SECRET.

New credentials use versioned AES-GCM with per-record salt and row-bound AAD. Legacy ciphertext stays readable.

On a VPS, put TLS in front of :5173 (or serve the web build behind your proxy) and set:

BETTER_AUTH_URL=https://app.example.com
WEB_ORIGIN=https://app.example.com
API_URL=https://app.example.com

Cookies and CORS follow those origins. SIGNUPS_ENABLED and SIGNUP_ALLOWLIST come from the operator’s environment on every start. An empty allowlist clears any old stored list; signup order never grants privileges.

With a nonempty signup allowlist and SMTP configured, users—including existing accounts—must verify their email to sign in. Without an allowlist, verification never blocks: a password signup signs in at once and is sent a verification link that only marks the address verified.

SMTP also enables sign-in with a six-digit emailed code, which expires after ten minutes and allows three attempts. A code can create an account, so it follows the same registration and allowlist rules as password signup. Signing in with a code to an existing unverified account removes that account’s password, SSO links and sessions, since nothing proved they belonged to the mailbox owner. This is what stops someone who registered another person’s address from keeping access.

Allowlisted signup requires mailbox verification and email delivery, including the first account. Without email delivery, use the server-side account bootstrap described below; public signup never marks an address verified or assigns server permissions.

Server configuration belongs in the operator’s environment. App accounts cannot change signup policy, server integration credentials, or deployment updates, and cannot read deployment security logs. SIGNUPS_ENABLED and SIGNUP_ALLOWLIST are authoritative on every start; an empty allowlist clears any old stored policy. There is no app-admin role, first-user privilege, or ADMINS list. Workspace owners manage only their own workspace.

For an installation without email, first set SIGNUPS_ENABLED=false in .env and start the stack. Create the intended account privately (the password stays in the shell environment):

read -r -p "Account email: " OWNER_EMAIL
read -r -s -p "Account password (at least 16 characters): " OWNER_PASSWORD
export OWNER_EMAIL OWNER_PASSWORD
docker compose --env-file .env -f docker-compose.images.yml exec -e OWNER_EMAIL -e OWNER_PASSWORD api node dist/cli/bootstrap-owner.js
unset OWNER_EMAIL OWNER_PASSWORD

For a source checkout, use -f infra/compose/docker-compose.yml instead. This creates an ordinary verified account and a personal workspace, with no server privileges. For an existing unverified account, its password must match before it can be verified; an existing verified account is left unchanged. Keep an installation without email on a trusted local network.

Optional OpenID Connect SSO

SSO works with a self-hosted or hosted OpenID Connect provider. Leave its settings unset to keep password authentication alone. Configure all three credentials together on the API:

OIDC_ISSUER=https://identity.example.com
OIDC_CLIENT_ID=replace-with-client-id
OIDC_CLIENT_SECRET=replace-with-client-secret
OIDC_NAME=SSO
OIDC_SCOPES=openid email profile
AUTH_PASSWORD_ENABLED=true
OIDC_ALLOW_SIGNUP_BYPASS=false

The issuer must be HTTPS and exactly match the discovery document’s issuer. Rakazo loads <issuer>/.well-known/openid-configuration, verifies ID tokens against discovery JWKS with issuer, audience and nonce checks, and uses authorization codes with PKCE. Additional scopes may be space- or comma-separated; openid email profile are always requested. OIDC_NAME is the button label, defaulting to “SSO”. Secrets remain on the API; Compose clears the worker’s OIDC credentials. The capabilities endpoint exposes only the label, discovery availability, and enabled authentication methods.

Register this redirect URI at the provider (using your public BETTER_AUTH_URL origin):

https://app.example.com/api/auth/callback/oidc

Web, Electron and mobile use the same provider redirect URI. Electron completes SSO in a sandboxed in-app popup sharing the app’s session, then returns to the main window. Mobile completes the callback on the API, then returns to rakazo://sign-in (or rakazo://account for linking and reauthentication) through Better Auth’s Expo authorization proxy and a native auth browser session. The rakazo app scheme is trusted by the auth server; never register a client secret in the mobile app. Mobile stores the resulting session with SecureStore, like password sign-in. Native builds need the Expo WebBrowser module.

Provider emails are verified only when email_verified is the boolean true. False or missing claims stay unverified, including on subsequent sign-ins. Sign-in never links accounts by email. If an email belongs to another account, sign in to that existing account and choose Link SSO in account settings. Linking requires an authenticated session, a verified provider email and matching email addresses. The issuer and provider subject identify the linked account thereafter. Changing issuers does not reuse an old identity. Old-issuer links remain stored but do not count as linked to the current provider, so Link SSO becomes available again. Link the new identity from the existing signed-in account; authenticated-session, verified-email and matching-email checks still apply. Restoring the old issuer makes its existing links usable again.

Ordinary SSO sign-in and linking may reuse the identity provider’s existing session. They do not grant fresh authentication proof for account deletion. The account settings’ explicit reauthentication flow requests prompt=login and max_age=0, binds the callback to the same linked issuer and subject, and requires the initiating session to remain valid. Only that callback grants proof to the resulting session, expiring five minutes after provider authentication (callback delays do not extend it). Its verified ID token must contain an auth_time consistent with that request (allowing 60 seconds of clock skew) and less than five minutes old. Missing, stale or invalid claims fail closed. OAuth callbacks are single-use, including concurrent requests. Password verification and emailed deletion codes remain available. Submit deletion codes through the account settings or the CSRF-protected POST /api/auth/delete-user endpoint; the GET deletion callback is disabled.

SSO signup follows closed registration and the deployment allowlist before creating an account. Allowlisted provider emails must be verified; signup never bypasses mailbox proof. OIDC_ALLOW_SIGNUP_BYPASS=true explicitly admits IdP identities without applying the allowlist or its email-verification admission requirement. Enable it only when the IdP controls who may join this deployment. It does not reopen closed registration, and does not change email verification claims or linking rules. Existing admitted accounts can sign in while registration is closed.

Set AUTH_PASSWORD_ENABLED=false for SSO-only operation. Password sign-in, signup, password reset and password mutation endpoints are disabled server-side, and sign-in forms are hidden. The API refuses to start in this mode unless all OIDC credentials are configured. It can still start while discovery is temporarily unavailable: the provider remains registered, sign-in returns a temporary error, and background retries recover without restarting. Discovery is lazy on first use, refreshed in the background, and failures back off up to one minute. Availability reflects discovery, not a guarantee that the provider’s token endpoint is currently reachable.

Account deletion keeps the existing password confirmation for password users. Users without a password may delete after explicit provider reauthentication within five minutes; Sign in again starts a fresh provider round-trip bound to the signed-in account. Choosing a different identity cannot confirm deletion or create another account in that flow. A stale or borrowed session alone cannot authorize deletion. With transactional email configured, Send deletion code sends a single-use code to the account email, valid for ten minutes. Enter it in account settings to confirm deletion. Email-code requests are rate-limited; invalid, expired or wrong-account codes fail. No password needs to be created for deletion.

Mobile push notifications

The API and worker send iOS and Android notifications through Expo’s push service. If the Expo project has enhanced security for push notifications enabled, every send needs an access token; without one Expo rejects all pushes. Create an access token in the Expo account that owns the project and set it for both services:

EXPO_ACCESS_TOKEN=replace-with-your-expo-access-token

Leave it unset when enhanced security is off. A missing or revoked token fails the push with an error that names EXPO_ACCESS_TOKEN; the token itself is never logged.

Verification and password recovery email

Password changes for signed-in users require no email configuration. Forgotten-password recovery appears on sign-in only when a transactional email provider is available. Rakazo uses a provider-neutral contract with Resend API and SMTP adapters. Templates render locally with React Email into HTML and plain text, independent of the sending provider.

To use the Resend HTTP API, verify a sending domain and create a sending-only API key restricted to that domain, then configure:

EMAIL_PROVIDER=resend
EMAIL_API_KEY=replace-with-sending-api-key
EMAIL_FROM=Rakazo <hello@mail.example.com>

Disable click and open tracking for account email in the provider’s domain settings. Resend delivery has a ten-second deadline and bounded retries with the same idempotency key, including when a response is lost. Errors omit provider response bodies and credentials.

For Amazon SES or another SMTP server, use:

SMTP_URL=smtps://smtp-user:replace-with-password@smtp.example.com:465
EMAIL_FROM=Rakazo <no-reply@example.com>

Set EMAIL_REPLY_TO to send replies somewhere other than EMAIL_FROM.

EMAIL_PROVIDER=smtp is optional: an existing SMTP_URL continues to select SMTP when no provider is specified. EMAIL_PROVIDER=none disables external sending. Explicit provider settings take precedence over an old SMTP_URL; incomplete or unknown provider configurations fail at startup.

Resend also supports SMTP: use smtp.resend.com, username resend, and an API key as the password. For Amazon SES, use the regional SMTP endpoint and SES SMTP credentials; these are different from ordinary AWS access keys. Verify the sender/domain with the provider before testing delivery. Keep credentials in .env, never in tracked files. smtps:// uses implicit TLS; smtp:// is also supported but requires STARTTLS. Rakazo rejects configuration that disables TLS or certificate verification.

Local source development can use the offline email emulator instead. It captures email without contacting a provider:

EMAIL_EMULATOR=true

The emulator is forcibly disabled when NODE_ENV=production and requires the API to bind to a loopback host. In NODE_ENV=development, captured messages are available from http://127.0.0.1:3100/api/dev/emails with cache disabled; the API logs only delivery metadata, never reset tokens. The inbox route is not registered in test, staging, or production.

Password policy

New passwords (signup, reset and change) must be 8 to 128 characters, with no composition rules. Optionally reject new passwords that appear in known breaches:

AUTH_BREACHED_PASSWORD_CHECK_ENABLED=true

This queries the Have I Been Pwned range API, sending only the first five characters of the password’s SHA-1 hash. It is off by default so the server makes no outside request. If the service is unreachable or takes longer than three seconds, the password is accepted and the failure logged.

Billing

Billing stays off unless STRIPE_SECRET_KEY and STRIPE_WEBHOOK_SECRET are both set (see .env.example). Enabling billing offers a monthly own-key plan for a personal model connection or monthly model credits in USD: $10, $20, $50, $100, $200, or $500. One-time top-ups use the same amounts. Monthly credits expire at the end of their service period; one-time top-ups do not expire. The hosted computer is included in both paths, and monthly credits include the own-key plan.

BYOK_MONTHLY_PRICE_CENTS in @rakazo/contracts sets the own-key plan’s price; null makes it free and ends paid own-key subscriptions at their period end. Runs check access on the server, so a lapsed organization’s bots and routines stop until it subscribes.

Configure the managed default model through the existing model connection settings. The default OpenRouter model is Luna 6; customers can choose another available model, with its usage deducted at model cost. In credit mode, the configured hosted provider uses Rakazo’s key even when a personal key for that provider is saved. Switching back to BYOK restores that connection. Personal connections to other providers continue to use their own keys and never debit Rakazo credits. Hosted calls reserve an upper cost estimate atomically before starting, then settle to reported or catalog-priced usage. Unknown usage retains its reservation for reconciliation.

STRIPE_PRICE_ID is optional and only needed to display the legacy seat price. Existing live seat subscriptions retain their access and billing portal. Upgrades charge the prorated difference immediately and add the amount collected to the credit balance; a failed payment leaves the plan unchanged. Downgrades lower the next renewal without charging now or removing credits. Credit purchases require settled payments; webhook replays do not add credits twice, and refunds reduce the remaining account balance.

Monthly funds are spent before top-ups. Each grant has an audited allocation lot, so reservations retain their funding source across renewal: unused monthly holds expire when released, while purchased top-ups remain available. Prorated upgrades share the original period end. Expiration creates a financial journal delta and runs before balance reads, reservations, settlements, and background recovery. Unclassified legacy balances are preserved as non-expiring funds.

Postgres owns accounting. billing_audit_events is an append-only journal enforced by database triggers; credit entries, usage rows, and balances are projections. Each managed call shares one callId across reservation, durable provider receipt, usage, and settlement. Provider generation IDs, token buckets, cost provenance, and pricing snapshots are retained. Usage and settlement commit together, and journal/usage history survives clearing messages or deleting a bot or organization. Corrections append evidence rather than rewriting earlier journal events. Existing rows receive explicit baseline events when the migration runs; earlier history cannot be recovered.

The API/worker reconciler recovers saved receipts and, when supported by the configured provider, fetches authoritative generation costs for interrupted calls and estimates. Retries use leases and idempotent settlement. Unverifiable calls retain their holds and become needs_review after six attempts. An hourly, paginated audit rebuilds drifted balance caches from the journal and records repairs. Investigate Model credit hold requires review logs using the shared call ID; do not turn missing provider evidence into a zero-cost settlement.

OpenRouter inference requests include the normal Rakazo user ID in user, HTTP-Referer (https://rakazo.com), and X-OpenRouter-Title (Rakazo). These values are sent only to OpenRouter’s HTTPS inference endpoints, never to compatible custom hosts. Recovery uses the existing configured deployment credential; no Redis or external accounting service is required.

The adapter pins Stripe API 2026-09-30.endive. Configure webhook destinations for your own account, using snapshot payloads on the same API version. Subscribe to all customer.subscription events, customer.deleted, checkout.session.completed, checkout.session.async_payment_succeeded, invoice.paid, invoice.payment_failed, charge.refunded, charge.dispute.created, and charge.dispute.closed. Credit Checkout restricts payment methods to cards. New subscriptions explicitly retain classic billing mode so API upgrades do not change invoice/proration behavior.

For local testing, forward Stripe test events to /api/v1/billing/webhook. Use the signing secret from stripe listen and test API credentials; keep them in an ignored environment file.

Pi tracing

Set OTEL_EXPORTER_OTLP_ENDPOINT on the worker to enable optional OpenTelemetry traces (for example, http://otel-collector:4318). The OTLP/HTTP exporter appends /v1/traces to the base endpoint and honors standard OTEL_EXPORTER_OTLP_HEADERS (comma-separated key=value pairs) and OTEL_SERVICE_NAME (default rakazo-worker). Compose workers read these settings from .env. With an empty or unset endpoint, no tracing SDK loads or initializes. No collector or hosted service is required to run Rakazo.

Traces contain a rakazo.agent.run span with pi.ai.request model-call spans and pi.agent.tool steps, including duration, token usage, cost when known, and success/error status. Each span carries rakazo.bot.id, rakazo.run.id, and rakazo.routine.id when the run belongs to a routine. Helper model calls inherit the same correlation. Pi 0.87.1 exposes built-in telemetry through its newer harness; Rakazo’s legacy Agent adapter maps its model and tool lifecycle through the same Pi telemetry contract.

Prompt, response, tool arguments and tool results are excluded by default. OTEL_CAPTURE_CONTENT=true includes them: enable only with a trusted collector and appropriate retention/access controls, since this content can contain secrets or personal data. Unknown attributes and exception messages/stacks stay excluded. Export is batched, best effort, and flushed when the worker shuts down; collector failures do not fail runs.

Product analytics

Product analytics is off by default. To enable the PostHog adapter, set ANALYTICS_PROVIDER=posthog, ANALYTICS_KEY to a project capture token, and ANALYTICS_HOST to the HTTPS ingestion host (default https://us.i.posthog.com) on API and worker. Also set ANALYTICS_PSEUDONYM_SECRET to a server-only random value (openssl rand -hex 32), identical on API and worker. Retain it when rotating the public capture token so user and event identities stay stable. Never send this secret to the browser or collector. Set ANALYTICS_PROVIDER=none to disable it. The public website uses its separate PUBLIC_POSTHOG_KEY configuration.

The backend sends account creation, session creation, fixed screen names, message counts, and run starts/outcomes. Run events include the shared trigger enum; analytics does not read or classify error text. User identifiers are pseudonymized with the server-only secret. No names, emails, prompts, responses, message text, DOM, URLs, search parameters, document contents, request metadata, or raw errors are exported. There is no browser or mobile analytics SDK, autocapture, session replay, or third-party script. Web, Electron, and mobile send only fixed screen/surface enums to the authenticated API; lifecycle events come from the shared backend. This works across clients without giving a collector access to their content. Local and self-hosted installations remain off until their backend is configured.

Delivery is best effort: memory is bounded, requests time out, and shutdown flushes pending events. Analytics failure does not fail product operations. This is not a durable audit trail, and events from before enablement are not backfilled.

Logging

Backend services write structured logs to stdout. LOG_LEVEL is debug, info, warn, error, or off (default info). Production defaults to LOG_FORMAT=json; development defaults to pretty unless you set json or pretty.

Axiom is optional. Set both AXIOM_TOKEN and AXIOM_DATASET for ingest to one shared dataset. Services set service.name (rakazo-api, rakazo-worker, rakazo-sandbox-supervisor, rakazo-updater). A partial Axiom config logs a one-time warning and stays off. AXIOM_EDGE is a regional hostname; AXIOM_EDGE_URL must be https and wins when both are set.

Computer usage is recorded at several boundaries:

• run.computer: one record per execution attempt, including runs that never request a computer (reason=none). reason is tool, takeover, or script otherwise; tool is an allowlisted built-in name and trigger identifies user, routine, or other run sources. Deduplicate run.id when counting runs, because a run can have several attempts.
• computer.requested: the first computer use in an attempt, with its reason, tool, and trigger.
• computer.provisioned: completed setup, with operation=create, resume, or reuse and durationMs. A setup request can reuse an already-running computer without spending a start.
• sandbox.start: Boat’s actual create, resume, and recover requests, including starts during preparation or after the provider expires a computer. outcome=accepted means the provider accepted the request, not that all later setup succeeded. rate_limited and failed are separate outcomes. Count accepted creates/resumes for start usage; recoveries are shown separately because their quota treatment depends on the provider.
• computer.suspended: successful idle or active-limit suspension, with the active duration.

These records use fixed metadata, opaque correlation IDs, durations, and booleans. They do not add prompts, responses, filenames, file contents, command arguments, or credentials. Request/job trace fields link non-run starts back to the API route or background job that initiated them. Axiom is an additional sink for the same logs; these events need no extra keys when ingest is configured.

For example, substitute your own dataset in these APL queries:

['YOUR_DATASET']
| where message == 'sandbox.start'
| summarize requests=count() by provider, operation, outcome

['YOUR_DATASET']
| where message == 'computer.requested'
| summarize requests=count() by trigger, reason, tool

['YOUR_DATASET']
| where message == 'run.computer'
| summarize attempts=count(), withoutComputer=countif(reason == 'none') by trigger

Track accepted starts per hour/day, rate-limit refusals, first-use tools, and the proportion of attempts needing no computer before tuning SANDBOX_IDLE_MS. Increasing the idle window reduces resumes between nearby uses but increases billed runtime; changing it does not remove unnecessary starts caused by tools. Provider rolling-window limits remain the source of truth for quota.

Error reporting to Sentry or a Sentry-compatible collector such as GlitchTip is optional:

• SENTRY_DSN reports backend error logs, with the exception chain, service, and request or job ids as tags. Uncaught exceptions are reported before exit.
• SENTRY_BROWSER_DSN reports web app errors. The browser reads it from /api/monitoring/config at startup and posts reports to /api/monitoring/tunnel; the API forwards only envelopes addressed to that DSN. Use a separate project from SENTRY_DSN.
• SENTRY_ENVIRONMENT names the deployment. SENTRY_RELEASE defaults to the image’s GIT_SHA.

Desktop and mobile use the connected server’s SENTRY_BROWSER_DSN. Electron sends crash metadata without memory dumps. Mobile native reporting requires a rebuilt app; Expo Go is unsupported. For readable mobile crash stacks, build with SENTRY_AUTH_TOKEN, SENTRY_ORG, SENTRY_PROJECT, and optional SENTRY_URL to upload symbols.

Neither SDK sends cookies, headers, request bodies, user identity, console output, or log context. When ANALYTICS_PSEUDONYM_SECRET is set, reports carry a keyed hash of the signed-in user’s id so the collector can count affected users; the raw id, email and IP address are never sent, and the browser receives its hash from the API. Reporting is off without a DSN; an invalid DSN logs a warning and stays off.

Compose passes these into the API, worker, supervisor, and updater. Computer containers and updater child commands do not receive them.

Optional:

SIGNUPS_ENABLED=true
SIGNUP_ALLOWLIST=you@example.com,@company.com
SANDBOX_PROVIDER=docker   # or none, e2b, novita, daytona, createos, boat. Keep fake only for pnpm test.
AGENT_RUNTIME=pi          # Keep scripted only for pnpm test.
WAKEUP_DRIVER=graphile
SANDBOX_IDLE_MS=600000    # pause or stop after 10 minutes idle; 0 disables idle sleep
# Set 0 when self-hosting with Docker, where an idle computer costs nothing, to keep long-running apps and sessions up.
# E2B, CreateOS and Boat still apply their own timeouts.
COMPUTER_MAX_ACTIVE_MS=7200000 # uptime cap for providers that bill while running; minimum 60000; 0 disables
SANDBOX_COMMAND_TIMEOUT_MS=300000 # stop a shell command after 5 minutes
MAX_TOOL_CALLS_PER_TURN=  # optional Pi turn tool-call fuse; unset/0 = unlimited
RAKAZO_MESSAGE_USER_MAX_CHARS=500 # message_user length cap; narration stays at 500
BOT_MESSAGE_MAX_HOPS=6    # bot-to-bot messages one chain may carry before the next is refused
E2B_API_KEY=              # when SANDBOX_PROVIDER=e2b
NOVITA_API_KEY=           # when SANDBOX_PROVIDER=novita; build its template first
DAYTONA_API_KEY=          # when SANDBOX_PROVIDER=daytona
CREATEOS_SANDBOX_API_KEY= # when SANDBOX_PROVIDER=createos
BOAT_API_KEY=             # when SANDBOX_PROVIDER=boat

RAKAZO_LIVE_MODEL_CATALOG=off disables provider and Codex live catalog probing; the default is on.

To use an operator-controlled OpenAI-compatible server such as Ollama, LM Studio, llama.cpp, or MLX, list its model IDs and an endpoint that both the API and worker processes can reach:

RAKAZO_LOCAL_MODELS=qwen3:4b,llama3.1:8b,qwen3-vl
RAKAZO_LOCAL_MODELS_URL=http://127.0.0.1:11434/v1
RAKAZO_LOCAL_CONTEXT_WINDOW=32768
RAKAZO_LOCAL_MAX_TOKENS=4096
# Optional: model ids on this endpoint that accept images (screenshot computer tools).
RAKAZO_LOCAL_VISION_MODELS=qwen3-vl

The loopback default is suitable when running Rakazo from a source checkout. From containers, prefer a stable LAN RFC1918 address (not Compose service DNS alone). On Docker Desktop, host.docker.internal also works. On Docker Desktop, a bot computer shell can often reach services bound to host 127.0.0.1 through that same hostname. Do not run sensitive unauthenticated services on loopback while bots run, or firewall / block that path. Linux does not get host.docker.internal the same way by default. Only configure an endpoint you control: prompts, attachments, and tool results sent to that model leave Rakazo through this URL. Leave RAKAZO_LOCAL_MODELS blank to disable the provider.

Ollama Cloud

In Settings → Models, choose Ollama Cloud, paste an API key from ollama.com, then Find models. This uses an OpenAI-compatible connection with no environment configuration, and stays available when RAKAZO_OPENAI_COMPAT_ALLOW_PUBLIC=0.

Each user can also connect their own OpenAI-compatible endpoint from Connect a model / Settings → Models on web and mobile. Choose OpenAI-compatible, enter the server base URL (for example http://127.0.0.1:8000/v1), the exact model id, and an optional API key. Add optional request headers under Advanced; values are stored encrypted, and only their names are shown afterwards. Public hosts and ordinary hostnames need HTTPS when they receive an API key or custom headers, and must resolve only to public addresses. Set RAKAZO_OPENAI_COMPAT_ALLOW_PUBLIC=0 on the API and worker to block them. Literal private IP, loopback, and host.docker.internal targets do not, but only the deployment owner can use them, checked on save, on model discovery, and on every run. Set MCP_ALLOW_PRIVATE_ENDPOINT=true on the API and worker to allow them for every user. For a self-hosted endpoint whose hostname resolves to private IPs, set RAKAZO_OPENAI_COMPAT_PRIVATE_HOSTS=models.example.test,other.example.test in the deployment environment for both API and worker, then recreate those containers. Entries are exact hostnames (case-insensitive, optional trailing dot), not URLs, suffixes, or wildcards. These hosts ignore RAKAZO_OPENAI_COMPAT_ALLOW_PUBLIC; every DNS answer must be private, and metadata and link-local destinations remain blocked. This applies to discovery and model requests. HTTPS still validates the hostname’s certificate. Like private IP endpoints, trusted hostnames also permit HTTP with API keys; use HTTPS unless the network is trusted. Only allow hosts you control. Private endpoints require MCP_ALLOW_PRIVATE_ENDPOINT=true for the deployment; no app account bypasses that policy. If that endpoint’s model accepts images, enable Supports images under Advanced when connecting so attachments and screenshot computer tools stay available. Existing connections default to disabled. For centrally managed endpoints, the deployment-wide fallback remains RAKAZO_OPENAI_COMPATIBLE_VISION_MODELS=gpt4o-vision,llava.

Client updates

Every web build is named by the UTC time it was built (YYYYMMDDHHMMSS). The web app sends its build on each /rpc request in x-rakazo-client-build (web/20261022093015; the mobile app sends mobile/ and the time its JavaScript bundle was built), and the API answers with the web build in its own image in x-rakazo-build. An open tab that sees a newer build reloads at a safe moment: while hidden, on its next navigation, or after a few idle minutes with nothing typed.

To retire old clients immediately, set RAKAZO_MIN_CLIENT_BUILD on the API, for example web=20261022093015 or web=20261022093015,mobile=20261001000000, and restart it. Older clients get HTTP 426 CLIENT_UPDATE_REQUIRED: the web app reloads, and the mobile app offers its update. Read the current build from the x-rakazo-build header of any /rpc response. Requests that name no build, such as webhooks and API tools, are never refused. A malformed value stops the API from starting rather than being ignored.

Client addresses

Sessions, logs, and security events carry the caller’s address. The API believes a forwarded header only when the TCP peer is in TRUSTED_PROXIES (IPs, CIDR ranges, or hostnames such as a Compose service name). It reads that header (CLIENT_IP_HEADER, default x-forwarded-for) right to left past trusted hops. Anyone else is the TCP peer, so a client cannot choose its own address. Unset, TRUSTED_PROXIES in a container is local-networks: loopback and every network the API has an interface on. Outside a container it is loopback only, since the host’s networks can hold direct clients. none trusts no proxy. In Compose those networks are the stack’s own, which hold the web container and the network gateway a reverse proxy on the Docker host connects from, so a host proxy that sets X-Forwarded-For (Caddy does; nginx needs proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for) is believed without configuration. Bot computers are never on those networks, so they cannot choose an address. Production Compose sets caddy instead. For any other layout, list every proxy between the internet and the API. Sign-in rate limits count per client address, so a proxy that is not believed makes every user share one limit.

Caddy resolves the client first. Caddyfile.prod takes the TCP peer and overwrites X-Forwarded-For with it. Behind a CDN, Caddy has to trust the CDN’s ranges and read its client header: Caddyfile.cloudflare.example does this with Cloudflare’s ranges and CF-Connecting-IP. For another CDN, copy that file and change the ranges and the header. If you copied the Cloudflare example before this change, copy it again.

Content-Security-Policy

The web server sends a strict Content-Security-Policy: scripts and connections are limited to the app’s own origin. When a CDN in front of the app injects a script, such as an analytics beacon, the browser blocks it until you add its origins. RAKAZO_CSP_SCRIPT_SRC extends script-src and RAKAZO_CSP_CONNECT_SRC extends connect-src. Each takes space-separated https:// origins, with an optional port and leading *. wildcard (https://*.example.com). Keywords such as 'unsafe-inline', bare schemes, *, http:// origins and paths are rejected, and the web server refuses to start until the value is fixed. Restart the web service after changing either one.

For example, Cloudflare Web Analytics loads its script from one host and reports to another:

RAKAZO_CSP_SCRIPT_SRC=https://static.cloudflareinsights.com
RAKAZO_CSP_CONNECT_SRC=https://cloudflareinsights.com

Add only origins you trust: a script these settings allow runs with the signed-in user’s access.

Browsers report what the policy blocks to /api/csp-report. The API writes one csp.violation warning per violation, with the directive, the blocked origin and the page path, so search your logs for it to find the origins to add.

Menu links

RAKAZO_NAV_LINKS adds items to the account menu that open a page from the same domain inside the web app, such as an internal tool your reverse proxy serves next to Rakazo. Set it on the API:

RAKAZO_NAV_LINKS='[{"id":"clients","label":"Clients","path":"/ext/clients/"}]'

Each id is lowercase letters, digits and dashes, and becomes /app/x/<id>. path must start with a single / and must not be under /app, /api, /rpc, /novnc or /health. Route it before the catch-all in your proxy, for example handle /ext/* { reverse_proxy office:8080 } in Caddyfile.prod. A malformed value stops the API from starting.

Pages open sandboxed with an opaque origin, so they cannot read Rakazo’s pages or act with the signed-in user’s session; that also blocks their own cookies and storage. Add "trusted": true to a link only for a service you control: a trusted page runs with the signed-in user’s access, like a script allowed by RAKAZO_CSP_SCRIPT_SRC. The page must allow framing by the app (frame-ancestors 'self'). Links appear in the web app only; the desktop and mobile apps hide them.

Security events

The API and worker append a row to security_events and write a security.event log line for:

• sign-up, sign-in, sign-out, and failed sign-in or sign-up
• session revocation, password and passkey changes, and account deletion
• model, MCP, connector, voice, memory, and secret credentials created, changed, or removed
• deployment settings changes, approval decisions, and approval policy changes
• refused private or internal destinations (egress.refused) and auth rate-limit hits

Each event has the user, space, client address, user agent, time, and a few identifiers. Events never include message content, secrets, or tokens. Rows older than SECURITY_EVENT_RETENTION_DAYS (default 90) are deleted, and Postgres rejects updates to the rows that remain.

Operators read recent events through server-side database/log tooling. For example, query Postgres:

docker compose --env-file .env -f infra/compose/docker-compose.prod.yml exec postgres \
  psql -U rakazo -d rakazo -c 'SELECT "createdAt", type, ip, "userId", "userAgent", metadata
  FROM security_events ORDER BY "createdAt" DESC LIMIT 50'

Caddy writes a JSON access-log line for each request to stdout. The lines exclude Authorization, Cookie, and Set-Cookie, and URL tokens are replaced with REDACTED. Docker rotates container logs and drops them when it recreates a container. To keep Caddy’s lines, ship container stdout to your log sink with a collector. Security events reach the sink through the API’s own log shipping, and Postgres keeps them too.

What to alert on

The queries below are pseudo-SQL over the JSON log lines, so translate them to your log sink. Field names are the literal keys in each line: message, security.event.type, client.ip, user_agent.original, user.id, and the security.event.metadata object. Tune the thresholds to your traffic.

Sign-up bursts from one address:

SELECT "client.ip", count(*) FROM logs
WHERE message = 'security.event' AND "security.event.type" = 'auth.sign_up'
GROUP BY "client.ip", bucket(1 hour) HAVING count(*) > 3

Password guessing, from one address and across many:

SELECT "client.ip", count(*) FROM logs
WHERE "security.event.type" IN ('auth.sign_in.failed', 'rate_limit.hit')
GROUP BY "client.ip", bucket(15 minutes) HAVING count(*) > 10;

SELECT count(DISTINCT "client.ip") FROM logs
WHERE "security.event.type" = 'auth.sign_in.failed'
GROUP BY bucket(15 minutes) HAVING count(*) > 50

Probes of internal addresses. Any hit deserves a look:

SELECT "user.id", "security.event.metadata.host", "security.event.metadata.reason", count(*)
FROM logs WHERE "security.event.type" = 'egress.refused'
GROUP BY 1, 2, 3

Scripted clients on account routes:

SELECT "client.ip", "user_agent.original", count(*) FROM logs
WHERE "security.event.type" LIKE 'auth.%'
  AND ("user_agent.original" IS NULL
    OR lower("user_agent.original") SIMILAR TO '%(curl|wget|python|go-http-client|headless|okhttp|java/)%')
GROUP BY 1, 2

Credential and settings changes from an address that user has not signed in from:

SELECT * FROM logs c
WHERE (c."security.event.type" LIKE 'credential.%'
    OR c."security.event.type" = 'deployment.settings.changed')
  AND NOT EXISTS (SELECT 1 FROM logs s WHERE s."security.event.type" = 'auth.sign_in'
    AND s."user.id" = c."user.id" AND s."client.ip" = c."client.ip")

Prompt cache capabilities

The model connection API accepts cacheCapabilities, including scope, retentionMs, minimumTokens, and retentionMode. Declare a positive retentionMs only when the endpoint supports prefix reuse for that window. For these connections with usable history-retrieval tools, the runtime automatically uses cache-aware context: recent history remains bounded on a cold request, and subsequent requests keep that selected prefix while it is likely warm and fits the context budget. Connections with unknown retention or retentionMode: "none" retain the default retrieval policy. An explicit executor context strategy still takes precedence. Callers without usable history-retrieval tools keep fitting original history under the model’s context budget instead of replacing it with snapshots.

Cache warmth is advisory. Changing instructions or tools, context pressure, worker eviction, or a process restart can require fresh prompt processing. These settings do not create backend KV snapshots or keep a model loaded; the inference server is responsible for retention and conversation switching.

Web search and page reading

Builtin web_search and web_fetch use keyless HTTP search (DuckDuckGo HTML) and local Readability extraction by default. To choose Parallel Search MCP, set WEB_PROVIDER=parallel in .env and restart both the API and worker. The same bot tools then return search excerpts and extracted page markdown through HTTPS MCP at https://search.parallel.ai/mcp. Search uses Fast mode without an account or API key.

This option sends search queries and requested page URLs to Parallel, which retrieves pages on its servers. Conversation accounting uses an HMAC of internal IDs keyed by ENCRYPTION_KEY (shared by the API and worker), so deployments with distinct secrets send distinct session IDs. Rotating that secret resets the session IDs; the existing development-only encryption-key fallback also applies here. Anonymous access has service limits; rate-limit and provider errors are returned to the bot without switching providers. Per-bot disabled web tools remain disabled. Set WEB_PROVIDER=keyless-http (or remove the setting) to return to the default.

To choose Keenable instead, set WEB_PROVIDER=keenable and restart both the API and worker. The same bot tools then return search snippets and extracted page markdown through HTTPS MCP at https://api.keenable.ai/mcp, without an account or API key. This option sends search queries and requested page URLs to Keenable, which retrieves pages on its servers. Anonymous access is rate limited per IP address; set KEENABLE_API_KEY on the API and worker to send a key and lift those limits. Errors, per-bot disabled tools and the way back to the default work as described for Parallel.

Self-hosted memory

Supermemory and Honcho can run on your own network. In Settings → Memory, choose Local and enter the instance’s base URL. The operator must set MCP_ALLOW_PRIVATE_ENDPOINT=true; workspace owners can then save a local endpoint. It must be loopback or resolve to private addresses (a Compose service name, an RFC1918 or unique-local address, or a .internal name such as Railway’s private network); Rakazo pins the resolved address for every request. Space members use the saved connection without seeing its key.

For Honcho, enable AUTH_USE_AUTH on the Honcho server and use an admin JWT from its scripts/generate_jwt.py --admin as the key, so each Space gets its own workspace. A workspace-scoped JWT works when you enter that workspace. Honcho needs its own PostgreSQL with pgvector, an LLM provider, and an embedding provider; set EMBEDDING_VECTOR_DIMENSIONS before its first start. Rakazo stores only explicitly saved facts in Honcho; conversation summaries stay in Rakazo. Bots can forget a saved fact on Honcho v3.2 or later.

Remote MCP servers and installed API / GraphQL connectors default to public HTTPS. The deployment owner can attach one on localhost, the same LAN, or a Docker network. Set MCP_ALLOW_PRIVATE_ENDPOINT=true on the API and worker to allow these for every user. Cloud metadata addresses stay blocked. Leave the flag unset on public installs.

Remote MCP servers that use OAuth normally register Rakazo with their provider on the first Connect OAuth. Some providers only accept a client you create in their console. For those, enter the client ID (and secret, for a confidential client) under Advanced when adding the server, and add the Redirect URI shown there (<your web origin>/mcp/oauth/callback) to the client’s allowed redirect URIs. The client is stored encrypted with the server’s other credentials and survives Disconnect; switching to a different client drops the current grant. On first use the client is bound to the authorization server the MCP server pointed to, and Rakazo refuses to send it anywhere else afterwards. Replacing the client clears the binding.

Stdio MCP servers can use OAuth too. They do not authorize themselves (the MCP spec has them read credentials from the environment), so Rakazo runs the flow for them. They are off unless the API and worker have MCP_STDIO_ENABLED=true and the command is listed in MCP_STDIO_ALLOWED_COMMANDS. When adding the server, enter the OAuth client, its Authorization server (the issuer, for example https://accounts.google.com) and the Scopes, then click Connect OAuth as for a remote server. Some providers issue a refresh token only when asked through extra parameters: for Google, set Authorization parameters to access_type=offline&prompt=consent. Pass the grant to the process through its environment variables:

• A value can reference {{oauth.access_token}}, {{oauth.refresh_token}}, {{oauth.client_id}}, {{oauth.client_secret}}, {{oauth.expires_at}} (epoch milliseconds), {{oauth.scope}} or {{oauth.token_type}}, alone or inside other text.
• A variable can be a File instead: its template (often JSON, with the same references) is written to a private temporary directory when the process starts, and the variable receives the file’s path. Use this for packages that read a credentials or token file path.

Rakazo refreshes the grant and restarts the process shortly before an access token it passed expires, so a long-running process never keeps a dead token. A process that only receives the refresh token refreshes on its own. Temporary files are removed when the process stops.

A server launched as npx [-y] <package>[@version] (or npx -p <package> <bin>) is not downloaded on every start. Rakazo installs the package once with npm install into MCP_PACKAGES_DIR (default DATA_DIR/mcp-packages, shared by the API and worker), then runs its bin directly; npx must still be in MCP_STDIO_ALLOWED_COMMANDS. The install has its own timeout, MCP_STDIO_INSTALL_TIMEOUT_MS (default 180000), separate from the handshake timeout MCP_STDIO_STARTUP_TIMEOUT_MS (default 30000). Pin a version so every start runs the same code; npm settings such as NPM_CONFIG_REGISTRY can be set in the server’s environment variables. Local paths, git and tarball URLs, and unknown npx flags fall back to running the command as configured. Each server runs as one process per user in each of the API and worker, shared by that user’s bots, and stops after MCP_STDIO_IDLE_MS (default 900000, 15 minutes) without use. When a server fails to start, the reason (including the end of its stderr, with credentials redacted) appears on the server in Settings, and its bots are told it is unavailable instead of silently losing its tools.

A run never waits long on one server. It gives each server MCP_DISCOVERY_TIMEOUT_MS (default 20000) to list its tools, then goes on without it; a package still installing finishes in the background and its tools appear on a later turn. After a failure, runs skip that server for 30 seconds, doubling up to 5 minutes while it keeps failing. A successful Test connection or discovery, or a change to the server’s settings, ends the wait at once.

For servers that accept standard reasoning_effort, enable Supports thinking under Advanced when connecting. The setting is saved on the connection (no env var or restart). Existing connections default to disabled. Reconnect former Qwen-list or deployment-local models via Settings → Models and turn it on; the old environment list is no longer read.

Enabled connections default to medium thinking. Web and desktop expose Thinking in a bot’s advanced settings; mobile inherits the same backend policy. Rakazo sends standard reasoning_effort (minimal, low, medium, high, or none when off); the server owns model-specific translation. Leave Supports thinking off when the server lacks standard effort support. Existing token limits still apply; effort is not a separate reasoning-token budget.

Do not commit .env. Never put COMPOSIO_API_KEY, OpenRouter keys, or provider tokens in git, logs, or chat.

Optional messaging platforms (iMessage, Slack, WhatsApp, Telegram, Feishu/Lark) mount when their env credentials are set — see .env.example. The Lark adapter and official SDK load only when configured. Point a Feishu/Lark bot event subscription at /api/v1/messaging/webhook/lark (webhook/HTTP inbound only; do not enable long connection). Personal group linking stays iMessage-only; Slack channels use workspace installations below. Telegram can run several bots side by side (TELEGRAM_BOT_TOKEN_<NAME> plus TELEGRAM_WEBHOOK_SECRET_TOKEN_<NAME>), for example one per space, so the same person can chat with a different Rakazo bot through each.

For Slack workspace installations, set SLACK_CLIENT_ID, SLACK_CLIENT_SECRET, and SLACK_SIGNING_SECRET on the API and worker. Register the OAuth redirect URL https://app.example.com/api/v1/messaging/install/slack/callback using your public API origin (API_URL). Enable distribution in the Slack app’s Manage Distribution settings so other workspaces can install it. Enable Slack Events API delivery to https://app.example.com/api/v1/messaging/webhook/slack, with app_mention, message.im, message.channels, message.groups, message.mpim, and app_uninstalled. Enable the app’s Messages tab. The install flow requests the message/history, channel/user lookup, file read and write, and reaction-write scopes it uses; reinstall the Slack app after changing scopes. Socket Mode is not used.

In Messaging settings, select Connect on the Slack row. The workspace answers as the first bot in your default space, whichever space is open. Each Slack workspace belongs to one Rakazo space and one installation owner; installing it from another space is rejected. To DM that bot, use Link DMs and send the code in a DM to the Slack app. Teammates in the same space can link their own bots the same way. Invite the Slack app to channels and mention it to receive replies there. The bot’s external conversations offer Mentions only, Listen, and room guidance. Workspace setup and personal linking are available on web, desktop and mobile. Mobile opens Slack authorization in the system browser; return to the app when it finishes. Channel listening settings currently live in the web/desktop interface.

Tokens are encrypted with the configured secret store. Workers resolve the installation for every outbound operation, including queued replies; disconnecting an installation stops further delivery and unlinks its personal DMs. Slack’s signed app_uninstalled event also removes the installation. Disconnect removes Rakazo’s local credentials; remove the app in Slack to revoke Slack’s installation itself. The installation owner must retain space membership and an active bot. Workspace installations require a workspace-level bot token without Slack token rotation; Enterprise Grid org-wide installs and rotating tokens are rejected explicitly.

The older SLACK_BOT_TOKEN plus SLACK_SIGNING_SECRET configuration still supports a single workspace. OAuth mode takes precedence when configured; existing static Slack identities/conversations must be relinked before using workspace installs. TEAM_CHAT_BOT_ID only selects a channel bot for the static configuration. It does not override workspace installations or personal DMs. Slack tools authorized from the Integrations screen are a separate connection from this messaging surface.

Settings with names you choose

The Compose files pass the API and worker only the settings each one reads, listed by name, rather than the whole .env. A setting whose name you choose, such as TELEGRAM_BOT_TOKEN_<NAME>, therefore needs its own line: add it under api and worker environment in an override file layered after the base file (and in RAKAZO_COMPOSE_FILE when the updater is enabled), with no value so Compose fills it from .env. install-images.sh names any .env setting that no container receives.

# docker-compose.override.yml
services:
  api:
    environment:
      TELEGRAM_BOT_TOKEN_SUPPORT:
      TELEGRAM_WEBHOOK_SECRET_TOKEN_SUPPORT:
  worker:
    environment:
      TELEGRAM_BOT_TOKEN_SUPPORT:
      TELEGRAM_WEBHOOK_SECRET_TOKEN_SUPPORT:

install-images.sh layers a docker-compose.override.yml next to it automatically. In your own commands, add -f docker-compose.override.yml after the base file.

Choosing a computer provider

The Electron desktop app is a client of the same API. Docker remains the default. For an app-managed local instance, the local operator can choose Use this Mac (macOS) or Use this computer (Windows/Linux). After native confirmation, an authenticated bridge sends commands from the Docker backend to Electron, where they run under your OS account. Computer Access… in the app menu changes the choice; Revoke Local Computer Access immediately disconnects active operations and revokes the saved permission. Local execution requires the desktop app to remain running. Connecting to an existing remote instance does not grant it access to your computer. Browser and mobile clients use the deployment’s configured computer; they do not expose their own host. SANDBOX_PROVIDER=desktop is a separate, development-only configuration for backends running directly on the intended host.

Native file reads and workspace exports are limited to 8 MiB per file. A larger workspace file fails the checkpoint and preserves the previous saved workspace; local files remain on the computer. Writes are bounded by the 12 MiB encoded request limit.

• Published images (docker-compose.images.yml) default to SANDBOX_PROVIDER=docker with a local supervisor and published ghcr.io/elie222/rakazo/computer image. No E2B account required. Optional: set e2b, novita, daytona, createos, or boat plus the matching API key for remote computers.
• Docker is the quick-start default for published images and for a source checkout / full local Compose stack. Workspace bots share a persistent Team Computer by default; Private computers are optional. Keep the supervisor private, as the included Compose files do.
• E2B runs bot computers away from the Rakazo host and is a good choice for public or multi-user production deployments. Rakazo checkpoints the portable workspace and browser-profile directory to DATA_DIR; the E2B disk is a runtime cache, not the durable source of truth.
• Daytona provides the same remote-computer contract through Daytona sandboxes. Configure DAYTONA_API_KEY and optionally DAYTONA_API_URL / DAYTONA_TARGET / DAYTONA_SNAPSHOT.
• CreateOS provides the same remote-computer contract through CreateOS desktop sandboxes. Configure CREATEOS_SANDBOX_API_KEY and optionally CREATEOS_SANDBOX_BASE_URL, CREATEOS_SANDBOX_SHAPE, or CREATEOS_SANDBOX_ROOTFS. Rakazo defaults to https://api.sb.createos.sh, s-2vcpu-2gb, and desktop:1.
• Boat provides a managed Linux desktop through BOAT_API_KEY and optionally BOAT_API_URL. Rakazo always creates or resumes sandboxes with noEnv: true, keeps the portable workspace under /home/user/rakazo-home, and refreshes a two-hour TTL. Boat uses the shared Linux desktop runtime and protected port routes for concurrent bot desktops. Each bot has its own persistent Chrome profile; logins are not shared between bots.
• Local computer access in the app-managed Electron instance runs commands on Windows, macOS, or Linux through the native bridge. Bots can read/change local files and launch installed apps through OS commands, subject to your account and OS permissions. Portable file tools and checkpoints remain scoped to bot workspaces; use commands for other local folders. Graphical screen control is unavailable. Permission is remembered locally and can be revoked from the app menu. Only enable this for a trusted personal instance: every bot on that instance can use your account’s permissions.
• Desktop provider (SANDBOX_PROVIDER=desktop) runs commands directly on the API/worker host, with only a minimal shell environment (no server secrets). The API and worker refuse it unless NODE_ENV is development or test, so a production server never runs bot commands on itself. Setting this inside a container does not grant access to the physical computer. Docker deployments use the authenticated native bridge when local computer access is selected; they never silently fall back to executing on the backend host.
• Fake is only an emulator for verification.
• None boots the product without a computer host (fallback when Docker/supervisor is not configured, or when a remote provider is selected without its API key).

For provider configuration and health checks, see the provider setup guide.

Content encryption

Rakazo encrypts chat messages, thread events, compacted history summaries, memory documents and their revisions, and uploaded or generated files before they reach PostgreSQL or DATA_DIR. Names, ids, timestamps and other metadata stay readable. Credentials are always encrypted, whatever this setting says. Details and the threat model are in content encryption.

The key. Content is encrypted under keys derived from ENCRYPTION_KEY (and ENCRYPTION_MASTER_KEYS, if set). Without them, a backup of the database and DATA_DIR cannot be read, by you or by anyone else. install-images.sh says so once, when it generates the key:

Back up ENCRYPTION_KEY from /path/to/rakazo/.env. Encrypted content cannot be recovered without it.

It never prints the key. Store a copy somewhere other than the server and its backups.

The switch. CONTENT_ENCRYPTION=on|off in .env:

With off, new content and files are stored readable, so you can query your own database directly and a lost key cannot lose them. Content encrypted earlier still reads with the key, so switching off never makes data unreadable. To convert it too, run content-encryption decrypt. Any other value stops the API and worker at startup.

To switch on later, set CONTENT_ENCRYPTION=on, restart, then encrypt what was stored while it was off. From the published-images folder (other stacks use their own Compose file):

docker compose -f docker-compose.images.yml exec api pnpm --filter @rakazo/api content-encryption migrate
docker compose -f docker-compose.images.yml exec api pnpm --filter @rakazo/api content-encryption status

Search works in both modes and while rows are mixed.

Reading your data. content-encryption export writes decrypted threads, messages, memory documents and revisions as JSON lines. It reads the whole database by default; narrow it with --space <id>, --thread <id> or --bot <id>. It only reads.

docker compose -f docker-compose.images.yml exec -T api \
  pnpm -s --filter @rakazo/api content-encryption export --bot <bot id> > rakazo-export.jsonl

The export is plaintext: store it like the database itself. Files are not included; use content-encryption decrypt to make them readable on disk.

Bot homes in object storage

Each bot computer has a home: its files, logged-in browser profiles and CLI tokens. By default homes live under DATA_DIR/homes on the server. Set OBJECT_STORAGE_BUCKET to keep them in any S3-compatible bucket instead, so DATA_DIR is only a cache and API and worker hosts no longer need a shared disk for homes. Leave it unset to keep homes on local disk, as before. The same connection also stores new artifacts; see shared storage migration for migrating existing files and push registrations.

OBJECT_STORAGE_BUCKET=rakazo-homes
OBJECT_STORAGE_ENDPOINT=https://storage.example.com
OBJECT_STORAGE_REGION=us-east-1
OBJECT_STORAGE_ACCESS_KEY_ID=replace-me
OBJECT_STORAGE_SECRET_ACCESS_KEY=replace-me
# Optional: a key prefix, when the bucket holds anything else.
OBJECT_STORAGE_PREFIX=
# Optional: true for services that address buckets in the path rather than the host name.
OBJECT_STORAGE_PATH_STYLE=false

Only the API and worker read these settings. Bot computers never receive the credentials or links to stored objects; files still reach them through the server, as before. The settings apply to every provider, including Docker computers, which keep mounting their cached home.

Encryption. Homes are encrypted on the server before upload, whatever CONTENT_ENCRYPTION says; the bucket only ever holds ciphertext. Each home has its own random data key, kept on the computer’s database row and wrapped by its space’s content key (see content encryption). Each stored revision is one object, sealed in authenticated segments bound to that home and revision, so a modified, truncated or swapped object fails to restore instead of restoring the wrong files. Object keys are random ids (homes/<id>/<revision>) that name no bot, user or space. Back up the database and the encryption keys as before: the bucket cannot be read without them.

The bucket. Use a private bucket, or a prefix of one, for this deployment alone: the worker deletes objects under homes/ that no home in this database owns. Block public access. The credentials need to read, write, list and delete objects in it, and nothing else. Turn on bucket versioning, with a lifecycle rule that expires noncurrent versions after your backup window, so a mistaken deletion can be undone. scripts/backup.sh does not copy the bucket; versioning, or the service’s own replication, is its backup.

What is stored. Each save uploads the whole home as one object (in parts of up to 16 MiB for large homes) and then deletes the previous one, so a home costs about one object however many files it has. Homes are saved when a run finishes or fails, before a computer stops, and before it sleeps, as before.

Deleting. Deleting a bot with its own computer, or a space (including the personal spaces removed with an account), deletes each home’s key with its computer’s row, so the stored objects become unreadable at once. The worker deletes those objects within the hour, along with any revision an interrupted save left behind.

The cache. DATA_DIR/homes holds a copy of homes in use. A copy whose revision matches the bucket is downloaded again whenever it is missing, so you can remove it. Docker computers work directly in their copy until their next save, so only remove the copy of a Docker computer that is stopped. Remote computers save straight from their files to the bucket; the worker removes their copies after an hour unread. Homes never saved to the bucket keep their only copy.

Saving large homes. A save streams files through DATA_DIR/staging on the data volume, so memory use does not grow with file or home size. A computer with more than 20 GiB or a million files, or more than the volume has free, is not saved and keeps running; the worker logs computer.sleep_deferred with reason too_large or too_many_files. Staging left by a killed process is removed at startup and by the worker within minutes.

Moving existing homes

After setting the bucket, restart the API and worker. Each home is uploaded the next time it is saved; until then its local copy is used. To upload the rest at once, run this where DATA_DIR is mounted. It skips homes already stored, so it can be stopped and run again. Each upload is downloaded and checked before the home counts as moved. It never deletes local files.

docker compose -f docker-compose.images.yml exec api pnpm --filter @rakazo/api home-storage migrate --dry-run
docker compose -f docker-compose.images.yml exec api pnpm --filter @rakazo/api home-storage migrate

Homes listed as having no computer belong to bots that no longer exist; they are left alone. Once a run reports no failures and nothing left to upload, remove the local copies: stop Docker computers, stop the API and worker, move DATA_DIR/homes and DATA_DIR/home-revisions aside, start them again and check a few bots’ files, then delete the moved directories.

Backup

./scripts/backup.sh

This dumps Postgres (pg_dump) and archives data/ into backups/<stamp>/. Backup directories have mode 0700, and both artifacts have mode 0600, including on reused paths. A missing data/ or RAKAZO_BACKUP_SKIP_HOMES=1 produces an empty archive that restore treats as omitted homes. Database or archive errors fail the backup. Discard the output directory of any failed run.

Moving an account between installs

To move from Rakazo Cloud to your own install (or back), use Settings → Export data on the old install and Import data on the new one. The export is a zip of every space you belong to: bots, chats, files, memory, routines, skills, bot workspace files and, in spaces you own, the board. Content is decrypted inside the zip, so keep it private. Importing creates new spaces you own. Routines arrive switched off. Credentials and connections (model keys, apps, MCP servers, messaging) stay behind, so connect them again on the new install. Workspace files over 8 MiB are left out, the export must stay under 4 GiB, and the upload must finish within the server’s request timeout (5 minutes by default in Node).

Public single-VM deployment

infra/compose/docker-compose.prod.yml runs the hosted product with Postgres, the API, worker, web app, and automatic HTTPS through Caddy. It uses E2B for bot computers, so the VM never exposes a Docker supervisor or browser containers — infra/compose/docker-compose.prod.docker.yml is the opt-in overlay that runs local Docker computers on the same stack (see below). The root-equivalent updater sidecar is an explicit opt-in profile.

Before deploying to a new Ubuntu host, create and verify a key-only deploy account, then apply the idempotent host-hardening baseline. It disables SSH passwords and root login, rate-limits SSH, allows only SSH/HTTP/HTTPS through UFW, enables fail2ban, unattended security updates, AppArmor, audit rules, and conservative kernel/network protections. Keep the provider console open until a fresh SSH login succeeds after the script reloads SSH.

sudo --preserve-env=SSH_CONNECTION DEPLOY_USER=deploy bash infra/compose/harden-host.sh

The production host also uses infra/compose/docker-daemon.json to enable live restore, bounded local container logs, default no-new-privileges, and the kernel NAT path instead of Docker’s userland proxy.

1. Point an A/AAAA record such as app.example.com at the VM and allow inbound TCP 80/443 and UDP 443. If you use Cloudflare, enable the proxy with Full (strict) TLS and copy Caddyfile.cloudflare.example to an operator-controlled path outside the public checkout. Set CADDYFILE_PATH to that absolute path. The example drops application requests that do not come from Cloudflare’s published IP ranges; reconcile those ranges whenever Cloudflare publishes a change. A Cloudflare Tunnel can replace the public web listeners.
2. Clone the repository on the VM and create a root .env with production-only values. At minimum set POSTGRES_PASSWORD, BETTER_AUTH_SECRET, ENCRYPTION_KEY, SCREEN_PROXY_SECRET, OPENROUTER_API_KEY, the API key for your selected sandbox provider, RAKAZO_HOST, and the three public origins. Set RAKAZO_DEPLOY_DIR when the checkout is not at the supported Linux default, /srv/rakazo. Use URL-safe random values for database credentials. If you enable the updater profile, also set a dedicated RAKAZO_UPDATER_TOKEN (at least 32 characters) that differs from BETTER_AUTH_SECRET, SANDBOX_SUPERVISOR_TOKEN, and SCREEN_PROXY_SECRET.
3. Keep registration allowlisted while the service is private:

NODE_ENV=production
RAKAZO_HOST=app.example.com
# Optional operator-owned override, for example the Cloudflare allowlist file:
# CADDYFILE_PATH=/etc/rakazo/Caddyfile.prod
BETTER_AUTH_URL=https://app.example.com
WEB_ORIGIN=https://app.example.com
API_URL=https://app.example.com
SIGNUPS_ENABLED=true
SIGNUP_ALLOWLIST=owner@example.com,reviewer@example.com
# e2b, novita, daytona, or boat
SANDBOX_PROVIDER=e2b
AGENT_RUNTIME=pi
WAKEUP_DRIVER=graphile
DATA_DIR=/data
# Absolute path of this checkout as the Docker daemon sees it. /srv/rakazo is the Linux default;
# set this explicitly for every other layout. See "The deploy directory must be one path" below.
RAKAZO_DEPLOY_DIR=/srv/rakazo
RAKAZO_IMAGE_TAG=local
# Optional: required only with `--profile updater`.
# RAKAZO_UPDATER_TOKEN=replace-with-32-plus-character-updater-token

4. Build the images from your checkout and start the stack, then verify its public health endpoint:

docker compose --env-file .env -f infra/compose/docker-compose.prod.yml \
  build --build-arg GIT_SHA=$(git rev-parse HEAD)
docker compose --env-file .env -f infra/compose/docker-compose.prod.yml \
  up -d --wait --pull never
curl --fail https://app.example.com/health

Build, do not pull, for a first deployment. RAKAZO_IMAGE_TAG ships as local, a tag no registry serves, so the commands above build api, worker, and web from the checkout you just cloned. The opt-in command under Updater sidecar builds updater when needed.

The public /health only reports liveness. Runtime, sandbox, and revision details stay on the API port at /internal/health, which the edge does not route:

docker compose --env-file .env -f infra/compose/docker-compose.prod.yml exec api \
  node -e "fetch('http://127.0.0.1:3100/internal/health').then(r=>r.text()).then(console.log)"

Passing GIT_SHA is what makes /internal/health report a "revision"; a locally built image has no other way to know its commit. Prebuilt images from the registry bake it in at publish time, so switching to a release tag needs nothing more: the containers report the revision the image was built from, not a GIT_SHA in .env.

Once a release has been published you can switch this host to prebuilt images by setting RAKAZO_IMAGE_TAG to that release tag and running pull followed by up -d --wait --pull never. See Published images and tags for the tag contract.

The root .env is excluded from both Git and the Docker build context. The database, application data, and Caddy certificates live in named Docker volumes. The webassets volume keeps the hashed web assets of recent builds (up to 20, each for a week after it is replaced), so open tabs keep loading after an update; it needs no setup and can be deleted at any time.

The production Compose file pins Postgres and Caddy to multi-architecture manifest digests, and the published application/updater builds pin their base-image digests. Refresh those pins deliberately when taking upstream security updates; changing only the visible major tag does not change the content while a digest is present.

For the single-VM production layout, install infra/compose/backup-prod.sh as /usr/local/sbin/rakazo-backup and enable the supplied rakazo-backup.timer. It creates a verified Postgres custom-format dump plus an application-data archive under /var/backups/rakazo, with mode 0600. It keeps up to RAKAZO_BACKUP_KEEP verified snapshots (default 3), bounded by RAKAZO_BACKUP_MAX_GB (default 50 GiB), and reserves RAKAZO_BACKUP_MIN_FREE_GB (default 5 GiB) of free disk space. Before starting, it removes abandoned partial snapshots and excess history, then checks capacity by scaling only the previous appdata archive by appdata source growth, plus the previous database dump size. Snapshots without source-size metadata use the raw source size for their initial estimate. It preserves the newest verified snapshot when capacity is insufficient, caps output file sizes, and removes a failed run’s partial snapshot. Local snapshots alone do not survive losing the host; encrypt them and copy them off-site as described below.

Encrypted and off-site backups

Backups are encrypted with age to public recipients only, so the host can write backups it cannot read. Generate the key pair on a machine you control, not on the server, and keep the identity file (the private key) offline, ideally with a second recipient held by another operator:

age-keygen -o rakazo-backup.key   # prints "Public key: age1..."

harden-host.sh installs age; on other hosts run apt-get install age. Then set the public keys in /etc/rakazo/backup.env, separated by spaces, or point at a recipients file with one key per line:

RAKAZO_BACKUP_AGE_RECIPIENTS=age1exampleprimaryrecipient age1examplerecoveryrecipient
# or: RAKAZO_BACKUP_AGE_RECIPIENTS_FILE=/etc/rakazo/backup-recipients.txt

With recipients set, each snapshot holds rakazo.dump.age, appdata.tgz.age, SOURCE_SIZE_KB, and SHA256SUMS. pg_dump and tar stream straight into age, so plaintext never touches disk. Verification needs no private key: pg_restore --list and tar -t read the same plaintext stream while it is encrypted, each file must carry an age header, and the checksums cover the encrypted files.

Without recipients the backup stays plaintext as before, logs a warning, and the host check alerts that the latest backup is not encrypted. After the first encrypted backup, a run without recipients fails instead of quietly writing plaintext. Set RAKAZO_BACKUP_ALLOW_PLAINTEXT=1 only to accept plaintext on purpose; it also silences the host-check alert.

For an off-site copy, install rclone and set RAKAZO_BACKUP_OFFSITE_REMOTE to any rclone remote and path, such as an S3-compatible bucket or an SFTP server. Configure the remote with rclone’s own settings in the same file, for example:

RAKAZO_BACKUP_OFFSITE_REMOTE=offsite:rakazo-backups
RAKAZO_BACKUP_OFFSITE_KEEP=14
RCLONE_CONFIG_OFFSITE_TYPE=s3
RCLONE_CONFIG_OFFSITE_PROVIDER=Other
RCLONE_CONFIG_OFFSITE_ENDPOINT=https://s3.example.com
RCLONE_CONFIG_OFFSITE_ACCESS_KEY_ID=replace-me
RCLONE_CONFIG_OFFSITE_SECRET_ACCESS_KEY=replace-me

After each verified local snapshot, the encrypted files are uploaded to <remote>/<stamp>/, checked with rclone check, and only then given SHA256SUMS, which marks the copy complete. Off-site copies require encryption. The remote keeps the newest RAKAZO_BACKUP_OFFSITE_KEEP complete snapshots (default 14), independent of RAKAZO_BACKUP_KEEP; only stamp-named directories are removed. Set it to 0 to leave deletion to the bucket’s lifecycle rules, so the credentials need list, read and write but no delete permission. A failed upload keeps the local snapshot but fails the service, so the backup alert fires.

Restoring an encrypted backup

Run these where the identity file lives, never leaving it on the server. Fetch a snapshot with rclone copy offsite:rakazo-backups/<stamp> ./<stamp> or scp, then run the drill, which checks the checksums and decrypts each file into pg_restore --list and tar -t without writing plaintext (it needs age, pg_restore of the server’s major version or newer, which is 18 for the production Compose file, and tar):

bash infra/compose/backup-drill.sh ./<stamp> rakazo-backup.key

Never extract appdata over the active volume. Files created after the snapshot would survive and mix with the restored database. Extract into a clean sibling directory, then preserve the old tree before replacing it. Pause automatic deploys and the updater during recovery. Stop any other processes that write to this database or volume. Keep the application stopped if any step fails.

Run these commands in one Bash session on the machine with the identity file. Replace the snapshot and volume names first. VOLUME must name the volume the API currently mounts at /data, which can differ when the stack uses another project name.

set -euo pipefail
umask 077
COMPOSE="sudo docker compose --env-file .env -f infra/compose/docker-compose.prod.yml"
VOLUME=rakazo-prod_appdata

ssh deploy@app.example.com "cd /srv/rakazo && $COMPOSE stop api worker web"
ROLLBACK_DUMP="$(mktemp "$PWD/before-restore.dump.XXXXXX")"
if ! ssh deploy@app.example.com \
  "cd /srv/rakazo && $COMPOSE exec -T postgres sh -c 'pg_dump -Fc -U \"\$POSTGRES_USER\" \"\$POSTGRES_DB\"'" \
  > "$ROLLBACK_DUMP"; then
  printf 'Pre-restore dump failed. Do not use the partial file at %s. Keep the application stopped.\n' "$ROLLBACK_DUMP" >&2
  exit 1
fi

APPDATA="$(ssh deploy@app.example.com "sudo docker volume inspect -f '{{.Mountpoint}}' '$VOLUME'")"
RESTORE="$(ssh deploy@app.example.com "sudo mktemp -d '${APPDATA%/*}/rakazo-restore.XXXXXX'")"
ssh deploy@app.example.com "sudo mkdir '$RESTORE/staged'"
printf 'Record this rollback pair:\nDatabase dump: %s\nData tree on deploy@app.example.com: %s/previous\n' \
  "$ROLLBACK_DUMP" "$RESTORE"
age -d -i rakazo-backup.key <stamp>/appdata.tgz.age | ssh deploy@app.example.com \
  "sudo tar -xzf - --numeric-owner -C '$RESTORE/staged'"

age -d -i rakazo-backup.key <stamp>/rakazo.dump.age | ssh deploy@app.example.com \
  "cd /srv/rakazo && $COMPOSE exec -T postgres sh -c 'pg_restore --clean --if-exists --no-owner --exit-on-error --single-transaction -U \"\$POSTGRES_USER\" -d \"\$POSTGRES_DB\"'"

ssh deploy@app.example.com \
  "sudo mv -- '$APPDATA' '$RESTORE/previous' && sudo mv -- '$RESTORE/staged' '$APPDATA'"
ssh deploy@app.example.com \
  "cd /srv/rakazo && $COMPOSE up -d --force-recreate --wait --pull never api worker web"

The archive must extract successfully before the SQL import starts. The clean staging directory and the rollback tree sit beside the volume’s active data directory, on the same filesystem. Recreating the application containers makes them mount the replacement directory. --single-transaction rolls back SQL errors, but it cannot include the directory switch. After the import commits, a failed switch or startup can leave the database and active tree out of sync. If startup fails, stop api worker web again, including containers that did start.

$ROLLBACK_DUMP names a private, plaintext rollback dump. mktemp creates a new mode-0600 file for each attempt without overwriting before-restore.dump, earlier rollback dumps, or existing symlink targets. If the dump fails, the commands stop with the application still stopped. The printed partial file is not usable rollback data. Preserve earlier rollback pairs and fix the dump failure before starting another recovery attempt.

Record the printed database dump path together with its matching remote $RESTORE/previous tree path. Keep that pair until you have checked the restored application. Do not remove the Docker volume while you need the rollback tree. For rollback after SQL commits, keep the application stopped and restore the recorded $ROLLBACK_DUMP into a freshly created empty database. Move the failed active tree to a separate path for diagnosis, then move that attempt’s recorded $RESTORE/previous tree back to $APPDATA and recreate the application containers. If the first directory move failed, the old tree is still at $APPDATA. Inspect both paths before moving anything. Switching the tree alone does not roll back the database.

Plaintext snapshots use cat in place of age -d and the filenames rakazo.dump and appdata.tgz. Use the same clean-directory procedure.

To restore into a new stack, create the rakazo_app role before the procedure above. The dump grants to it, and on a new stack only the migration step would create it, so pg_restore stops without it. Do not run the migrations first: tables newer than the snapshot would block the restore. The procedure’s final start applies them.

ssh deploy@app.example.com "cd /srv/rakazo && $COMPOSE up -d --wait postgres"
ssh deploy@app.example.com "cd /srv/rakazo && $COMPOSE exec -T postgres sh -c 'psql -U \"\$POSTGRES_USER\" -d \"\$POSTGRES_DB\" -c \"CREATE ROLE rakazo_app NOLOGIN NOSUPERUSER NOBYPASSRLS NOCREATEDB NOCREATEROLE\"'"

The scheduled backup uses /srv/rakazo by default. For another deployment directory, set RAKAZO_DEPLOY_DIR=/absolute/path/to/checkout in a root-owned /etc/rakazo/backup.env (mode 0600). The service reads this optional file on each run; the script uses the selected checkout’s .env and production Compose file. If the stack was started with a custom -p, set the same COMPOSE_PROJECT_NAME in that file. Set the retention and space settings there too. When updating an existing installation, reinstall the script and units, then run systemctl daemon-reload; pulling the repository does not update /usr/local/sbin copies.

For a dedicated backup disk, mount it persistently and bind its backups directory onto /var/backups in /etc/fstab. Add x-systemd.requires-mounts-for=/mnt/your-backup-volume to the bind mount options. Set RAKAZO_BACKUP_REQUIRE_VOLUME=1 and add a service drop-in containing RequiresMountsFor=/var/backups and AssertPathIsMountPoint=/var/backups under [Unit]. The script then refuses to fall back to root storage if the mount is missing or points at the root filesystem.

Monitoring is opt-in. Install host-alert.py as /usr/local/sbin/rakazo-host-alert and check-host.sh as /usr/local/sbin/rakazo-host-check, and check-services.py as /usr/local/sbin/rakazo-check-services (all executable), then install the supplied host-check service and timer and enable rakazo-host-check.timer. Checks run every five minutes and report low disk space, failed backups, a verified backup older than 36 hours, or missing configured IPv4 SSH rules. To also check API and worker containers, set RAKAZO_MONITOR_COMPOSE_PROJECT in backup.env to the Compose project name (rakazo-prod for production). The worker probe checks heartbeat and reconciliation freshness. Leave the variable unset if container monitoring is managed elsewhere.

The dedicated-volume check is enabled only by RAKAZO_BACKUP_REQUIRE_VOLUME=1 in backup.env. Identical alerts are suppressed for a day; recovery sends a separate notification.

Alerts go to the system journal by default; no external service is required. For optional Slack delivery, create a root-only /etc/rakazo/alerts.json containing {"slack_webhook":"https://hooks.slack.com/services/REPLACE/ME/SECRET"}. Never commit this file. To notify immediately on backup failure, also install rakazo-backup-alert.service and copy infra/systemd/rakazo-backup-alerts.conf.example to /etc/systemd/system/rakazo-backup.service.d/alerts.conf. Run systemctl daemon-reload after installing units or drop-ins. The base backup service does not require the monitoring tools.

Host hardening preserves existing firewall rules and installs firewall-change.sh as /usr/local/sbin/rakazo-firewall-change. For subsequent firewall edits, use sudo --preserve-env=SSH_CONNECTION /usr/local/sbin/rakazo-firewall-change apply COMMAND.... This requires SSH_CONNECTION to be preserved, saves both /etc/ufw and /etc/default/ufw, and arms a three-minute rollback before running the command. Open a new SSH connection and run the printed confirm TOKEN command; confirmation from the original connection is rejected. An unconfirmed change restores the saved rules and default policies. The guard does not protect arbitrary direct ufw commands or SSH daemon configuration changes.

To deploy from CI, install infra/compose/deploy-launcher.sh as /usr/local/sbin/rakazo-deploy-main and give CI a key restricted to it in the deploy user’s authorized_keys (restrict,command="/usr/local/sbin/rakazo-deploy-main" ssh-ed25519 …). CI deploys a main commit only after its checks pass and ci.yml has published its images, so the host pulls <RAKAZO_DEPLOY_IMAGE>:sha-<commit> instead of building next to live traffic. RAKAZO_DEPLOY_IMAGE defaults to ghcr.io/<owner>/<repo>/app for the GitHub repository the checkout tracks, so a fork pulls what its own CI published. If the pull fails or takes longer than RAKAZO_DEPLOY_PULL_TIMEOUT (default 5m), the deploy logs a warning and builds the checkout on the host instead; set RAKAZO_DEPLOY_IMAGE= (empty) to always build. This repository’s images are public, so the host needs no registry login; for a private fork package, docker login ghcr.io as root with a token that can only read packages. The deploy starts each new API, web and worker container beside the running one and stops the old one only once the new one is healthy, so a deploy does not take the site offline or pause chat, then waits for https://$RAKAZO_HOST/health and rolls back on failure by restoring the image the previous containers ran. Old and new API containers serve together for a few seconds after migrations run, so a migration must keep working with the previous release: add first, and drop or rename only in a later deploy. Pull, build and start are time-limited so a stuck step fails the deploy instead of holding its lock; a deploy that finds the lock held exits non-zero. A deploy refuses to start with less than RAKAZO_DEPLOY_MIN_FREE_GB (default 5) free, and after a healthy rollout it keeps the current and previous app images (the previous ones tagged rakazo-rollback), removes untagged images, and trims the build cache that fallback builds leave to RAKAZO_DEPLOY_BUILD_CACHE_GB (8 GB by default). For a checkout outside /srv/rakazo, put RAKAZO_DEPLOY_DIR=/absolute/path (and any of these settings) in a root-owned /etc/rakazo/deploy.env readable by the deploy user. The launcher runs infra/compose/deploy-main.sh from origin/main, so deploy changes apply on the next deploy; only a change to the launcher itself needs sudo install -m 0755 infra/compose/deploy-launcher.sh /usr/local/sbin/rakazo-deploy-main.

Point-in-time recovery

The nightly backup can lose up to a day. Point-in-time recovery streams the database’s write-ahead log (WAL) to S3-compatible object storage and takes a base backup every 24 hours, so the database can be restored to any moment in the last PITR_RETAIN_DAYS (default 7) and a lost server costs about a minute of writes. It covers the database only; keep the nightly backup for application data. It is off by default and needs the production Compose file with Postgres 18.

The optional pitr service streams WAL with pg_receivewal over the internal data network, using a replication-only login and a replication slot, and uploads from its own network, so Postgres keeps no internet access. Postgres closes a WAL segment at least every minute while anything is written (archive_timeout). Each segment and base backup is encrypted with age to public recipients before it leaves the container, as the nightly backup is, so the server cannot read its own backups. You can use the same recipients.

The slot keeps WAL that has not reached the sidecar, so a sidecar restart leaves no gap. Disk use stays bounded if uploads stall: the sidecar holds at most PITR_MAX_WAL_GB (default 4) of compressed WAL waiting to upload, then pauses streaming, and Postgres keeps at most that much again for the slot (max_slot_wal_keep_size). Beyond that Postgres drops the slot’s WAL, recovery has a gap, and the sidecar takes a new base backup once it reconnects.

Use a bucket of its own. Its credentials need to list, read, write and delete objects in it; deletes only remove backups older than the retention window. With PITR_RETAIN_DAYS=0 nothing is deleted, so no delete permission is needed and the bucket’s lifecycle rules handle expiry. Add to .env:

COMPOSE_PROFILES=pitr
PITR_DATABASE_PASSWORD=replace-with-openssl-rand-hex-24
PITR_BUCKET=example-rakazo-pitr
PITR_ENDPOINT=https://s3.example.com
PITR_REGION=example-region
PITR_ACCESS_KEY_ID=replace-me
PITR_SECRET_ACCESS_KEY=replace-me
PITR_AGE_RECIPIENTS=age1exampleprimaryrecipient age1examplerecoveryrecipient
# Optional: PITR_PREFIX (a key prefix), PITR_PATH_STYLE=true (path-style bucket addressing),
# PITR_RETAIN_DAYS, PITR_MAX_WAL_GB.

COMPOSE_PROFILES is comma-separated; with the updater it is updater,pitr. Leave PITR_ENDPOINT unset for AWS S3. Then turn it on from the checkout:

sudo infra/compose/pitr.sh enable

This creates the rakazo_pitr replication login, allows it in pg_hba.conf, sets archive_timeout and max_slot_wal_keep_size, and reloads Postgres without restarting it. The sidecar then takes its first base backup. Run enable again after changing PITR_DATABASE_PASSWORD or PITR_MAX_WAL_GB, or after moving Postgres to a new cluster. Uploads are grouped by the cluster’s system identifier, so a new cluster never overwrites an old one’s WAL.

The sidecar’s health check fails when an upload has waited over ten minutes, streaming is paused, nothing is streaming from the slot, or the last base backup is over 30 hours old. With RAKAZO_MONITOR_COMPOSE_PROJECT set, the host check alerts on it. For details:

sudo docker compose --env-file .env -f infra/compose/docker-compose.prod.yml exec pitr rakazo-pitr health

Restores run on any machine with Docker and this checkout. The identity comes on stdin and stays in memory inside the restore container. Prove the latest point restores, into a scratch volume that is removed afterwards. Run it after enabling, after changing the PITR scripts, and periodically:

sudo infra/compose/pitr.sh drill < rakazo-backup.key

Away from the server, set PITR_ENV_FILE to a file holding only the PITR_ storage settings. To recover production to a moment, stop everything that writes to the database and restore into its volume. The current cluster is moved aside inside the volume, not deleted. The time is read as UTC unless it names a zone:

COMPOSE="sudo docker compose --env-file .env -f infra/compose/docker-compose.prod.yml"
ssh deploy@app.example.com "cd /srv/rakazo && $COMPOSE stop api worker web pitr postgres"
ssh deploy@app.example.com \
  "cd /srv/rakazo && sudo infra/compose/pitr.sh restore '2026-01-31 09:41:00+00' rakazo-prod_postgres" \
  < rakazo-backup.key
ssh deploy@app.example.com "cd /srv/rakazo && $COMPOSE up -d --wait"

The restore uses the newest base backup that finished before the target, replays WAL up to it, and promotes the result, which starts a new timeline; the sidecar takes a fresh base backup for it. Pause CI deploys during recovery. Once the restored database checks out, remove the old cluster: $COMPOSE run --rm --no-deps --entrypoint sh postgres -c 'rm -rf "$PGDATA".before-restore-*'.

To turn it off, remove pitr from COMPOSE_PROFILES, then run sudo infra/compose/pitr.sh disable. It removes the sidecar, the slot (so Postgres stops keeping WAL for it), the login and the settings. Uploaded backups stay in the bucket.

Host audit trail

harden-host.sh installs audit rules for writes to SSH keys, sudoers, accounts, /etc/rakazo, the deployment’s .env, /usr/local/sbin, systemd units, cron, and the Docker daemon config. A rule whose directory does not exist yet, such as the .env before the first clone, is skipped with a notice; rerun the script once it exists. Anyone with root can wipe local logs, so the record only helps once it has left the host.

To ship it, create a root-only /etc/rakazo/audit-ship.env and put the sink’s token in /etc/rakazo/audit-ship.token, then rerun harden-host.sh. Only then does it install audit-ship.py as /usr/local/sbin/rakazo-audit-ship and enable rakazo-audit-ship.service. The sink is any HTTPS endpoint that accepts a JSON array of events with a bearer token, for example Axiom’s https://api.axiom.co/v1/datasets/<dataset>/ingest.

RAKAZO_AUDIT_SINK_URL=https://ingest.example.com/v1/events
# Optional: name the SSH keys you expect (fingerprints from `ssh-keygen -lf key.pub`).
RAKAZO_AUDIT_KNOWN_KEYS=SHA256:replace-with-fingerprint=laptop SHA256:replace-with-fingerprint=ci
# Optional: RAKAZO_AUDIT_TOKEN_FILE (default /etc/rakazo/audit-ship.token), RAKAZO_AUDIT_HOST
# (default: the hostname), and RAKAZO_AUDIT_HEARTBEAT_SECONDS (default 10800, three hours).

Use a token that can only write to that one destination, so a compromised host cannot read or delete what it already sent. The shipper follows /var/log/auth.log and the audit log and sends ssh.login (with key_name, unknown for keys not listed), ssh.failed, sudo.command, su, identity.change, and file.change (with the rule’s watch key and the changed paths), plus a shipper.heartbeat on start and every RAKAZO_AUDIT_HEARTBEAT_SECONDS. Every event carries event, host, source, message, and _time. Offsets advance only after the sink accepts a batch, and failures retry with backoff, so events arrive at least once. On first start it ships the current auth log and starts the audit log at its end. With monitoring installed, the host check alerts when a sink is configured but the shipper is not running.

Alert in the destination, not on the host. Starting points, written as Axiom APL here but plain filters in any log store:

Docker computers on the production stack

Layer infra/compose/docker-compose.prod.docker.yml after the base file in every Compose invocation to run bot computers as local Docker containers instead of a remote provider:

# Pull only the pull-only dependency images (postgres, caddy, busybox);
# --pull never fails when they are absent locally.
docker compose --env-file .env \
  -f infra/compose/docker-compose.prod.yml \
  -f infra/compose/docker-compose.prod.docker.yml \
  pull --ignore-buildable
docker compose --env-file .env \
  -f infra/compose/docker-compose.prod.yml \
  -f infra/compose/docker-compose.prod.docker.yml \
  up -d --build --wait --wait-timeout 300 --pull never

In this topology api, worker, web, supervisor, and computer always build from the checkout — pull --ignore-buildable only covers dependencies that have no build section. To deploy published app images instead of building, use the base file without the overlay. If docker compose up --help lacks --wait-timeout (older Compose), drop --wait and verify with docker compose ... ps instead; --wait alone can hang on one-shot services.

The overlay adds the supervisor (app image, user: root, Docker socket), the one-shot computer image build and data-init ownership fix, points the API and worker at http://supervisor:7091, and makes SANDBOX_PROVIDER=docker the default. It needs a dedicated SANDBOX_SUPERVISOR_TOKEN in .env; RAKAZO_COMPUTER_* and the SANDBOX_* limits apply as documented in .env.example. Like the updater, the supervisor is root-equivalent on the host: it publishes no port, joins only the internal app network, and Caddy has no route to it. Bot computers are sibling containers on per-bot networks that only the supervisor and the web screen proxy join.

When the updater profile is enabled too, give the sidecar the same file list and recreate set so updates do not leave the supervisor on the previous app image:

RAKAZO_COMPOSE_FILE=infra/compose/docker-compose.prod.yml:infra/compose/docker-compose.prod.docker.yml
RAKAZO_UPDATE_SERVICES=supervisor

The updater pulls and recreates services but cannot rebuild the computer stub image (pull_policy: build is not pullable). Rebuild it on the host when the computer image should change — for example after updating the checkout:

docker compose --env-file .env \
  -f infra/compose/docker-compose.prod.yml \
  -f infra/compose/docker-compose.prod.docker.yml \
  build computer

Restore

For backups created by scripts/backup.sh, use an empty rakazo database in the development Compose stack. The script validates and extracts the homes archive in a clean sibling directory before touching the database, then stops the API, worker, and supervisor. It rejects a data/ path that is a symlink or a non-directory. Stop bot computers and any other processes that write to this database or tree before running the script.

The SQL import runs in one transaction and stops on the first error, including conflicts with existing tables. An invalid archive or failed import leaves the current data tree unchanged. After SQL succeeds, the script moves the current tree to data.rollback.<random>/data, prints that path, and replaces data/ with the saved tree. Files absent from the snapshot do not remain in the active tree. A missing homes archive or an empty archive from omitted homes leaves data/ unchanged. An archive containing an empty data/ directory replaces it with an empty tree.

Application services start only after the import and tree replacement succeed. Once writer shutdown starts, a later failure or interruption makes the script attempt to stop the API, worker, and supervisor again. This includes services that started before final startup failed. The script preserves the original failure status even if cleanup fails. If Docker cannot stop the services, stop them before recovery.

A failure after the SQL commit cannot roll back the database automatically. Leave the application stopped, preserve both trees, and recover a matching database before moving the rollback tree back to data/. Keep a separate database backup from before recovery if you need that rollback. This script does not consume the production snapshot’s custom-format rakazo.dump or appdata.tgz.

./scripts/restore.sh backups/<stamp>

Upgrade

A Compose deployment on a published release tag upgrades by moving that tag:

docker compose --env-file .env -f infra/compose/docker-compose.prod.yml pull api worker web
docker compose --env-file .env -f infra/compose/docker-compose.prod.yml \
  up -d --wait --pull never api worker web

A deployment on the default local tag has no registry to pull from, so it upgrades by rebuilding the checkout instead:

git pull
GIT_SHA=$(git rev-parse HEAD) docker compose --env-file .env -f infra/compose/docker-compose.prod.yml \
  up -d --wait --pull never --build api worker web

up --wait does not report success until the new API is healthy and the worker and web containers are running. The API starts only after the migrate container finishes, so a migration failure keeps the new API from starting. A failed CLI recreate does not auto-roll back; recover with the previous RAKAZO_IMAGE_TAG (or rebuild local) and up -d --wait --pull never.

The updater sidecar has its own image and tag so an update never recreates the process performing it. Move it deliberately by setting RAKAZO_UPDATER_IMAGE_TAG to the full sha-<commit> tag, then running docker compose … pull updater && docker compose … up -d --wait --pull never updater. Sidecar /apply and /rollback recover a failed recreate by redeploying the previously cached image when possible; if that also fails, they report a possible mixed-version runtime.

Source checkouts (not Compose) still upgrade the old way: pull, rebuild with GIT_SHA=$(git rev-parse HEAD), run pnpm db:migrate, then restart API and worker. Product contracts stay compatible across cloud and self-hosted.

Upgrading to the separate app database login

Releases with database roles need POSTGRES_APP_PASSWORD in .env. The desktop app, install-images.sh, the updater sidecar and infra/compose/deploy-main.sh add it when it is missing. Elsewhere, add it yourself (openssl rand -hex 16) before you pull, or Compose stops at config validation with Set POSTGRES_APP_PASSWORD in .env. On first start, the API creates the rakazo_app login, and the API and worker then connect as it. Nothing else changes, and the upgrade does not rewrite any data.

If a feature misbehaves under row-level security, run ALTER ROLE <login> BYPASSRLS; on the login in DATABASE_URL (rakazo_app in Compose) as a superuser (the Compose owner is one), then restart the API and worker. Role attributes are not inherited, so granting it to rakazo_app does nothing for a separate login. The app then runs as before, with a startup warning, and later migrations leave that login alone. Undo it with NOBYPASSRLS and another restart, then search the API and worker logs for violates row-level security and without a tenant or system context. An older release ignores the extra variable and connects as the owner, but releases from the same day also encrypt content at rest: run content-encryption decrypt before rolling back past them (content encryption), or older releases show ciphertext.

Upgrading to Postgres 18

The production and source-checkout Compose files run Postgres 18. Installs created before that keep their data in a Postgres 16 volume (pgdata), which Postgres 18 cannot open. Until the data is copied to the new postgres volume, the database container exits and logs that it found PostgreSQL data in /var/lib/postgresql/16/docker, and production deploys stop before replacing anything. Nothing is deleted or started empty. New installs need nothing. The published-images file (docker-compose.images.yml, also used by the desktop app) stays on Postgres 16.

Take a backup and run the drill on it first (encrypted backups). Then update the checkout and run the upgrade as root:

cd /srv/rakazo
git pull --ff-only
sudo infra/compose/upgrade-postgres.sh

When the stack uses other Compose files, such as the Docker computers overlay or the source-checkout stack, pass all of them: sudo RAKAZO_DEPLOY_DIR="$PWD" infra/compose/upgrade-postgres.sh infra/compose/docker-compose.yml. Set COMPOSE_PROJECT_NAME if the stack runs under a custom -p, and on restricted networks set RAKAZO_POSTGRES_16_IMAGE to your mirror of the previous Postgres 16 image (POSTGRES_IMAGE must now name a Postgres 18 image).

The script stops the stack, copies every role (with its password) and database from Postgres 16 with pg_dumpall, and compares roles, memberships, ownership, grants, row-level security policies, sequences and the row count of every table. It also checks that rakazo_app signs in with POSTGRES_APP_PASSWORD and does not bypass row-level security. Only then does it switch Postgres 18 over and restart the services that were running. The site is down for the whole copy, which takes about as long as dumping and restoring the database: a few minutes for a small one. It needs free disk for one more copy of the database plus 1 GiB. Running it again after it succeeded does nothing. If any step fails, it stops with the stack down and the Postgres 16 data unchanged; fix the error and run it again. On the production host, then rerun the deploy that stopped.

The Postgres 16 volume stays as it was, mounted read-only. To roll back, check out the previous release and start the stack: it runs Postgres 16 on the original data. Anything written after the upgrade exists only on Postgres 18, so roll back right away or not at all, and pause CI deploys until you upgrade again. Once you no longer need the rollback, stop postgres, remove the <project>_pgdata volume and start it again; Compose recreates that volume empty.

Space privacy-boundary migration

Before upgrading across migration 20260830200000_space_scope_names_and_user_credentials, check that every existing model and voice credential still belongs to a member of its Space. This query uses the pre-migration workspaceId column name and must return no rows:

SELECT 'model' AS credential_type, credential."id", credential."userId",
       credential."workspaceId" AS "spaceId"
FROM "user_model_credentials" AS credential
LEFT JOIN "space_members" AS membership
  ON membership."spaceId" = credential."workspaceId"
 AND membership."userId" = credential."userId"
WHERE membership."id" IS NULL
UNION ALL
SELECT 'voice', credential."id", credential."userId", credential."workspaceId"
FROM "user_voice_credentials" AS credential
LEFT JOIN "space_members" AS membership
  ON membership."spaceId" = credential."workspaceId"
 AND membership."userId" = credential."userId"
WHERE membership."id" IS NULL;

The migration renames columns used by the API and worker and is therefore a coordinated cutover, not an online rolling migration. Stop the old API and worker, apply the migration, and start the new versions together. Its lock waits are bounded so contention fails the migration instead of leaving application traffic queued indefinitely.

Published images and tags

.github/workflows/publish-server-image.yml publishes to ghcr.io/<owner>/<repo>/…, derived from ${{ github.repository }} rather than hardcoded, so a fork’s CI fills the fork’s own namespace. For this repository that is:

infra/compose/docker-compose.images.yml is the no-checkout path for those app and computer tags plus Postgres. The supervisor runs from the app image on the internal network only (not a separate published supervisor image, and no host port). Production Compose (docker-compose.prod.yml) can also pull the same app tags once RAKAZO_IMAGE_TAG is set to a published value.

If you deploy from your own fork, set RAKAZO_IMAGE and RAKAZO_UPDATER_IMAGE to your namespace — your CI cannot publish into someone else’s.

Every publish, including edge from main merges, is multi-arch (amd64 + arm64): each architecture builds natively, on Depot when the DEPOT_PROJECT_ID repository variable names a project and on GitHub’s runners otherwise, and one manifest is assembled per image. Depot builds publish and attest each image in the same job; native GitHub builds combine their architecture artifacts in a separate publication job. Main CI revisions validate independently. Publication locks are per image, and a superseded main build keeps its commit tag without moving edge. Image and release queues retain pending jobs so a slower older run cannot displace a newer one. Production deployment remains serialized and deploys the exact revision that passed its checks. Before any tag points at a new app image, CI fails the publish if it holds a .env file, a .git directory or a private key (infra/compose/check-image.sh). Until a stable vX.Y.Z has been published, GHCR may only have edge and sha-* tags; do not pin latest unless that tag exists in the registry.

Building the images yourself does not need QEMU. docker compose up --build builds for the host’s own architecture, and a fork publishing multi-arch images should do what publish-server-image.yml does: build each architecture on a native runner (GitHub Actions provides ubuntu-24.04-arm for public repositories) and merge the digests into one manifest. QEMU emulation (docker/setup-qemu-action, binfmt) still works if you have no native arm64 machine, but it is many times slower, hours rather than minutes for the computer image.

The updater resolves the newest stable vX.Y.Z source tag but deploys its sha-<full-commit> image, not latest or a moving minor tag. A registry tag is not an OCI digest and GHCR package writers can replace it, so the trust boundary remains this repository’s publishing credentials. The workflow reduces that boundary by using SHA-pinned actions, read-only pull-request jobs, digest-pinned base images, SBOM/provenance output, and a GitHub build attestation. Operators who require registry-level content addressing can pin RAKAZO_IMAGE outside the automatic updater to a verified digest.

Rollback never contacts the registry: it redeploys the previous tag from the local Docker cache, so a later tag move cannot change rollback content. Do not prune the previous application image until the next update has been accepted. If it is missing, rollback fails closed instead of pulling new content under an old tag.

To populate the registry the first time, run the workflow manually (workflow_dispatch) or push a v* tag. A manual run produces sha-<full-commit>; only a stable vX.Y.Z tag (no prerelease suffix) produces latest, and any v* tag produces semver tags. The updater ignores prereleases and refuses the official path until a stable vX.Y.Z exists.

Updater sidecar

Compose production deployments offer an opt-in updater profile on a private control network. Normal deployments do not start it or require its credential. To enable it, set a dedicated RAKAZO_UPDATER_TOKEN and explicitly start the profile:

docker compose --env-file .env -f infra/compose/docker-compose.prod.yml \
  --profile updater up -d --build updater

It exposes /health, /state, /plan, /apply, and /rollback at http://updater:7092 with RAKAZO_UPDATER_TOKEN. Operator CLI upgrades above do not need it; the sidecar is for automated apply/rollback over that private HTTP API. Only deployment tooling calls it; app accounts have no update endpoint, and the application API does not receive the updater token.

The API cannot update itself — its image has no .git, and nothing inside the container would restart it — so the work happens in a separate updater container that outlives the recreate:

• Official repository: resolves the newest stable release and its source commit with git ls-remote --tags, pins the corresponding full sha-<commit> image tag in .env, keeps the outgoing tag in RAKAZO_IMAGE_TAG_PREVIOUS, explicitly pulls the new image, then runs up -d --wait --pull never. No build runs on the server.
• Fork (Advanced): a fork has no published images, so the sidecar fast-forwards the checkout in RAKAZO_DEPLOY_DIR and runs up -d --build. This builds on the server and takes minutes rather than seconds. Point it only at a fork you control and have reviewed — the sidecar runs that Compose file through a root-equivalent Docker socket.

Updates and rollbacks run one at a time. A failed pull leaves running services alone; a failed recreate restores the previous environment pin and attempts to redeploy the cached previous image. A failed fork build also restores the pre-update branch and commit (including when checkout succeeded but merge did not) so a later manual --build cannot deploy the rejected or unintended revision. Database migrations are not reversed. The sidecar never recreates itself, never touches Postgres or Caddy, and never runs migrations — that ordering belongs to the API’s dependency on the migrate service.

Only https:// and ssh:// git remotes are accepted. Merges are fast-forward only. A dirty or untracked source tree fails closed before anything runs (the application Dockerfile uses COPY . .).

The deploy directory must be one path

RAKAZO_DEPLOY_DIR is bind-mounted into the updater at the same path it is read from (${RAKAZO_DEPLOY_DIR}:${RAKAZO_DEPLOY_DIR}), and that is load-bearing rather than tidy. Production Compose defaults both sides to /srv/rakazo; set the variable for any other layout. When the updater runs docker compose -p <project> --file $RAKAZO_DEPLOY_DIR/infra/compose/docker-compose.prod.yml up -d, the Compose CLI inside the container expands this file’s relative bind mounts — ../../.env, ./Caddyfile.prod — against that path and hands the results to the daemon. The daemon has to be able to resolve the same strings, or it silently creates empty directories where your .env and Caddyfile should be. Compose makes the effective -p value available for interpolation but does not automatically put it in a container’s environment, so the production file explicitly assigns COMPOSE_PROJECT_NAME to the updater. A standalone sidecar can instead set RAKAZO_COMPOSE_PROJECT_NAME; the final fallback is rakazo-prod. Without that propagation, a stack started with -p something-else would be left alone while a second project with a new empty Postgres volume came up beside it.

Deployments that layer a Compose overlay

RAKAZO_COMPOSE_FILE takes a list, separated the way Compose’s own COMPOSE_FILE is (: by default, or whatever COMPOSE_PATH_SEPARATOR says). Each entry becomes its own --file, in the order given, so the updater reconciles the same stack the operator runs by hand:

RAKAZO_COMPOSE_FILE=infra/compose/docker-compose.prod.yml:ops/compose/overlay.yml

Every entry is validated separately and must stay inside RAKAZO_DEPLOY_DIR.

If the overlay adds a service built from the application image, name it in RAKAZO_UPDATE_SERVICES (comma separated) so it is pulled, recreated and rolled back with the rest. Otherwise an update leaves that service running the previous code:

RAKAZO_UPDATE_SERVICES=supervisor

These names are appended to the built-in api, worker, web, never substituted for them, so no value here can drop a core service from an update.

The value therefore has to be the path the daemon sees, which is not always the path your shell sees:

• Linux. The daemon shares the host filesystem, so the checkout path is the answer: /srv/rakazo is the default and supported production layout. Set RAKAZO_DEPLOY_DIR explicitly when the checkout is elsewhere.
• Docker Desktop (Windows/macOS). The daemon runs in a VM that mounts your drive somewhere else. On Windows, C: appears at /run/desktop/mnt/host/c, so a checkout at C:\Users\you\rakazo is RAKAZO_DEPLOY_DIR=/run/desktop/mnt/host/c/Users/you/rakazo. Host Git may use core.autocrlf=true; the updater ignores CR-only diffs so that does not block /apply. Verify the mount before deploying:

docker compose --env-file .env -f infra/compose/docker-compose.prod.yml \
  --profile updater run --rm updater git -C "$RAKAZO_DEPLOY_DIR" log --oneline -1

That must print your checkout’s HEAD. The two tempting wrong answers both fail: a native Windows path is rejected by the daemon (mount denied: … too many colons, because the drive letter’s colon collides with the bind-mount separator), and /mnt/c/... fails silently — the container starts, the mount is an empty directory, and the updater simply reports no checkout.

The updater’s privileges

The updater holds the Docker socket, which is root-equivalent on the host. It is scoped as narrowly as that allows:

• No ports, so nothing is published on the host.
• Only on the dedicated control network shared with the API. Caddy is not attached, so the reverse proxy has no route to the updater.
• Every route except /health requires the shared bearer token, compared in constant time.
• The process environment carries only updater settings (RAKAZO_UPDATER_TOKEN, deploy path, image name, project name). Application secrets stay in the bind-mounted .env that Compose reads for interpolation; they are not loaded into this container.
• The Docker CLI lives only in the updater image. The api, worker, and web containers keep cap_drop: ALL and no socket.

Enabling the updater profile requires RAKAZO_UPDATER_TOKEN to be a dedicated random value (at least 32 characters in production). It must differ from BETTER_AUTH_SECRET, SANDBOX_SUPERVISOR_TOKEN, and SCREEN_PROXY_SECRET. Leave the profile disabled if you would rather not grant the capability.

Other deployment layouts

API and worker need always-on processes; serverless request handlers are not sufficient. Use a Node.js version supported by the root package.json, Postgres 16 or newer and a persistent DATA_DIR volume shared by API and worker, with encrypted off-host backups. Homes stay on that volume unless they are kept in object storage; other DATA_DIR files still need it shared.

Use the same HTTPS origin for the web app, /api, and /rpc. Preserve the authenticated screen proxy routes. Choose a computer provider appropriate to the service’s trust boundary. SIGNUPS_ENABLED and SIGNUP_ALLOWLIST apply from the environment on every API start. The optional marketing site in apps/www can be hosted separately.

Image and video generation

Bot media tools are disabled by default. Configure the worker (and API for inline execution) with a provider and its generic HTTP connection:

MEDIA_GENERATION_PROVIDER=fal
MEDIA_GENERATION_BASE_URL=https://queue.fal.run
MEDIA_GENERATION_API_KEY=replace-with-media-api-key
MEDIA_GENERATION_IMAGE_MODEL=fal-ai/flux/schnell
MEDIA_GENERATION_VIDEO_MODEL=fal-ai/wan/v2.2-a14b/text-to-video

MEDIA_GENERATION_PROVIDER accepts disabled (the default), fal, or fake (for offline testing only). The base URL is optional and defaults to the URL above. Only set the image/video model fields for capabilities you want enabled. Compatible servers can use the same connection fields; models must support the corresponding fal queue API and Flux/Wan input and output shapes. Credentials stay on the server. Set these in your deployment secret configuration; never commit actual credentials.

Bots use generate_image to attach one PNG or generate_video to submit a four-second MP4. The video tool returns a job ID and queued/running status; call it again with jobId to check progress and attach the completed video. Saved jobs can be polled after worker restarts by the same owner/bot/thread. The first version requires polling through the bot; it does not automatically deliver jobs in the background. Use the existing per-bot tool switches and action approval rules to control access. Both tools require approval by default. Ticket board runs keep these chat-only media tools unavailable so ticket work stays on the board.

Each run allows at most two submissions (including failures). Submissions are never retried automatically: a lost response can mean the provider accepted and charged for the request. Provider errors omit remote response bodies. Configure account budgets and rate limits with your provider; the run cap is not a currency budget. Provider 429 responses are returned without automatic retries. Outputs must fit both the 10 MiB media ceiling and the configured attachment limit; larger outputs are rejected.

Images use existing inline chat rendering. MP4 files can be previewed with native video controls on web and Electron and opened/downloaded through the existing native file flow on mobile. No new mobile player or settings screen is required.

Connect mobile clients

The iOS and Android app can also point at a self-hosted origin at runtime. On the sign-in screen, tap Use a custom server and enter the same HTTPS origin as WEB_ORIGIN (for example https://app.example.com). Store builds still default to EXPO_PUBLIC_API_URL; the in-app setting is an override for people running their own API. Changing the server signs the device out of any previous session.

Plain http:// is accepted only for a private IP, localhost, or a .local name. A hostname reached over a VPN such as WireGuard or Tailscale needs HTTPS with a certificate the phone trusts, for example a real domain with a Let’s Encrypt certificate issued through a DNS challenge.

Passkeys

Passkeys are on whenever WEB_ORIGIN is an HTTPS domain (or localhost in development); the relying party is that origin’s host, so existing passkeys keep working only while WEB_ORIGIN stays on the same domain. They are off when password sign-in is disabled for SSO-only deployments. On web, a browser offers them where it has a platform authenticator. The macOS desktop app uses the system passkey sheet, which only serves the domain the release is associated with (app.rakazo.com), so it hides passkeys for other servers.

Native apps can use a domain’s passkeys only when the domain vouches for them and the app was built for that domain (the app’s associated domain comes from EXPO_PUBLIC_API_URL at build time). To let a build use this server’s passkeys, set:

# iOS and the macOS desktop app (dev.rakazo.desktop): <Apple Team ID>.<bundle id>, comma-separated.
# Served as /.well-known/apple-app-site-association.
PASSKEY_APPLE_APP_IDS=ABCDE12345.com.rakazo.app,ABCDE12345.dev.rakazo.desktop
# Android: SHA-256 fingerprints of the app signing certificates, comma-separated.
# Served as /.well-known/assetlinks.json and accepted as passkey origins.
PASSKEY_ANDROID_CERT_SHA256=AA:BB:...:FF

Until a platform’s setting is set, the native app on that platform hides passkeys for this server. Both files are served through the web origin, so a reverse proxy in front of the web container must pass /.well-known/ through to it.

macOS voices

When the API runs on macOS, Voice settings also offers macOS, which needs no key. The API renders each reply with the built-in /usr/bin/say into an audio clip, so it plays on the listener’s device and stops like any other voice. Docker and Linux hosts do not list it.

Spoken summaries

Off by default. Set VOICE_SUMMARY_MIN_CHARS (for example 600) to have a voice read a bot reply longer than that many characters as a short summary written by that bot’s own model. Each summary is an extra model call billed to the bot. The chat keeps the full reply, and if the summary call fails the full reply is read.

Upload size

MAX_UPLOAD_SIZE_MB sets the per-file chat attachment and computer file transfer limit in MiB (1 MiB = 1,048,576 bytes). It defaults to 10 and must be a positive integer. Set it in .env and restart the API, worker, and Docker supervisor (when used). Web, desktop, and mobile use the limit advertised by the server. If a reverse proxy limits request bodies, allow room for base64 encoding (about 4/3 of the file size) and the JSON envelope.

Shared storage migration

Push notification registrations now live in Postgres, encrypted with ENCRYPTION_KEY. The same key must be configured on every API and worker replica. Registrations remain bound to their authenticated session; deleting an account also deletes its registration.

With OBJECT_STORAGE_BUCKET configured, new artifact contents use the existing S3-compatible connection alongside computer homes. Artifact contents are always encrypted before upload using the space’s content key and authenticated segments bound to the artifact identity. This applies even with CONTENT_ENCRYPTION=off. Metadata and access checks remain in Postgres; clients and sandboxes receive neither bucket credentials nor public object URLs. Without object storage, artifacts continue using the local volume.

Existing artifact rows continue reading their local files until migrated. Before spreading API or worker replicas across machines, migrate files from every original application volume. Back up the database and volume first. Deploy the schema migration and the new application version, then stop all API and worker processes, including old releases, while running the following commands from a maintenance container with the original volume, DATABASE_URL, ENCRYPTION_KEY, content master keys, and object storage configuration:

# Published image (run from /app/apps/api):
node dist/cli/shared-storage.js artifacts --dry-run
node dist/cli/shared-storage.js artifacts
node dist/cli/shared-storage.js push-tokens --dry-run
node dist/cli/shared-storage.js push-tokens

# Source checkout equivalents:
pnpm --filter @rakazo/api shared-storage artifacts
pnpm --filter @rakazo/api shared-storage push-tokens

The artifact migration uploads and decrypts each file to verify it before conditionally updating its database storage key. It keeps the original local files and skips rows already migrated, so interrupted runs can resume. A database failure after upload can leave an unreferenced object; do not delete objects blindly after a failed migration. Keep local originals until the migration is complete and verified. Files absent from the mounted volume are counted as pending while migration continues for files present on that volume. Run the command on each original volume, then run artifacts --dry-run again and require pending: 0 before starting independent replicas. Object-backed artifacts require object storage to remain configured; old application versions cannot read their new keys. The local content-encryption CLI operates on local artifacts only; object artifacts stay encrypted and retain access through the key versions referenced in their headers.

The push import preserves registrations with a live session, skips missing/expired or sessionless registrations, accounts already registered in Postgres, and devices owned by any existing database registration. Import never hands a device back to an older account. It archives processed files with a .migrated suffix. Those archives are never used as a runtime fallback, so they cannot restore a registration after logout. Clients skipped during migration register again when they next open the app. Remove the archived token files after verifying the import. Restart API and worker processes after both commands finish. A schema-only rollback remains compatible with the previous release, but the old code does not read database registrations or object-backed artifacts.