Identity and the Gateway

Volumes I and II ended at the same place: Minimal trusts five headers, and something in front of it has to turn a real person into those five values before a request is allowed through. Volume II's closing chapter described that boundary in general terms and named three identity providers it could be built against. This volume builds it, with one of them, to the point where an app on a phone can sign in, get an identity, and call an API — and where the next app can be added the same way in an afternoon.

The three pieces are Zitadel for identity, Apache APISIX for the boundary, and Minimal for the API. They share one Postgres instance. Every step in this book was run before it was written down, on a Mac for the development environment and on a Debian 12 virtual machine standing in for the production hosts; where a step could not be run — anything that needs a real domain, a real Apple developer account, or a real Google project — the text says so at the step.

The apps, and what identity means to them

The apps this book serves are consumer apps: someone installs one from an app store, opens it, and starts using it without signing in. On first launch the app mints its own identity — a ULID for the device and a key pair whose private half never leaves the device. Everything the app does alone, it does with that local identity and nothing else.

Registration happens later, and only when the person reaches for a feature that needs a server: a shared list, a backup, a second device. At that point the app sends the person to sign in with Google, GitHub or Apple. Zitadel handles that sign-in and hands the app a token. That token is the whole of what the app carries to the API from then on: no password, no API key, no Minimal access token.

Three rules about identity hold everywhere in this book, and each has a chapter that enforces it:

• The server assigns the user id. Minimal's X-User-Id is a ULID the server minted the first time it issued a token for that person. It is never the Zitadel subject, never the device's own ULID, and never anything the app chose. The device's ULID and public key are data the app registers under the server's id, if the app needs them server-side at all. Chapter 4 mints it; Chapter 5 copies it into the header; Chapter 7 shows the whole path.
• One organisation per app in Zitadel. A person who registers in two of your apps has two accounts, one in each app's organisation, even with the same email. Signing in to one never signs them in to the other. Chapter 4 sets this up and Chapter 8 proves it with a second app.
• Nothing reaches Minimal without a token the gateway verified. There is no anonymous route. A request with no token, a forged token, a token for another app, or a token from another issuer is refused at the gateway with 401 or 403 and never reaches Minimal. Chapter 5 builds this and shows each refusal.

The shape of it

graph LR
    subgraph Device["Phone, tablet or Mac"]
        App["App<br/>local ULID + key pair"]
    end
    subgraph AppHost["app-1 — the application host"]
        GW["APISIX<br/>api.example.com · auth.example.com<br/>the only public listener"]
        ZL["Zitadel Login V2<br/>127.0.0.1:3000"]
        ZT["Zitadel<br/>127.0.0.1:8081"]
        MN["Minimal<br/>127.0.0.1:3045"]
    end
    subgraph DBHost["db-1 — the database host"]
        PG["Postgres 18<br/>zitadel · minimal · healthyme<br/>one role per database"]
    end
    App -->|"1. sign in (OIDC, PKCE)"| GW
    GW -->|"/ui/v2/login"| ZL
    GW -->|"everything else on auth."| ZT
    ZL -.->|"session API"| ZT
    App -->|"2. Authorization: Bearer JWT"| GW
    GW -->|"3. X-Org-Id · X-Project-Id · X-Space-Id<br/>X-User-Id · X-User-Roles"| MN
    ZT -->|TLS| PG
    MN -->|TLS| PG

Two hosts. The app talks only to APISIX. APISIX is the only process anyone outside can reach; Zitadel, its login, and Minimal all bind to loopback on the same host and are reachable only through it. Postgres sits on its own host and accepts each service under its own role, over TLS, into its own database.

Read the numbered arrows as one person's first day:

1. The app opens the system browser at https://auth.example.com/oauth/v2/authorize with a PKCE challenge. APISIX sends that to Zitadel, which redirects to its own login page, which APISIX sends to the Login V2 process. The person taps "Continue with Google". When they come back, the app has an authorization code and exchanges it — still through APISIX — for a JWT access token.
2. The app calls https://api.example.com/minimal/... with that JWT in the Authorization header.
3. APISIX checks the signature against Zitadel's published keys, checks the issuer and the audience, deletes every identity header the app may have sent, and sets the five headers itself from the token's claims. Minimal sees a request from X-User-Id: 01M2F0KNNC4M7Z1HEJHHG80JGE holding X-User-Roles: member, and does what Volumes I and II describe.

The token in step 1 carries the server-assigned id because Zitadel minted it: an action that runs every time Zitadel issues an access token looks the person up, assigns a ULID the first time, stores it against the account, and puts it on the token as a claim named minimal_user_id. Chapter 4 writes that action; it is twenty lines.

Names used throughout

One fictional app, Healthy Me, carries the whole book; Chapter 8 adds a second, Sleep Well, to show that nothing about the first was special. The names below recur in every chapter and every configuration file. Replace the placeholders once, consistently, and the files in Appendix D work as printed.

Three of those deserve a sentence each.

The Minimal organisation is yours, not the customer's. Every person using Healthy Me is a row in Healthy Me's tables, identified by their server-assigned ULID; none of them is a Minimal organisation, project or space. The Minimal hierarchy here is your deployment structure — one organisation for the company, one project per app, one space per release channel — exactly as Volume I, Chapter 3 describes it for Acme.

The Zitadel organisation is the app. Zitadel's own hierarchy is instance → organisation → project → application, and a user belongs to one organisation. Putting each app in its own organisation is what gives you separate user bases: the same email address registering in Healthy Me and in Sleep Well creates two unrelated Zitadel users.

The roles are Zitadel's, copied by the gateway. A person holds member or premium because Zitadel says so on the token, and APISIX writes exactly those names into X-User-Roles. Minimal's permission templates then name the same strings. Nothing on the Minimal side stores who holds which role; Chapter 6 sets the templates so the three systems agree on the vocabulary.

Two environments, and only two

There is no staging. A change is tried on the Mac and then applied to production, and the Minimal space is the release channel: Volume II's definition-copy route promotes a definition from a next space to live without touching anything else. Chapter 9 builds dev; Chapters 2 to 7 build production, and say where dev differs.

What is fixed before you start

These were current on the day this book was verified, 14 September 2026, and every command below is written against them:

Two of those choices were made against alternatives, and the reasons matter later:

• APISIX rather than a Go gateway. KrakenD is a single Go binary that validates JWTs against a JWKS and, unlike most gateways, forwards no client header unless a route lists it — a genuinely good fit for Minimal's five-header contract. It was rejected because it is an API gateway for JSON backends and cannot front Zitadel's console and login, which need a general reverse proxy with HTTP/2. One gateway that does both beat two gateways. The cost is that APISIX is Lua on OpenResty and, as its own build files state, runs only on Linux — which is why dev keeps it in a VM.
• Binaries and systemd rather than containers. Every piece ships a Linux binary or a Node bundle, so the production host runs four units — apisix, zitadel, zitadel-login, minimal — under their own unprivileged users, with the hardening options systemd offers, and nothing else in between. The Docker Compose files Zitadel publishes are a fine reference for environment variable names; the ones this book uses are transcribed from them and from the bundle's own startup script, and each was tested.

What you need before Chapter 2

• Two Debian 12 hosts with a private network between them, sudo on both, and the two public DNS names pointing at app-1. Ubuntu 22.04 or 24.04 works with the same commands; the vendor repository lines in Chapters 2 and 5 name the distribution and you change those.
• A Mac with Homebrew, Postgres 18 already running, Node 22 or later, and Go, for Chapter 9.
• Developer accounts for the sign-in providers you will offer — a Google Cloud project, a GitHub OAuth app, and an Apple Developer team with a Services ID — for Chapter 4. The rest of the book does not wait on them: the dev organisation signs in with a password until they exist.
• A Minimal release build, and Volumes I and II within reach. This volume does not re-explain organisations, permission templates, lock masks or row-level security; it uses them.

How each chapter is written

Every chapter is a sequence of steps, each of which ends in something you can check: a command whose output is shown, a query whose rows are shown, a request whose status code is shown. The checks are the point. When a step has a failure mode that was actually hit while writing this book — a configuration key in the wrong file, a Lua pattern that silently never matches, a port forwarded from the wrong place — the step says what the failure looks like so you recognise it in a minute rather than an hour.

Configuration files are printed in full once, in Appendix D, and shown in the chapters as the fragments each step changes. Secrets in printed files are placeholders in angle brackets; a value generated by a command, such as a masterkey or a personal access token, is shown as the command that generated it and never as a literal.

Exercises

1. Draw the request path for one person opening Healthy Me for the first time and, a week later, turning on a shared list. Mark on it the moment the server-assigned ULID comes into existence, and the moment Minimal first learns of this person. They are not the same moment; say what happens between them.

2. Using the names table, write the same table for your own first app: the domains, the Postgres role and database it gets, its Zitadel organisation and project, its Minimal project and space, and its role names. Keep it; Chapters 2 to 7 will ask for each entry in turn.

3. This chapter says a person who registers in two of your apps has two accounts. Write down what would have to change — in Zitadel, in the gateway, and in the apps — if you later wanted one account shared across apps, and which chapter each change lands in. Chapter 8 gives one answer; compare yours to it.

One Postgres cluster on db-1 holds three databases: zitadel, where the identity provider keeps its event store and projections; minimal, Minimal's own definition store from Volume II, Chapter 9; and healthyme, the app's data, which Minimal reaches as a registered tenant database. Nothing about Postgres makes sharing one cluster unsafe. What makes it unsafe is a single role that can reach all three, a pg_hba.conf line that says all all, or a connection that crosses the private network in the clear. This chapter builds the cluster so that none of those is true, and ends with the queries that prove it.

The rules, before the commands:

• One role per database, and the role owns nothing outside it. zitadel owns zitadel, minimalist owns minimal, healthyme_app owns healthyme. None is a superuser, none can create databases or roles, and each has a connection limit sized to the pool that will use it.
• A role can connect only to the databases pg_hba.conf names for it, and only over TLS. The file lists one line per (database, role) pair. There is no all anywhere on a TCP line. PUBLIC loses its default CONNECT on every database, so a role that reaches the server cannot even open a session against a database it was not granted.
• Minimal reads the app's database as a second role, not as the owner. minimalist gets SELECT, INSERT, UPDATE and DELETE on healthyme's tables through default privileges, and is what the project's registered connection uses in Chapter 6. Schema changes to the app's tables run as healthyme_app. If Minimal's DDL surface is wanted on that database later, that is a deliberate grant, made then.
• Extensions and pg_cron are the superuser's job, done once. Minimal's schema needs vector, pg_trgm and pg_cron; the last can live in exactly one database per cluster, and this cluster says which.

Step 1 — Install Postgres 18 from the PGDG repository

Debian 12 ships an older Postgres. The PostgreSQL project's own repository carries 18 for bookworm and trixie, with the two extensions Minimal needs as separate packages:

sudo apt install -y postgresql-common
sudo /usr/share/postgresql-common/pgdg/apt.postgresql.org.sh -y
sudo apt update
sudo apt install -y postgresql-18 postgresql-18-cron postgresql-18-pgvector

Check that the cluster is up and that the extensions are available to it — available, not yet installed into any database:

systemctl is-active postgresql@18-main
sudo -u postgres psql -Atc "select version()"
sudo -u postgres psql -Atc "select name, default_version from pg_available_extensions
                            where name in ('vector','pg_cron','pg_trgm') order by 1"

active
PostgreSQL 18.6 (Debian 18.6-1.pgdg12+2) on aarch64-unknown-linux-gnu, ...
pg_cron|1.6
pg_trgm|1.6
vector|0.8.6

pg_trgm is part of Postgres itself; the other two came from the packages just installed.

Step 2 — Server settings: listen on the private address, load pg_cron, require SCRAM

Debian keeps postgresql.conf under /etc/postgresql/18/main/ and reads every file in its conf.d directory after it. Put the deployment's settings in one file there rather than editing the distribution's:

sudo tee /etc/postgresql/18/main/conf.d/90-minimal.conf >/dev/null <<'EOF'
# db-1 — one cluster, three databases, three roles. See Book of Minimal, Volume III, Chapter 2.
listen_addresses = 'localhost,10.0.0.10'   # loopback for the superuser, the private address for app-1
shared_preload_libraries = 'pg_cron'
cron.database_name = 'minimal'             # pg_cron lives in exactly one database per cluster
password_encryption = 'scram-sha-256'
ssl = on
ssl_cert_file = '/etc/postgresql/18/main/server.crt'
ssl_key_file  = '/etc/postgresql/18/main/server.key'
log_connections = on
log_disconnections = on
EOF

cron.database_name is the line to read twice. pg_cron runs its scheduler in one database, and CREATE EXTENSION pg_cron succeeds only there. Minimal's schema creates that extension and schedules two jobs in it, so the database has to be minimal. A second Minimal store on the same cluster cannot have pg_cron; Chapter 9 says what that means for the Mac.

ssl = on needs a certificate. Debian's package installs a self-signed "snakeoil" pair and points the default configuration at it, which is enough for the connection to be encrypted but not for a client to verify who it is talking to. Issue a real one for db-1 from whatever certificate authority your hosts already trust — an internal CA is the usual answer — and install it as the two files named above, readable by the postgres user only:

sudo install -o postgres -g postgres -m 0600 server.key /etc/postgresql/18/main/server.key
sudo install -o postgres -g postgres -m 0644 server.crt /etc/postgresql/18/main/server.crt

With a certificate the clients can verify, Chapters 3 and 6 set sslmode=verify-full on their connections and name the CA. With the snakeoil pair they can only say require, which encrypts but does not authenticate the server. Both are shown; use the first.

Step 3 — pg_hba.conf: one line per pair, TLS only on TCP

Replace the distribution's file. The superuser keeps its local socket, and every TCP line is a hostssl line naming one database, one role and the one address that will connect:

sudo cp /etc/postgresql/18/main/pg_hba.conf /etc/postgresql/18/main/pg_hba.conf.dist
sudo tee /etc/postgresql/18/main/pg_hba.conf >/dev/null <<'EOF'
# TYPE  DATABASE   USER           ADDRESS        METHOD
local   all        postgres                      peer
hostssl zitadel    zitadel        10.0.0.20/32   scram-sha-256
hostssl minimal    minimalist     10.0.0.20/32   scram-sha-256
hostssl healthyme  minimalist     10.0.0.20/32   scram-sha-256
hostssl healthyme  healthyme_app  10.0.0.20/32   scram-sha-256
EOF
sudo systemctl restart postgresql@18-main

Then confirm the three settings the rest of the chapter relies on came from the file you wrote and not from a default:

sudo -u postgres psql -Atc "show shared_preload_libraries; show cron.database_name; show ssl;
                            show password_encryption;"

pg_cron
minimal
on
scram-sha-256

There is deliberately no line for a database administrator over TCP. Administration happens on db-1 over the local socket as postgres, through whatever access you already control to that host. If you need psql from app-1 for operations, add one hostssl line for one named role with the grants that work needs, and nothing broader.

Step 4 — Roles and databases

As the superuser, on the local socket. Each role gets LOGIN, a strong password, every dangerous attribute switched off explicitly, and a connection limit that matches its pool: Zitadel's default pool is 10, Minimal's store pool is whatever definition_store.max_open_connections says (20 in the sample file) plus the tenant pool it opens to healthyme, and the numbers below leave headroom for psql sessions during an incident.

sudo -u postgres psql -v ON_ERROR_STOP=1 <<'SQL'
CREATE ROLE zitadel       LOGIN PASSWORD '<zitadel-password>'
  NOSUPERUSER NOCREATEDB NOCREATEROLE NOINHERIT CONNECTION LIMIT 20;
CREATE ROLE minimalist    LOGIN PASSWORD '<minimalist-password>'
  NOSUPERUSER NOCREATEDB NOCREATEROLE NOINHERIT CONNECTION LIMIT 60;
CREATE ROLE healthyme_app LOGIN PASSWORD '<healthyme-password>'
  NOSUPERUSER NOCREATEDB NOCREATEROLE NOINHERIT CONNECTION LIMIT 10;

CREATE DATABASE zitadel   OWNER zitadel;
CREATE DATABASE minimal   OWNER minimalist;
CREATE DATABASE healthyme OWNER healthyme_app;

REVOKE CONNECT ON DATABASE zitadel   FROM PUBLIC;
REVOKE CONNECT ON DATABASE minimal   FROM PUBLIC;
REVOKE CONNECT ON DATABASE healthyme FROM PUBLIC;
GRANT  CONNECT ON DATABASE healthyme TO minimalist;
SQL

The three REVOKE CONNECT ... FROM PUBLIC lines are the second half of what pg_hba.conf does. The first half refuses the TCP connection; this half refuses the database even to a role that arrives some other way — a future pg_hba line written too broadly, a local-socket session, a role added later and forgotten. Belt and braces, both cheap.

minimalist gets CONNECT on healthyme because Minimal will register that database against the Healthy Me project in Chapter 6 and read rows from it. It gets nothing on zitadel, and zitadel gets nothing outside its own database, and neither is changed by anything later in this book.

Step 5 — Inside each database: schemas, extensions, default privileges

Still as the superuser. In minimal, create the three extensions Minimal's schema file expects and let its owner use pg_cron; in healthyme, take the public schema away from PUBLIC (Postgres 15 and later already do this, but it costs nothing to say so) and arrange that tables healthyme_app creates in future are readable and writable by minimalist without a grant per table:

sudo -u postgres psql -v ON_ERROR_STOP=1 <<'SQL'
\c minimal
CREATE EXTENSION IF NOT EXISTS vector;
CREATE EXTENSION IF NOT EXISTS pg_trgm;
CREATE EXTENSION IF NOT EXISTS pg_cron;
GRANT USAGE ON SCHEMA cron TO minimalist;
REVOKE ALL ON SCHEMA public FROM PUBLIC;
GRANT  ALL ON SCHEMA public TO minimalist;

\c healthyme
REVOKE ALL ON SCHEMA public FROM PUBLIC;
GRANT  ALL   ON SCHEMA public TO healthyme_app;
GRANT  USAGE ON SCHEMA public TO minimalist;
ALTER DEFAULT PRIVILEGES FOR ROLE healthyme_app IN SCHEMA public
  GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO minimalist;
ALTER DEFAULT PRIVILEGES FOR ROLE healthyme_app IN SCHEMA public
  GRANT USAGE, SELECT ON SEQUENCES TO minimalist;
SQL

CREATE EXTENSION for vector and pg_cron needs the superuser; pg_trgm is marked trusted and its owner could create it, but there is no reason to split the three. GRANT USAGE ON SCHEMA cron is what lets minimalist call cron.schedule when the schema file runs next — without it the two scheduling statements at the end of that file fail with permission denied for schema cron.

The ALTER DEFAULT PRIVILEGES pair is scoped to tables healthyme_app creates. A table created by anyone else — the superuser, say, during an incident — does not get the grant, and Minimal will read permission denied from it until someone grants it by hand. That is the intended shape: the owner role is the one that makes tables, and only its tables are automatically Minimal's to read.

Step 6 — Apply Minimal's schema as its owner, over TLS

Minimal's Postgres schema is the postgres.sql in the database repository's minimal directory. Two things about it before running it. Its head says "Tables only" — the database and role were the bootstrap script's job, and you have just done that job by hand, so the bootstrap script is not run here or anywhere in this book. And the file creates pg_cron with IF NOT EXISTS and schedules two jobs, which is why Step 5 created the extension and granted cron first.

Run it as minimalist, from app-1 or from db-1 over TCP, so that the run itself proves the hostssl line, the password and the grants all hold:

PGPASSWORD='<minimalist-password>' psql \
  "host=10.0.0.10 dbname=minimal user=minimalist sslmode=verify-full sslrootcert=/etc/ssl/certs/internal-ca.pem" \
  -v ON_ERROR_STOP=1 -q -f postgres.sql

The two already exists notices for the extensions are expected. Then count what landed and read the two jobs back:

PGPASSWORD='<minimalist-password>' psql "host=10.0.0.10 dbname=minimal user=minimalist sslmode=verify-full sslrootcert=/etc/ssl/certs/internal-ca.pem" \
  -Atc "select count(*) from information_schema.tables where table_schema='public';
        select jobname, schedule from cron.job order by 1;"

42
create_audit_partition|0 2 * * *
drop_old_audit_partitions|0 3 * * *

Forty-two tables is the number for the schema as of this writing; the count you get is the one to record. The two jobs are Minimal's audit-partition maintenance from Volume II, Chapter 6, and they are the reason pg_cron is in this cluster at all.

Zitadel's schema is not applied here. Chapter 3's zitadel init zitadel step creates it, as the zitadel role, into the empty database this chapter created — that is the whole point of having pre-provisioned the role and database: the identity provider never needs a superuser.

The healthyme database stays empty until Chapter 6, where healthyme_app creates the first table.

Step 7 — Prove the isolation

Every one of these must fail, and the message tells you which layer refused it:

for t in "zitadel healthyme" "healthyme_app zitadel" "minimalist zitadel"; do
  set -- $t
  PGPASSWORD="<its-password>" psql "host=10.0.0.10 dbname=$2 user=$1 sslmode=require" -Atc "select 1"
done

FATAL:  no pg_hba.conf entry for host "10.0.0.20", user "zitadel", database "healthyme", SSL encryption
FATAL:  no pg_hba.conf entry for host "10.0.0.20", user "healthyme_app", database "zitadel", SSL encryption
FATAL:  no pg_hba.conf entry for host "10.0.0.20", user "minimalist", database "zitadel", SSL encryption

Those refusals come from pg_hba.conf; the address in the message is whichever one the connection came from. To see the second layer, add a permissive line temporarily and watch REVOKE CONNECT refuse instead — FATAL: permission denied for database "zitadel" — then take the line out again. And a plaintext connection must fail even for a pair that is allowed:

PGPASSWORD='<minimalist-password>' psql "host=10.0.0.10 dbname=minimal user=minimalist sslmode=disable" -Atc "select 1"

FATAL:  no pg_hba.conf entry for host "10.0.0.20", user "minimalist", database "minimal", no encryption

Finally, once Chapters 3 and 6 have their services running, this query on db-1 is the standing check that every application session is actually encrypted:

sudo -u postgres psql -Atc "select a.usename, a.datname, s.ssl from pg_stat_ssl s
  join pg_stat_activity a using (pid) where a.usename <> 'postgres' group by 1,2,3"

minimalist|minimal|t
zitadel|zitadel|t

A row with f in the last column is a service whose connection string lost its sslmode.

Backups, and the one thing about Zitadel's

pg_dump per database, as the superuser on the local socket, on a timer, to storage off db-1, is enough for this deployment's size and is what the rest of this book assumes:

sudo -u postgres pg_dump -Fc zitadel   > /var/backups/postgres/zitadel.$(date +%F).dump
sudo -u postgres pg_dump -Fc minimal   > /var/backups/postgres/minimal.$(date +%F).dump
sudo -u postgres pg_dump -Fc healthyme > /var/backups/postgres/healthyme.$(date +%F).dump

Zitadel is event-sourced: the table that matters is eventstore.events2, and every projection can be rebuilt from it. Its production guidance says to back that table up and treat the rest as derivable; a whole-database dump covers it and is simpler to restore. Minimal's store is encrypted at the column level by the key Chapter 6 configures — a dump without that key restores rows nobody can read, so the key is part of the backup set, kept apart from the dumps.

What dev does differently

On the Mac, the cluster already exists: Homebrew's Postgres 18 on port 5433, with pg_cron already preloaded and cron.database_name already minimal, and a minimalist role that already owns the minimal store. Chapter 9 adds only the zitadel and healthyme_app roles and their two databases, with the same REVOKE CONNECT lines, and leaves pg_hba.conf as it is — the Mac's cluster listens on loopback only and is trusted locally, which is the Volume I, Chapter 2 argument about a machine only you can reach. Everything in Steps 4 and 5 runs unchanged there; Steps 2, 3 and 7 are production's.

Is the cluster ready? A closing checklist

• Three roles, three databases, three owners, and \du shows no role but postgres with Superuser, Create DB or Create role.
• pg_hba.conf has no all on any TCP line and no host line at all — only hostssl, one per pair, each naming app-1's private address.
• PUBLIC cannot connect to any of the three databases, shown by the permission denied for database refusal, not assumed from the REVOKE having run.
• cron.database_name is minimal and the two jobs exist in cron.job.
• The server certificate is one the clients can verify, and Chapters 3 and 6 use verify-full. If it is still the snakeoil pair, that is written down somewhere someone will read.
• pg_stat_ssl shows t for every application session, once there are any.
• The dumps run, and a restore of one of them into a scratch database has actually been done once. A backup that has never been restored is a hypothesis.

Exercises

1. Add a fourth role and database for a second app, following Steps 3 to 5 exactly, and run Step 7 for every pair of the four roles against the four databases. Sixteen combinations; count how many should succeed before you run them, then check.

2. Deliberately misspell the database name on one hostssl line, restart, and try to connect as that role. Record the exact message. Then fix the line but revoke the role's CONNECT instead, and record that message. You should be able to tell the two layers apart from the text alone.

3. Run the schema file a second time against minimal. Note which statements succeed silently, which raise a notice, and whether the two cron.schedule calls create duplicate jobs. Decide from that whether "re-run the schema file" is a safe recovery step on this cluster, and write down the answer where the on-call person will find it.

Zitadel v4 is two processes. The first is a single Go binary that serves every API, the OpenID Connect endpoints and the management console. The second is the login: a Next.js application that renders every screen a person sees while signing in, talks to the first process over its API, and is required on a v4 instance unless you switch it off. The release page ships both — a Linux binary and a zitadel-login.tar.gz bundle that runs on Node — so neither needs a container. This chapter installs both on app-1 as systemd units, bound to loopback, under their own users, with the identity provider's database role and the two service accounts everything later depends on.

The reverse proxy that puts a public name in front of them is Chapter 5. Until then Zitadel is reached only from app-1 itself, and only with a Host header that names its public domain, because Zitadel resolves which instance a request belongs to from that header and answers 404 to any other.

How Zitadel starts, and why the order matters

The binary has three phases, each its own subcommand, and the flags they take decide what runs as whom:

Chapter 2 already created the role and the empty database, so the superuser phase is never run. The docs call the schema-only step init schema; in v4.17.3 the subcommand is init zitadel, and its help text says "initialize ZITADEL internals". It is the one this chapter uses.

Two facts about setup cost time when they are not known in advance. The first instance — its name, its organisation, its admin user, the service accounts and where their tokens are written — is configured in a steps file passed with --steps, not in the configuration file passed with --config. A FirstInstance: block in the config file is read as nothing, and setup silently creates the default instance named ZITADEL with zitadel-admin@zitadel.localhost and the password Password1!. And the token files setup writes are written by the user setup runs as, into the path you name, so that path has to be writable by that user — a directory owned by root is not.

Step 1 — Users, directories, binaries

sudo useradd --system --home-dir /var/lib/zitadel       --create-home --shell /usr/sbin/nologin zitadel
sudo useradd --system --home-dir /var/lib/zitadel-login --create-home --shell /usr/sbin/nologin zitadel-login
sudo install -d -m 0750 -o root    -g zitadel /etc/zitadel
sudo install -d -m 0750 -o zitadel -g zitadel /var/lib/zitadel
sudo install -d -m 0755 /opt/zitadel-login

Fetch the release, verify it against the release's own checksum file, and install:

V=v4.17.3; ARCH=amd64   # arm64 on an ARM host
cd /tmp
curl -fsSLO https://github.com/zitadel/zitadel/releases/download/$V/zitadel-linux-$ARCH.tar.gz
curl -fsSLO https://github.com/zitadel/zitadel/releases/download/$V/zitadel-login.tar.gz
curl -fsSLO https://github.com/zitadel/zitadel/releases/download/$V/checksums.txt
grep -E "zitadel-linux-$ARCH.tar.gz|zitadel-login.tar.gz" checksums.txt | sed 's#.artifacts/pack/##' | sha256sum -c -
tar xzf zitadel-linux-$ARCH.tar.gz
sudo install -m 0755 zitadel-linux-$ARCH/zitadel /usr/local/bin/zitadel
sudo tar xzf zitadel-login.tar.gz -C /opt/zitadel-login
sudo chown -R root:root /opt/zitadel-login
zitadel --version

zitadel-linux-amd64.tar.gz: OK
zitadel-login.tar.gz: OK
zitadel version v4.17.3

The checksum file lists paths under .artifacts/pack/, which is why the sed strips that prefix before sha256sum looks for the files. The login bundle unpacks to apps/login/server.js, a node_modules directory, public/, and an entrypoint.sh that reads the token file — it is the standalone output of a Next.js build, and it runs with nothing but Node.

Node 22 from the NodeSource repository, since Debian 12's own Node is too old for the bundle:

curl -fsSL https://deb.nodesource.com/setup_22.x | sudo bash -
sudo apt install -y nodejs
node -v

v22.23.2

Step 2 — The masterkey and the configuration file

The masterkey is a 32-character secret Zitadel uses to encrypt what it stores — keys, secrets, the personal access tokens it issues. It cannot be changed once the instance exists, so generate it once, keep it in the backup set with the database dumps, and never print it:

sudo sh -c "head -c 32 /dev/urandom | base64 | tr -dc 'A-Za-z0-9' | head -c 32 > /etc/zitadel/masterkey"
sudo chown root:zitadel /etc/zitadel/masterkey && sudo chmod 0640 /etc/zitadel/masterkey

The configuration file names the public domain, the internal port, the database, and switches the login requirement on with the public URL of the login. Every key here has an environment-variable twin, but a file the zitadel group can read and nobody else is the better place for a password:

sudo tee /etc/zitadel/config.yaml >/dev/null <<'EOF'
Log:
  Level: info

Port: 8081                    # loopback; APISIX is the only thing that connects
ExternalDomain: auth.example.com
ExternalPort: 443
ExternalSecure: true          # the public side is HTTPS, terminated by APISIX
TLS:
  Enabled: false              # ... and the hop from APISIX to this port is plain h2c

Database:
  postgres:
    Host: 10.0.0.10
    Port: 5432
    Database: zitadel
    MaxOpenConns: 10
    MaxIdleConns: 5
    User:
      Username: zitadel
      Password: <zitadel-password>
      SSL:
        Mode: verify-full
        RootCert: /etc/ssl/certs/internal-ca.pem

DefaultInstance:
  Features:
    LoginV2:
      Required: true
      BaseURI: "https://auth.example.com/ui/v2/login"
  OIDCSettings:
    AccessTokenLifetime: 1h
    IdTokenLifetime: 1h
    RefreshTokenIdleExpiration: 720h
    RefreshTokenExpiration: 2160h
EOF
sudo chown root:zitadel /etc/zitadel/config.yaml && sudo chmod 0640 /etc/zitadel/config.yaml

ExternalDomain, ExternalPort and ExternalSecure are recorded into the instance at setup. They are the issuer every token carries and the host every request must name; changing them later is an instance-domain migration, not a config edit. Get them right here. SSL.Mode: verify-full is the Chapter 2 server certificate doing its job; with the snakeoil pair it has to be require, and a disable here fails at start against Chapter 2's hostssl-only pg_hba.conf with no pg_hba.conf entry ... no encryption.

The OIDCSettings block shortens the access token from Zitadel's default of twelve hours to one. The gateway in Chapter 5 validates tokens locally against the signing keys and never asks Zitadel whether a given token has been revoked, so a token's lifetime is the longest a revoked account keeps working. One hour with a ninety-day refresh token is the usual trade for a phone app.

Step 3 — The steps file: first instance, admin, two service accounts

sudo tee /etc/zitadel/steps.yaml >/dev/null <<'EOF'
FirstInstance:
  InstanceName: littlebit
  DefaultLanguage: en
  # Written by the setup step, which runs as the zitadel user; moved to /etc/zitadel in Step 4.
  PatPath: /var/lib/zitadel/provisioner.pat
  LoginClientPatPath: /var/lib/zitadel/login-client.pat
  Org:
    Name: Littlebit
    Human:
      UserName: admin
      FirstName: Instance
      LastName: Admin
      Email:
        Address: admin@example.com
        Verified: true
      Password: "<a strong initial password>"
      PasswordChangeRequired: true
    Machine:
      Machine:
        Username: provisioner
        Name: Provisioning service account
      Pat:
        ExpirationDate: "2030-01-01T00:00:00Z"
    LoginClient:
      Machine:
        Username: login-client
        Name: Login V2 client
      Pat:
        ExpirationDate: "2030-01-01T00:00:00Z"
EOF
sudo chown root:zitadel /etc/zitadel/steps.yaml && sudo chmod 0640 /etc/zitadel/steps.yaml

Three accounts come out of this, and the rest of the book uses each for one thing:

• admin, a human in the first organisation with the instance-owner role. It is for the console, by a person, and for nothing automated. PasswordChangeRequired: true makes the first console login set a real password.
• provisioner, a machine account with the instance-owner role and a personal access token written to provisioner.pat. Chapter 4 uses it for every API call that creates an organisation, a project, an app; Chapter 8's command-line tool is built on it. It can do anything on the instance, which is why its token lives in a root-only file and is never given to a running service.
• login-client, a machine account whose token the Login V2 process uses to drive the session API on people's behalf. The steps file's LoginClient block is the documented way to create it, and the login works with the token it writes; Zitadel's own compose file creates it the same way.

The first organisation, Littlebit, is where these accounts live. It is not an app's organisation and no app user is ever created in it.

Step 4 — Schema, then setup

Both as the zitadel user, so that nothing about this ever needed the superuser:

sudo -u zitadel zitadel init zitadel --config /etc/zitadel/config.yaml
sudo -u zitadel zitadel setup --config /etc/zitadel/config.yaml --steps /etc/zitadel/steps.yaml \
  --masterkeyFile /etc/zitadel/masterkey --tlsMode external --init-projections

The first command ends on verify unique constraints; the second, after a page of projection starts prefilling lines, on setup completed. --tlsMode external is the combination the config file already states — HTTPS outside, plain inside — and the flag overrides the file, so say it on every phase and never disagree with yourself. --init-projections fills the read models during setup rather than on first start, which makes the first start fast.

If setup fails part-way and you run it again, it fails again with Errors.Instance.Domain.AlreadyExists, because the instance was created before whatever failed. The clean recovery at this stage, before the instance holds anything, is to drop and recreate the database as Chapter 2 did and run both commands again. Do not reach for setup cleanup here; it is for stuck migrations on a live instance.

Now move the two tokens where they belong. The login process needs to read its own; the provisioner's is for a person or a tool with root on this host:

sudo install -o root -g zitadel-login -m 0640 /var/lib/zitadel/login-client.pat /etc/zitadel/login-client.pat
sudo install -o root -g root          -m 0600 /var/lib/zitadel/provisioner.pat  /etc/zitadel/provisioner.pat
sudo rm /var/lib/zitadel/*.pat
sudo ls -la /etc/zitadel

-rw-r-----  1 root zitadel        467 config.yaml
-rw-r-----  1 root zitadel-login  278 login-client.pat
-rw-r-----  1 root zitadel         32 masterkey
-rw-------  1 root root           278 provisioner.pat
-rw-r-----  1 root zitadel        828 steps.yaml

And confirm what setup created, straight from the projections, before starting anything:

sudo -u postgres psql -h 10.0.0.10 -d zitadel -Atc "select name from projections.instances;
  select username, type from projections.users14;"

littlebit
provisioner|2
admin@example.com|1
login-client|2

Type 1 is a human, type 2 a machine. Three users, one instance, none of them the default.

Step 5 — Two units

sudo tee /etc/systemd/system/zitadel.service >/dev/null <<'EOF'
[Unit]
Description=Zitadel identity server
After=network-online.target
Wants=network-online.target

[Service]
User=zitadel
Group=zitadel
WorkingDirectory=/var/lib/zitadel
ExecStart=/usr/local/bin/zitadel start --config /etc/zitadel/config.yaml --masterkeyFile /etc/zitadel/masterkey --tlsMode external
Restart=on-failure
RestartSec=5
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
ReadWritePaths=/var/lib/zitadel

[Install]
WantedBy=multi-user.target
EOF

The login reads its settings from an environment file. The names below are the ones the bundle's server.js and entrypoint.sh actually read: ZITADEL_API_URL for where the API is, and ZITADEL_SERVICE_USER_TOKEN_FILE for the token. The second is read by entrypoint.sh, which exports the file's contents as ZITADEL_SERVICE_USER_TOKEN and then runs whatever command follows it — so the unit starts the login through that script. Started as a bare node apps/login/server.js, the process comes up, answers its health route, and fails every sign-in with Flow initiation failed, because it never had a token and never tried to reach the API; that cost an hour in writing this chapter.

sudo tee /etc/zitadel/login.env >/dev/null <<'EOF'
NODE_ENV=production
PORT=3000
HOSTNAME=127.0.0.1
ZITADEL_API_URL=http://127.0.0.1:8081
ZITADEL_SERVICE_USER_TOKEN_FILE=/etc/zitadel/login-client.pat
EOF
sudo chown root:zitadel-login /etc/zitadel/login.env && sudo chmod 0640 /etc/zitadel/login.env

sudo tee /etc/systemd/system/zitadel-login.service >/dev/null <<'EOF'
[Unit]
Description=Zitadel Login V2
After=network-online.target zitadel.service
Wants=network-online.target

[Service]
User=zitadel-login
Group=zitadel-login
WorkingDirectory=/opt/zitadel-login
EnvironmentFile=/etc/zitadel/login.env
ExecStart=/opt/zitadel-login/entrypoint.sh /usr/bin/node apps/login/server.js
Restart=on-failure
RestartSec=5
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now zitadel zitadel-login
sleep 5; systemctl is-active zitadel zitadel-login

active
active

HOSTNAME=127.0.0.1 is what makes the login bind loopback; without it Next.js listens on every interface. The bundle's own prod script sets the same variable, and the Debian firewall is not what should be keeping port 3000 private.

How the login tells Zitadel which instance it is calling about is worth one paragraph, because it is not the Host header. On every API call the login sends x-zitadel-instance-host and x-zitadel-public-host, both copied from the Host of the browser request it is serving — which, through Chapter 5's gateway, is auth.example.com. Zitadel's configuration lists exactly those two header names, under InstanceHostHeaders and PublicHostHeaders, as the ones it reads first when resolving an instance. So the login needs no Host override of its own; it needs the gateway to deliver the public Host to it, which pass_host: pass in Chapter 5 does. (The bundle also honours a CUSTOM_REQUEST_HEADERS variable for adding headers to its API calls. Do not use it to set Host: Node's HTTP client replaces that header with the connection's own address, and the setting is silently ineffective.)

Step 6 — Prove it answers, and only for its own name

curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:8081/.well-known/openid-configuration
curl -s -H "Host: auth.example.com" http://127.0.0.1:8081/.well-known/openid-configuration | python3 -c "import sys,json; print(json.load(sys.stdin)['issuer'])"
curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:3000/ui/v2/login/healthy

404
https://auth.example.com
200

The first line is the fact this chapter opened with: without a Host that names the instance, Zitadel has no instance to serve and says so with 404, logging unable to set instance beside it. Chapter 5's gateway passes the public Host through unchanged, which is what turns that 404 into a working identity provider. The second line is the issuer every token will carry, and it is the string Chapter 5's gateway will demand. The third is the login process's own health route, which the gateway can also use.

On db-1, Chapter 2's standing check now has its first row:

zitadel|zitadel|t

The console, once the gateway exists

The management console is served by the same process at /ui/console and reached, from Chapter 5 onward, at https://auth.example.com/ui/console. Sign in there once as admin, set the real password, and turn on two things the API cannot: the SMTP settings under the instance's notification settings, so that verification and recovery emails can be sent, and the instance's branding if you want any. Nothing in Chapter 4 requires the console — every step there is an API call with the provisioner's token — but a person who wants to see what those calls made will find it all there.

What dev does differently

Chapter 9 runs both binaries from a terminal on the Mac against the Homebrew cluster, with ExternalDomain: localhost, ExternalPort: 8080, ExternalSecure: false and --tlsMode disabled, and the login started as node apps/login/server.js with the same six variables exported. The steps file is the same shape, and its PatPath values point somewhere the terminal user can write. The one trap that is the same on both: a config file with a FirstInstance: block and no --steps gives you the ZITADEL instance with Password1!, quietly.

Is Zitadel ready? A closing checklist

• Setup created your instance, not ZITADEL, shown by projections.instances, and the three users are admin, provisioner and login-client.
• The masterkey is backed up with the database dumps and nowhere else.
• provisioner.pat is root-only and has never been pasted into a service's environment.
• Both units are active after a reboot, and both bind loopback: ss -ltn shows 127.0.0.1:3000 and [::]:8081 or 127.0.0.1:8081, and no public address.
• The database session is TLS — pg_stat_ssl says t for zitadel.
• The issuer is exactly the public URL, https://auth.example.com, with no port and no path.
• Access token lifetime is deliberate, not the twelve-hour default.

Exercises

1. Stop zitadel-login and run through Step 6 again. The API still answers; what does a person see if they start a sign-in? Then stop zitadel and start only zitadel-login; read its journal. Write down which of the two a monitoring check has to watch to know that sign-in works.

2. Run zitadel setup a second time on the finished instance, with the same flags. Record what it prints and whether anything changed in projections.users14. Then read the help for setup cleanup and say in one sentence when you would use it.

3. Change AccessTokenLifetime in the config file to 5m, restart, and check whether tokens issued afterwards actually carry a five-minute exp — the setting is under DefaultInstance, and the instance already exists. If they do not, find the instance-level API or console setting that does apply to a running instance, and note which of the two places is the one that matters after day one.

Everything an app needs from the identity provider is created in this chapter, with the provisioner's token, as API calls — eleven of them, in an order that matters. By the end, a person can register in Healthy Me with Google, GitHub or Apple and nothing else, gets a server-assigned ULID on their very first token, and cannot use that account to sign in to any other app you add later. The console shows every object afterwards; none of it needs the console to be made.

The calls below are shown against production, https://auth.example.com, through the gateway that Chapter 5 builds. If you are reading straight through, run them from app-1 against http://127.0.0.1:8081 with -H "Host: auth.example.com" added to every one; that is exactly how they were verified, and it is what Chapter 9 does on the Mac with Host: localhost:8080.

Zitadel's API comes in two generations and this chapter uses both. Organisations, users and their tokens are v2 resources: JSON under /v2/..., or, for the newest services, a Connect-RPC style POST /zitadel.<service>.v2.<Service>/<Method> that takes the same JSON. Login policies, identity providers, project roles, role grants and actions are still v1 management resources under /management/v1/..., and every v1 call needs an x-zitadel-orgid header naming the organisation it acts in. The v2 calls carry the organisation in the body instead.

Z=https://auth.example.com
PAT=$(sudo cat /etc/zitadel/provisioner.pat)
H=(-H "Authorization: Bearer $PAT" -H "Content-Type: application/json")

Step 1 — The organisation

curl -s "${H[@]}" -X POST $Z/v2/organizations -d '{"name":"Healthy Me"}'

{
  "details": {
    "sequence": "4",
    "changeDate": "2026-09-14T03:47:03.862275Z",
    "resourceOwner": "390676976144220428"
  },
  "organizationId": "390676976144220428"
}

Keep the id; every later call needs it. ORG=390676976144220428 below.

This organisation is the app. Its users are the app's users; its login policy decides how they sign in; its identity providers are the ones the app offers. A person who registers here does not exist in any other organisation on the instance, which is what "one organisation per app" buys and what Chapter 8 demonstrates with a second one.

Step 2 — The project, and the two settings that keep other apps' users out

curl -s "${H[@]}" -X POST $Z/zitadel.project.v2.ProjectService/CreateProject \
  -d "{\"organizationId\":\"$ORG\",\"name\":\"Healthy Me\",\"projectRoleAssertion\":true,
       \"authorizationRequired\":true,\"projectAccessRequired\":true}"

{
  "projectId": "390677090732605708",
  "creationDate": "2026-09-14T03:48:12.162217Z"
}

PROJ=390677090732605708. The three booleans are the whole reason to read this step rather than skim it:

• projectRoleAssertion puts the person's roles on the token, under two claims named urn:zitadel:iam:org:project:roles and urn:zitadel:iam:org:project:<PROJ>:roles. Chapter 5's gateway reads the first.
• authorizationRequired refuses to issue a token to a person who holds no role in this project.
• projectAccessRequired refuses to issue a token to a person whose organisation has not been granted this project.

Without the last two, any user on the instance can complete a sign-in against this app. That was tested: with both off, a Healthy Me user obtained a valid token for a second app in a different organisation; with both on, the same attempt is refused at the moment the login finishes, with Errors.User.ProjectRequired. A project created earlier without them is fixed with an update:

curl -s "${H[@]}" -X POST $Z/zitadel.project.v2.ProjectService/UpdateProject \
  -d "{\"projectId\":\"$PROJ\",\"authorizationRequired\":true,\"projectAccessRequired\":true}"

Step 3 — Roles

Roles are strings on a project. Minimal's permission templates in Chapter 6 name the same strings, and APISIX in Chapter 5 copies them verbatim into X-User-Roles, so choose them once, here:

for r in member:Member premium:Premium; do
  curl -s "${H[@]}" -H "x-zitadel-orgid: $ORG" -X POST $Z/management/v1/projects/$PROJ/roles \
    -d "{\"roleKey\":\"${r%%:*}\",\"displayName\":\"${r##*:}\",\"group\":\"healthyme\"}"
done

Each answers with a details object and a rising sequence. A person with no role at all is refused by authorizationRequired, so every registered person has to hold at least member; Step 8 arranges that automatically.

Step 4 — The native application

One application per platform gives each its own client id and its own redirect list, which is worth having the day one platform's build is compromised and you want to revoke it alone. The book creates one, for iOS, and Android and Mac are the same call with a different name and scheme:

curl -s "${H[@]}" -H "x-zitadel-orgid: $ORG" -X POST $Z/zitadel.application.v2.ApplicationService/CreateApplication -d "{
  \"projectId\": \"$PROJ\",
  \"name\": \"Healthy Me iOS\",
  \"oidcConfiguration\": {
    \"redirectUris\": [\"com.littlebit.healthyme://auth/callback\"],
    \"postLogoutRedirectUris\": [\"com.littlebit.healthyme://auth/signedout\"],
    \"responseTypes\": [\"OIDC_RESPONSE_TYPE_CODE\"],
    \"grantTypes\": [\"OIDC_GRANT_TYPE_AUTHORIZATION_CODE\", \"OIDC_GRANT_TYPE_REFRESH_TOKEN\"],
    \"applicationType\": \"OIDC_APP_TYPE_NATIVE\",
    \"authMethodType\": \"OIDC_AUTH_METHOD_TYPE_NONE\",
    \"version\": \"OIDC_VERSION_1_0\",
    \"developmentMode\": false,
    \"accessTokenType\": \"OIDC_TOKEN_TYPE_JWT\",
    \"accessTokenRoleAssertion\": true,
    \"idTokenRoleAssertion\": false,
    \"idTokenUserinfoAssertion\": false,
    \"skipNativeAppSuccessPage\": true,
    \"loginVersion\": {\"loginV2\": {\"baseUri\": \"https://auth.example.com/ui/v2/login\"}}
  }}"

{
  "applicationId": "390677214766563596",
  "creationDate": "2026-09-14T03:49:26.090323Z",
  "oidcConfiguration": {
    "clientId": "390677214766629132"
  }
}

CLIENT=390677214766629132. There is no client secret in the answer and there should not be: a native app is a public client, authMethodType NONE says so, and PKCE is what protects the code exchange. Three settings are load-bearing:

• accessTokenType: OIDC_TOKEN_TYPE_JWT. Zitadel's default access token is an opaque string that only Zitadel's introspection endpoint can read. A JWT is what lets the gateway verify every request locally against the published keys, with no call to Zitadel on the request path. Leave this at the default and Chapter 5 fails on every request with token is signed by unexpected algorithm.
• accessTokenRoleAssertion: true puts the roles on the access token — the token the gateway sees — not only on the id token the app keeps.
• developmentMode: false refuses http:// redirect URIs. Chapter 9's dev app turns it on so that a local callback can be http://localhost:4200/auth/callback; production never does.

The redirect scheme is the app's bundle identifier reversed, which is what iOS, Android and macOS all accept for a custom scheme. skipNativeAppSuccessPage sends the person straight back to the app after sign-in instead of showing a "you may close this window" page first.

Step 5 — The login policy: sign-in only through a provider

A new organisation inherits the instance's default policy, which allows usernames and passwords. Give Healthy Me its own:

curl -s "${H[@]}" -H "x-zitadel-orgid: $ORG" -X POST $Z/management/v1/policies/login -d '{
  "allowUsernamePassword": false,
  "allowRegister": true,
  "allowExternalIdp": true,
  "forceMfa": false,
  "passwordlessType": "PASSWORDLESS_TYPE_NOT_ALLOWED",
  "hidePasswordReset": true,
  "ignoreUnknownUsernames": true,
  "allowDomainDiscovery": false,
  "disableLoginWithEmail": false,
  "disableLoginWithPhone": true,
  "passwordCheckLifetime": "864000s",
  "externalLoginCheckLifetime": "864000s",
  "mfaInitSkipLifetime": "2592000s",
  "secondFactorCheckLifetime": "64800s",
  "multiFactorCheckLifetime": "43200s"
}'

The five lifetimes are protobuf durations and have to be written in seconds with an s — "240h" is refused with invalid google.protobuf.Duration value. allowRegister: true is what lets a person who has never been seen create an account by signing in with a provider; with allowUsernamePassword: false there is no other way in. Chapter 9's dev organisation sets that one true so that a password user can exercise the whole path without a provider account.

Read it back with GET $Z/management/v1/policies/login and check isDefault is now false.

Step 6 — Identity providers

Each provider is created on the organisation, then switched on in the login policy. The options block is the same for all three and it is where the "register by signing in" behaviour actually lives: isCreationAllowed and isAutoCreation together create the Zitadel user on first sign-in from the provider's claims, isAutoUpdate refreshes name and email on later sign-ins, and isLinkingAllowed: false refuses to attach a provider identity to an existing account — which, with passwords off, cannot happen anyway, but say so.

Google. In the Google Cloud console, create an OAuth client of type Web application — the sign-in happens in a browser, so from Google's side this is a web client even though your app is native — and give it the authorized redirect URI https://auth.example.com/idps/callback. That is Zitadel's own callback handler, served by the API process under /idps, and it is the same URL for every provider on the instance; the console shows it on each provider's settings page, and it is the one to copy rather than the older /ui/login/login/externalidp/callback form that belongs to the retired login. Then:

curl -s "${H[@]}" -H "x-zitadel-orgid: $ORG" -X POST $Z/management/v1/idps/google -d '{
  "name": "Google",
  "clientId": "<client-id>.apps.googleusercontent.com",
  "clientSecret": "<client-secret>",
  "scopes": ["openid", "profile", "email"],
  "providerOptions": {"isLinkingAllowed": false, "isCreationAllowed": true,
                      "isAutoCreation": true, "isAutoUpdate": true,
                      "autoLinking": "AUTO_LINKING_OPTION_UNSPECIFIED"}}'

GitHub. Under Developer settings → OAuth Apps, register an app whose authorization callback URL is the same Zitadel callback, and generate a client secret. GitHub does not standardise the profile fields it returns, so first and last name may arrive empty; email arrives if the person has a public one or the user:email scope is granted.

curl -s "${H[@]}" -H "x-zitadel-orgid: $ORG" -X POST $Z/management/v1/idps/github -d '{
  "name": "GitHub",
  "clientId": "<client-id>",
  "clientSecret": "<client-secret>",
  "scopes": ["openid", "profile", "email"],
  "providerOptions": {"isLinkingAllowed": false, "isCreationAllowed": true,
                      "isAutoCreation": true, "isAutoUpdate": true,
                      "autoLinking": "AUTO_LINKING_OPTION_UNSPECIFIED"}}'

Apple. Three objects in the Apple Developer account: an App ID with Sign in with Apple enabled; a Services ID that is the client id, configured with your domain and the same Zitadel callback as its return URL; and a key with Sign in with Apple enabled, downloaded once as a .p8 file. The request carries the key's contents base64-encoded:

curl -s "${H[@]}" -H "x-zitadel-orgid: $ORG" -X POST $Z/management/v1/idps/apple -d "{
  \"name\": \"Apple\",
  \"clientId\": \"com.littlebit.healthyme.signin\",
  \"teamId\": \"<team-id>\",
  \"keyId\": \"<key-id>\",
  \"privateKey\": \"$(base64 -w0 AuthKey_<key-id>.p8)\",
  \"scopes\": [\"name\", \"email\"],
  \"providerOptions\": {\"isLinkingAllowed\": false, \"isCreationAllowed\": true,
                        \"isAutoCreation\": true, \"isAutoUpdate\": true,
                        \"autoLinking\": \"AUTO_LINKING_OPTION_UNSPECIFIED\"}}"

Apple requires HTTPS and a domain it has verified; there is no way to exercise it against localhost, which is why Chapter 9's dev organisation keeps passwords on. Apple also returns the person's name only on the very first authorisation; if that sign-in fails part-way, the name is gone until the person removes the app from their Apple ID and starts again.

Each of the three answers with an id. Switch each on for the organisation:

for ID in <google-id> <github-id> <apple-id>; do
  curl -s "${H[@]}" -H "x-zitadel-orgid: $ORG" -X POST $Z/management/v1/policies/login/idps \
    -d "{\"idpId\":\"$ID\",\"ownerType\":\"IDP_OWNER_TYPE_ORG\"}"
done
curl -s "${H[@]}" -H "x-zitadel-orgid: $ORG" $Z/management/v1/policies/login \
  | python3 -c "import sys,json; p=json.load(sys.stdin)['policy']; print([i['idpName'] for i in p['idps']], p['allowUsernamePassword'])"

['Google', 'GitHub', 'Apple'] False

The three creation calls and the activation were run with placeholder credentials to confirm the request shapes, and the three buttons then appeared on the login page; Zitadel does not contact the provider when the provider is created, only when a person signs in with it. What was not run in writing this book is a sign-in through a real Google, GitHub or Apple account, because the verification environment has no public domain. The callback path is read from Zitadel's source for this version; check the URL the console shows on the provider's page before you paste it into Google or Apple.

Step 7 — The action that assigns the server's user id

This is the twenty lines Chapter 1 promised. Zitadel runs it every time it is about to mint an access token for anyone in this organisation. It reads the person's metadata; if there is no minimal_user_id yet, it mints a ULID and stores one; either way it puts the value on the token as a claim of that name.

/**
 * Complement Token, trigger "Pre access token creation".
 * Gives every user a server-assigned ULID the first time a token is minted for
 * them, keeps it in user metadata, and asserts it as a claim on every token.
 */
function minimalUserId(ctx, api) {
  var KEY = 'minimal_user_id';
  var existing = null;
  var md = ctx.v1.user.getMetadata();
  if (md && md.metadata) {
    for (var i = 0; i < md.metadata.length; i++) {
      if (md.metadata[i].key === KEY) { existing = md.metadata[i].value; break; }
    }
  }
  if (!existing) {
    existing = ulid();
    api.v1.user.setMetadata(KEY, existing);
  }
  api.v1.claims.setClaim(KEY, existing);
}

function ulid() {
  var A = '0123456789ABCDEFGHJKMNPQRSTVWXYZ';
  var t = Date.now();
  var out = '';
  for (var i = 9; i >= 0; i--) { out = A.charAt(t % 32) + out; t = Math.floor(t / 32); }
  for (var j = 0; j < 16; j++) { out += A.charAt(Math.floor(Math.random() * 32)); }
  return out;
}

Save it as minimal-user-id.js and register it, then bind it to the trigger. Actions are per organisation; flow type 2 is "Complement Token" and trigger type 5 is "Pre access token creation", the two numbers Zitadel's own flow listing gives back:

python3 -c "import json; print(json.dumps({'name':'minimalUserId','script':open('minimal-user-id.js').read(),'timeout':'5s','allowedToFail':False}))" > action.json
curl -s "${H[@]}" -H "x-zitadel-orgid: $ORG" -X POST $Z/management/v1/actions -d @action.json

{
  "details": {
    "sequence": "1",
    "creationDate": "2026-09-14T03:50:02.513180Z",
    "resourceOwner": "390676976144220428"
  },
  "id": "390677275885961484"
}

curl -s "${H[@]}" -H "x-zitadel-orgid: $ORG" -X POST $Z/management/v1/flows/2/trigger/5 \
  -d '{"actionIds":["390677275885961484"]}'
curl -s "${H[@]}" -H "x-zitadel-orgid: $ORG" $Z/management/v1/flows/2 | head -c 300

{"flow":{"type":{"id":"2","name":{"key":"Action.Flow.Type.CustomiseToken","localizedMessage":"Complement Token"}},...,"triggerActions":[{"triggerType":{"id":"5","name":{"key":"Action.TriggerType.PreAccessTokenCreation","localizedMessage":"Pre access token creation"}},"actions":[{"id":"390677275885961484",...,"name":"minimalUserId"

Four things the script's shape settles, because each was the alternative:

• It runs at token time, not at registration. Zitadel has a "post creation" trigger on the external-authentication flow, but that flow belongs to the old login and does not fire under Login V2. Token creation fires for every login on every version. Minting lazily also means a user created any other way — by an administrator, by an import — gets an id the first time they sign in.
• allowedToFail: false. If the action throws, no token is issued. A token without the claim would reach the gateway, which refuses it with 403 anyway; failing at the source is clearer.
• The metadata value is JSON-encoded. api.v1.user.setMetadata stores the string with its quotes, so reading it back through the metadata API returns "01M2..." base64-encoded with the quotes inside. The claim on the token is the bare 26 characters, which is what the gateway reads; the metadata is the durable copy and is never parsed by anything but this script.
• Math.random is enough. A ULID's random half exists to make ids unique, not secret; the server-assigned id is not a credential, it is a name. The 48-bit timestamp half makes ids from the same millisecond collide only if their 80 random bits match.

Actions V1 is the mechanism that runs JavaScript inside Zitadel. It still works in v4 and is on a deprecation path for v5, where Actions V2 — an HTTP call to an endpoint you host, with the same preaccesstoken trigger — replaces it. The migration is the same script behind an HTTP endpoint; nothing in the gateway or in Minimal changes.

Step 8 — Every new person gets member

authorizationRequired refuses a token to anyone with no role, so someone has to grant member to each person the provider just created. In this book that is a grant per user, made by whatever creates the user:

curl -s "${H[@]}" -H "x-zitadel-orgid: $ORG" -X POST $Z/management/v1/users/<user-id>/grants \
  -d "{\"projectId\":\"$PROJ\",\"roleKeys\":[\"member\"]}"

For provider sign-ins nothing of yours creates the user, so the grant has to be automatic. Zitadel offers two hooks for it, and neither is used here: Actions V1's post-creation trigger belongs to the external-authentication flow, which does not run under Login V2, and Actions V2's user.human.added event needs an HTTP target you host. This book does not ship either, and says so plainly: the verification environment made its grants by hand, and the right place to put the automatic one is Chapter 8's provisioning tool, which already holds the provisioner's token. Until then, a person who registers and holds no role is refused at login with Errors.User.ProjectRequired, which is a loud failure rather than a quiet one.

Step 9 — The first token, by hand

Nothing above has been proven until a token comes out of it with the claim on. Zitadel's session API lets a script do what the login page does, which is how every token in this book was obtained and how Chapter 8's smoke test works. With a dev user that has a password — Chapter 9's alice — the flow is four calls: start the authorization request, create a session with the login-client's token, finish the request with that session, exchange the code.

LOGIN_PAT=$(sudo cat /etc/zitadel/login-client.pat)
VERIFIER=$(head -c 48 /dev/urandom | base64 | tr -dc 'A-Za-z0-9' | head -c 64)
CHALLENGE=$(printf %s "$VERIFIER" | openssl dgst -sha256 -binary | base64 | tr '+/' '-_' | tr -d '=')
SCOPE="openid profile email offline_access urn:zitadel:iam:org:id:$ORG urn:zitadel:iam:org:project:id:$PROJ:aud urn:zitadel:iam:org:project:roles"

# 1. the authorization request, which redirects to the login with an id
AUTH=$(curl -s -o /dev/null -w '%{redirect_url}' "$Z/oauth/v2/authorize?client_id=$CLIENT&redirect_uri=com.littlebit.healthyme://auth/callback&response_type=code&scope=$(python3 -c "import urllib.parse,sys;print(urllib.parse.quote(sys.argv[1]))" "$SCOPE")&code_challenge=$CHALLENGE&code_challenge_method=S256&prompt=login" | sed 's/.*authRequest=//')
# 2. a session for the person, as the login client
SESSION=$(curl -s -H "Authorization: Bearer $LOGIN_PAT" -H "Content-Type: application/json" -X POST $Z/v2/sessions \
  -d '{"checks":{"user":{"loginName":"alice@healthyme.localhost"},"password":{"password":"<her password>"}}}')
SID=$(echo "$SESSION" | python3 -c "import sys,json;print(json.load(sys.stdin)['sessionId'])")
STOK=$(echo "$SESSION" | python3 -c "import sys,json;print(json.load(sys.stdin)['sessionToken'])")
# 3. finish the request with that session; the answer is the app's callback URL carrying the code
CODE=$(curl -s -H "Authorization: Bearer $LOGIN_PAT" -H "Content-Type: application/json" -X POST $Z/v2/oidc/auth_requests/$AUTH \
  -d "{\"session\":{\"sessionId\":\"$SID\",\"sessionToken\":\"$STOK\"}}" | sed 's/.*code=//; s/".*//')
# 4. the exchange, exactly as the app does it
curl -s -X POST $Z/oauth/v2/token -d "grant_type=authorization_code&code=$CODE&redirect_uri=com.littlebit.healthyme://auth/callback&client_id=$CLIENT&code_verifier=$VERIFIER" \
  | python3 -c "import sys,json,base64; t=json.load(sys.stdin); p=t['access_token'].split('.')[1]; print(json.dumps(json.loads(base64.urlsafe_b64decode(p+'='*(-len(p)%4))), indent=1))"

{
 "aud": ["390677214766629132", "390677090732605708"],
 "client_id": "390677214766629132",
 "exp": 1789401040,
 "iat": 1789357840,
 "iss": "https://auth.example.com",
 "jti": "V2_390677338834075916-at_390677338834141452",
 "minimal_user_id": "01M2F0KNNC4M7Z1HEJHHG80JGE",
 "nbf": 1789357840,
 "sub": "390677090900377868",
 "urn:zitadel:iam:org:project:390677090732605708:roles": {"member": {"390676976144220428": "healthy-me.example.com"}},
 "urn:zitadel:iam:org:project:roles": {"member": {"390676976144220428": "healthy-me.example.com"}}
}

Read it against the three rules of Chapter 1. minimal_user_id is the server's ULID, and running the flow again returns the same value: the action found the metadata the second time. sub is Zitadel's own user id, present and unused. The roles are there as a claim keyed by role name. aud carries the client id and the project id, and iss is the public URL — the two things Chapter 5 checks before it believes any of the rest.

The scope string asked for three Zitadel-specific scopes, and the app's scope list carries all three, always. urn:zitadel:iam:org:id:<ORG> names the organisation: it makes the login page show this organisation's policy and providers — without it the hosted login shows the instance's default organisation, which has no providers and a password field, and that is what you will see the first time you forget it — and it makes Zitadel refuse the sign-in unless the person belongs to that organisation, which is a second lock on the door Step 2 closed. urn:zitadel:iam:org:project:id:<PROJ>:aud adds the project to the audience. urn:zitadel:iam:org:project:roles asks for the roles claim; an app that forgets it gets a token with no roles, which the gateway turns into an empty X-User-Roles and Minimal refuses everywhere.

What the app does with all this

The app's side is a standard OpenID Connect authorization-code flow with PKCE, in the system browser — ASWebAuthenticationSession on Apple platforms, Custom Tabs on Android — against https://auth.example.com, with the client id from Step 4, the redirect scheme from Step 4, and the scope string from Step 9, organisation scope included. Through the gateway, that request lands on a page headed "Welcome back!" with a username field, "Register new user", and three buttons — Google, GitHub, Apple — under "or sign in with"; on production the username field is absent because Step 5 turned passwords off. Two settings on the app's side are part of the "one app, one sign-in" rule, because the login domain is shared by every app on the instance and its cookie would otherwise sign a person into the second app silently:

• send prompt=login on every authorization request, which makes Zitadel ask for credentials even when its cookie holds a session, and
• open the browser session as ephemeral (prefersEphemeralWebBrowserSession on Apple platforms), so the login domain's cookie is not shared with the other app's session in the first place.

The app keeps the refresh token in the platform keychain, refreshes the access token silently before it expires, and sends only the access token to the API. It never sees a Minimal access token and never sets an X- header.

Is the app ready in Zitadel? A closing checklist

• The project has authorizationRequired and projectAccessRequired on, and a user from another organisation is refused with Errors.User.ProjectRequired — tested, not assumed.
• The native app issues JWT access tokens with roles asserted, and its redirect list holds only the app's scheme.
• The login policy has passwords off, registration on, and the three providers active, and isDefault is false.
• The action is bound to flow 2, trigger 5, and two consecutive tokens for the same person carry the same minimal_user_id.
• The app's scope string carries the audience and roles scopes, and prompt=login is on every request.
• Whatever grants member to a new person exists, or you have written down that it does not yet.

Exercises

1. Remove urn:zitadel:iam:org:project:roles from the scope in Step 9 and decode the token again. Then put it back and remove projectRoleAssertion from the project instead. Which of the two claims survives each change, and which one does the gateway in Chapter 5 actually read?

2. Create a second native application in the same project, "Healthy Me Android", with its own scheme. Obtain a token with its client id and compare aud and client_id with the iOS token. Say what the gateway would need to know to accept both, and what it would need to revoke one.

3. Deactivate the action and obtain a token. Confirm the claim is gone, then confirm what the gateway does with that token once Chapter 5 is built. Reactivate the action and check that the next token carries the same ULID as before — the metadata outlived the action.

APISIX is the only process on app-1 with a public address. It terminates TLS for two names, auth.example.com and api.example.com; sends the first to Zitadel and its login; and, for the second, verifies the JWT on every request, throws away every identity header the caller sent, sets the five headers Minimal trusts from the verified claims, and forwards. This chapter installs it as a systemd unit, configures it from one file with no etcd and no admin API, and then proves each of Volume II, Chapter 10's four requirements with a request that tries to break it.

The whole gateway is about a hundred lines of YAML and forty of Lua. It is printed in full in Appendix D; the sections below show it as it was built, one route at a time.

Step 1 — Install from the vendor's repository

The Apache APISIX project publishes Debian packages through the repos.apiseven.com repository, with separate paths for amd64 and arm64. The package carries its own OpenResty under /usr/local/openresty, the gateway under /usr/local/apisix, and a systemd unit:

curl -fsSL http://repos.apiseven.com/pubkey.gpg | sudo gpg --dearmor -o /usr/share/keyrings/apisix.gpg
ARCH=$(dpkg --print-architecture)
if [ "$ARCH" = arm64 ]; then REPO=http://repos.apiseven.com/packages/arm64/debian; else REPO=http://repos.apiseven.com/packages/debian; fi
echo "deb [signed-by=/usr/share/keyrings/apisix.gpg] $REPO debian12 main" | sudo tee /etc/apt/sources.list.d/apisix.list
sudo apt update
sudo apt install -y apisix
apisix version
systemctl is-enabled apisix

3.18.0
disabled

Installed but not enabled, and with a default configuration that expects etcd. Do not start it yet. The unit it installed is Type=forking, runs /usr/bin/apisix start from /usr/local/apisix, and reloads with apisix reload; it needs no change.

Step 2 — Standalone mode: one file of rules, no etcd, no admin API

Two files under /usr/local/apisix/conf/. The first, config.yaml, is the process configuration: which ports to listen on and where the rules come from. Replace the package's copy:

# /usr/local/apisix/conf/config.yaml
apisix:
  node_listen:
    - 80
  enable_http2: true
  ssl:
    enable: true
    listen:
      - port: 443
  enable_admin: false
deployment:
  role: data_plane
  role_data_plane:
    config_provider: yaml
nginx_config:
  error_log_level: warn

config_provider: yaml is standalone mode: every route, upstream and certificate is read from apisix.yaml beside this file, re-read within a second of any change, and applied without restarting a worker. The Admin API does not exist in this mode, which is exactly right for a gateway whose configuration should be a file under version control and nothing else. enable_http2 has to be at this level; the older per-listener form is refused at apisix test with a message saying so.

The second file, apisix.yaml, holds the rules and must end with a line reading #END, which is how APISIX knows the file was written completely before it reads it. Start with the upstreams and nothing else, and check that the process comes up at all:

# /usr/local/apisix/conf/apisix.yaml
upstreams:
  - id: zitadel
    nodes:
      "127.0.0.1:8081": 1
    type: roundrobin
    pass_host: pass
  - id: zitadel-grpc
    nodes:
      "127.0.0.1:8081": 1
    type: roundrobin
    scheme: grpc
    pass_host: pass
  - id: zitadel-login
    nodes:
      "127.0.0.1:3000": 1
    type: roundrobin
    pass_host: pass
  - id: minimal
    nodes:
      "127.0.0.1:3045": 1
    type: roundrobin
    pass_host: node

routes: []
#END

cd /usr/local/apisix && sudo apisix test
sudo systemctl enable --now apisix
systemctl is-active apisix

configuration test is successful
active

pass_host: pass on the three Zitadel upstreams is Chapter 3's fact at work: Zitadel resolves the instance from the Host header, so the gateway must hand it the public name unchanged. Minimal does not care, and node sends it the upstream address instead.

Step 3 — TLS for two names

APISIX takes certificates as inline PEM in the ssls section, matched to requests by SNI. That is inconvenient for files certbot renews on disk, so the arrangement is: certbot obtains and renews into /etc/letsencrypt, and a deploy hook renders the PEM into apisix.yaml, which APISIX picks up without a restart.

certbot's HTTP challenge needs port 80, which APISIX holds. Give certbot its own port and let APISIX proxy the challenge path to it — one more upstream and one route, permanent:

upstreams:
  # ... the four above, then:
  - id: acme
    nodes:
      "127.0.0.1:8402": 1
    type: roundrobin

routes:
  - id: acme-challenge
    uri: /.well-known/acme-challenge/*
    priority: 100
    upstream_id: acme

sudo apt install -y certbot
sudo certbot certonly --standalone --http-01-port 8402 --non-interactive --agree-tos -m ops@example.com \
  -d auth.example.com -d api.example.com

One certificate for both names, under /etc/letsencrypt/live/auth.example.com/. The deploy hook, which certbot runs after every successful renewal, replaces the ssls block:

sudo tee /etc/letsencrypt/renewal-hooks/deploy/apisix.sh >/dev/null <<'EOF'
#!/bin/bash
# Render the live certificate into APISIX's rules file. APISIX re-reads the file on its own.
set -euo pipefail
LIVE=/etc/letsencrypt/live/auth.example.com
RULES=/usr/local/apisix/conf/apisix.yaml
python3 - "$LIVE" "$RULES" <<'PY'
import re, sys
live, rules = sys.argv[1], sys.argv[2]
ind = lambda t: "\n".join("      " + l for l in t.strip().splitlines())
cert = open(f"{live}/fullchain.pem").read(); key = open(f"{live}/privkey.pem").read()
block = ("ssls:\n  - id: public\n    snis:\n      - auth.example.com\n      - api.example.com\n"
         "    cert: |\n" + ind(cert) + "\n    key: |\n" + ind(key) + "\n")
s = open(rules).read()
s = re.sub(r"(?ms)^ssls:\n.*?(?=^upstreams:)", "", s)      # drop the old block, if any
s = s.replace("upstreams:", block + "\nupstreams:", 1)
open(rules, "w").write(s)
PY
EOF
sudo chmod 0755 /etc/letsencrypt/renewal-hooks/deploy/apisix.sh
sudo /etc/letsencrypt/renewal-hooks/deploy/apisix.sh

The key ends up inside a file readable by the APISIX process, which is the same exposure the Nginx it is built on has always had. Keep apisix.yaml at mode 0640, owned by root and the process's group.

What was verified in writing this: the ssls shape above, with a self-signed certificate for the two test names, served over HTTP/2 with the right certificate chosen by SNI. What was not: a real Let's Encrypt issuance, since the verification hosts have no public name. The certbot line and the hook directory are from certbot's own documentation; run the issuance once by hand and read what it prints before trusting the timer.

Step 4 — The identity host: Zitadel and its login

Three routes, all matched on the auth.example.com host. The login is a path prefix; native gRPC callers — Zitadel's own SDKs and tools, not the console — need an h2c upstream, which APISIX selects by the request's content type; everything else is plain HTTP/1.1 to the same port, which is how the console, the OIDC endpoints and the REST APIs are served:

routes:
  - id: zitadel-login
    host: auth.example.com
    uri: /ui/v2/login/*
    upstream_id: zitadel-login
  - id: zitadel-grpc
    host: auth.example.com
    uri: /*
    vars: [["http_content_type", "~~", "^application/grpc"]]
    priority: 10
    upstream_id: zitadel-grpc
  - id: zitadel
    host: auth.example.com
    uri: /*
    upstream_id: zitadel

Now the 404 from Chapter 3 becomes a working identity provider, because the public Host reaches it:

curl -s https://auth.example.com/.well-known/openid-configuration | python3 -c "import sys,json; print(json.load(sys.stdin)['issuer'])"
curl -s -o /dev/null -w "%{http_code} http/%{http_version}\n" https://auth.example.com/ui/v2/login/healthy
curl -s -o /dev/null -w "%{http_code}\n" https://auth.example.com/ui/console/

https://auth.example.com
200 http/2
200

Step 5 — The API host: verify, strip, set, forward

This is the route the volume exists for. It is one route with five plugins, and the order they run in is fixed by APISIX rather than by the order written, so the order is stated here and then relied on:

1. serverless-pre-function, in the rewrite phase, runs first and deletes every header a caller could use to claim an identity — the five Minimal headers, LB-Access-Token, and the three headers the next plugin will itself set — and drops the lb-access-token query parameter.
2. openid-connect, in the access phase, requires a bearer token, verifies its signature against Zitadel's published keys, checks the issuer, checks that the audience contains this app's client id, and, on success, writes the token's claims into an X-Userinfo request header as base64 JSON.
3. serverless-post-function, also in the access phase but after the previous plugin by priority, reads that header, refuses the request unless it carries a well-formed minimal_user_id, sets the five headers from the claims, and deletes X-Userinfo and Authorization so that nothing but the five reaches Minimal.
4. limit-count, keyed on the user id the previous step just set.
5. The upstream.

  - id: minimal-api
    host: api.example.com
    uri: /minimal/*
    upstream_id: minimal
    plugins:
      serverless-pre-function:
        phase: rewrite
        functions:
          - |
            return function(conf, ctx)
              -- 1. Nothing a caller sends may ever reach Minimal as identity.
              for _, h in ipairs({"X-Org-Id","X-Project-Id","X-Space-Id","X-User-Id","X-User-Roles",
                                  "LB-Access-Token","X-Userinfo","X-Access-Token","X-ID-Token"}) do
                ngx.req.clear_header(h)
              end
              local args = ngx.req.get_uri_args()
              if args["lb-access-token"] ~= nil then
                args["lb-access-token"] = nil
                ngx.req.set_uri_args(args)
              end
            end
      openid-connect:
        client_id: "390677214766629132"          # the native app's client id, Chapter 4 Step 4
        client_secret: "not-used-for-bearer-validation"
        discovery: "https://auth.example.com/.well-known/openid-configuration"
        bearer_only: true
        use_jwks: true
        token_signing_alg_values_expected: RS256
        claim_validator:
          issuer:
            valid_issuers: ["https://auth.example.com"]
          audience:
            required: true
            match_with_client_id: true
        set_userinfo_header: true
        set_access_token_header: false
        set_id_token_header: false
        set_refresh_token_header: false
        unauth_action: deny
      serverless-post-function:
        phase: access
        functions:
          - |
            return function(conf, ctx)
              -- 2. Translate the verified claims into the five headers Minimal trusts.
              local core = require("apisix.core")
              local raw = ngx.req.get_headers()["X-Userinfo"]
              if not raw then return 401, {message = "no verified identity"} end
              local claims = core.json.decode(ngx.decode_base64(raw))
              local uid = claims and claims["minimal_user_id"]
              if type(uid) ~= "string" or #uid ~= 26 or not uid:match("^[0-9A-HJKMNP-TV-Z]+$") then
                return 403, {message = "token carries no server-assigned user id"}
              end
              local roles = {}
              local r = claims["urn:zitadel:iam:org:project:roles"]
              if type(r) == "table" then
                for k, _ in pairs(r) do roles[#roles + 1] = k end
              end
              table.sort(roles)
              ngx.req.clear_header("X-Userinfo")
              ngx.req.clear_header("Authorization")
              ngx.req.set_header("X-Org-Id", "lbl")
              ngx.req.set_header("X-Project-Id", "healthyme")
              ngx.req.set_header("X-Space-Id", "live")
              ngx.req.set_header("X-User-Id", uid)
              ngx.req.set_header("X-User-Roles", table.concat(roles, ","))
            end
      limit-count:
        count: 600
        time_window: 60
        key_type: var
        key: http_x_user_id
        rejected_code: 429
        policy: local

Some of that is not obvious from reading it, and each of these was learned by running it:

• client_secret is required by the plugin's schema even in bearer mode, where it is never sent anywhere. Any string satisfies it; a value that says so is kinder to the next reader.
• use_jwks with bearer_only is the local-verification path. The plugin fetches the discovery document once, caches the JWKS, and checks the signature itself; there is no call to Zitadel per request. The discovery URL is the public one, through this same gateway, which is fine — it is one request at startup and on key rotation.
• The audience check is real and it is 403. A token minted for a different application on the same instance is signed by the same key and carries the same issuer; only aud tells them apart. With match_with_client_id, a Sleep Well token on this route is refused with {"error":"mismatched audience"} before any Lua runs. Without it, the post-function would refuse it anyway for lacking the claim — unless that app's organisation also runs the action, in which case a Sleep Well member would reach Healthy Me's API as a Healthy Me member. Set it.
• Lua patterns have no {n} quantifier. The first version of the ULID check was written ^[0-9A-HJKMNP-TV-Z]{26}$, which is a valid pattern that matches nothing, and every real token was refused with the 403 above. Check the length separately.
• X-Org-Id, X-Project-Id and X-Space-Id are constants of the route, because this route is this app. Chapter 8 adds a second route with its own three constants and its own client id; that is the whole per-app configuration on the gateway.
• The roles claim is an object keyed by role name, whose values say which organisation granted the role. The gateway takes the keys, sorts them, and joins with commas — the form Volume I, Chapter 4 says X-User-Roles takes. Zitadel puts the claim on the token only if the app asked for the roles scope, so an app that forgets it arrives with an empty header and is refused by every template.
• limit-count keyed on the user id, not the address. By the time it runs, X-User-Id is the verified ULID; six hundred requests a minute per person is generous for a phone app and a wall for a script.

apisix test and the file is live within a second. There is no restart in this book after Step 2.

Step 6 — Prove it, with requests that try to break it

Every claim in Step 5 was checked with the requests below, and the outputs are the ones to expect. $T is a token from Chapter 4, Step 9.

No token, a garbage token, a token from another issuer:

curl -s -o /dev/null -w "%{http_code}\n" https://api.example.com/minimal/system/api/v1/org
curl -s -o /dev/null -w "%{http_code}\n" https://api.example.com/minimal/system/api/v1/org -H "Authorization: Bearer nope"
curl -s -o /dev/null -w "%{http_code}\n" https://api.example.com/minimal/system/api/v1/org \
  -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IngifQ.eyJpc3MiOiJodHRwczovL290aGVyLmV4YW1wbGUiLCJzdWIiOiIxIn0.c2ln"

401
401
401

The gateway's error log names each: No bearer token found in request, token is signed by unexpected algorithm "HS256", and a signature failure. None reached Minimal.

A token for the other app: 403 with {"error":"mismatched audience"} — Chapter 8 produces the token.

A valid token together with every forged header a caller could think of. To see what actually arrives at the upstream, point a copy of the route at a tiny echo server for the duration of the test — APISIX reloads the file in a second, so this is a two-minute exercise:

curl -s "https://api.example.com/minimal/echo/v1?lb-access-token=mus:smuggled&keep=1" \
  -H "Authorization: Bearer $T" \
  -H "X-Org-Id: evil" -H "X-Project-Id: evil" -H "X-Space-Id: evil" -H "X-User-Id: evil" -H "X-User-Roles: admin" \
  -H "LB-Access-Token: mus:smuggled" -H "X-Userinfo: eyJtaW5pbWFsX3VzZXJfaWQiOiJFVklMIn0=" -H "X-Access-Token: evil"

What the echo upstream received, every header:

path: /minimal/echo/v1?keep=1
  Accept: */*
  Host: api.example.com
  User-Agent: curl/8.7.1
  X-Forwarded-For: ...
  X-Forwarded-Host: api.example.com
  X-Forwarded-Port: 443
  X-Forwarded-Proto: https
  X-Org-Id: lbl
  X-Project-Id: healthyme
  X-Real-IP: ...
  X-Space-Id: live
  X-User-Id: 01M2F0KNNC4M7Z1HEJHHG80JGE
  X-User-Roles: member

The five headers carry the gateway's values, not the caller's. LB-Access-Token, X-Userinfo, X-Access-Token and the query parameter are gone. Authorization is gone too; Minimal's own middleware would have dropped it, and now it never arrives.

The same forged request against Minimal itself. Reading the organisation record needs a role in org_read_roles, which is admin; the caller claims admin in a header and smuggles a Minimal admin token minted directly on app-1:

curl -s -o /dev/null -w "%{http_code}\n" "https://api.example.com/minimal/system/api/v1/org?lb-access-token=$ADMIN_TOKEN" \
  -H "Authorization: Bearer $T" -H "X-User-Roles: admin" -H "LB-Access-Token: $ADMIN_TOKEN"

403

Minimal refused it as member, which is what the token says. The same admin token presented to Minimal's port directly on app-1 answers 200 — it is a real credential, and the only reason it is harmless from outside is that outside cannot reach that port. That is Volume II's "be the only path in", and it is the next step.

Step 7 — Be the only path in

Minimal, Zitadel and the login all bind loopback (127.0.0.1:3045, :8081, :3000), which is what the units in Chapters 3 and 6 do and what ss -ltn should show. The host firewall is the second layer; with nftables or ufw, allow 22 from your administrative network, 80 and 443 from anywhere, and nothing else inbound. Postgres on db-1 accepts only app-1's private address, per Chapter 2.

sudo ss -ltnp | awk 'NR==1 || /LISTEN/' | grep -vE "127.0.0.1|\[::1\]"

The only lines left should be APISIX on 80 and 443, and sshd.

What dev does differently

The same two files, with node_listen: 8080, no ssls, http://localhost:8080 in every URL, no host: on the routes because everything is localhost, and the four upstreams pointing at the Mac through the VM's host.lima.internal name. Chapter 9 has the file. The Lua is byte-identical.

Is the boundary load-bearing? A closing checklist

Volume II, Chapter 10's list, answered for this gateway:

• It strips the five headers before it sets its own — shown by the echo, not by reading the config. Repeat the echo after any change to the route.
• Minimal's listener has no route from outside — ss -ltn on app-1 and the firewall rules, read this week.
• The channel decision is made from something the identity provider controls. Every caller through this route is a person on the direct channel; there is no mcp: or m2m: prefix on any header it sets, and agent_identity is left out of Minimal's configuration in Chapter 6. A machine caller gets its own route, its own Zitadel service account and its own constants, when there is one.
• Raw headers, deliberately. This deployment sets the five headers on every request and never mints a Minimal access token for a person. The trade Volume II names is accepted: the gateway's translation is trusted on every request, and in exchange a revoked Zitadel account stops working at the end of its last access token's hour rather than whenever someone remembers to suspend a Minimal token.
• mcp.behind_reverse_proxy is irrelevant, because the MCP listener is not fronted by this gateway and is not enabled in Chapter 6.

Exercises

1. Remove match_with_client_id from the route, obtain a token for Chapter 8's second app, and send it to this route. Record the status and the reason. Then run Chapter 4's action on the second app's organisation too and repeat. Put the setting back before going on.

2. Change the ULID check back to the {26} form and send a valid token. Watch it fail, read the gateway's error log, and note that the log says only which plugin exited with 403. Write the one line you would add to the Lua to make that failure name its cause.

3. The limit-count plugin counts per X-User-Id. Send six hundred and one requests with one token inside a minute and record the status and headers of the last. Then decide whether the login host needs a limit too, what it would be keyed on before anyone is signed in, and add it.

Minimal is the piece this book changes least. It runs as it did in Volume II, Chapter 9, with a configuration file whose values are chosen for a listener that only the gateway can reach, and it holds one organisation, one project and one space for the app, set up exactly as Volume I, Chapter 3 set up Acme. What is new is the contract at its front door: every request arrives with the five headers the gateway wrote, X-User-Id is always a server-assigned ULID, and X-User-Roles is always a subset of the role names Chapter 4 created in Zitadel. This chapter installs the service, writes the configuration that fits that contract, bootstraps the app, and puts the first table behind a template and a row scope so that a person sees only their own rows.

Step 1 — The service

A user, a directory for its state, a directory for its configuration, and the binary:

sudo useradd --system --home-dir /var/lib/minimal --create-home --shell /usr/sbin/nologin minimal
sudo install -d -m 0750 -o root -g minimal /etc/minimal
sudo install -m 0755 minimal /usr/local/bin/minimal

The configuration file is the sample from Volume II, Chapter 9 with the values below changed, and nothing else. Every changed line is a decision this deployment has made:

# /etc/minimal/config.yml -- the lines that differ from sample/config.yml
service:
  host: "127.0.0.1"                 # only APISIX, on this host, ever connects
  port: "3045"
  environment: "production"
  server_key: "<a real secret, generated for this deployment>"
  log_level: "info"
  debug_mode: false
  show_request_details: false
  strict_local_host: true

definition_store:
  type: postgres
  host: 10.0.0.10
  port: "5432"
  schema: minimal
  username: minimalist
  password_source: file
  password: '<minimalist-password>'
  ssl_mode: "verify-full"            # Chapter 2's certificate, verified
  max_open_connections: 20
  encrypt_database: true
  encryption_key_source: env         # MINIMAL_ENCRYPTION_KEY, from the unit's environment file
  encryption_key: ''

auto_api:
  permission_fail_open: false        # a lookup failure is a refusal, not a pass

access_token:
  max_per_user: 5                    # nobody mints person-tokens here; keep the ceiling low
  default_expiry_days: 30
  retain_dead_days: 30
  sweep_interval_secs: 86400

# agent_identity: left out entirely. No caller reaches this listener with an mcp: or m2m:
# prefix, because the gateway never writes one; leaving the section out means such a prefix
# would not be recognised even if something did.

mcp:
  enabled: false

Three of those deserve the sentence Volume II already gave them. permission_fail_open: false is the setting the sample ships true and calls temporary; a production deployment whose tables all carry a template turns it off, and this one's tables will. agent_identity is omitted rather than configured, per Volume II, Chapter 10: no caller through this gateway is an agent or a machine, and a section that is absent cannot be matched by accident. mcp.enabled: false because the MCP listener is not fronted by this gateway and has no place in a consumer app's deployment; when it is wanted, it gets its own boundary and its own chapter.

sslmode on the store connection is verify-full, with the system's CA bundle carrying Chapter 2's internal CA. If the cluster still has the snakeoil certificate, this has to be require, and the closing checklist says to write that down.

The encryption key and the unit:

sudo sh -c 'echo "MINIMAL_ENCRYPTION_KEY=$(openssl rand -hex 16)" > /etc/minimal/minimal.env'
sudo chown root:minimal /etc/minimal/minimal.env /etc/minimal/config.yml
sudo chmod 0640 /etc/minimal/minimal.env /etc/minimal/config.yml

sudo tee /etc/systemd/system/minimal.service >/dev/null <<'EOF'
[Unit]
Description=Minimal API server
After=network-online.target
Wants=network-online.target

[Service]
User=minimal
Group=minimal
WorkingDirectory=/var/lib/minimal
EnvironmentFile=/etc/minimal/minimal.env
ExecStart=/usr/local/bin/minimal -c /etc/minimal/config.yml
Restart=on-failure
RestartSec=5
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
ReadWritePaths=/var/lib/minimal

[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now minimal
sleep 3; systemctl is-active minimal
curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:3045/minimal/system/api/v1/org

active
400

400 is Volume I, Chapter 4's "no credential at all" — the listener is up and refusing a request that carries no identity, which is the only kind a caller who reached it directly could send. The encryption key seals every definition, every registered database password and every cached table shape from the first write onward; it is part of the backup set with the database dumps, and it is not a value to regenerate.

On db-1, Chapter 2's standing check now has both rows:

minimalist|minimal|t
zitadel|zitadel|t

Step 2 — Bootstrap the app: organisation, project, space

Every call in this step is made on app-1, straight to 127.0.0.1:3045, with the five headers typed by hand, because the gateway does not carry administrative identities and never will: the administrator here is X-User-Id: appctl holding X-User-Roles: admin, and the only place that identity is ever asserted is a shell on this host or Chapter 8's tool running from one. Zitadel does not know appctl exists, and Minimal does not check it — it is a name in an audit trail.

The organisation is the company, created with the server key as every organisation is:

M=http://127.0.0.1:3045
SK=$(sudo grep -E '^\s*server_key:' /etc/minimal/config.yml | sed -E 's/.*: *"?([^"]+)"?/\1/')
curl -s -X POST $M/minimal/system/api/v1/org -H "X-Server-Key: $SK" -H "X-User-Id: sys:appctl" -H "X-User-Roles: admin" \
  -H "Content-Type: application/json" -d '{
  "org_id": "lbl", "name": "Littlebit Labs", "abbreviation": "lbl",
  "admin_email": "ops@example.com", "admin_name": "Ops", "address": "-", "state": "-", "country": "IN", "pin_code": "000000",
  "org_read_roles": "admin", "org_write_roles": "admin", "org_delete_roles": "admin", "project_create_roles": "admin"}'

{"org_id": "lbl", "note": "please keep this ID saved"}

Every role list on the organisation is admin alone. No app user holds admin — Chapter 4 created member and premium and nothing else — so no request through the gateway can read, change or delete the organisation record, create a project, or reach any other organisation-level surface.

The project is the app, and it registers the app's database as minimalist, the second role Chapter 2 granted on healthyme:

ADM=(-H "X-Org-Id: lbl" -H "X-Project-Id: healthyme" -H "X-Space-Id: live" -H "X-User-Id: appctl" -H "X-User-Roles: admin" -H "Content-Type: application/json")
curl -s -X POST $M/minimal/system/api/v1/project "${ADM[@]}" -d '{
  "org_id": "lbl", "project_id": "healthyme", "name": "Healthy Me", "abbreviation": "healthyme",
  "description": "Backend of the Healthy Me apps",
  "database": [{
    "host_details": [{"host": "10.0.0.10", "port": "5432"}],
    "username": "minimalist", "password": "<minimalist-password>",
    "db_name": "healthyme", "db_type": "postgres", "ssl_mode": "verify-full",
    "connect_timeout_secs": 30, "idle_timeout_secs": 600, "query_timeout_secs": 60,
    "max_open_connections": 20, "max_idle_connections": 10}],
  "project_read_roles": "admin,member,premium",
  "project_write_roles": "admin",
  "project_delete_roles": "admin"}'

{"org_id": "lbl", "project_id": "healthyme"}

project_read_roles includes the two app roles, and the two write columns do not. A member can therefore read the project record and list what they own, and cannot define a permission template, assign one, re-index a table or change the project — every one of those is a project_write_roles check, per Volume I, Chapter 4. The database password is stored sealed under the encryption key and comes back masked on every read.

The space is the release channel:

curl -s -X POST $M/minimal/system/api/v1/space "${ADM[@]}" -d '[{"space_id":"live","org_id":"lbl","project_id":"healthyme","name":"Live","description":"What the apps call","email":"ops@example.com"}]'

[
  {
    "name": "Live",
    "owner_id": "appctl",
    "project_id": "healthyme",
    "space_id": "live"
  }
]

X-Space-Id: live is one of the three constants the gateway writes on every request. A second space named next is where Volume II, Chapter 1's definitions are written and tried before the copy route promotes them to live; the gateway never points at it.

Step 3 — The first table, and who owns it

The app's tables are made by healthyme_app, the owner, so that Chapter 2's default privileges hand minimalist its read and write grants automatically. The first table is the one Chapter 7 needs: the device a person registered, keyed by the ULID the app minted, holding the public key the app generated, and scoped to the person by the server's ULID:

PGPASSWORD='<healthyme-password>' psql "host=10.0.0.10 dbname=healthyme user=healthyme_app sslmode=verify-full" -v ON_ERROR_STOP=1 <<'SQL'
CREATE TABLE device (
  device_id   CHAR(26)    PRIMARY KEY,           -- minted by the app, never re-minted here
  user_id     CHAR(26)    NOT NULL,              -- the server-assigned ULID, set by the row scope
  public_key  TEXT        NOT NULL,              -- the app's public key, base64
  platform    TEXT        NOT NULL,
  created_at  TIMESTAMPTZ NOT NULL DEFAULT now(),
  last_seen   TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX device_user_idx ON device (user_id);
SQL

Both identifiers are CHAR(26), which is what a ULID is in text and what X-User-Id carries. There is no email column and no Zitadel subject column: the identity provider holds the email, and the subject is a Zitadel-internal number that nothing in this database needs to know.

Now tell Minimal about it — Volume II, Chapter 8's sequence, condensed:

curl -s -X POST "$M/minimal/system/api/v1/project/table/index?db_type=postgres&schema=healthyme&table=device" "${ADM[@]}"

{
  "db_name": "healthyme",
  "db_type": "postgres",
  "indexed": "refreshed",
  "lock_mask": 4091,
  "org_id": "lbl",
  "project_id": "healthyme",
  "table_name": "device",
  "table_type": "TABLE"
}

lock_mask: 4091 is a freshly indexed table's state: everything locked except direct-channel read. It stays that way until the template and the lock mask are set on purpose.

Step 4 — Template, lock mask, row scope

The template names Chapter 4's roles and no others, and opens nothing on the agent or machine channels:

curl -s -X POST $M/minimal/system/api/v1/permission/template "${ADM[@]}" -d '{
  "name": "member-own-rows",
  "description": "A member may create, read and update rows on the direct channel; agents and machines get nothing.",
  "auto_api_create_roles": "member,premium,admin",
  "auto_api_read_roles":   "member,premium,admin",
  "auto_api_update_roles": "member,premium,admin",
  "auto_api_delete_roles": "admin",
  "mcp_create_roles": "", "mcp_read_roles": "", "mcp_update_roles": "", "mcp_delete_roles": "",
  "m2m_create_roles": "", "m2m_read_roles": "", "m2m_update_roles": "", "m2m_delete_roles": "",
  "override_similar_template_warning": false}'

{
  "name": "member-own-rows",
  "org_id": "lbl",
  "project_id": "healthyme",
  "template_id": "01M2F0VR0HSV42S9FMCXN3D0GS"
}

Assign it, open the direct channel only, and bind the row scope:

TID=01M2F0VR0HSV42S9FMCXN3D0GS
curl -s -X PUT $M/minimal/system/api/v1/project/table/permission "${ADM[@]}" \
  -d "{\"schema\":\"healthyme\",\"table\":\"device\",\"db_type\":\"postgres\",\"template_id\":\"$TID\"}"
curl -s -X PUT $M/minimal/system/api/v1/project/table/lock "${ADM[@]}" -d '{
  "db_type": "postgres", "schema": "healthyme", "table": "device",
  "table_create_locked": false, "table_read_locked": false, "table_update_locked": false, "table_delete_locked": false,
  "mcp_create_locked": true, "mcp_read_locked": true, "mcp_update_locked": true, "mcp_delete_locked": true,
  "m2m_create_locked": true, "m2m_read_locked": true, "m2m_update_locked": true, "m2m_delete_locked": true}'
curl -s -X PUT $M/minimal/system/api/v1/project/table/rls "${ADM[@]}" \
  -d '{"db_type":"postgres","schema":"healthyme","table":"device","rls_column_name":"user_id","rls_header_key":"X-User-Id"}'

{
  "db_name": "healthyme",
  "db_type": "postgres",
  "org_id": "lbl",
  "project_id": "healthyme",
  "rls_column_name": "user_id",
  "rls_header_key": "X-User-Id",
  "table_name": "device",
  "updated_at": "2026-09-14T03:55:04.682745Z"
}

The last call is the one that makes the whole design hold. Volume II, Chapter 4's row scope binds device.user_id to X-User-Id: every read is filtered to the caller's rows, and every insert has the column set from the header regardless of what the body says. The header is the server's ULID, written by the gateway from a verified token. So a person can only ever read and write their own devices, not because the app remembers to filter, and not because a definition's SQL says WHERE user_id = ?:x-user-id, but because the table itself is scoped to whoever the gateway says is calling. Every user-scoped table gets this binding, at the moment it is indexed.

Step 5 — The first request through the front door

With Chapter 4's token for alice, through the gateway, the app registers its device. The body carries a user_id on purpose, and it is wrong on purpose:

curl -s -X POST "https://api.example.com/minimal/api/rest/auto/v1/lbl/healthyme/pg/healthyme/device" \
  -H "Authorization: Bearer $T" -H "Content-Type: application/json" -w " [%{http_code}]" \
  -d '[{"device_id":"01M2F0000000000000DEV1CE01","user_id":"SPOOFED_VALUE_IGNORED_BY_RLS",
        "public_key":"MCowBQYDK2VuAyEAyT9pQ2rJ0xTf4NcB8HvKdXw3PsRe0aQfLmZ1u7Yg2kM=","platform":"ios",
        "created_at":null,"last_seen":null}]'
curl -s "https://api.example.com/minimal/api/rest/auto/v1/lbl/healthyme/pg/healthyme/device?ps=10&pg=0" \
  -H "Authorization: Bearer $T" -w " [%{http_code}]"

{"rows_affected": 1} [201]
[{"created_at":"2026-09-14T09:25:04.707274+05:30","device_id":"01M2F0000000000000DEV1CE01","last_seen":"2026-09-14T09:25:04.707274+05:30","platform":"ios","public_key":"MCowBQYDK2VuAyEAyT9pQ2rJ0xTf4NcB8HvKdXw3PsRe0aQfLmZ1u7Yg2kM=","user_id":"01M2F0KNNC4M7Z1HEJHHG80JGE"}] [200]

And on db-1, what the row holds:

01M2F0000000000000DEV1CE01|01M2F0KNNC4M7Z1HEJHHG80JGE|ios

The spoofed value never reached the table. user_id is the ULID from the token, the same one X-User-Id carried, the same one the metadata in Zitadel holds. That is the chain, end to end, and Chapter 7 walks it from the app's side.

The identity contract, stated once

For anyone writing a definition or a template against this deployment from now on:

No request through the gateway ever carries admin, an mcp: or m2m: prefix, or an LB-Access-Token. A definition that binds ?:x-user-id gets the ULID. A template that names member and premium covers every person; one that names admin covers only the shell on app-1. A row scope on user_id bound to X-User-Id is the way a user-scoped table is made, and this deployment has no user-scoped table without one.

What dev does differently

The same bootstrap, against a Minimal started from a terminal on the Mac with the sample configuration's port and a store of its own; Chapter 9 has the file and says why the store is separate from the one your Postman runs use. permission_fail_open stays false there too — a dev environment that passes where production refuses is not a dev environment.

Is Minimal ready? A closing checklist

• The listener is loopback, environment is production, debug_mode and show_request_details are off, and the server key is not the sample's.
• permission_fail_open is false, and every table reachable through the gateway carries a template that names only Chapter 4's roles on the direct channel.
• Every user-scoped table has a row scope bound to X-User-Id, checked by an insert with a wrong user_id in the body, as above.
• agent_identity is absent and mcp.enabled is false.
• The store connection is TLS — pg_stat_ssl shows minimalist with t — and the app database is registered as minimalist, not as its owner.
• The encryption key and the server key are in the backup set, apart from the dumps.

Exercises

1. Add a second table, preference, owned by healthyme_app, with a user_id column. Index it and, before assigning a template or a row scope, read it through the gateway as alice. Explain the status you get from the lock mask alone. Then assign the template but not the row scope, insert two rows as two different users, and read as one of them.

2. Send a request through the gateway whose token carries premium as well as member, and one that carries neither. Confirm which template columns each is checked against, and what an empty X-User-Roles does on a table whose template names no empty role.

3. Register a second database on the project as healthyme_app instead of minimalist and run a DROP TABLE through Volume II, Chapter 3's DDL route with the admin identity. Then say, in one sentence, why this book registered the database as the role that cannot do that.

Chapters 2 to 6 each proved their own piece. This chapter walks one person through all of them in order, from the app's side, and names at each step which system is doing the work and which is idle. It then answers the question Chapter 1 left open — where the app's own ULID and public key go, and whether Minimal is involved in registration at all — and closes with the smoke test that exercises the whole path from a shell, with no phone.

Day one: the app alone

A person installs Healthy Me and opens it. The app generates a ULID — call it the device id — and a key pair, stores both in the platform keychain, and gets on with being an app. Nothing has been sent anywhere. Zitadel has never heard of this person; neither has Minimal; the gateway has seen no request. The device id is the app's name for this installation, and the private key is what will later prove that a message came from it.

This state can last forever. A person who never touches a shared feature never registers, and that is by design: there is no anonymous route on the gateway and no beacon on first launch, so a person who has not chosen to register has left nothing on the server.

Day nine: the person reaches for a shared feature

The app needs a server now. It starts a sign-in: an authorization-code request with PKCE, opened in the system browser as an ephemeral session, with prompt=login, against https://auth.example.com.

sequenceDiagram
    participant A as App
    participant G as APISIX
    participant Z as Zitadel
    participant L as Login V2
    participant P as Google / GitHub / Apple
    A->>G: GET /oauth/v2/authorize (PKCE challenge, scopes)
    G->>Z: same, Host: auth.example.com
    Z-->>A: 302 → /ui/v2/login/login?authRequest=…
    A->>G: GET /ui/v2/login/login?authRequest=…
    G->>L: same
    L-->>A: sign-in page: Google · GitHub · Apple
    A->>P: the person chooses a provider and signs in there
    P-->>L: callback with the provider's identity
    L->>Z: create the user (first time) and a session; finish the auth request
    Z-->>A: 302 → com.littlebit.healthyme://auth/callback?code=…
    A->>G: POST /oauth/v2/token (code, PKCE verifier)
    G->>Z: same
    Z->>Z: action: minimal_user_id — mint on first token, store, assert
    Z-->>A: access token (JWT, 1h) + refresh token (90d)

Registration is the first sign-in. Zitadel creates the account from the provider's claims, Chapter 4's action mints the server-assigned ULID as it issues the first token, and the app has everything it needs before it has spoken to the API once.

Three things happened inside Zitadel during that exchange, and only inside Zitadel:

1. The account was created, because the organisation's Google, GitHub or Apple provider has isAutoCreation on and the login policy has allowRegister on. Zitadel now holds the person's email, their display name, and the link to the provider's identity. This is the only place any of that is stored.
2. The server-assigned ULID was minted. The action on "pre access token creation" found no minimal_user_id in the new account's metadata, made one, wrote it to the metadata, and put it on the token. Every token this person ever gets carries that same value.
3. A role check ran. With authorizationRequired on the project, a person holding no role is refused at the last step — so whatever grants member (Chapter 4, Step 8) has to have run by the time the token is requested, or the person sees a refusal and no token is issued.

The app now holds a JWT. What it does with it is the whole of the app's job on the identity side: keep the refresh token in the keychain, refresh the access token quietly before its hour is up, put the access token in the Authorization header of every API request, and never put anything in an X- header.

The same day: the first API call

The app's first request is to register its device — the ULID and public key from day one — under the person's account:

POST /minimal/api/rest/auto/v1/lbl/healthyme/pg/healthyme/device HTTP/1.1
Host: api.example.com
Authorization: Bearer <the access token>
Content-Type: application/json

[{"device_id":"01M2F0000000000000DEV1CE01","user_id":null,
  "public_key":"MCowBQYDK2VuAyEAyT9pQ2rJ0xTf4NcB8HvKdXw3PsRe0aQfLmZ1u7Yg2kM=",
  "platform":"ios","created_at":null,"last_seen":null}]

sequenceDiagram
    participant A as App
    participant G as APISIX
    participant M as Minimal
    participant D as Postgres (healthyme)
    A->>G: POST …/device, Authorization: Bearer JWT
    G->>G: strip X-*, LB-Access-Token, ?lb-access-token
    G->>G: verify signature (JWKS), iss, aud
    G->>G: X-User-Id ← minimal_user_id · X-User-Roles ← roles · 3 constants
    G->>M: POST …/device with the five headers only
    M->>M: lock mask · template (member may create) · row scope sets user_id
    M->>D: INSERT … user_id = '01M2F0KNNC4M7Z1HEJHHG80JGE'
    D-->>M: 1 row
    M-->>A: 201 {"rows_affected": 1}

The gateway is the only party that reads the token. Minimal never sees it, and never sees a user_id the app chose.

user_id is null in the body, or anything else; Chapter 6's row scope overwrites it with the X-User-Id the gateway set, which is the ULID from the token. The response is 201 with a row count, as Volume I, Chapter 2 says every insert answers. A read back —

GET /minimal/api/rest/auto/v1/lbl/healthyme/pg/healthyme/device?ps=10&pg=0 HTTP/1.1
Host: api.example.com
Authorization: Bearer <the access token>

— returns exactly this person's devices and nobody else's, because the row scope filters the read by the same header. A second device registered from a second phone lands under the same user_id, because the second phone signed in as the same person and got a token with the same ULID.

Where the device id and public key live, and why

Chapter 1 promised an answer to "does registration involve Minimal". It does not, and now the reason can be stated precisely: registration is complete the moment Zitadel issues the first token. The account exists, the server-assigned id exists and is durable, and the person can sign in again on any device and get the same id. Nothing in Minimal has to happen for any of that to be true, and if the API were down on day nine the person would still be registered.

The device id and public key are a different matter, and the answer depends on what the app's own features need:

• If nothing server-side ever uses the key — the app just wanted an account, for backup or sync of data it encrypts itself — then the key never needs to leave the device at all, and the device id is a value the app can send as device_id on rows it writes, so that sync can attribute changes. No registration call, no device table.
• If a server-side feature needs to know a person's devices — sending an encrypted item to another person's devices, approving a new device from an old one, listing and revoking devices — then the key is data the app stores about itself, and it belongs in the app's own database under the person's server-assigned id, which is exactly the device table Chapter 6 made and the insert above. That is data the app writes through Minimal like any other row; it is not identity, and Zitadel is the wrong place for it.

The one thing not to do is put the device id or the key into Zitadel as user metadata. Metadata is where Zitadel keeps the server-assigned ULID because that value has to be on every token; a key that another person's device will look up belongs in a table that person's device can query, scoped by Chapter 6's row scope to the owner, and readable by the other party through a definition that says so. Volume II, Chapter 1 is where that definition gets written; this book stops at the table.

If a feature genuinely needs the server to prove something to the app — an acknowledgement the app can verify came from your server and not from the network — that is a signature over the response with a key the server holds, and it is a definition's job, using whatever signing library the deployment adopts. Nothing in this chapter's flow needs it: the token the app holds is Zitadel's signed statement that the person is who they are, and the row that came back is Minimal's answer over TLS.

Day ten onward

Every later launch, the app has a refresh token and gets a fresh access token without the person seeing anything. Every token carries the same minimal_user_id, because the action finds it in the metadata now and mints nothing. Every API request is the same shape as the first. When the person gets a new phone, they sign in with the same provider, Zitadel matches the provider identity to the existing account, and the new phone's token carries the old ULID — the device table gains a row, the person's data does not move.

When a person leaves, deleting their Zitadel user ends their ability to get a token within the hour; their rows in healthyme are still keyed by a ULID that no token will ever carry again, and whether those rows are deleted, retained or anonymised is a product decision the app enforces with a definition, not something the identity layer decides.

The smoke test: the whole path from a shell

Chapter 4, Step 9 showed that a script can do what the login page does, through Zitadel's session API, with the login client's token. Put that together with the insert above and you have a test that exercises every piece in this book, from one shell, in a few seconds, with no phone and no provider account — against a user who signs in with a password, which is why the dev organisation in Chapter 9 keeps passwords on for exactly one such user.

The test, in outline (Chapter 9 ships it as smoke.sh):

1. Start an authorization request through the gateway; expect the 302 to the login and capture the request id.
2. Create a session for the test user through the gateway with the login client's token; expect 201.
3. Finish the request with that session; expect 200 and a callback URL carrying a code.
4. Exchange the code; expect 200, a JWT whose minimal_user_id is 26 characters and whose iss is the gateway's public URL.
5. Insert a device row through the gateway with a wrong user_id in the body; expect 201.
6. Read the table back through the gateway; expect 200 and rows whose user_id is the claim's value, none other.
7. Read the organisation record through the gateway; expect 403 — a member is not an admin.
8. Send the same read with no token; expect 401.

Each step that fails names the chapter to reopen: 1 and 3 are Chapter 5's identity host and Chapter 3's units, 2 is the login client's token, 4 is Chapter 4's application settings and action, 5 and 6 are Chapter 6's template and row scope, 7 is Chapter 6's organisation roles, 8 is Chapter 5's API route. Run it after every deployment and after every change to any of the three systems.

Exercises

1. Run the smoke test twice with the same user and once with a second user. Compare the three minimal_user_id values and the three device reads. Then delete the second user in Zitadel, run the test with the first user, and read the device table on db-1 directly: whose rows are still there, and what would it take for anyone to ever read them through the API again?

2. Register a device from a second "phone" — a second run of the test with a different device_id and the same user — and then write, without running it, the definition that would let another person fetch this person's public keys for an item they are sharing. Say which header it binds and which template column the caller needs.

3. The chapter says registration is complete when the first token is issued, even with the API down. Stop minimal on app-1, run steps 1 to 4 of the smoke test, and confirm they pass. Then stop zitadel instead, and confirm which steps a person with an existing, unexpired access token can still complete.

Nothing in Chapters 4 to 6 was specific to Healthy Me except the names. This chapter adds a second app, Sleep Well, by listing every object the first one needed and making each again; proves that the two are as separate as Chapter 1 promised; and then writes down the shape of a small Go tool that does the whole list from one description file, so that the third app takes a minute.

What one app is made of

Counted from the chapters, an app is exactly these objects, in this order, because each one's id is an input to the next:

Two things are not on the list, because they are made once for the deployment and shared: the Minimal organisation lbl, and the Zitadel instance with its provisioner and login-client accounts. Everything else is per app.

Sleep Well, by hand

Rows 1 and 2 are Chapter 2's SQL with sleepwell in place of healthyme, plus one hostssl line for each of the two roles on the new database. Rows 3 to 9 are Chapter 4's calls with the name changed; the ids they return are the ones below:

ORG2=390678055808401676    # organisation "Sleep Well"
PROJ2=390678055858733324   # project "Sleep Well", authorizationRequired + projectAccessRequired
APP2=390678055925907724    # native app "Sleep Well iOS", client id

Row 10 is a second route in apisix.yaml, which is the Healthy Me route with four values changed: the id, the client id in openid-connect, and the three constants in the post-function. Everything else — the Lua, the plugin settings, the rate limit — is identical, and the cleanest way to keep it so is to keep the two routes next to each other and diff them when either changes:

  - id: sleepwell-api
    host: api.example.com
    uri: /minimal/*
    vars: [["http_x_app", "==", "sleepwell"]]     # see below
    priority: 5
    upstream_id: minimal
    plugins:
      # identical to minimal-api, except:
      openid-connect:
        client_id: "390678055925907724"
        # ...
      serverless-post-function:
        # ... X-Org-Id lbl · X-Project-Id sleepwell · X-Space-Id live

Both apps call the same host and the same path prefix, so something has to say which route a request belongs to. Two honest options: a second API host (api.sleepwell.example.com), which is one more name on the certificate and no other change; or one host and a header the app sends (X-App: sleepwell, matched by vars as above). A wrong or missing value is not a security problem in either case — the token's audience still has to match the route's client id, so a Sleep Well token on the Healthy Me route is 403 mismatched audience whichever route it lands on — it is only a routing problem. This book uses the header.

Rows 11 to 13 are Chapter 6's calls with sleepwell in place of healthyme.

Proving the two are separate

Three facts, each one request:

A Healthy Me user cannot sign in to Sleep Well. Chapter 4's headless flow, with Sleep Well's client id and alice's credentials, is refused at the finishing step:

3. finalize -> 403 {"code":7,"message":"Errors.User.ProjectRequired (OIDC-foSyH49RvL)", ...}

That is projectAccessRequired on the Sleep Well project: alice's organisation has not been granted it. Before that setting was on, the same flow issued a token — which is the reason Chapter 4 puts the two booleans on the project and says so twice.

A Sleep Well token cannot reach Healthy Me's API. With bob, a user in the Sleep Well organisation, a valid Sleep Well token on the Healthy Me route:

curl -s -o /dev/null -w "%{http_code}\n" https://api.example.com/minimal/system/api/v1/org -H "Authorization: Bearer $BOB_TOKEN"

403

{"error":"mismatched audience"} from the gateway; Minimal never saw it.

Two accounts, even for one person. Register the same email in both apps and Zitadel holds two users with two sub values in two organisations, each with its own minimal_user_id — the action runs per organisation and mints per user. Nothing links them, and nothing in Minimal can tell they belong to the same person, because nothing does. If one day the product wants one account across apps, that is a different Zitadel layout — one organisation, one project per app inside it, and a project grant model — and it is a rebuild of rows 3 to 9, not a change to the gateway or to Minimal, whose contract with the world is still five headers.

A tool for the list

Thirteen rows, four systems, three credentials, and every id from one row typed into the next: this is what a small command-line tool is for. The tool described here was not built in writing this book; its shape is fixed by the rows above and by what was verified in running them by hand, and it is specified closely enough to be built as the next piece of work.

Input is one file per app:

# healthyme.app.yaml
name: Healthy Me
slug: healthyme
scheme: com.littlebit.healthyme
platforms: [ios, android, mac]
roles: [member, premium]
providers:
  google: {client_id: ..., client_secret: ...}
  github: {client_id: ..., client_secret: ...}
  apple:  {client_id: com.littlebit.healthyme.signin, team_id: ..., key_id: ..., private_key_file: AuthKey.p8}
minimal:
  org: lbl
  space: live
  database: {host: 10.0.0.10, port: 5432, name: healthyme, user: minimalist}

Credentials come from the environment or root-only files, never from the input: ZITADEL_URL and the provisioner's token, Minimal's URL and server key and the appctl identity, the minimalist and <app>_app database passwords.

Commands, one per group of rows, each idempotent — it looks the object up by name before creating it, and prints the id either way:

Output is the ids, in the same YAML shape, appended to a <app>.state.yaml the next command reads — which is how the tool stays idempotent and how the Sleep Well ids above would be recorded rather than remembered.

Three decisions about the tool are worth making now rather than at the keyboard:

• It never holds a database superuser. Rows 1 and 2 are printed, not executed. Chapter 2's whole argument is that the superuser is used once, by a person, on db-1; a tool with that credential on app-1 undoes it.
• It never edits apisix.yaml in place. It renders a block; a person puts it in the file that is under version control and watches the diff. The gateway is the security boundary and its configuration is reviewed, not generated.
• It is the one place the member grant is automated. Chapter 4 left that as the open item; the tool holds the provisioner's token and can watch for new users — POST /v2/users with a filter by organisation, or an Actions V2 event target when the deployment moves to those — and grant member to each one it has not seen.

Where it lives is a decision for its owner; what it must not be is a second copy of any of these systems' logic. Every command above is a thin call onto an API this book already showed, and a command that needs an API none of them offers is a question to raise, not a route to invent.

Exercises

1. Do Sleep Well by hand, from the table, recording each id in a file as you go. Time it. Then write down which of the thirteen rows you got wrong on the first attempt, and whether the tool as specified would have caught it.

2. Switch the routing choice: give Sleep Well its own API host instead of the header. List every file that changes, run Chapter 5's echo test on both hosts, and say which choice you would ship.

3. The tool's smoke command needs a test user with a password in each app's organisation, and Chapter 4's production login policy forbids passwords. Decide how the test user signs in on production — a separate organisation, a provider account the tool holds, the session API with a different check — and write the one-paragraph justification that goes in the tool's README.

Every production step in Chapters 2 to 7 has a development twin on the Mac, and this chapter is the list of them in the order you run them. The point of dev is that the whole path — sign-in page, token, gateway, Minimal, row — works on one machine with no domain, no certificate and no provider account, so that a change to any of the three systems can be tried before it goes near production. Everything here was run on a Mac with Homebrew, an Apple-silicon processor, and the Postgres 18 that was already serving Minimal's local store.

Two things are different in kind, not just in value. Postgres is the cluster already on the Mac, on port 5433, and the book adds to it rather than building one. And APISIX does not run on macOS — its own build files refuse the platform — so the gateway runs inside a small Debian virtual machine managed by Lima, exactly the Debian package Chapter 5 installs, with the Mac's services reached from inside it as host.lima.internal. Zitadel, its login, and Minimal run natively from a terminal.

The layout, and the port each piece uses:

3047 rather than 3045 so that the Minimal you already run for Postman keeps its port; change it if nothing is there. Zitadel's public name is localhost:8080, which is the gateway, and its issuer is therefore http://localhost:8080.

Step 1 — Postgres: two roles, two databases, on the cluster you have

The Homebrew cluster already preloads pg_cron for the minimal store and already has vector. Chapter 2, Steps 4 and 5, with the local names:

psql -p 5433 -d postgres -v ON_ERROR_STOP=1 <<'SQL'
CREATE ROLE zitadel       LOGIN PASSWORD 'zitadel-dev-pw'   NOSUPERUSER NOCREATEDB NOCREATEROLE NOINHERIT CONNECTION LIMIT 20;
CREATE ROLE healthyme_app LOGIN PASSWORD 'healthyme-dev-pw' NOSUPERUSER NOCREATEDB NOCREATEROLE NOINHERIT CONNECTION LIMIT 30;
CREATE DATABASE zitadel   OWNER zitadel;
CREATE DATABASE healthyme OWNER healthyme_app;
REVOKE CONNECT ON DATABASE zitadel   FROM PUBLIC;
REVOKE CONNECT ON DATABASE healthyme FROM PUBLIC;
GRANT  CONNECT ON DATABASE healthyme TO minimalist;
\c healthyme
REVOKE ALL ON SCHEMA public FROM PUBLIC;
GRANT  ALL   ON SCHEMA public TO healthyme_app;
GRANT  USAGE ON SCHEMA public TO minimalist;
ALTER DEFAULT PRIVILEGES FOR ROLE healthyme_app IN SCHEMA public GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO minimalist;
ALTER DEFAULT PRIVILEGES FOR ROLE healthyme_app IN SCHEMA public GRANT USAGE, SELECT ON SEQUENCES TO minimalist;
SQL

The isolation probes from Chapter 2, Step 7 give permission denied for database here rather than a pg_hba.conf refusal, because the Mac's cluster trusts loopback and only REVOKE CONNECT is standing between the roles — which is the second layer doing its job on its own.

Minimal's store for dev is a separate database, because the one your Postman runs use should not gain a lbl organisation. pg_cron can live only in minimal, so the schema file is applied with its three pg_cron lines removed; the two audit-partition jobs are not part of dev:

psql -p 5433 -d postgres -c "CREATE DATABASE minimal_book OWNER minimalist"
psql -p 5433 -d minimal_book -c "CREATE EXTENSION IF NOT EXISTS vector; CREATE EXTENSION IF NOT EXISTS pg_trgm;"
grep -vE "CREATE EXTENSION IF NOT EXISTS pg_cron|SELECT cron\.schedule" database/minimal/postgres.sql > /tmp/postgres-no-cron.sql
psql -p 5433 -U minimalist -d minimal_book -v ON_ERROR_STOP=1 -q -f /tmp/postgres-no-cron.sql
psql -p 5433 -d minimal_book -Atc "select count(*) from information_schema.tables where table_schema='public'"

42

Step 2 — Zitadel and its login, from a terminal

The macOS binary and the login bundle from the same release as Chapter 3, into a working directory of your choosing (~/dev/zitadel below):

V=v4.17.3; mkdir -p ~/dev/zitadel && cd ~/dev/zitadel
curl -fsSLO https://github.com/zitadel/zitadel/releases/download/$V/zitadel-darwin-arm64.tar.gz
curl -fsSLO https://github.com/zitadel/zitadel/releases/download/$V/zitadel-login.tar.gz
tar xzf zitadel-darwin-arm64.tar.gz && mkdir -p login && tar xzf zitadel-login.tar.gz -C login
./zitadel-darwin-arm64/zitadel --version
head -c 32 /dev/urandom | base64 | tr -dc 'A-Za-z0-9' | head -c 32 > masterkey
brew install node@22

The configuration is Chapter 3's with the dev values — no TLS anywhere, the gateway's name and port as the public ones, the login's base URI through the gateway:

# ~/dev/zitadel/config.yaml
Log:
  Level: info
Port: 8081
ExternalDomain: localhost
ExternalPort: 8080
ExternalSecure: false
TLS:
  Enabled: false
Database:
  postgres:
    Host: 127.0.0.1
    Port: 5433
    Database: zitadel
    MaxOpenConns: 10
    MaxIdleConns: 5
    User:
      Username: zitadel
      Password: zitadel-dev-pw
      SSL:
        Mode: disable
DefaultInstance:
  Features:
    LoginV2:
      Required: true
      BaseURI: "http://localhost:8080/ui/v2/login"

The steps file is Chapter 3's with the token paths in the working directory and a dev password that does not have to be changed on first login:

# ~/dev/zitadel/steps.yaml
FirstInstance:
  InstanceName: littlebit-dev
  DefaultLanguage: en
  PatPath: /Users/<you>/dev/zitadel/provisioner.pat
  LoginClientPatPath: /Users/<you>/dev/zitadel/login-client.pat
  Org:
    Name: Littlebit
    Human:
      UserName: admin
      FirstName: Instance
      LastName: Admin
      Email:
        Address: admin@littlebit.localhost
        Verified: true
      Password: "Dev-Admin-Passw0rd!"
      PasswordChangeRequired: false
    Machine:
      Machine:
        Username: provisioner
        Name: Provisioning service account
      Pat:
        ExpirationDate: "2030-01-01T00:00:00Z"
    LoginClient:
      Machine:
        Username: login-client
        Name: Login V2 client
      Pat:
        ExpirationDate: "2030-01-01T00:00:00Z"

Schema, setup, start — the same three commands, and the same trap if FirstInstance is in the wrong file:

Z=./zitadel-darwin-arm64/zitadel
$Z init zitadel --config config.yaml
$Z setup --config config.yaml --steps steps.yaml --masterkeyFile masterkey --tlsMode disabled --init-projections
ls *.pat
$Z start --config config.yaml --masterkeyFile masterkey --tlsMode disabled

login-client.pat  provisioner.pat
... level=INFO msg="server is listening" ... address=[::]:8081

In a second terminal, the login, through its own entrypoint so that the token file is read. Started as a bare node apps/login/server.js it comes up, answers its health route, and every sign-in page says only Internal server error while its log says Flow initiation failed: it has no token and never tries to reach the API. Proxy variables in the shell are unset as well, since Node's HTTP client would otherwise consult them for 127.0.0.1:

cd ~/dev/zitadel/login
env -u HTTP_PROXY -u HTTPS_PROXY -u http_proxy -u https_proxy \
  HOSTNAME=127.0.0.1 PORT=3000 ZITADEL_API_URL=http://127.0.0.1:8081 \
  ZITADEL_SERVICE_USER_TOKEN_FILE=$HOME/dev/zitadel/login-client.pat NODE_ENV=production \
  ./entrypoint.sh $(brew --prefix node@22)/bin/node apps/login/server.js

▲ Next.js 16.2.11
- Local:         http://127.0.0.1:3000
✓ Ready in 0ms

The MetadataLookupWarning the login prints on a Mac is its OpenTelemetry resource detector looking for a Google Cloud metadata server; it is harmless. Chapter 3's checks apply: 8081 without a Host is 404; with Host: localhost:8080 the issuer is http://localhost:8080; 3000 answers /ui/v2/login/healthy with 200.

Step 3 — The app in Zitadel, dev variant

Chapter 4, with Z=http://127.0.0.1:8081 and -H "Host: localhost:8080" on every call until the gateway exists, and three differences:

• The native app has developmentMode: true and a second redirect URI, http://localhost:4200/auth/callback, so that a local web build can use it. The custom scheme stays.
• The login policy has allowUsernamePassword: true, so that one test user can sign in without a provider. Everything else about the policy is the same.
• One human user with a password, alice@healthyme.localhost, is created with POST /v2/users/new and granted member — she is the user the smoke test signs in as, and the reason passwords are on.
• The three providers are created with placeholder client ids and secrets. They appear on the login page, which is what you want to see; signing in with one against localhost cannot work, and Apple cannot even be configured for it.

Step 4 — The gateway in a VM

Lima runs a Linux VM from a template with one command and forwards every port a guest process listens on to the Mac's localhost, which is exactly the two things the gateway needs:

brew install lima
limactl start --name=book --tty=false --cpus=2 --memory=4 --disk=20 template://debian-12
limactl shell book -- getent hosts host.lima.internal

192.168.5.2     host.lima.internal

Inside the VM, Chapter 5, Step 1 verbatim — the arm64 repository path on an Apple-silicon Mac — then the two files, dev variant. config.yaml listens on 8080 and is otherwise Chapter 5's:

# /usr/local/apisix/conf/config.yaml (dev)
apisix:
  node_listen:
    - 8080
  enable_admin: false
deployment:
  role: data_plane
  role_data_plane:
    config_provider: yaml
nginx_config:
  error_log_level: warn

apisix.yaml is Chapter 5's with no ssls, no host: on the routes, host.lima.internal in every upstream, http://localhost:8080 in the discovery URL and the issuer, and the dev app's client id. The Lua is byte-identical. The full file is in Appendix D; the delta is four upstream addresses and three strings.

limactl copy config.yaml book:/tmp/config.yaml
limactl copy apisix.yaml book:/tmp/apisix.yaml
limactl shell book -- sudo bash -c 'install -m 0644 /tmp/config.yaml /tmp/apisix.yaml /usr/local/apisix/conf/ && cd /usr/local/apisix && apisix test && systemctl enable --now apisix'
curl -s http://localhost:8080/.well-known/openid-configuration | python3 -c "import sys,json; print(json.load(sys.stdin)['issuer'])"

http://localhost:8080

That line is the whole dev gateway working: the Mac's localhost:8080 is the VM's APISIX, which sent the request to the Mac's Zitadel with the Host it needs. Open http://localhost:8080/ui/console in a browser and sign in as admin to see the instance.

One trap with Lima's port forwarding, met while writing this: it forwards every guest port. If you use the same VM to rehearse the production units — Zitadel, the login, Minimal on Debian — those processes take over the Mac's 8081, 3000 and 3047 and quietly answer instead of the native ones, with a different instance and a different store. Keep the gateway VM to the gateway, or stop the rehearsal units before you go back to dev.

Step 5 — Minimal, from your working tree

The sample configuration with the store, the port and the key changed, run from a scratch directory so that the config.yml it reads is this one:

mkdir -p ~/dev/minimal && cd ~/dev/minimal
cp <repo>/sample/config.yml config.yml
# service.port 3047, mcp.service.port 3048, definition_store.schema minimal_book,
# definition_store.password '<the minimalist password>', auto_api.permission_fail_open false
openssl rand -hex 16 > enc.key
MINIMAL_ENCRYPTION_KEY=$(cat enc.key) <repo>/minimal -c config.yml
curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:3047/minimal/system/api/v1/org

400

Then Chapter 6, Steps 2 to 4, against M=http://127.0.0.1:3047, with the database registered at 127.0.0.1:5433 as minimalist with ssl_mode: "" — the Mac's cluster has no certificate to verify — and the device table created as healthyme_app.

Step 6 — The smoke test

smoke.sh is Chapter 7's eight steps as a script. It reads the two things that differ per environment from its environment — the gateway URL and where the login client's token is — and everything else from the app's ids:

#!/bin/bash
# smoke.sh -- the whole path: sign-in, token, gateway, Minimal, row scope.
set -euo pipefail
G=${G:-http://localhost:8080}
LOGIN_PAT=$(cat ${LOGIN_PAT_FILE:-$HOME/dev/zitadel/login-client.pat})
ORG=390676976144220428; PROJ=390677090732605708; CLIENT=390677214766629132
USER_LOGIN=alice@healthyme.localhost; USER_PASS='Alice-Test-Passw0rd!'
REDIRECT=com.littlebit.healthyme://auth/callback
TABLE=$G/minimal/api/rest/auto/v1/lbl/healthyme/pg/healthyme/device

VERIFIER=$(head -c 48 /dev/urandom | base64 | tr -dc 'A-Za-z0-9' | head -c 64)
CHALLENGE=$(printf %s "$VERIFIER" | openssl dgst -sha256 -binary | base64 | tr '+/' '-_' | tr -d '=')
SCOPE=$(python3 -c "import urllib.parse;print(urllib.parse.quote('openid profile email offline_access urn:zitadel:iam:org:id:$ORG urn:zitadel:iam:org:project:id:$PROJ:aud urn:zitadel:iam:org:project:roles'))")

step() { printf '%-4s %-44s ' "$1" "$2"; }
ok()   { echo "ok  $1"; }
fail() { echo "FAIL $1"; exit 1; }

step 1 "authorize redirects to the login"
AUTH=$(curl -s -o /dev/null -w '%{redirect_url}' "$G/oauth/v2/authorize?client_id=$CLIENT&redirect_uri=$REDIRECT&response_type=code&scope=$SCOPE&code_challenge=$CHALLENGE&code_challenge_method=S256&prompt=login")
[[ "$AUTH" == *"/ui/v2/login/login?authRequest="* ]] && ok "${AUTH##*=}" || fail "$AUTH"
AUTH=${AUTH##*authRequest=}

step 2 "session for the test user"
S=$(curl -s -H "Authorization: Bearer $LOGIN_PAT" -H "Content-Type: application/json" -X POST $G/v2/sessions \
   -d "{\"checks\":{\"user\":{\"loginName\":\"$USER_LOGIN\"},\"password\":{\"password\":\"$USER_PASS\"}}}")
SID=$(echo "$S" | python3 -c "import sys,json;print(json.load(sys.stdin)['sessionId'])") && ok "$SID" || fail "$S"
STOK=$(echo "$S" | python3 -c "import sys,json;print(json.load(sys.stdin)['sessionToken'])")

step 3 "finish the auth request"
CB=$(curl -s -H "Authorization: Bearer $LOGIN_PAT" -H "Content-Type: application/json" -X POST $G/v2/oidc/auth_requests/$AUTH \
   -d "{\"session\":{\"sessionId\":\"$SID\",\"sessionToken\":\"$STOK\"}}")
CODE=$(echo "$CB" | python3 -c "import sys,json,urllib.parse;u=json.load(sys.stdin)['callbackUrl'];print(urllib.parse.parse_qs(urllib.parse.urlparse(u).query)['code'][0])") && ok "code" || fail "$CB"

step 4 "exchange the code; claim present, issuer right"
T=$(curl -s -X POST $G/oauth/v2/token -d "grant_type=authorization_code&code=$CODE&redirect_uri=$REDIRECT&client_id=$CLIENT&code_verifier=$VERIFIER" \
   | python3 -c "import sys,json;print(json.load(sys.stdin)['access_token'])")
MUID=$(python3 -c "import sys,json,base64;p='$T'.split('.')[1];d=json.loads(base64.urlsafe_b64decode(p+'='*(-len(p)%4)));assert d['iss']=='$G';print(d['minimal_user_id'])") && ok "$MUID" || fail "bad token"
[ ${#MUID} -eq 26 ] || fail "claim is not a ULID"

step 5 "insert a device with a wrong user_id in the body"
DEV=$(python3 -c "import secrets;print('01SMOKE'+secrets.token_hex(10).upper()[:19])")
R=$(curl -s -o /dev/null -w '%{http_code}' -X POST "$TABLE" -H "Authorization: Bearer $T" -H "Content-Type: application/json" \
   -d "[{\"device_id\":\"$DEV\",\"user_id\":\"SPOOFED\",\"public_key\":\"smoke\",\"platform\":\"test\",\"created_at\":null,\"last_seen\":null}]")
[ "$R" = 201 ] && ok 201 || fail "$R"

step 6 "read back: every row is mine"
ROWS=$(curl -s "$TABLE?ps=100&pg=0" -H "Authorization: Bearer $T")
python3 -c "import sys,json;r=json.loads('''$ROWS''');assert r and all(x['user_id']=='$MUID' for x in r);print(len(r))" >/dev/null && ok "$(echo "$ROWS" | python3 -c "import sys,json;print(len(json.load(sys.stdin)),'rows')")" || fail "$ROWS"

step 7 "org record refused to a member"
R=$(curl -s -o /dev/null -w '%{http_code}' $G/minimal/system/api/v1/org -H "Authorization: Bearer $T"); [ "$R" = 403 ] && ok 403 || fail "$R"

step 8 "no token refused at the gateway"
R=$(curl -s -o /dev/null -w '%{http_code}' $G/minimal/system/api/v1/org); [ "$R" = 401 ] && ok 401 || fail "$R"

1    authorize redirects to the login             ok  V2_390680877182550284
2    session for the test user                    ok  390680877216039180
3    finish the auth request                      ok  code
4    exchange the code; claim present, issuer right ok  01M2F0KNNC4M7Z1HEJHHG80JGE
5    insert a device with a wrong user_id in the body ok  201
6    read back: every row is mine                 ok  2 rows
7    org record refused to a member               ok  403
8    no token refused at the gateway              ok  401

Against production, G=https://api.example.com for the API steps and the login client's token from app-1 — with the caveat Chapter 8's third exercise raises, that the production login policy has no password user to sign in as, which is a decision to make before the test can run there.

Sign in like a person, once

The smoke test proves the machinery; a browser proves the page. Open, in the Mac's browser:

http://localhost:8080/oauth/v2/authorize?client_id=390677214766629132&redirect_uri=com.littlebit.healthyme%3A%2F%2Fauth%2Fcallback&response_type=code&scope=openid%20profile%20email%20urn%3Azitadel%3Aiam%3Aorg%3Aid%3A390676976144220428&code_challenge=E9Melhoa2OwvFrEMTJguCHaoeK1t8URWbuGJSstw-cM&code_challenge_method=S256&prompt=login

The page is the hosted login, through the gateway: "Welcome back!", a username field, "Register new user", and Google, GitHub and Apple under "or sign in with". Leave the organisation scope out of the URL and the three buttons disappear, because the page is then the instance's default organisation — which is the mistake to recognise, not the page to debug. Sign in as alice — username, then password — and the flow ends on the login's own page, "Welcome Alice Tester! You are signed in." The callback to com.littlebit.healthyme://auth/callback?code=... is what a device with the app installed receives at that moment; a browser with nothing registered for the scheme has nowhere to deliver it, and the smoke test is the way to see the code and exchange it from a Mac.

Is dev complete? A closing checklist

• The smoke test passes, all eight steps, from a fresh shell.
• psql -p 5433 -l shows zitadel, healthyme and minimal_book, and your Postman store is untouched.
• The VM runs APISIX and nothing else, and limactl shell book -- ss -ltn shows only 8080.
• The three native processes are started the way this chapter says — the login through entrypoint.sh, with no proxy variables; Minimal with permission_fail_open: false.
• The dev app's developmentMode and the dev organisation's password login are the only two settings that differ from Chapter 4, and both are listed here so that nobody carries them to production.

Exercises

1. Stop the VM (limactl stop book) and run the smoke test. Read which step fails and what it prints, and compare it with stopping Zitadel or Minimal instead. Three different first failures for three different outages.

2. Point the dev gateway's minimal upstream at your Postman Minimal on 3045 instead of 3047, without changing anything else, and run the smoke test. Explain the failure from the store the two instances use — and then say what would have happened if permission_fail_open had been true on that instance.

3. Write the Lima template that forwards only port 8080 from the VM, start a second VM from it, and confirm the rehearsal trap in Step 4 can no longer happen. Decide whether the book's one-command limactl start was the right default.

Three services, one database cluster, and a boundary that is only as good as its configuration on the day someone asks. This chapter is what changes after the first deployment: the secrets and how each is rotated, what each system tells you when something is wrong, what breaks when one piece is down, how each piece is upgraded, and the one list to read before saying the deployment is done.

The secrets, and what each one guards

Nothing in that table is a person's password or a person's token. People authenticate at the provider; their tokens live an hour on their own devices; the servers never store either. That is the property to protect when the next change is proposed.

What each piece says when it is unhappy

APISIX writes /usr/local/apisix/logs/error.log. Every refusal at the boundary is one line there naming the plugin and the reason — No bearer token found in request, token is signed by unexpected algorithm, audience list does not contain the client id, serverless-post-function exits with http status code 403 — and a rise in any of them is the earliest sign of a client build that is wrong or a token that is not what the gateway expects. Whether access.log is written is nginx_config.http.enable_access_log in config.yaml, which the book's file leaves at the package default; before relying on it, set it explicitly and give it a format that omits the Authorization header.

Zitadel logs to its journal, and one line matters more than the rest: unable to set instance means a request arrived with a Host that names no instance — a gateway misconfiguration, a health check hitting the port directly, or a new domain not yet added. Its metrics are at /debug/metrics on the internal port, OpenTelemetry format, never routed through the gateway. The login's journal shows Flow initiation failed when it cannot reach the API with the token it has, and its /ui/v2/login/healthy says only that the process is up.

Minimal logs to its journal as Volume II, Chapter 6 describes, and its audit trail is the record of who did what: every request through the gateway lands there with the ULID as the actor, and the appctl identity is the only non-ULID actor that should ever appear. A sys: or admin actor in the trail that is not a shell on app-1 is the alarm.

Postgres has log_connections on from Chapter 2, so every session start is a line in its log with the role and database, and pg_stat_ssl is the standing check that each one was encrypted.

Two health checks, from outside, cover the whole thing: GET https://auth.example.com/.well-known/openid-configuration answers 200 when the gateway, Zitadel and the instance domain are all right; GET https://api.example.com/minimal/system/api/v1/org with no token answers 401 when the gateway and its route are up, and 502 when Minimal is not. Chapter 9's smoke test is the deeper check, on a timer, from a host that is not app-1.

What breaks when one piece is down

The second row is the one to remember. The gateway's local verification is what makes an identity provider outage a "no new sign-ins for a while" event rather than an "app is down" event, and the one-hour access token is the price — a revoked account keeps working for up to an hour either way.

Upgrading each piece

Zitadel releases carry migrations. The order is: back up the database; stop zitadel-login and zitadel; replace the binary; run zitadel setup --config ... --steps ... --masterkeyFile ... --tlsMode external as the zitadel user, which applies the migrations and creates nothing new; start both units; check the issuer and the login's health route. Replace the login bundle with the same release's, always — the two are built together. Read the release notes for a change to InstanceHostHeaders, to the Actions V1 sunset, or to Login V2's environment, because those are the three things this book depends on that Zitadel has said it will change.

APISIX upgrades through apt. The package replaces /usr/local/apisix, and the two files under conf/ are yours: keep them in version control, and after the upgrade diff the package's new config-default.yaml against the previous one for a renamed key, then apisix test before systemctl restart apisix. The enable_http2 placement in Chapter 5 was one such rename.

Minimal is Volume II, Chapter 9's process: a new binary, the same configuration, a restart, and the schema file's changes applied to the store as minimalist as Chapter 2 did. Run the smoke test after.

Postgres minor versions are apt upgrades with a restart; a major version is a pg_upgradecluster with the extensions reinstalled for the new major first, and it is the one upgrade to rehearse on the Mac before doing on db-1.

Node, for the login, follows NodeSource's current LTS; the bundle states which major it was built for in its package.json.

Adding a platform, adding a provider, adding a role

Three routine changes, each smaller than it sounds:

• A platform is Chapter 4, Step 4 again — a second native application in the same project with its own scheme — and then either the same route (both client ids are in the same audience family: add the second to a claim_validator.audience check that accepts either, or run two routes) or a second route keyed on a header. The gateway's client_id is the setting that changes.
• A provider is Chapter 4, Step 6 again, and nothing else: the app already shows whatever the organisation's login policy activates.
• A role is Chapter 4, Step 3 plus Chapter 6's templates that should name it. The gateway copies whatever roles the token carries; it does not have a list.

The closing checklist for the whole deployment

Each chapter had its own; this is the one to read once, at the end, against the live system.

• Only APISIX has a public port on app-1, and only app-1's private address can reach Postgres on db-1.
• Every application session to Postgres is TLS and each role can reach exactly one database — pg_stat_ssl and the three refused connections from Chapter 2.
• The issuer is https://auth.example.com and the gateway's valid_issuers says the same.
• The Healthy Me project has authorizationRequired and projectAccessRequired, its login policy has passwords off, and a user from another organisation is refused.
• The action is bound, and a fresh user's first token carries a 26-character minimal_user_id.
• The echo test from Chapter 5 shows the five headers with the gateway's values and nothing the caller sent.
• Minimal runs with permission_fail_open: false, agent_identity absent, mcp.enabled: false, and every user-scoped table has a row scope on X-User-Id.
• The smoke test passes on production, from a host that is not app-1.
• The backup set holds the dumps, the masterkey, the encryption key and the server key, in two places, and one restore has been rehearsed.
• The two .pat files have never been copied off app-1, and the provisioner's has never been given to a service.

Exercises

1. Rotate the minimalist password end to end: ALTER ROLE, the store connection in Minimal's configuration, and the registered database on the Healthy Me project — which is a PUT on the project carrying the whole database list, per Volume I, Chapter 3. Time it, and note which of the three you forgot on the first pass.

2. Take zitadel down for ten minutes while the smoke test runs on a one-minute timer. Record which step fails and when, then bring it back and record when the test passes again. Compare with the table above.

3. Upgrade Zitadel to the next patch release on the Mac first, following the order given, and write down every line the release notes contain about the three things this book depends on. If there are none, write that down too — it is what makes the production upgrade a fifteen-minute job.

Every configuration file the chapters showed in fragments, complete, for production first and then the dev variants where they differ. Secrets are placeholders in angle brackets. Paths are the ones the chapters install to. Where a file is Volume II's sample with a few values changed, the changed values are listed and the sample is not reprinted.

db-1 — Postgres (Chapter 2)

/etc/postgresql/18/main/conf.d/90-minimal.conf

listen_addresses = 'localhost,10.0.0.10'
shared_preload_libraries = 'pg_cron'
cron.database_name = 'minimal'
password_encryption = 'scram-sha-256'
ssl = on
ssl_cert_file = '/etc/postgresql/18/main/server.crt'
ssl_key_file  = '/etc/postgresql/18/main/server.key'
log_connections = on
log_disconnections = on

/etc/postgresql/18/main/pg_hba.conf

# TYPE  DATABASE   USER           ADDRESS        METHOD
local   all        postgres                      peer
hostssl zitadel    zitadel        10.0.0.20/32   scram-sha-256
hostssl minimal    minimalist     10.0.0.20/32   scram-sha-256
hostssl healthyme  minimalist     10.0.0.20/32   scram-sha-256
hostssl healthyme  healthyme_app  10.0.0.20/32   scram-sha-256

Roles, databases and grants — run once as postgres on the local socket:

CREATE ROLE zitadel       LOGIN PASSWORD '<zitadel-password>'   NOSUPERUSER NOCREATEDB NOCREATEROLE NOINHERIT CONNECTION LIMIT 20;
CREATE ROLE minimalist    LOGIN PASSWORD '<minimalist-password>' NOSUPERUSER NOCREATEDB NOCREATEROLE NOINHERIT CONNECTION LIMIT 60;
CREATE ROLE healthyme_app LOGIN PASSWORD '<healthyme-password>' NOSUPERUSER NOCREATEDB NOCREATEROLE NOINHERIT CONNECTION LIMIT 10;
CREATE DATABASE zitadel   OWNER zitadel;
CREATE DATABASE minimal   OWNER minimalist;
CREATE DATABASE healthyme OWNER healthyme_app;
REVOKE CONNECT ON DATABASE zitadel   FROM PUBLIC;
REVOKE CONNECT ON DATABASE minimal   FROM PUBLIC;
REVOKE CONNECT ON DATABASE healthyme FROM PUBLIC;
GRANT  CONNECT ON DATABASE healthyme TO minimalist;
\c minimal
CREATE EXTENSION IF NOT EXISTS vector;
CREATE EXTENSION IF NOT EXISTS pg_trgm;
CREATE EXTENSION IF NOT EXISTS pg_cron;
GRANT USAGE ON SCHEMA cron TO minimalist;
REVOKE ALL ON SCHEMA public FROM PUBLIC;
GRANT  ALL ON SCHEMA public TO minimalist;
\c healthyme
REVOKE ALL ON SCHEMA public FROM PUBLIC;
GRANT  ALL   ON SCHEMA public TO healthyme_app;
GRANT  USAGE ON SCHEMA public TO minimalist;
ALTER DEFAULT PRIVILEGES FOR ROLE healthyme_app IN SCHEMA public GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO minimalist;
ALTER DEFAULT PRIVILEGES FOR ROLE healthyme_app IN SCHEMA public GRANT USAGE, SELECT ON SEQUENCES TO minimalist;

Then Minimal's postgres.sql from the database repository, applied as minimalist to minimal.

app-1 — Zitadel (Chapter 3)

/etc/zitadel/config.yaml — root:zitadel, 0640

Log:
  Level: info

Port: 8081
ExternalDomain: auth.example.com
ExternalPort: 443
ExternalSecure: true
TLS:
  Enabled: false

Database:
  postgres:
    Host: 10.0.0.10
    Port: 5432
    Database: zitadel
    MaxOpenConns: 10
    MaxIdleConns: 5
    User:
      Username: zitadel
      Password: <zitadel-password>
      SSL:
        Mode: verify-full
        RootCert: /etc/ssl/certs/internal-ca.pem

DefaultInstance:
  Features:
    LoginV2:
      Required: true
      BaseURI: "https://auth.example.com/ui/v2/login"
  OIDCSettings:
    AccessTokenLifetime: 1h
    IdTokenLifetime: 1h
    RefreshTokenIdleExpiration: 720h
    RefreshTokenExpiration: 2160h

/etc/zitadel/steps.yaml — root:zitadel, 0640; used once, by zitadel setup --steps

FirstInstance:
  InstanceName: littlebit
  DefaultLanguage: en
  PatPath: /var/lib/zitadel/provisioner.pat
  LoginClientPatPath: /var/lib/zitadel/login-client.pat
  Org:
    Name: Littlebit
    Human:
      UserName: admin
      FirstName: Instance
      LastName: Admin
      Email:
        Address: admin@example.com
        Verified: true
      Password: "<a strong initial password>"
      PasswordChangeRequired: true
    Machine:
      Machine:
        Username: provisioner
        Name: Provisioning service account
      Pat:
        ExpirationDate: "2030-01-01T00:00:00Z"
    LoginClient:
      Machine:
        Username: login-client
        Name: Login V2 client
      Pat:
        ExpirationDate: "2030-01-01T00:00:00Z"

/etc/zitadel/masterkey — root:zitadel, 0640, 32 characters from /dev/urandom. /etc/zitadel/provisioner.pat — root:root, 0600. /etc/zitadel/login-client.pat — root:zitadel-login, 0640. Both written by setup to /var/lib/zitadel/ and moved.

/etc/zitadel/login.env — root:zitadel-login, 0640

NODE_ENV=production
PORT=3000
HOSTNAME=127.0.0.1
ZITADEL_API_URL=http://127.0.0.1:8081
ZITADEL_SERVICE_USER_TOKEN_FILE=/etc/zitadel/login-client.pat

/etc/systemd/system/zitadel.service

[Unit]
Description=Zitadel identity server
After=network-online.target
Wants=network-online.target

[Service]
User=zitadel
Group=zitadel
WorkingDirectory=/var/lib/zitadel
ExecStart=/usr/local/bin/zitadel start --config /etc/zitadel/config.yaml --masterkeyFile /etc/zitadel/masterkey --tlsMode external
Restart=on-failure
RestartSec=5
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
ReadWritePaths=/var/lib/zitadel

[Install]
WantedBy=multi-user.target

/etc/systemd/system/zitadel-login.service

[Unit]
Description=Zitadel Login V2
After=network-online.target zitadel.service
Wants=network-online.target

[Service]
User=zitadel-login
Group=zitadel-login
WorkingDirectory=/opt/zitadel-login
EnvironmentFile=/etc/zitadel/login.env
ExecStart=/opt/zitadel-login/entrypoint.sh /usr/bin/node apps/login/server.js
Restart=on-failure
RestartSec=5
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true

[Install]
WantedBy=multi-user.target

Zitadel — the action (Chapter 4)

minimal-user-id.js, registered on the app's organisation and bound to flow 2, trigger 5

/**
 * Complement Token, trigger "Pre access token creation".
 * Gives every user a server-assigned ULID the first time a token is minted for
 * them, keeps it in user metadata, and asserts it as a claim on every token.
 */
function minimalUserId(ctx, api) {
  var KEY = 'minimal_user_id';
  var existing = null;
  var md = ctx.v1.user.getMetadata();
  if (md && md.metadata) {
    for (var i = 0; i < md.metadata.length; i++) {
      if (md.metadata[i].key === KEY) { existing = md.metadata[i].value; break; }
    }
  }
  if (!existing) {
    existing = ulid();
    api.v1.user.setMetadata(KEY, existing);
  }
  api.v1.claims.setClaim(KEY, existing);
}

function ulid() {
  var A = '0123456789ABCDEFGHJKMNPQRSTVWXYZ';
  var t = Date.now();
  var out = '';
  for (var i = 9; i >= 0; i--) { out = A.charAt(t % 32) + out; t = Math.floor(t / 32); }
  for (var j = 0; j < 16; j++) { out += A.charAt(Math.floor(Math.random() * 32)); }
  return out;
}

The app's scope list, verbatim, with the two ids from Chapter 4:

openid profile email offline_access urn:zitadel:iam:org:id:<ORG> urn:zitadel:iam:org:project:id:<PROJ>:aud urn:zitadel:iam:org:project:roles

app-1 — APISIX (Chapter 5)

/usr/local/apisix/conf/config.yaml

apisix:
  node_listen:
    - 80
  enable_http2: true
  ssl:
    enable: true
    listen:
      - port: 443
  enable_admin: false
deployment:
  role: data_plane
  role_data_plane:
    config_provider: yaml
nginx_config:
  error_log_level: warn

/usr/local/apisix/conf/apisix.yaml — root:root, 0640; the ssls block is rendered by the certbot deploy hook and shown here with placeholders

ssls:
  - id: public
    snis:
      - auth.example.com
      - api.example.com
    cert: |
      <fullchain.pem>
    key: |
      <privkey.pem>

upstreams:
  - id: zitadel
    nodes:
      "127.0.0.1:8081": 1
    type: roundrobin
    pass_host: pass
  - id: zitadel-grpc
    nodes:
      "127.0.0.1:8081": 1
    type: roundrobin
    scheme: grpc
    pass_host: pass
  - id: zitadel-login
    nodes:
      "127.0.0.1:3000": 1
    type: roundrobin
    pass_host: pass
  - id: minimal
    nodes:
      "127.0.0.1:3045": 1
    type: roundrobin
    pass_host: node
  - id: acme
    nodes:
      "127.0.0.1:8402": 1
    type: roundrobin

routes:
  - id: acme-challenge
    uri: /.well-known/acme-challenge/*
    priority: 100
    upstream_id: acme

  - id: zitadel-login
    host: auth.example.com
    uri: /ui/v2/login/*
    upstream_id: zitadel-login
  - id: zitadel-grpc
    host: auth.example.com
    uri: /*
    vars: [["http_content_type", "~~", "^application/grpc"]]
    priority: 10
    upstream_id: zitadel-grpc
  - id: zitadel
    host: auth.example.com
    uri: /*
    upstream_id: zitadel

  - id: minimal-api
    host: api.example.com
    uri: /minimal/*
    upstream_id: minimal
    plugins:
      serverless-pre-function:
        phase: rewrite
        functions:
          - |
            return function(conf, ctx)
              for _, h in ipairs({"X-Org-Id","X-Project-Id","X-Space-Id","X-User-Id","X-User-Roles",
                                  "LB-Access-Token","X-Userinfo","X-Access-Token","X-ID-Token"}) do
                ngx.req.clear_header(h)
              end
              local args = ngx.req.get_uri_args()
              if args["lb-access-token"] ~= nil then
                args["lb-access-token"] = nil
                ngx.req.set_uri_args(args)
              end
            end
      openid-connect:
        client_id: "<the native app's client id>"
        client_secret: "not-used-for-bearer-validation"
        discovery: "https://auth.example.com/.well-known/openid-configuration"
        bearer_only: true
        use_jwks: true
        token_signing_alg_values_expected: RS256
        claim_validator:
          issuer:
            valid_issuers: ["https://auth.example.com"]
          audience:
            required: true
            match_with_client_id: true
        set_userinfo_header: true
        set_access_token_header: false
        set_id_token_header: false
        set_refresh_token_header: false
        unauth_action: deny
      serverless-post-function:
        phase: access
        functions:
          - |
            return function(conf, ctx)
              local core = require("apisix.core")
              local raw = ngx.req.get_headers()["X-Userinfo"]
              if not raw then return 401, {message = "no verified identity"} end
              local claims = core.json.decode(ngx.decode_base64(raw))
              local uid = claims and claims["minimal_user_id"]
              if type(uid) ~= "string" or #uid ~= 26 or not uid:match("^[0-9A-HJKMNP-TV-Z]+$") then
                return 403, {message = "token carries no server-assigned user id"}
              end
              local roles = {}
              local r = claims["urn:zitadel:iam:org:project:roles"]
              if type(r) == "table" then
                for k, _ in pairs(r) do roles[#roles + 1] = k end
              end
              table.sort(roles)
              ngx.req.clear_header("X-Userinfo")
              ngx.req.clear_header("Authorization")
              ngx.req.set_header("X-Org-Id", "lbl")
              ngx.req.set_header("X-Project-Id", "healthyme")
              ngx.req.set_header("X-Space-Id", "live")
              ngx.req.set_header("X-User-Id", uid)
              ngx.req.set_header("X-User-Roles", table.concat(roles, ","))
            end
      limit-count:
        count: 600
        time_window: 60
        key_type: var
        key: http_x_user_id
        rejected_code: 429
        policy: local
#END

/etc/letsencrypt/renewal-hooks/deploy/apisix.sh — 0755

#!/bin/bash
set -euo pipefail
LIVE=/etc/letsencrypt/live/auth.example.com
RULES=/usr/local/apisix/conf/apisix.yaml
python3 - "$LIVE" "$RULES" <<'PY'
import re, sys
live, rules = sys.argv[1], sys.argv[2]
ind = lambda t: "\n".join("      " + l for l in t.strip().splitlines())
cert = open(f"{live}/fullchain.pem").read(); key = open(f"{live}/privkey.pem").read()
block = ("ssls:\n  - id: public\n    snis:\n      - auth.example.com\n      - api.example.com\n"
         "    cert: |\n" + ind(cert) + "\n    key: |\n" + ind(key) + "\n")
s = open(rules).read()
s = re.sub(r"(?ms)^ssls:\n.*?(?=^upstreams:)", "", s)
s = s.replace("upstreams:", block + "\nupstreams:", 1)
open(rules, "w").write(s)
PY

app-1 — Minimal (Chapter 6)

/etc/minimal/config.yml — root:minimal, 0640. Volume II, Chapter 9's sample/config.yml with these values; every other key is the sample's:

service:
  host: "127.0.0.1"
  port: "3045"
  environment: "production"
  server_key: "<a real secret, generated for this deployment>"
  log_level: "info"
  debug_mode: false
  show_request_details: false
  strict_local_host: true

definition_store:
  type: postgres
  host: 10.0.0.10
  port: "5432"
  schema: minimal
  username: minimalist
  password_source: file
  password: '<minimalist-password>'
  ssl_mode: "verify-full"
  max_open_connections: 20
  encrypt_database: true
  encryption_key_source: env
  encryption_key: ''

auto_api:
  permission_fail_open: false

access_token:
  max_per_user: 5
  default_expiry_days: 30
  retain_dead_days: 30
  sweep_interval_secs: 86400

# agent_identity: omitted

mcp:
  enabled: false

/etc/minimal/minimal.env — root:minimal, 0640

MINIMAL_ENCRYPTION_KEY=<openssl rand -hex 16>

/etc/systemd/system/minimal.service

[Unit]
Description=Minimal API server
After=network-online.target
Wants=network-online.target

[Service]
User=minimal
Group=minimal
WorkingDirectory=/var/lib/minimal
EnvironmentFile=/etc/minimal/minimal.env
ExecStart=/usr/local/bin/minimal -c /etc/minimal/config.yml
Restart=on-failure
RestartSec=5
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
ReadWritePaths=/var/lib/minimal

[Install]
WantedBy=multi-user.target

The app's first table, created as healthyme_app:

CREATE TABLE device (
  device_id   CHAR(26)    PRIMARY KEY,
  user_id     CHAR(26)    NOT NULL,
  public_key  TEXT        NOT NULL,
  platform    TEXT        NOT NULL,
  created_at  TIMESTAMPTZ NOT NULL DEFAULT now(),
  last_seen   TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX device_user_idx ON device (user_id);

dev — the variants (Chapter 9)

~/dev/zitadel/config.yaml

Log:
  Level: info
Port: 8081
ExternalDomain: localhost
ExternalPort: 8080
ExternalSecure: false
TLS:
  Enabled: false
Database:
  postgres:
    Host: 127.0.0.1
    Port: 5433
    Database: zitadel
    MaxOpenConns: 10
    MaxIdleConns: 5
    User:
      Username: zitadel
      Password: zitadel-dev-pw
      SSL:
        Mode: disable
DefaultInstance:
  Features:
    LoginV2:
      Required: true
      BaseURI: "http://localhost:8080/ui/v2/login"

~/dev/zitadel/steps.yaml — Chapter 3's with InstanceName: littlebit-dev, the two token paths under ~/dev/zitadel/, and PasswordChangeRequired: false.

The login, from ~/dev/zitadel/login:

env -u HTTP_PROXY -u HTTPS_PROXY -u http_proxy -u https_proxy \
  HOSTNAME=127.0.0.1 PORT=3000 ZITADEL_API_URL=http://127.0.0.1:8081 \
  ZITADEL_SERVICE_USER_TOKEN_FILE=$HOME/dev/zitadel/login-client.pat NODE_ENV=production \
  ./entrypoint.sh $(brew --prefix node@22)/bin/node apps/login/server.js

APISIX in the Lima VM, /usr/local/apisix/conf/config.yaml:

apisix:
  node_listen:
    - 8080
  enable_admin: false
deployment:
  role: data_plane
  role_data_plane:
    config_provider: yaml
nginx_config:
  error_log_level: warn

and apisix.yaml: the production file with no ssls block and no acme upstream or route, no host: on any route, these four upstream addresses —

upstreams:
  - id: zitadel
    nodes:
      "host.lima.internal:8081": 1
    type: roundrobin
    pass_host: pass
  - id: zitadel-grpc
    nodes:
      "host.lima.internal:8081": 1
    type: roundrobin
    scheme: grpc
    pass_host: pass
  - id: zitadel-login
    nodes:
      "host.lima.internal:3000": 1
    type: roundrobin
    pass_host: pass
  - id: minimal
    nodes:
      "host.lima.internal:3047": 1
    type: roundrobin
    pass_host: node

— and, in the minimal-api route, discovery: "http://localhost:8080/.well-known/openid-configuration", valid_issuers: ["http://localhost:8080"], and the dev app's client id. The Lua is unchanged.

Minimal, ~/dev/minimal/config.yml: the sample with service.port "3047", mcp.service.port "3048", definition_store.schema minimal_book, the minimalist password, and auto_api.permission_fail_open false; started with MINIMAL_ENCRYPTION_KEY from a file kept beside it.

smoke.sh is printed in Chapter 9, Step 6, and is the same file for both environments.