An evening of “the server is fine” while the screen said otherwise — what actually broke each time, and the one thing all four had in common.
The thread. Four separate faults, and in every one of them the evidence that would have identified it in seconds sat somewhere I could not reach: a Cloudflare build log, a browser tab, an address bar, a config file on a phone. Every time, I checked the things I could reach, found them healthy, and said so — which was true, useless, and misleading in roughly equal measure.
Moving the API alias to api.katomia.com silently broke the front-end deploy. build-static.sh computed the service-binding name as ${ALIAS%%.*} — “the first label of the alias's workers.dev host” — which was true only while the alias was a workers.dev host. The custom domain made it yield api, and no Worker is called that.
Service binding 'HAPI' references Worker 'api' which was not found [10143]pilot · both dists matching their sourceA patch script hit a failed assertion and exited before its write step, so five edits were lost while the explanatory note describing them was saved. The page then documented behaviour it did not have. I reported the change as shipped because the commit existed.
A browser left open across a deploy keeps running the JavaScript it loaded. Nothing tells it otherwise, and nothing tells the person looking at it either.
The dashboard resolves the API base as new URL('../') — which is /future/ from /future/v3/ and / from /future/v3. The router matched both forms and served the page for each, so the slash-less URL loaded a working-looking dashboard whose every call went to the site root, which 302s to the home page. The page parsed HTML as JSON.
got a web page, not data, from /?…action=signersThese are four different bugs — a shell expansion, a script exiting early, browser caching, a URL normalisation. They do not share a cause. They share a diagnostic shape:
| Could I see it? | What I concluded instead | |
|---|---|---|
| Cloudflare build log | No — token is read-only and cannot see Workers Builds | “the deploy must be slow” |
| The browser's console | No | “the server is healthy, so it must be cache” |
| The URL being visited | No — I assumed the one in an old screenshot | “I cannot reproduce it” |
| A config file on a phone | No | correct, but only after three claims had expired |
In each case I had a green signal from the half of the system I could observe, and treated it as evidence about the whole. That is the actual failure, and it is mine: a check that cannot fail on the broken half is not evidence, and reporting it as reassurance is worse than saying nothing.
Exactly one thing broke the deadlock, and it was not more server-side probing. It was making the client say what it tried and what came back. That message was written an hour before it was needed, for a different reason, and it identified the fault on its first run. Every hour of this evening's investigation could have been replaced by it.
getJson now reads the body before deciding. A non-2xx carries its status, the path and the start of the response; a 2xx that is not JSON quotes what did arrive and says a sign-in or error page would look like this. The two likeliest causes both send HTML, and both used to surface as Unexpected token '<' — a message that names the symptom and hides the cause.
It compares the build stamp it booted with against build-info.json on each refresh and offers a reload once when they differ. Its own failure is swallowed deliberately: a courtesy must not take down the thing it decorates.
The router redirects /future/v3, /future/v3/help and /future/v3/help/business to their slash forms, preserving the query string — the convention /qa already followed. And the page derives its base from a known anchor in its own path rather than ../, so it survives any host that serves it without one.
Not derived from a hostname. A Worker's name and the address it answers on are independent facts, and the build script now says so rather than assuming.
ops/clash-check.sh reports whether the base branch moved and separates a source clash from a generated one. build-static.sh with no argument builds both dists — the stale dist-gcp caught two sessions in one day, which is a trap in the tool rather than two mistakes.
This is the biggest remaining hole. The katui workflow already has a smoke job meant to curl the live routes after a push, and it did not catch an hour of failed deploys. Give it something version-shaped to assert: build-info.json already carries source_sha, so “the bundle in git” and “the bundle being served” can be compared rather than assumed equal. Roughly ten lines, and it closes the single worst case in this list. kpayuk and kmigrate have the same blind spot and have not been bitten yet.
The lost-edits case was a script whose write came after its last assertion. Any edit script should verify its result after writing — grep for what it claimed to add — rather than trusting that reaching the end meant success. This is a habit, not a tool, and it is mine to hold.
Three of the four cases would have been shorter if the first message had included the exact URL being visited. The page could show its own resolved API base in the raw-links row it already has, so a screenshot carries it without anyone having to think of it.
This is the obvious answer to “I cannot see the client”, and it has bitten this estate before. Worth being precise about which part is dangerous, because the useful part and the dangerous part are separable.
An open POST /log that accepts arbitrary text is an unauthenticated write endpoint with unbounded storage, it will faithfully record whatever happens to be in scope — tokens, account ids, query strings, a signing key someone pasted into a form — and a log viewer that renders it becomes stored XSS aimed squarely at an admin. That is three separate incidents waiting in one convenience.
Do not ship logs. Ship a fingerprint: on a failure, the page reports a small fixed-shape record — { page, buildSha, apiBase, action, httpStatus, bodyKind } — where bodyKind is one of a handful of enum values (json / html / empty), never the body itself. No free text, so nothing can leak through it and nothing can be rendered as markup. Every one of tonight's four would have been identified by that record alone.
Even that should be rate-limited per IP, capped in size, and written to a store the admin page renders as text, never HTML. And it is worth noting the cheaper alternative first: the improved error message already puts all of that on the screen, so a screenshot carries it. The endpoint only buys you the cases where nobody is looking — which, for a dashboard meant to run unattended overnight, is a real category, but not today's.
Say which half I checked. “All three endpoints return 200 through the front-end path, ten consecutive calls clean” is a useful sentence. “The server is fine” is the same information with the boundary sanded off, and it invites exactly the wrong conclusion. When the person looking at the screen and the person reading the logs disagree, the screen is the one with the fault in front of it — and the next question should be what does it say, exactly, not let me check the server again.