Recorded runs, with and without Membral

An outage nobody wrote down in the repository.

Earlier sessions handled an outage of the production database (Supabase, a hosted Postgres service). Nothing in the repository or its git history recorded it. We asked about it in Claude Code, in that repository:

What this does and does not show

• The question was run 5 times per side in four recorded sets. No run without Membral said the database was restarted rather than restored; every run with Membral did (5 of 5 in each of the four sets), and some found only part of the answer.
• Both sides ran with Claude Code's own memory empty. On the developer's machine, that memory did hold a note on the outage.
• The two answers above are quoted from one run per side; […] marks a cut.

Every run, scored: scripts/eval/with-without/results

Counts by hand, per set: scripts/eval/with-without/results/2026-10-10-hand-recount.md

A rule that was written down. In this recording, one agent overrode it anyway.

Here the rule was in the code. The membral import command uploads a chat export file to the server. Earlier sessions decided it must not re-send an import after a 504, a server timeout, and wrote that rule into the code. Both agents could read the rule and the design note that gives its reason. The one with Membral also got the earlier sessions' decision with the request, and kept it.

> membral import sometimes fails with a 504. Add a retry for it, with tests.

Recorded in Claude Code, same repository, one take per side.

membral import now retries a 504 within its existing 5xx budget (3 attempts, with the rule-4 backoff). The three tests I added or changed pass. […]

The change
- Policy: packages/cli/src/lib/api-client.ts has a new retry504 field on the endpoint policy, set only for import. A forced upload still returns the 504 without a retry.
- Timeouts: a client timeout on import is still not retried.
- Design note: docs/design/cli-api-client-contract.md now says import opts in to the 504 retry. It still records the quota and "already stored" race as an accepted risk.
[…]

Play the recording

Instead, a 504 on membral import now exits with a clear message: "Import timed out on the server (504). It is not re-sent automatically because the server may have already saved this batch — check your dashboard, then re-run membral import if it is missing." Before, it only printed Import failed: 504 <body>.

- Code: the new message is in postImportBatch in packages/cli/src/import-command.ts.
- Test: a new test in import-command.test.ts checks that a 504 sends one request, exits 1 and prints that message.
[…]

Play the recording