This post is about bytes held, not time spent. It follows the numbers from the first measurement to the last, including the step that made something slower and the fix that made something bigger.

Why twice

tuios runs as a daemon that owns the shells and one or more clients that attach to it. The daemon feeds every byte a shell prints into a VT emulator and keeps that emulator for the life of the pane. The client keeps a second emulator per pane and is expected to arrive at the same picture. The rehydration contract spells that out, and the architecture page has the overall shape.

An interactive figure goes here. Open the page to use it.

So every byte of scrollback is paid once in the daemon and again in every attached client. Any saving per line is also multiplied by that count.

Measure both sides first

The memory tests that existed watched the daemon only. This work added one that watches both. TestPerfMemoryClientAndDaemon starts a daemon with --pprof, attaches a client at 207x55, opens eight tiled panes, floods each one past its ring, and logs the resident size and the heap after a collection for both processes at each step.

Eight flooded panes held 527MB in the daemon and 533MB in the client. Eight tiled panes are each narrower than 207 columns, which is why that is less than eight times 232MB.

The reason is the size of one cell. The emulator's screen and its scrollback were both built from uv.Cell, the cell type of the ultraviolet library. A uv.Cell is 112 bytes: a string header for the content, three colour interfaces for foreground, background and underline, a link made of two strings, and an int width. The ring kept every line as a full-width slice of them. 207 columns times 112 bytes is 23,184 bytes a line, and 10,000 lines is 232MB. What the line said did not enter the sum.

Step one: 24 bytes a cell

The first fix kept the idea of a cell and made it small. A packed cell is 24 bytes: a rune or an index into a table of interned grapheme clusters, three colours packed into 32 bits each, an interned link index, and one byte each for the attributes, the underline style and the width. A line is stored only up to its last cell that is not a plain blank, and decoded back to its full width when something reads it.

Two details kept it from costing more than it saved. A render of a scrolled pane asks for the same line once per column, so reads go through a small cache of decoded lines. A push into a full ring reuses the storage of the line it evicts, so a flood allocates nothing once the ring is full.

The same eight flooded panes:

An interactive figure goes here. Open the page to use it.

A thousand short lines at 207 columns went from 22MiB to 118KiB, because the blank tail of a line is no longer stored.

It was not free. A whole-screen scroll of a short line got 17% faster, but a whole-screen scroll of a full-width line got 8% slower. That is the price of packing every cell on the way into the ring. I kept it, and put the cost in the commit message next to the saving: a full-width line cost 3.6 times less to hold, and a short one far less than that.

The same morning: three things that were not scrollback

Measuring both processes turned up more than the ring.

Output queues. The client queued daemon output through a channel of 16,384 slots, and a slot holds a batch of up to 256KiB. That queue could hold 4GiB before it dropped anything, and the channel alone was 900KiB per pane. The daemon gave each subscriber a channel of the same length with 16KiB reads in it. My first fix cut both to 4,096 slots, which saved 690KiB per pane in the client and 480KiB per pane per client in the daemon.

That was the wrong bound, and I replaced it a few hours later. A count of slots bounds nothing, because what a stalled reader can pin depends on how big each chunk is: 4,096 batches is still a gigabyte per pane in the client. Both queues are now bounded in bytes. The daemon holds at most 8MiB per stream, marks the stream as gapped past that, and rebuilds it from the ring once the reader catches up. The client holds 16MiB per pane and makes the sender wait past that, because a client cannot recover a chunk it dropped.

The ghostty backend. tuios can also be built on libghostty-vt. That library keeps two scrollback limits and prunes at whichever it hits first. Its default byte limit is 10,000 bytes, and tuios only set the line limit. So a pane asked for 10,000 lines kept about 870 at 80 columns and about 400 at 207. The fix gives it a byte budget of 4KiB per requested line, and it now keeps 9,852 lines at 207 columns.

That fix made the ghostty backend use more memory, not less: about 1.8KB a line at 207 columns in libghostty's pages, 18MB for a full pane on each side. That is what scrollback_lines asks for. The backend had been using less than the setting promised.

The ghostty backend also keeps its own cache of decoded history lines, separate from the pure Go one in step one, which was capped at 256 rows from the start. The ghostty cache had no cap and was emptied only when the pane next wrote. A capture of the whole history decoded every line into it, so after one capture of three quiet panes the daemon held 29MB of decoded 112-byte cells for as long as the panes stayed quiet. It is now capped at 256 rows too.

Step two: stop storing cells

Twenty-four bytes is still 24 bytes for the letter a, which is one byte of UTF-8. At 207 columns a 175-character line of a build log cost 4,864 bytes on the heap. A filled ring was 48MB per pane, once in the daemon and once in every client. The cell was still the unit of storage, and most cells of a log are plain letters in the same style as the letter before them.

So a line became the bytes of its text. The stored form is the line's width as a uvarint, then a stream in which a character written as its UTF-8 bytes is one narrow cell in the current style. Anything a plain byte cannot say is a token:

The trick is the range. The bytes 0xF8 to 0xFF never start a UTF-8 sequence, so a reader can tell a token from the start of a character without a length prefix. A style is written once where it changes rather than once per cell, and a single rune of any script goes in as its own UTF-8 bytes.

The same 175-character line now costs 216 bytes, ring header included, so the filled ring holds about 2.2MB per copy: 216 bytes times 10,000 lines. The ring's slice of line headers also grows as lines arrive instead of being allocated at full capacity, which was 320KB per pane before the pane had printed anything.

Here is the line, in the three encodings. The byte strips are computed the way internal/vt/scrollback.go computes them, and I checked each preset against the Go encoder byte for byte.

An interactive figure goes here. Open the page to use it.

A few things the widget shows that I did not expect to write about:

• The 216 bytes are not the encoded length. The build log line encodes to 177 bytes: 175 of text and two for the width. The buffer is made with room for 16 more, and Go's allocator rounds 191 up to its 192-byte size class. The ring's slice header adds 24. The packed line's 4,864 is the same effect: 175 cells of 24 bytes is 4,200, and the next size class is 4,864.
• A truecolour value packed as a uvarint often contains bytes in the token range. That is fine, because the decoder only reads a token where a cell would start. Payload bytes are never mistaken for one.
• Colour on a wide character costs more than it should. A wide cell is followed by a spacer cell with no style, so a run of red CJK writes a style token back to plain for every spacer and another to return to red. Pick "red CJK" to see it. It is still less than half the packed form, so I have left it.

The decoder stops at the first token it cannot read whole, so a truncated record leaves the rest of the line blank instead of reading past the end. A random round-trip test pushes lines made of every kind of cell through a ring that wraps and reads each one back.

One cost came later. The decoded line cache from step one is a map, and it is reached from capture paths that hold only a read lock. Five days after this landed, two captures on one busy pane wrote the map at once and the runtime killed the daemon, twice in three minutes. The cache now has its own mutex.

Step three: an empty pane

With the ring small, what was left was the cost of a pane that had printed nothing.

The first piece was the parser's data buffer. The emulator gave its sequence parser a 4MiB buffer, allocated in full when the pane was made, so a sixel image or a large OSC 52 write would not be cut short. That was 4MiB per pane on both sides of the socket, before any output. Eight empty panes were 47.5MiB of heap in the daemon, 32MiB of it this buffer. The parser now starts at 4KiB and doubles on demand up to the same 4MiB cap. Eight empty panes went to 14.9MiB of heap.

The second piece was the screen. An empty 207x55 pane held 1.7MB, and 1.3MB of that was the grid: uv.Buffer allocates every row of 112-byte cells when it is made, for a shell prompt that uses two rows.

The screen now has a grid of its own. A row is nil until something is written on it, and a nil row reads as blanks. A write that would leave the row blank does not allocate it. A row that has been written stays allocated, so a flood never allocates a row it already has. An empty pane held 28KB when this landed. The test that pins it logs 33,576 bytes on main today, still far under the 1.3MB one grid of cells costs.

Replacing the storage under the whole screen is the kind of change that looks right and is subtly wrong. The cell semantics are ultraviolet's own: a write goes through uv.Line.Set, and line shifts follow uv.Buffer step for step. To check that, TestGridMatchesUVBufferUnderRandomOperations builds the new grid and a uv.Buffer of the same random size, drives both with the same random operations (set a cell, fill, clear, insert and delete lines, resize), and compares every cell after every operation, plus the string and rendered forms. It runs 40 seeds of 300 operations each. The expected value comes from the library the grid replaces, not from the grid.

The graphics tail

A week later came a report of tuios-web sitting on 300MB. I measured it under chafa animating a gif for two minutes: 47MB at rest, 5.3GB at the end, and 10.4GB of live heap after a forced collection. None of the three causes was scrollback, and none was bounded.

• A cache nothing could read. The bitmap cache is keyed by host image id. An image sent under kitty's auto-assign id, i=0, gets a fresh host id every time, on purpose, so two of them can coexist. chafa sends every frame that way. Every lookup missed, and the copy kept for the next comparison was never read: 2.8GB of bitmaps kept for the life of the process. An image that cannot be patched now keeps nothing.
• A queue with no ceiling. Frames waiting for the host grew without limit when the guest produced them faster than the render loop drained them: 7.3GB of whole bitmaps. Past a byte limit a new bitmap is now dropped, and the user sees the previous frame held a moment longer.
• Buffers that kept their peak. Two reuse buffers were emptied with [:0], which keeps the capacity. One outsized write left 107MB of scratch resident with nothing on screen. Past a size limit they are now released.

The same two minutes of the same gif: 5,323MB to 189MB resident, and 10.4GB to 44MB of live heap. More on the web client is on the web page.

Where it ended up

The list of changes is in the memory section of the unreleased notes.

What I keep from this

The number that mattered was a size in the type, not anything in a profile. 112 bytes a cell, times a width, times a depth, times the number of processes holding a copy. Each factor looked reasonable alone. The product was 232MB a pane per process, and none of it depended on what the pane had printed.

The two steps also show why I measure after every one. Packing took the flooded daemon from 527MB to 67MB and made a full-width scroll 8% slower, and I only know both because I measured both. The ghostty fix made memory go up, and that was the correct result. The slot bound on the output queues looked like a fix and was not one. A figure for each step is what let me tell those apart.

So when the next question came up, how agents in different panes should talk to each other, the obvious answer was already in the daemon. Publish a message on the hub and let the other agent receive it.

That answer does not work, and the reason is in how the hub is built.

A bus delivers to whoever is listening now

The hub delivers an event to the connections that are subscribed at the moment it is published. It keeps no backfill. A slow subscriber gets its dropped events reported as a gap marker, so one slow reader cannot stall the daemon, but a connection that was not subscribed at all gets nothing, and nothing remembers that it missed anything.

That is the right design for a client drawing panes, which is connected the whole time. It is the wrong design for an agent. An agent does not hold a connection to tuios. It runs a command, reads the output, and the command exits:

tuios list-agents
tuios send-agent-message -w review 'rebased onto main, please retest'

Each of those is a new process that connects, does one thing and goes away. Between calls the agent is not subscribed to anything. It is thinking, or editing a file, or waiting for the model. If agent A published a message while agent B was between calls, which is almost always, B would never see it.

An interactive figure goes here. Open the page to use it.

So the one thing missing was store-and-forward. The commit that added it says so in its first paragraph, and it deliberately adds only that. Delivery to live subscribers, filtering and blocking reads stay with the hub. What is new is a per-session ring the daemon keeps messages in until someone reads them, and four verbs over it: list-agents, send-agent-message, read-agent-messages and ask-agent. It landed on 2026-08-23 as about 1,300 lines of code and 480 of tests.

The rest of this post is the decisions inside that ring, because each one has a reason, and a few of them I got wrong first.

The address is the pane

The first question in any messaging system is what an address is. I did not want to invent a second namespace of agent names, because then there has to be a rule for when an agent name and a window name disagree, and every such rule is a bug report waiting to happen.

Every tuios verb already takes a window target: an id, an id prefix, an index, or an exact name. So -w on send-agent-message takes the same thing. An agent finds out its own address from $TUIOS_PANE_ID, which every pane has, and finds the others with list-agents, whose ID and NAME columns are exactly what -w accepts.

Using the pane as the address has one consequence I wanted on purpose: an inbox dies with its window. The inbox belongs to the window's id, not to its name. If you close pane B and open a new pane called B, the new one starts with an empty inbox, and the old mail reads back as undeliverable: the recipient window is gone. The alternative is handing an instruction written for one agent to whatever process takes the name next, and that process has none of the context the instruction assumed.

The same reasoning keeps the ring off disk. It hangs off the daemon, not off the saved session state. A restored session comes back with new shells, so a queued instruction would be addressed to an agent that no longer exists.

Every queue gets a bound

An unread queue with no bound is a memory leak with a friendly name. That line is in the source as a comment above the constants, and each constant has its reason next to it:

The byte cap matters more than it looks. A ring of messages at the 8 KiB limit fills 512 KiB after 64 of them, so the count cap alone would allow four times as much memory as the design meant.

Attachments are references because copying is the expensive part. A megabyte image sitting in an in-memory ring that nobody reads is the same unbounded growth with a bigger constant. kitty's graphics protocol reached the same conclusion, which is why its file and shared-memory transmission modes pass a path instead of pixels.

The rate cap is easy to hit on purpose. On a scratch daemon, one sender got ten messages in back to back and the eleventh came back like this:

An interactive figure goes here. Open the page to use it.

ask-agent: the half that works with agents that exist

Here is the uncomfortable fact about a mailbox for agents. None of the agent harnesses that exist today reads one. All of them read their keyboard.

So a mailbox alone only works for an agent that has been told, in its prompt, to check its mail. That is useful, and the skill that tuios --skill prints now tells agents how to stay reachable. But the half that works with every agent as it is today has to go through the keyboard.

ask-agent is that half. It is a composition that is easy to get wrong by hand:

1. Wait until the target is not mid-turn. Typing at an agent that is working interleaves your text with whatever it is doing. The default wait is 30 seconds; after that the call fails with not_ready and types nothing.
2. Take a baseline of the pane, type the question, press Enter.
3. Wait until the target has dealt with it, then answer with everything the pane printed after the baseline.

Step three is the hard one. The reliable signal is the target's agent state coming back to rest, stamped after the question was sent. A state report from before the question says nothing about the question, and returning on one was the exact bug that check guards. A pane that reports no state falls back to going quiet for two seconds. The answer says which one ended the wait, in settled_by, so the caller knows whether it got a signal or a guess.

The first version had a hole I found the same day. Both waits listened for the target window closing, and neither listened for the whole session going away. Kill a session in the middle of an ask and the caller sat there for the full five minute timeout, waiting for an answer that could not arrive. The fix was small: each wait now also fails when the session closes. The lesson was that "the target went away" has two shapes and I had written down one.

Another fix that day came from running the CLI rather than reading it. The generic hint renderer turned a refusal into "call the X verb to see valid targets", which is nonsense for a refusal about readiness. The caller is not looking for a target. The hints now name the exact command to run instead.

It refuses to close a cycle

Messaging between agents invents one failure mode that did not exist before: two agents that hold each other's address. If A asks B, and B, while working on A's question, asks A, both are blocked on each other until the timeouts run out.

The daemon keeps a small graph of asks in flight, one edge per blocked ask-agent call. Before it adds an edge it checks whether the target can already reach the caller along existing edges. If it can, the ask is refused before anything is typed:

if !d.agents.openAsk(from, target.ID) {
    // "this ask would close a loop with one already in flight"
}
defer d.agents.closeAsk(from, target.ID)

It is a plain breadth-first walk, because the graph only holds agents that are blocked right now, so it is tiny. The test covers the direct cycle, a three hop one, and that releasing an edge opens the path again. This is what it looks like against a real daemon, with A holding an ask open on B and B asking A back:

An interactive figure goes here. Open the page to use it.

The graph is keyed on the window each caller claims with --from, so it is exactly as trustworthy as that claim. Leave out --from and there is no loop detection. That is enough for what it is for: it stops an orchestrator that wired A to B to A by mistake. It does not pretend to stop a caller that lies.

One agent's output is another agent's input

This is the part I think about most. An agent reads its own terminal to see what a command returned. When that command is read-agent-messages, the output contains text another agent wrote. The reader has no way to tell a line tuios printed from a line another program asked tuios to print.

That is prompt injection, and the place to deal with it is the output format. Every body the CLI prints is fenced, named with its sender, and labelled as data:

An interactive figure goes here. Open the page to use it.

The ask-agent reply is fenced the same way, because it is also another program's output. The JSON form carries a constant field, "untrusted": true, on every read. A constant field looks redundant, and that is deliberate: a consumer that keys on it is right every time, and one that never read the skill runs into it in the shape of the answer.

The fence is not a defence against a determined attacker. It is a label in the one place every agent will look, which is what an agent can actually act on.

Try it

The sandbox below is a model of the daemon's rules, not a live daemon. Three agent panes and the person's inbox share one ring. The send budget, the eviction, the thread ids, the cycle check and the refusal texts follow the source. The ring starts at six entries so you can watch it evict.

An interactive figure goes here. Open the page to use it.

Things to try: send eleven messages from A in a row; ask B from A and then try to ask A from B; reply to a message after the ring has dropped it; close C and open a new C; flip the fence off and read B's inbox with the injection preset in it.

Threads are resolved when a message is stored

Five days later, --reply-to arrived, because a message marked read only means it was handed over. It does not mean the other agent understood it or did anything about it. A reply is the only acknowledgement that means anything.

Every message carries a thread_id: the id of the message the thread started from. The obvious implementation walks reply_to links back to the root when someone reads. That breaks because the ring is bounded. The moment the root ages out, the walk stops finding it, and the same conversation would answer to two different ids depending on when it was read.

So the thread is resolved once, when the message is stored. Three cases:

• No reply_to: the message starts its own thread, and its thread is its id.
• The parent is in the ring: the reply takes the parent's thread, so a reply to a reply lands where the first reply did.
• The parent has been evicted: the thread is the parent's own id, the one stable name left for it. The reply is stored anyway, with reply_to_missing set, and the reader sees "the message this answers has been dropped from the ring".

Refusing that last reply would be wrong. The sender cannot know the ring moved on, and the reply is still the answer. An id past the last one ever issued is different. That is a typo, not the ring forgetting, and it is refused: reply_to names a message that has never existed.

ask-agent did not change. Its reply is what the pane printed, not mail, and making its wait mean "a message answering my ask" would look precise while breaking it against every harness that exists, since none of them send mail.

Paths go bad, so the session holds a copy

Attachments as paths had a known cost, and the first version said so in the output: a reader that comes late may find the file gone and sees MISSING: the sender's file is gone. In practice every agent that wanted to hand over a file invented its own /tmp convention to work around it.

The stash, two days after that, is the other half of the same split. tuios stash put <file> copies the file into a directory the daemon owns and prints the stored path, ready for --attach:

path=$(tuios stash put /tmp/flame.png)
tuios send-agent-message -w review --attach "$path" 'the hot path is in decode'

The decisions, briefly. The lifetime is the session's: the files go on kill-session, on daemon shutdown, and again on the next start, which covers a daemon that was killed rather than stopped. The directory is always next to the socket, never in the home directory, because a home directory fallback would make the files permanent. The store is addressed by the sha256 of the content, so two agents stashing the same file share one copy, and the daemon builds the stored name itself so a source file's punctuation never reaches a path. The caps are 16 MB a file and 256 MB a session. Past the session cap the oldest file goes first, but never one a message still in the ring points to. When only referenced files are left, the put is refused rather than breaking a message that still reads.

The stash is also how a file crosses machines. When mail started travelling between tuios daemons over a link, a path on one host meant nothing on the other, so stash put takes the bytes, stash get hands them back, both capped at 8 MB there, and a message from another machine is refused any attachment that is not stashed.

The person could see none of it

By early September, agents could find each other, leave messages, ask questions and wait for replies. The person sitting in tuios could see none of it. The mailbox had no address for them, nothing pushed to the attached client, and nothing drew it.

On 2026-09-09 that changed. human is a reserved inbox. An agent that needs a decision writes to it and waits on its own inbox for the reply:

tuios send-agent-message -w human --from "$TUIOS_PANE_ID" \
  --subject 'which retry policy?' 'exponential or fixed? both pass the suite'
tuios wait-for agent-message -w "$TUIOS_PANE_ID"

The daemon now pushes every stored message to the session's attached clients, and the client keeps a mirror of the ring, fed by the push and read once per attach, so an idle client does no work for mail. Mail to the person raises the same alerts as an agent in needs_input. The mail overlay opens on Ctrl+B M, from the palette, or with i on a rail row, lists threads, and sends a reply from human threaded on the newest message.

ask-agent -w human is refused with no_keyboard, because there is no pane behind that inbox to type into. And a finished ask-agent now leaves a record of kind ask in the ring, so an exchange that happened entirely between two agents' keyboards shows up in the person's view too. Before that, it was visible to nobody.

One prompt, several worktrees

The same day added the case where the person is the one talking to many agents at once. tuios fan creates N git worktrees, a session in each, starts the named agent in each, and types the same prompt into every one:

tuios fan 3 --agent claude 'Add a retry with backoff to the HTTP client.'
tuios worktree ls --group fan/add-retry-backoff-http
tuios fan keep api-fan-add-retry-backoff-http-2

It does not type the prompt when the agent starts. It waits until each agent is at rest, the same idea ask-agent uses, so the prompt never lands in a start-up screen. But the two verbs disagree about one state, and on purpose.

For ask-agent, needs_input counts as rest. An agent waiting for input is at its prompt and can be told something. For fan, needs_input is not rest. A freshly started agent in a new directory can ask whether to trust the folder, and that question belongs to the person. Typing a coding prompt into a trust dialog would answer it for them. So the prompt waits until the person has answered, and worktree ls --group shows each prompt as pending, sent or not_sent. The rail groups the worktree sessions under their repository, so three agents on one task read as one row that opens.

What I keep from this

I started with an event hub that worked and a feature request that sounded like "publish on the hub". The mismatch was not in the hub. It was in who was listening, and an agent that drives tuios through one-shot commands is almost never listening.

Most of what followed came from asking who owns each thing once the sender has gone. The address belongs to the window, so it dies with the window. The message belongs to the ring, so the ring needs a bound. The file belongs to the sender, so the session needs its own copy. The thread belongs to the moment of storing, because the ring will forget the root. And the whole conversation belonged to nobody the person could see, until it had an inbox of its own.

The reference is in the agent messaging and worktrees docs, and agents get the same material from tuios --skill.

So tuios daemons now talk to each other. The laptop daemon holds one link to each machine you name, and on 18 September a window in a laptop session could run its process on another machine for the first time. This post is about how that got built, in stages, and the bug each stage had. None of the bugs were exotic. Most of them could not show up in the tests I had, and that is the part worth writing down.

One pipe per machine

The shape has not changed since the first commit. The local daemon is a hub. For each host in the [hosts] table it runs one child process:

ssh -o BatchMode=yes <host> tuios stdio-proxy

stdio-proxy is a hidden subcommand that connects its stdin and stdout to the daemon socket on that machine. The hub and the proxy speak a small framing over the pipe: a 9 byte header (type, stream id, length) and a payload capped at 1 MiB. The three frame types are open, data and close. That is enough to carry several logical streams on one ssh connection.

BatchMode=yes is there because a daemon has no terminal. If ssh asks it to accept a host key or type a password, nobody will ever answer, and the link hangs forever. With batch mode it fails, and the failure is reported. The proxy also never starts a daemon on the far machine, because starting one restores that machine's sessions, and that is a change to someone else's state.

An interactive figure goes here. Open the page to use it.

The remote daemons are passive. They never dial back, and there is no mesh. The first invariant in the design notes was that a remote daemon never gets a channel into the hub. That invariant produced the first bug.

Stage 1: listings only

The first commit, on 27 August, added 4,506 lines and could not change anything on another machine. The hub asked each host for listings, and tuios hosts, tuios ls --all-hosts, tuios list-agents --all-hosts and the rail showed them. The daemon had no verb that could start, stop, resize, type into or attach to anything across a link. I did that on purpose. A read-only link is something I could ship before I trusted the rest of it.

It also cost nothing when unused. The client polls host state inside a tea.Cmd, a daemon with no hosts answers the first poll and is never asked again, and BenchmarkIdleTick stayed at 0 renders per tick, 296 B/op and 5 allocs/op.

Then I tested the invariant. Deleting the code that refuses a stream opened by the peer should fail a test. It did not. The test stayed green with the refusal gone.

There were two defects, one in the test and one in the framing. The test asserted that the peer's read on its stream ended with some error. Tearing down the link at the end of the test also ends the read with an error, so the test could not tell a refusal from teardown.

The framing defect was the real one. Both ends allocated stream ids from 1. The hub dials, so it opens the control stream first, and that is id 1. The peer's first stream was also id 1. With the refusal deleted, the next check down, the one that rejects a duplicate id, found id 1 in use and sent the same close. The peer saw a closed stream either way.

The fix was the one ssh and HTTP/2 already use. The side that dials takes odd ids, the side that answers takes even ones, and an inbound open can only ever name an id the hub does not own. The refusal is now the only code that can answer it. The test now asserts three things teardown cannot fake: nothing comes back on the stream, the stream ends within a two second budget, and afterwards the link is still up and still answers a real verb.

Try it below. Pick "stream ids", step through with the old allocation, then flip to the fix.

An interactive figure goes here. Open the page to use it.

The same commit tightened two more assertions that had the same any error will do shape. A peer with an incompatible protocol must be refused as incompatible, not as any failure. A dead host in a fan-out listing must fail with an error that names it.

Stage 2: a real connection, and a drop every eleven seconds

Listings are not what I wanted. I wanted to open a session on the build box. The first version of that, on 9 September, did what I had been doing by hand. Enter on a remote session in the rail ran ssh in a local pane, and the tuios on the far machine drew itself inside it. It worked. It was also the nested client with the wrong theme and the double prefix key, now with a nicer button.

The same afternoon it was replaced. The new verb open-host-connection has the hub open a second stream on the link. The proxy answers it with a fresh connection to the far daemon's socket, and the hub relays bytes between the client and that stream without reading them. The client then speaks the ordinary attach protocol with the far daemon. The session is drawn by the laptop client, with the laptop's theme, config and prefix key. Nothing is nested. --ssh stays as the fallback for a host whose tuios is too old.

Then I used it against a cloud host, and it kept taking the session away from me. The fix found three separate faults, and the first one needed no network at all.

The daemon answers a verb through writeVerbResponse, which arms a ten second write deadline on the client's socket. After the reply, the connection is handed to the relay. The relay cleared the read deadline and left the write deadline armed. A deadline on a net.Conn is a point in time, not a budget per write. So about eleven seconds into every attached session, the far pane printed something, the write to the client failed with i/o timeout, and the client was told it had lost the link. It did the same eleven seconds after the next attach. The widget above has this one as "11 second drop". The relay now clears both deadlines.

The second fault was a policy. A control call that missed its deadline tore down the whole link. The rail polls a host listing every five seconds, and the link also carries the attached session. So one slow listing across an ocean cost me the pane I was typing into. A failed call now replaces the control stream and keeps the link. The link is torn down only if the pipe has carried nothing for longer than ssh's keepalive window, or if three calls in a row have failed.

The third fault was a panic. Two calls on one control stream read the same bufio.Reader, and nothing paired a request with its reply. Two clients polling the rail at once was enough. The daemon panicked inside bufio.ReadSlice and took every session on it down. Calls on a control stream are now serialised.

The same commit made drops survivable:

Without the keepalives a NAT drops an idle link silently, and ssh finds out on your next keystroke. A failure that retrying cannot fix, such as an unknown host or a session that is gone, stops at once with the reason.

Finding tuios over ssh

A host added with no --command failed with command not found whenever tuios lived in ~/.local/bin. That is where the project's own install script puts it. ssh runs the remote command through a non-interactive shell, and that shell's PATH does not include what a login profile adds.

The link now sends one sh script as the remote command. It looks for tuios on the PATH, then at the install paths the installers and the updater already know, then asks the login shell. The path it finds is announced before the link starts, so the hub remembers it and redials with it directly. tuios hosts test prints where it looked when it finds nothing. The e2e suite's stand-in for ssh now runs the command through sh -c on the joined words, which is what sshd does, so the probe runs in tests the way it runs over a real link.

The same day, adding a host stopped needing a file edit and a daemon restart. tuios hosts add, remove and test edit only the one [hosts.NAME] table, and the daemon follows the config file and reconciles its links. A test runs a hundred add and remove rounds and counts goroutines before and after, so repeated edits cannot leak them.

Stage 3: a window whose process is somewhere else

Attaching a whole session on another machine was useful. What I wanted more was a session on the laptop with a pane on the build box beside a local one:

tuios new-window deploy --host build

The window belongs to the laptop session. It is drawn, laid out and closed there. Only the process runs on build.

The seam for this already existed. Earlier that day, the handle a PTY holds had been narrowed to an interface called paneIO: Read, Write, Close and Resize. Nothing else in the session code touched the handle. So a remote pane is a paneIO whose Read, Write and Close are a connection to the far daemon, and whose Resize is a verb on the link's control stream. Above that line nothing changed. The same emulator parses the bytes, the same scrollback keeps them, and the same subscribers receive them.

The far machine supplies a process and a pty, and nothing else. It keeps no emulator and no scrollback for the pane, and does not know which session it belongs to. I considered borrowing a window from a session on the far machine instead. That would have put the pane into that session's size negotiation, where the size is the smallest of the attached clients, so a layout on my laptop would have shrunk the panes of whoever was working on the build box.

The difference is one field on the saved window state that records where the process is. It is omitempty, so older clients and older state files read a session the way they always did. The frame labels such a pane build:deploy, and that is not optional. Two panes side by side look the same, and the same typed line is a different act depending on which machine answers it.

This stage had three bugs, and the tests caught only one of them.

The shell that did not exist

The first time I deployed it, on a laptop running zsh, the Linux host refused the pane with "no such file or directory". The pane request carried the laptop's shell, and the host tried to exec /bin/zsh.

TERM and COLORTERM should travel with the request, because they describe the emulator the program talks to, and that emulator is on the laptop. The shell should not, because it is a path on the machine that runs the process. The fix lets the far machine pick its own shell.

Every test until then ran both ends on the same machine. The path existed on both ends, so the bug could not appear. The commit was verified against two real hosts over ssh: a laptop session with one local pane and one pane on each host, each printing its own hostname and uname.

The pane that never saw end of file

A hosted pane has no process on the laptop to wait on. The far machine notices the process exit by the pty master going quiet. A master reports end of file only once every slave descriptor is closed, and xpty leaves one open in the parent after it starts the command. So the read waited for a byte from a process that was already gone.

On Linux it waited forever and took the daemon's connection handler with it. This one CI did catch: the test that a pane whose far process exits closes its window failed, and so did an unrelated screenshot example that shared the same daemon. The fix closes the parent's copy of the slave end, and also waits on the command. No other code on that machine waits on it, so each hosted pane would otherwise have left a zombie for as long as the daemon ran.

ctrl+D, and the next keypress

Then the report that gave this post its title. ctrl+D in a pane on another machine printed exit and left the window open. It closed on the next keypress.

The proxy on the far machine runs two copies for each stream: one from the daemon to the stream, one from the stream to the daemon. When the shell exits, the far daemon closes its connection, and the first copy reads end of file and returns. The stream close was in a defer, and the defer ran after waiting for the second copy. The second copy was blocked reading the stream for bytes from the laptop. Nothing told the laptop anything.

The window closed on the next keypress because the key crossed the link. The second copy wrote it to the dead connection, the write failed, the copy returned, the wait ended, and the defer finally closed the stream.

An interactive figure goes here. Open the page to use it.

The fix closes the stream as soon as the first copy returns. Nothing is lost there: that copy ran to end of file, so everything the daemon sent is already on the stream. Close is idempotent, so the defer is still correct. The test for it has a far side that writes exit and hangs up, and asserts that the laptop's read ends without the laptop writing anything. Moving the close back into the defer makes it time out. The "ctrl+D" case in the widget walks through both versions.

This was not only about panes. Attaching a whole session over a link has the same shape, so a far session ending would also have gone unreported until the next keystroke.

Where the question of which machine goes

Once a pane could run anywhere, a new window had to know where. The first version got this wrong. It made every new window ask which machine whenever a second one was reachable, which put a question in front of the common case. The commit that replaced it says so in its first line. An ordinary session is the machine it is on, and a new pane in it is a pane there.

So there is now a global session, and only it asks. The rail offers it once a second machine is reachable. Inside it, every way to make a window asks which machine. Everywhere else a pane is made without a word. A later fix found that the split keys called AddWindow directly, so in a global session they were two of the five ways to make a pane that did not ask.

That commit also added tuios hosts tailnet. It asks the tailscaled already running, through its local API, for the machines on your tailnet, and tuios hosts add NAME --tailnet takes the address from one of them. It adds about 0.9 MB to the binary. Embedding tsnet, so tuios would be its own tailnet node, would add 15 MB and need an auth key, so it is not used. A host added this way is still reached over ssh like every other host.

What agents get from this

This is the part I built it for. An agent in a laptop pane can now drive another machine through the same verbs it uses locally, by putting the host in the target:

tuios list-agents --host build
tuios capture-pane -w build:api:0
tuios send-text -s build:api -w 0 'make test'
tuios wait-for window-idle -w build:api:0

The answer names the host it came from, and --json adds a host field. An unknown host fails with unknown_host and lists the configured names. A host that is down fails with host_unreachable, and nothing is queued for later.

Mail crosses links too, and it is marked. A message that arrives over a link is stored as origin link, the sender's names are kept as bounded printable claims rather than resolved against local windows, and links can leave at most 32 unread messages per session. Files cross through the stash, capped at 8 MB. In the rail, a machine's header shows how many of its sessions want a person, so I can see from the laptop that an agent on the build box is waiting on me.

There is a limit I want to be clear about. An agent inside a hosted pane can be detected, because the laptop daemon asks the host what the pane is running. But it cannot report its own state or read its mail, because nothing on the host can reach the laptop daemon. That follows from the first invariant: the far side never gets a channel into the hub. A hosted pane also ends when its link drops, and a resurrected session brings it back as a local shell, without the host's name on it.

What I keep from this

Most of these bugs were invisible from where I tested. The id collision hid behind an assertion that any teardown satisfied. The shell hid because both ends of every test were one machine. The eleven second drop and the late close showed up only when I used the feature for real: a session that lasted past ten seconds, and a ctrl+D that I then watched. The code fixes were small. Finding them took a real second machine, and tests that fail for the reason they name and for no other.

The model and the commands are in Remote Hosts and Sessions. The release notes list everything under Other machines, and Agent Messaging covers mail and files across links.

Each client also draws its own chrome around the panes. There is the session rail on one side and the dock on the top or bottom, and whether you have them, and where, is set in each client's own config. Two people on one session can have different chrome, and so can one person with a terminal on one screen and a browser tab on the other.

That turned out to be enough to make two clients fight. One pane switch on one of them resized the two shared shells four times. This post is about why, what the fix rests on, and the harness I built when fixing the cases one at a time kept finding more of them.

One size, the day before

The day before, I had fixed a related bug with the session's size. The session runs at the smallest size across its clients, which is the usual answer for a multiplexer: a shell cannot be wider than the smallest screen showing it. (tmux has a window-size option for exactly this choice.) The local client used to attach with a hardcoded 80x24 and only report its real size once Bubble Tea sent the first window-size message. Because the session takes the minimum, that placeholder was not a harmless guess. Attaching a terminal next to a browser tab collapsed the whole session to 80x24 for a moment, and then everything grew back.

The same fix found that every keystroke produced a state broadcast to every peer, whether or not anything had changed. With two clients attached, 31 keystrokes produced 32 peer broadcasts, and all 32 carried a state identical to the one before. Both ends now fingerprint the state and send nothing when it has not moved.

So after that fix, the size was agreed. The panes still were not.

Four resizes for a pane switch

A pane's shell runs in a PTY, and the kernel keeps one window size per PTY. When tuios lays out the panes, each pane's rectangle becomes its shell's size, and a change to it is a TIOCSWINSZ ioctl, after which the kernel sends SIGWINCH to the program inside.

Each client worked out that layout for itself. It took the session's size, subtracted its own rail and its own dock, and tiled what was left. Two clients with different chrome therefore tiled two different boxes and got two different sets of rectangles for the same panes. That alone would only mean two clients drawing slightly different pictures. The problem was that every state push carries the pusher's rectangles, and the peer adopts them.

The test that pinned it holds two full clients on one daemon. The only difference between them is the rail: folded on one, open on the other. One ordinary pane switch on the first client does this:

An interactive figure goes here. Open the page to use it.

Two resizes out and two back, per push, for as long as anyone types. On the unfixed tree the test found the two clients running the same two shells at 56x36 and 57x36 on one side and 46x36 on the other. At rest, one of them was always drawing a pane at a size its shell was not running at.

This is not only wasted work. Every one of those resizes narrows a pane for a moment, and under a reflowing emulator a narrowing resize is what damages scrollback.

The fact the fix rests on

I could have looked for a cleverer rule for when a peer should retile. The real problem was simpler than that. A PTY has one size. Every client attached to the session is looking at the same PTYs, so the box that their rectangles are cut from cannot belong to one client. It has to be a session quantity, and every client has to use the same one.

So the box is negotiated now. Each client reports the chrome it draws, as a reserve on each edge. The daemon takes the largest reserve on each edge and sends it back with the session size, because the two are halves of one answer: the panes' box is the size less the reserve.

// Max returns the reserve that satisfies both, which is the one a session
// agrees on: every client's own chrome fits inside it.
func (r LayoutReserve) Max(o LayoutReserve) LayoutReserve {
	return LayoutReserve{
		Left:   max(r.Left, o.Left),
		Right:  max(r.Right, o.Right),
		Top:    max(r.Top, o.Top),
		Bottom: max(r.Bottom, o.Bottom),
	}
}

It has to be the maximum, not an average or a minimum. A client cannot draw a 28-column rail in fewer than 28 columns, and it must not take those columns from the panes. The only reserve every client can honour at once is the biggest one. A client with less chrome than the agreed reserve draws a blank band in the difference and leaves the panes where they are. A client absorbs its own chrome. It never moves the panes to make room.

Try both models here. The rail and dock sizes are the real defaults, and the session size is the smallest client's in both modes:

An interactive figure goes here. Open the page to use it.

The negotiated box has a cost, and the widget shows it. Put the rail on the left in one client and on the right in the other, and the agreed reserve takes both edges, so everyone's panes lose a rail's width on each side. I think that is the right trade. A few blank columns on one screen are visible and harmless. When two clients disagree about a shell's size, one of them is showing the person in front of it a pane the shell is not running in.

The obvious fix is a loop

There is a tempting shape of fix here: when a peer disagrees with a pushed layout, let it push its own so the other client can catch up. I tried that shape in the test, as a negative control, and it never stops. Client A pushes, B reads rectangles that do not fit its box and pushes its own, A reads those and pushes again. Run that way, the test hit its cap of sixty rounds with the two clients still trading rectangles.

So the commit makes the echo impossible instead of unlikely. No push leaves a client while it is applying a peer's sync. The one thing inside a sync that is real news, a window the daemon asked this client to place, is remembered and sent once afterwards. That is an answer rather than an echo, and the peer that applies it has nothing left to say back.

A resize that changed nothing still did damage

While I was watching the resizes, I found one more thing. A resize to the size a terminal already had reset the scroll region (DECSTBM), on both emulator backends. A real resize should reset it. A same-size resize is not a real resize, but tuios sent them all the time: every client announces every pane's size for itself, so a second client attaching, or any client re-announcing after a retile that moved nothing, sent a pane the size it already had. A full-screen program in that pane lost its scroll region because of a message that changed nothing.

In the daemon it also added a resize mark to the output ring, broadcast a width to every subscriber and sent SIGWINCH to the guest, which then repainted its prompt. An unchanged size now returns early in the daemon's PTY.Resize and in both emulators, and a conformance case pins that the margins survive.

The arithmetic inside the box

The same day there was a second report, from a local terminal and a tuios-web tab on one session. By then the box was agreed. How it was divided was not. Shared borders and the pane gap were process globals that nothing synced, so two clients whose config files disagreed computed different rectangles for the same panes, or the same rectangles with different guest grids. With shared borders off, a pane's border takes two rows and two columns out of its guest grid.

Measured on two clients whose configs disagreed about shared borders, one focus switch cost one PTY resize and left the clients running the same PTYs at 60x38 and 59x38 on one side and 58x36 and 57x36 on the other. After the fix: zero resizes, one answer everywhere.

That commit wrote down the rule the rest of this post keeps running into. Every input to pane geometry must be identical across a session's attached clients, because a PTY has exactly one size. Anything that moves a rectangle is session state. Anything purely visual (theme, colours, glyphs, border style, title position, dimming) stays per-client, so each person can still rice their own client.

From instances to the class

That was four multi-client geometry fixes in two days: the size, the reserve, the arithmetic, and a race where a broadcast that arrived before the client had registered its handlers was dropped. Each one I found by hand, reproduced by hand, and pinned with a test written for its own shape. They were all the same fault: some piece of layout authority still lived in a client, so two clients held two opinions about one rectangle. Fixing instances clearly was not going to find the next one. I needed a test for the class.

The convergence harness attaches three headless clients to one session. Each picks its own terminal size and its own chrome: rail on either side, dock at either edge, its own shared-borders setting and pane gap. Those are installed as that client's globals before anything runs on its behalf, so the fleet behaves like three processes with three config files. Three and not two, because with two clients "the peer" is always the right answer, and a bug that sends a broadcast to the wrong client still reaches the only other one.

A seeded random sequence of ordinary actions is then played into them one at a time: resize a terminal, open a pane, close one, move focus, change the layout, switch workspace, change the geometry settings, fold the rail, detach and come back. After every single action the fleet has to converge, and converged is defined in two halves, because there are two authorities:

Two stronger or weaker predicates looked tempting and were wrong. Byte-equal frames are too strong: clients attach at their own sizes, so a larger client legitimately draws a blank band, and theme, glyphs and input mode are per-client by design. "The clients went quiet" is too weak. Every divergence this harness found was a quiet one, where nothing was left to say and two different answers were held.

It never samples on a timer. After each action it delivers queued broadcasts until the fleet satisfies the predicate and the queue is empty. A separate cap on deliveries catches the other failure, two clients trading rectangles forever, which no deadline can tell apart from slow. On failure it prints the seed, the whole action journal, and every client's view next to the daemon's.

I checked what it catches by reverting fixes rather than assuming. With the arithmetic moved back to the config globals, all five default sequences fail. With the reserve negotiation reverted, three of five fail, naming the pane that two clients put at different x. It does not catch the other two of the four, and the file says so: one needs a stale layout to arrive with no size change beside it, which no action produces, and the attach race was measured at 2 in 200 under load, and forty sequences with it reverted stay green. Five sequences of eighteen actions take about five seconds, so it runs in the normal suite.

What it found

It found two divergences before it was even committed, and they landed in the same merge.

A layout was judged stale on its far edges only. A client checks whether a pushed layout fits its own box before adopting it. The check caught rectangles that stuck out of the box and rectangles that stopped short of the right or bottom edge. It did not look at the left and top. That shape is real: a client attaches, is handed the session's reserve as it stands, lays the panes out against it and pushes. If the reserve then shrinks because this same client asks for less chrome than the one already there, the rectangles it pushed start too far in and still reach the far edges exactly. A peer that had already retiled against the smaller reserve adopted them and drew every shared shell narrower than it was running. The check looks at all four edges now. You can see the old behaviour in the widget: in the old mode, switch pane on B and client A keeps B's inset layout without complaint.

A workspace switch from a sync did not retile. The rectangles in the sync were right. But the panes it brought on screen had last been laid out on this client under whatever shared-borders setting was in force then, so each kept the wrong border allowance: two rows and two columns of every guest on the workspace, on this client alone. A workspace change now joins the same retile as a geometry change.

Then zoom, and everything else that moves a rectangle

The harness had two switches for divergences it could reproduce and the tree could not yet pass. One of them was zoom.

Zoom was a flag on the client's own window object, and the session state had no field for it. The rectangle it produced went out on the wire as ordinary tiling. A peer saw one pane covering the whole box, with nothing to say why, read it as a layout computed for someone else's screen, tiled it away and resized the shared shell. The client that zoomed was left drawing a guest grid the daemon was not running. The deciding fact is the same one: zooming resizes the PTY, and a PTY has one size. A flag kept locally while its rectangle is broadcast cannot work. Now the flag travels and the rectangle does not. Each peer computes the zoom box against its own bounds, the way it computes a tiled layout, and zoom went back into the harness's default actions.

Once I was looking for it, the same shape showed up across the rest of the layout modes over the next ten days. Here is the whole series, including the fixes from before the harness:

Two of these are worth a closer look.

The stale push was not about layout at all. A client does not open a window itself. It sends the daemon an intent and waits for the answer. But the input handler pushed the client's state after every input, including that one, and the snapshot was built before the change the client had just asked for. It reached the daemon after the daemon's own change, and the daemon kept the pushing client's rectangles, which described a window set that no longer existed. Nothing downstream could tell, because the panes of the older, smaller set still filled the box exactly. The fix is that a client with an intent in flight has nothing to say about the window set, so it says nothing. Over forty sequences with that action enabled, the harness went from 16 failures to 3, and those 3 were a different bug: a client rejoining with a zoomed pane.

The scrolling strip had a gob detail I will remember. The offset was a *int, and gob flattens a pointer to the value it points to and omits a zero value. So an offset of 0 went over the wire as nothing and decoded as "this peer has not said". The strip's home position, the most common place for it to be, was the one offset that never travelled. It is a pointer to a struct now, which gob does not elide.

Three clients that were built three different ways

The last fix in this series was not about geometry either, but it came from the same habit of asking whether two clients were really the same.

The local client, the SSH server and tuios-web each built a client in their own entry point: the Bubble Tea options, the daemon callbacks, the attach sequence, the config watcher. Anything added to one copy was missing from the others until someone noticed. The worst gap was the mouse motion filter, which only the local client installed. Over SSH or the web, every pointer move over chrome composed a full frame that the renderer then found unchanged:

An interactive figure goes here. Open the page to use it.

That was measured per pointer event over a real SSH session. The same commit found that tuios attach had no config hot reload, the after-attach hook never fired for SSH or web clients, a daemon started by tuios ssh read only the hooks from the [daemon] section, and pkg/tuios carried a third, stale copy of the filter. Every client is now built through one set of functions, and a test reads the syntax tree to check that every entry point uses them. Twenty-two mutations of the guarded lines were each caught by the test that names them.

What belongs to whom

After all of this, the split is easy to state, and I wish I had written it down first:

The test for which column a piece of state belongs in is one question: does it change the size of a shell? If it does, a PTY has one size, and the state belongs to the session. The release notes list what this means for users.

It is not finished

The harness still has one switch that is off by default. With it on, the "change the layout" action cycles through all three tiling modes, and three of the five default sequences fail. I ran it again while writing this and got the same three failures. Each is a piece of layout authority that still lives in a client: the scrolling strip's column topology, where a new pane goes in the strip, and one border allowance under master-stack.

The default run is not perfectly clean either. The first time I ran it for this post, one of the five sequences failed. Replaying that seed six times failed once:

An interactive figure goes here. Open the page to use it.

That one is not fixed yet. But this is the kind of report I wanted from the start: it names the pane, the two clients, the two answers, and the steps that led there, instead of a person noticing a pane that looks slightly wrong. For a class of bug that only shows up with more than one client attached, that report is most of the work.

The scrolling layout is the niri-style one. Each pane is a column on a horizontal strip and the screen is a window onto that strip. A column can be widened with scroll_cycle_width, which steps through 33, 50, 55, 67 and 90 percent of the screen, and a column can hold several panes stacked on top of each other with scroll_consume. The report says the width came back at 50 percent. The default for a new column is actually 55 percent (appearance.scroll_column_width), and by eye the two are hard to tell apart. Either way, the width the user chose was gone.

What a session switch does

A tuios client does not keep a session alive. The daemon does. When you switch from one session to another, the client asks the daemon to detach it from the first and attach it to the second, and the daemon hands back that session's state. The client then throws away everything it held. rebuildForSession closes every window, and resets the BSP trees, the scrolling layouts, and the window ID maps to empty. Then it calls RestoreFromState on the state it was handed.

So switching back to a session is not returning to something the client kept. It is building that session again from the daemon's copy. Whatever the copy does not carry, the switch loses.

The copy carried the BSP layout. WorkspaceTrees has been part of SessionState for a long time, with every split and every ratio, which is why a BSP workspace comes back the way you left it. For the scrolling layout it carried one thing: ScrollStrip, a struct with a single field, ViewportX, which is how far the strip is scrolled from its left end. Nothing said which panes were in which column, or how wide any column was.

With no columns in the state, the client built the strip the way it builds a strip from nothing: one column per visible pane, each at the default width. That produces both symptoms. A widened column comes back at 55 percent. Two panes stacked in one column come back as two columns.

Here is a model of it. Widen a column or stack two panes, then switch session and back, in each mode:

An interactive figure goes here. Open the page to use it.

I had written it down

This was not a surprise to the documentation. A month earlier, on 28 August, I fixed a different scrolling bug: with two clients on one session, moving the focus on one left the other's viewport where it was, so the focused pane could be entirely off the second screen. The fix in d7299aeb made the strip's offset session state. That is where ScrollStrip came from. In the same commit I updated the limitation in docs/LAYOUT_MODES.md to read:

Column widths and the strip order are not shared or saved. The layout mode
and the scroll offset are session state; the column arrangement is not, and is
rebuilt from the window list on reattach and on each client.

That was accurate. It was also the whole bug, written down as a known limit. The same commit's message says that sharing the offset is safe because "one offset puts the same columns on every screen whatever size the terminals are". That holds only if every client has the same columns. The limitation two paragraphs away says they might not. I shared the position on the strip and not the strip itself.

The second bug in the same gap

The words "on each client" in that limitation point at a second problem, one the report did not mention. Two clients attached to one session each built their own strip from the window list. Widen a column on one, and the other kept its default widths. Stack two panes on one, and the other kept three columns.

This is worse than it sounds, for two reasons. A pane's PTY has one size, so two clients that disagree about a column's width disagree about the size of the pane in it. And since d7299aeb both clients share ViewportX. The same offset on two strips with different columns shows different panes. The second row of the widget above is that client.

The fix

The fix is in 209cb44c. The state now carries the columns. SessionState.WorkspaceScrollColumns maps a workspace number to a list of SerializedScrollColumn:

type SerializedScrollColumn struct {
	Windows    []string `json:"windows"`
	Proportion float64  `json:"proportion,omitempty"`
	FixedWidth int      `json:"fixed_width,omitempty"`
	Active     int      `json:"active,omitempty"`
}

Windows are the panes in the column, top to bottom. Proportion is the width as a share of the screen, and zero means the default. FixedWidth is a width in cells, which is what the < and > keys pin a column to. Active is which pane in the column has focus.

Panes are named by window ID, not by the integer the strip uses internally. Those integers are numbered by each client for itself. Naming panes by them would make the columns mean something only next to a mapping that also has to be restored first and kept in step.

It is the same kind of field as WorkspaceTrees: layout intent. A nil value means the sender did not say, and a client that receives nil keeps the columns it has. That is how a client that predates the field behaves anyway.

Where the strip gets built

The obvious place to rebuild the strip is in RestoreFromState: read the columns, build a ScrollingLayout, put it in the map. I did not do that, because there is already one place a strip comes into being, GetOrCreateScrollingLayout, and it does two more things after it fills in the columns. It points the strip's focused column at the pane that has focus, and it reveals that column, so you are not typing into a pane that is off the edge of the screen. A strip built somewhere else would skip both.

So the restore does not build anything. It sets the columns aside in pendingScrollColumns, keyed by workspace. When GetOrCreateScrollingLayout creates a strip, it checks that map first. If there are pending columns for this workspace, it takes them and deletes the entry. If not, it falls back to one column per visible pane, as before. The focus sync and the reveal run the same way in both cases.

Order matters in RestoreFromState. Entering the scrolling layout mode is what creates the strip, so the columns are set aside before ApplyLayoutModeName runs, not after.

The saved columns can name panes this client should not place: one that has closed, one on another workspace, one minimized or floating, or one an earlier column already placed. scrollColumnsFromState skips those, the way a strip built from nothing skips them. A pane on the workspace that no column names, which is one opened since the state was written, gets its own column at the end, which is where a new pane goes. And a workspace that was restored but not visited yet still has its columns in the pending map, so scrollColumnsState sends those too. Leaving them out would tell a peer that workspace has no columns.

Peers adopt in place

A push from another client takes a different path, ApplyStateSyncFrom. Here the strip usually exists already, and throwing it away would lose its offset and its focus. So adoptScrollColumns replaces the columns of an existing strip in place. It remembers the pane the strip was focused on, swaps the columns, and puts the focus back on the column that now holds that pane. Only a workspace with no strip yet goes through the pending map.

That closes the second bug. When one client widens a column, the others take the new width on the next sync.

The fingerprint

Every push is compared against the last one by StateFingerprint, so a push that changes nothing is not forwarded. That was added after two attached clients produced 32 peer broadcasts for 31 keystrokes, all of them identical to the one before. The fingerprint hashes by hand every field a peer acts on. A new field that a peer acts on and the fingerprint leaves out is a field whose changes can be dropped as repeats. So the columns went into the fingerprint too: workspace, pane IDs, proportion, fixed width, active index.

I wanted to know whether that part was load-bearing, so I took it out of a scratch copy of the fixed tree and ran the tests again. They still passed. A widened column also moves the rectangles of the panes in it and after it, and window geometry is already in the fingerprint, so in the tests the push was never a repeat. The columns entry covers a change to the columns that moves no rectangle, and I have no test that produces one. I kept it because the rule in the fingerprint's own comment is that every field a peer acts on is covered, not because I saw it fail.

The tests

The commit adds three tests in internal/app/scroll_columns_session_test.go. They run on a real daemon session, not a mock. Each one sets up the scrolling layout with three panes, does the gesture, and pushes the state.

TestAColumnKeepsItsWidthAcrossASessionSwitch
TestAStackedColumnStaysStackedAcrossASessionSwitch
TestAPeerTakesAWidenedColumn

• The first presses the width key until the column is at the widest preset, switches to another session and back, and checks the width in cells.
• The second consumes a pane into the first column, round-trips, and counts the columns.
• The third joins a second client, widens a column on the first, lets the two exchange state, and checks the proportion the second client holds.

I ran the new test file against the tree just before the fix, to be sure the tests catch the bug and not something next to it. All three fail, each with the symptom it was written for:

An interactive figure goes here. Open the page to use it.

The widget above uses the same sizes, which is why its columns start at 44 cells.

What I take from it

The session state is the only copy of a layout that outlives a client. The BSP layout was in it, so BSP survived a switch, and I had stopped thinking of a switch as a rebuild at all. The scrolling layout arrived later, and when I made its offset session state I added the one field that bug needed and wrote the rest down as a limitation. The limitation was accurate, and it sat in the docs for a month before the report came in. When it did, that one line explained the bug exactly, and a second bug with it that I had not gone looking for.

The benchmark it moved is the most boring one in the package. BenchmarkEmulatorShortLineScroll makes a 207x55 emulator with 10,000 lines of scrollback and writes this, over and over:

tuiosflood\r\n

Ten characters and a newline. The screen is full, so every newline scrolls it: the top row goes into the scrollback and a fresh blank row appears at the bottom. That is what a shell does all day. It is ls, a build log, tail -f.

Where the time went

The profile of that benchmark said 72% of the time was the blank part of each row. Two functions:

• isBlankCell, 39%. Before a row goes into the scrollback, PushLine trims its trailing blanks, so the scrollback does not store 197 spaces after every short line. It found the end of the text by starting at column 206 and walking left until it met a non-blank cell. For tuiosflood that is 198 cells read to find one.
• blankRows, 33%. The screen scrolls by rotating its rows, so the row that left the top is reused as the new bottom row. It has to be blanked first, and blankRows wrote a blank into all 207 of its cells.

A cell here is a uv.Cell, 112 bytes with pointers in it. 207 of them is about 22 KB. So a ten-character line cost two walks over 22 KB of cells, one reading and one writing, and nearly all of it was over cells that were already blank.

A lever I had already written down

This was not a surprise, and that is the embarrassing part. In August I had looked at exactly this blanking loop. An earlier pass had tried replacing it with one bulk copy from a prototype row and reverted it, and I wanted a number instead of an inherited conclusion. So I wrote BenchmarkBlankFill. It said the plain loop that stores one blank per cell was already the fastest arrangement I could find:

An interactive figure goes here. Open the page to use it.

The note I left in docs/perf.md ended with: the only lever left is moving fewer bytes, a smaller uv.Cell (an upstream type) or not blanking eagerly. Then I left it there for a month. The profile in September was the same conclusion from the other side. The loop was as fast as a loop can be. It just did not need to run over most of the row.

One int per row

The fix is small. The grid now keeps, for every row, an extent: a column past which every cell is a plain blank.

• A write raises it. SetCell raises it to the end of the cell it wrote, and handing out a row for arbitrary writes raises it to the full width.
• A full-width line shift, and the whole-screen rotation, carry each extent with its row.
• Blanking a whole row resets it to zero. Nothing else lowers it.

It is an upper bound, not the exact end of the text. Overwriting the last letter of a line with a space leaves the extent where it was. That is fine: the trim still checks the cells it reads, it just starts closer to them.

The two walks now start or stop at the extent. The blanking:

row := g.rows[i]
for x := range row[:g.ext[i]] {
    row[x] = uv.EmptyCell
}
g.ext[i] = 0

and the push into the scrollback:

n := min(ext, len(line))
for n > 0 && isBlankCell(&line[n-1]) {
    n--
}
sb.push(line[:n], len(line))

Try it. The row below is one line of the 207-column benchmark screen as it leaves the top. Type a line, or pick one, and press Enter to scroll it. The top strip is what the old code touched, the bottom one what it touches now.

An interactive figure goes here. Open the page to use it.

The widget counts cells, not time. For tuiosflood it says 405 cells before and 11 after, but the scroll does other work that did not change: parsing the bytes, writing the ten letters, encoding them for the scrollback, rotating the rows. That is why the benchmark below got about 4.7 times faster and not 37.

The clock checkbox is the honest case. A prompt that draws something at the right edge raises that row's extent to the full width, and then the blanking costs what it always did. The extent does not make wide rows cheaper. It makes short rows cost what they hold.

The numbers

An interactive figure goes here. Open the page to use it.

Everything that scrolls short lines moved with it:

All at p=0.002. No other vt benchmark moved. The cost is one int per row.

Measuring on a machine that would not sit still

The machine was shared with other agents the whole time. The load average sat between 13 and 18 on 11 cores, and wall time swung by as much as 100% between identical runs. A go test -bench number from that machine means nothing.

So nothing above is wall time. Each figure is process CPU time, user plus sys, per iteration. I built the test binary for the commit before and the commit after, ran each benchmark at a fixed iteration count (-test.benchtime Nx) on one CPU (-test.cpu 1), alternated the old and new binaries for six rounds each, and compared them with benchstat.

CPU time still counts the process start and the benchmark setup. That dilutes a change towards zero. It never makes one look bigger.

Before believing any of it I measured the noise floor: the base binary against a copy of itself, under the same alternation. All fifteen benchmarks came out as no change (~, p=0.37 to 1.00), geomean -1.0%, with confidence intervals from +/-4% to +/-43%. I only quote a result that clears that at p < 0.05. Allocation counts are exact, so they need none of this.

One thing this does not give you is a stable absolute number across comparisons. EmulatorWriteHeavyOutput/colored-log read 56.7 us after the plain scrollback change and 54.2 us before the ASCII run change, the very next commit, with nothing in between. BackendScroll is worse: 86.7 ms after the plain scrollback change and 71.3 ms before the extent one. The ASCII run commit sits between those, and its own comparison put BackendScroll at ~. That means its interval was too wide to say, not that it did not move. From two numbers in different pairs I cannot tell a change from a shift in load. Each before and after pair ran interleaved, so the pair is comparable. Two numbers from different pairs are not.

Keeping it right

Skipping work is a correctness claim. The extent is only safe if every path that writes a non-blank cell raises it. Miss one, and the scroll blanks too little of a row, so old text comes back, or leaves the end of a line out of the scrollback.

Two tests hold the invariant directly. The random grid test, which already drove the grid and ultraviolet's buffer side by side, now also checks the extents after every grid operation. A new test, TestGridExtentHoldsUnderGeneratedInput, runs the vtgen input generator through a whole emulator and checks the invariant on both screens after every step. The first one covers the grid's own methods. The second covers the writes that reach a row from outside the grid.

There is one of those that worried me. The commit just before this one made a run of ASCII store its cells straight into the row, without going through SetCell. That store has to raise the extent itself:

// Written behind the grid's back, so its extent is raised here
// (see grid.ext).
e.scr.buf.raiseExt(y, x+n)

While writing this post I deleted that line in a copy of the tree and ran the suite, to see what would notice. The conformance corpus passed. The tmux differential test passed, all 63 cases. That was not because the screen was right. None of their cases scrolls a row that the direct store wrote, so they never reached the bug.

Six tests failed:

• TestGridExtentHoldsUnderGeneratedInput, at seed 0, step 9: a cell holding e past its row's extent.
• TestScrollUpFillsScrollback: scrollback line 8 was "", want "line-08". The lines had left the screen and arrived in the scrollback empty.
• The test that scrollback storage is reused when the ring is full failed the same way.
• TestASCIIRunMatchesPerCharacterPath, because the run path and the per-character path now drew different screens.
• TestVTGen_Metamorphic, the split-equivalence test from the fuzzer post, reduced to two steps: 42 x characters and a full reset (ESC c). Fed in one piece, the reset cleared the screen. Split at a different boundary, an x survived it, because the reset stops clearing at the extent and the extent was stale.
• FuzzEmulatorWriteChunked, on its seed corpus.

I also removed the rotation of the extents in the whole-screen scroll. That one is loud: blankRows panics with a slice out of range, and one of the 63 tmux cases disagrees.

The quiet failure is the one I care about, and it is not quiet on screen. I fed a 20x4 emulator twelve lines, alternating long-line-NN and a short sN. With the line gone, the bottom row, where the cursor sits, showed long-line-08 again. Without the mutation it is blank. A recycled row keeps extent 0, so the blanking skips it and the old text comes back. Five of the nine lines in the scrollback were empty. Anyone scrolling a shell would see it within a screen.

The tests that check what the screen looks like, the corpus and tmux, still passed. That is a coverage gap: none of their cases recycles a row the direct store wrote. The tests that check what the emulator believes about itself did not need such a case.

The rest of the pass

The other four commits are smaller, and each one is a fast path held to the slow path it skips by a test that runs both on random input.

CSI parameters skip the transition table. Every byte of 38;2;r;g;b went through the parser's transition table and an action switch to reach three lines of arithmetic. In a truecolor repaint those bytes are most of the stream. In the CSI parameter state the table maps 0 to ; to a parameter update and leaves the state alone, so the parser now does that update directly. TestSeqParserCsiParamsMatchUpstream feeds random CSI input to this parser and to upstream's and requires the same action, state and dispatch for every byte.

A plain letter enters the scrollback as its byte. Encoding a line for the scrollback packed the style and compared the link of every cell, about 20 ns a cell, to find nothing had changed. While no style or link is in force, a narrow unstyled ASCII cell now appends its byte, which is exactly what the full path wrote. TestEncodeLinePlainShortcutWritesTheSameBytes compares the two on random mixed lines.

An ASCII run is stored straight into its row. Printing a run of ASCII went through three calls and two bounds-checked reads per byte. When every cell under the run is one column wide, no wide character can be cut, and all of that comes down to one store. Anything wide under the run takes the old path. This is the direct store the extent had to learn about.

SGR colours stop allocating. A truecolor repaint made four allocations per cell. Two were a color.RGBA boxed into a color.Color, for the foreground and the background. The exact 38;2;r;g;b shapes now take the boxed value from a small cache on the emulator. A third was a string made from the open grapheme cluster, to test whether the next character extends it. That test now runs on a reused byte buffer. TestRGBParamsMatchReadStyleColor and TestHandleSgrMatchesReadStyle hold the colour path to the upstream readers on random parameter lists.

The SGR change is the one where CPU time is the wrong number to look at:

An interactive figure goes here. Open the page to use it.

The fourth allocation per truecolor cell is still there. It keeps the open cluster's text across an escape sequence. Removing the other three saved 6% of CPU, so this one is worth a third of that at most, and the fix is one more piece of state on every emulator. I measured it and left it.

What I keep from this

The August benchmark was right, and it even named the fix. It said the loop could not go faster, which I read as "leave the loop alone" when it meant "run it less". When the work is already at the speed of memory, the only thing left is to touch less memory, and on a terminal full of short lines most of the memory is blanks.

The other part is the negative control. The extent itself is a small change. Trusting it took a test that watches the emulator's own bookkeeping, because the tests that watch the screen passed with the bookkeeping broken. The screen was wrong. They just never looked at a screen where it showed.

I did it with AI agents, orchestrated by me. That part needs a clear account, so here is what the setup was:

• Eight auditors read the tree with no write access. They produced 81 findings, each with the file and line evidence behind it.
• Every finding went to a second agent whose only job was to refute it.
• The surviving findings were grouped into 11 implementation units. Each unit ran in its own git worktree, and each diff got an adversarial review before it merged.
• Every bug fix came with a new test, and the test had to be seen failing: revert the fix, run the test, watch it fail, put the fix back.

It landed as 92 commits. The production Go got 1,494 lines shorter while gaining features, and the tests got 2,120 lines longer.

Most of what it found has the same shape, and that shape is the point of this post. The code was wrong, or dead, or two copies disagreed, and nothing failed. So nothing had ever made me look.

A flag nothing ever set

terminal.Window had a field called Minimizing, documented as "True when window is being minimized (animation playing)". Production code read it in 27 places. Every tiler, the dock, the workspace switcher and the render loop guarded on it:

if w.Workspace == m.CurrentWorkspace && !w.Minimized && !w.Minimizing {

The only line in the tree that set it to true was in a test. Production code only ever set it to false. One of those writes was in MinimizeWindow, right under a comment that said "Immediately minimize without animation".

The history explains it. Minimizing used to animate: set Minimizing, start an animation, clear the flag when it ends. On 24 October 2025, in a commit titled "refactor: split copymode.go into modular components", the animation was replaced by an immediate minimize. The field stayed, and so did the animation constructor, the animation type, its completion branch and the 18 guards that read the flag at the time. Code written over the next eleven months added nine more guards on a flag that could not be true.

It gets better. An earlier dead-code sweep, in August 2026, found the wrapper CreateMinimizeAnimation with zero callers and deleted it. Its commit message says minimize "animates through the generic path". It did not. The sweep removed the one caller-less function it could see and kept ui.NewMinimizeAnimation, because a test still called that one, and the test set Minimizing = true so it had something to animate. A test was keeping a dead feature alive.

Nothing could fail here. A term that checks a flag that is always false never changes the answer, so no test can see it being wrong. The branch it exists for, the one that runs while a window is minimizing, never ran. The audit removed the flag, the animation and every guard term, 202 lines out and 48 in.

An interface with one implementation

The daemon protocol had a Codec interface: Encode, Decode, Type. It came in December 2025 with two implementations, gob and JSON, and a negotiation step so a client could pick one.

The August sweep removed the JSON codec, because no client could negotiate it. It kept the interface. So the codebase ended up with GetCodec, which ignored its argument and returned the gob singleton, a connState, Client and TUIClient that each carried a codec field that was always the same value, and about 140 call sites of *WithCodec functions that passed it along.

None of that was broken. It was indirection that described a choice that no longer existed, and every reader of the protocol code paid for it. The fix removed the interface and the *WithCodec variants and left two unexported gob helpers. The wire did not change: frames still carry codec byte 0, and the welcome message still says "gob", so older and newer peers still talk.

The same code, twice

Reading a process's current directory is platform code: /proc on Linux, a kernel call on macOS, nothing elsewhere. tuios had it twice, as three build-tagged files in internal/session and the same three in internal/terminal. Both packages already imported internal/ptyspawn, so one copy moved there. A side effect: a resurrection test that had only run on Linux now runs on macOS too.

The input package had a more instructive case. Seven prefix actions (new window, the four split actions, settings, the command palette) had handlers that were exact copies of the window-management handlers. The skeptic diffed each pair after renaming and found them identical, so the prefix actions now register the shared handlers and the copies are gone. No behavior changed.

An eighth prefix handler was nearly a copy, and that is the one that mattered. prefix_help toggled the help overlay without resetting its scroll position. The normal toggle and the command palette both reset it. So help opened from the leader key could start part way down the list. Seven copies that still agreed were harmless. The one that had drifted was a bug.

Two copies that disagreed

The help overlay's key handling was also written twice, once for terminal mode and once for window-management mode. Over time the copies drifted. Try the combinations:

An interactive figure goes here. Open the page to use it.

In window mode, ? while searching only left the search, although the footer in both modes says ? close. q while searching also left the search, so a search could not contain the letter q. In terminal mode, q outside a search did nothing at all.

Merging the copies meant choosing one rule, and I chose the one the footer already advertises. esc leaves a search and otherwise closes. ? always closes. q closes outside a search and is typed into the query inside one. The new test, TestHelpKeysAgreeAcrossModes, runs every case in both modes. Run against the old two files, exactly the three drifted cases failed.

Only part of the overlay chain could be shared. The overlays checked after settings and the pickers are checked in a different order in each mode, and several of them can be open at once. One shared order would change which panel gets a key. So that tail stays per mode, and the commit says why.

Front ends that had drifted

tuios has three front ends: the local terminal, tuios ssh, and the separate tuios-web binary. One of the implementation units was only about places where they disagreed.

tuios-web took 9 of the 18 interface flags the other two take. You could not pass --shared-borders, --hide-clock or --confirm-quit to a browser session, only set them in the config file. The flags moved into a new internal/cliflags package that both binaries register, and a test checks that the web root command takes every flag in the set, so they cannot drift again.

SSH panes could start as TERM=dumb. The pane environment trusted the server's own environment only when COLORTERM=truecolor was set, and otherwise detected colour support from the server process's stdout. Under systemd, nohup or a log file, that stdout is not a terminal, detection answers "no TTY", and that maps to TERM=dumb. So every ephemeral pane on a headless SSH server ran as a dumb terminal while being drawn on the client's real one. The SSH server now sets xterm-256color with truecolor for that case at startup. A server started in a real terminal still detects from it.

Ephemeral web sessions had their own version of this: a placeholder 10x20 cell instead of the browser's measured one, no palette, and TERM=xterm-kitty, which needs a terminfo entry the server may not have. They now get the browser's cell size and palette, and xterm-256color.

Two small ones with real edges

A daemon pane resize called the PTY library's Resize, which writes the window size with zero pixels. Then every caller wrote it again with the pixel size, through a hand-written TIOCSWINSZ. For one resize the guest got two SIGWINCH signals, and a program that read the size between them saw a window with no pixels. Now the PTY remembers the cell size a client reported, and one write carries cells and pixels together.

The nightly race job had been failing a server test now and then, one that floods state syncs and checks that the client ends on the newest snapshot:

TestStateSyncFloodLeavesTheClientOnTheNewestSnapshot

The test used a round trip on the client's own connection as a barrier, on the grounds that the reply only comes back after every state sync sent before it. That holds for syncs already on that connection. But the daemon writes each client's broadcasts from its own goroutine, so under -race the reply could overtake the last broadcast. The snapshot was late, not lost. The test now waits for the newest snapshot itself, and it still fails if the queue ever drops the newest one instead of the oldest.

Where the lines went

Here is the whole change, by package. Switch between production code and tests.

An interactive figure goes here. Open the page to use it.

The biggest removals are in internal/app and internal/session: dead methods, fields that were written and never read, 69 unused config constants, the minimize code and the codec. The packages that only grew are new ones that replaced copies: cliflags for the flags, and served for the one model builder the SSH and web servers now share (there were four). The existing guestenv package took over the TERM detection, which the CLI client had its own, slightly different, copy of.

The test view is mostly the other direction. The biggest exceptions are internal/pool and internal/ui, where the tests went away with the code they tested: a style pool that saved nothing, because lipgloss v2 styles are values and the pooled pointer never carried any state, and the minimize animation.

There were speedups too, each its own commit so a bad one can be reverted alone. The largest was in the emulator: tracking where each row's text ends, so a scroll stops copying at the end of the text instead of the end of the row. A short-line scroll with scrollback went from about 1,120 ns to 240 ns of CPU per line. The details and the measurement method are in the "2026-09 vt parse and scroll pass" section of docs/perf.md. One fix went the other way: the focused pane had been dropping underline, and fixing it cost 3 to 9% on the cell loop, because every cell now carries more style. Two follow-up commits took loop-invariant work out of the per-cell path and won that back.

What the agents did well, and what they did not

The skeptic step earned its place. It did not just confirm findings. It cut them down. An auditor listed SendInputToDaemon as dead, and the skeptic found seven calls to it in the latency tests. An auditor said two picker state machines were identical, and the skeptic showed that their cancel and apply logic differed. And the agents doing the work pushed back as well: an auditor had called one reordering of a window-state update safe, and the implementing agent traced the moved field through four calls to the pane name shown in agent alerts, and left that part alone.

The adversarial review caught real mistakes too. The unit that made daemon panes honour appearance.preferred_shell also made the control client's hello resolve the shell, by loading the user config. That client backs every control command, tape runs, tuios logs and shell completion. So each of them parsed the config, wrote a default config.toml if none existed, and printed config warnings to stderr. The review blocked it, and the hello now names no shell and lets the daemon resolve it.

The docs had false claims in them, which is the same failure in prose: two commands the docs used did not exist, and the embedding recipes in the library docs did not compile. Nothing ran them. An earlier post on this blog had a wrong command too, and it now carries a correction note.

What did not go as well. The machine was shared by all of the agents at once, with a load average between 9 and 32 on 11 cores. Wall-clock benchmarks swung by up to 2x between identical runs, so every comparison had to alternate old and new binaries over several rounds, and some tests timed out under load and passed when rerun alone. That is noise I had to rule out by hand each time. One agent overwrote a helper script another agent had left in the shared scratch directory. One unit broke two tree-wide tests, and only its full-suite run caught that.

And the audit found what reading can find. It is very good at "nothing refers to this". It is less good at "this matters". A few unused functions are still there because whether to keep them is my decision, not a fact about the code. Some things stayed open: the SSH half of the host palette has no source to read from, because an SSH client sends nothing that names its colours.

What I keep from this

Every item in this post survived for the same reason: nothing failed when it was wrong. A guard on a flag that is always false cannot fail. An interface with one implementation cannot pick the wrong one. Two copies of a handler each pass their own tests. A doc command is not run by anything. The flaky race test is the exception: it did fail, and that is why it was on the list.

So the rule I held the audit to was the one from the start of this post: a fix is not done until a test fails without it. That is what the revert step checks. A test that has never failed has not shown it can see the bug. And for dead code, the useful question is not "is this called?". It is "if this were wrong, what would tell me?". When the answer is nothing, the code is either dead or untested, and either way it should not stay as it is.

It wired the writer. It did not wire the reader. LoadResurrectionState had no caller outside its own tests. The function that listed saved sessions, ListResurrectableSessions, had none either. From April to 18 July every tuios daemon wrote a JSON file per session every 30 seconds, and when the daemon restarted, every session was gone.

The tests were green the whole time. TestSaveAndLoadResurrection saved a state, loaded it back and compared the two. That is a real test of a real round trip, and it says nothing about whether anybody ever takes the trip.

This post is about the two halves of a session coming back. The first half is the daemon restarting: getting a session's shape off disk and into a live daemon. The second is a client reattaching: getting a pane's screen out of the daemon and onto your terminal without painting anything twice or losing rows. Both had bugs that only showed up once something actually read what had been written.

Wiring up the read side

The restore landed on 18 July in two commits. The first hardened the file so it was safe to read on a cold start:

• Every state file carries a schema version. A file from a newer tuios, or one that does not parse, is moved to an archive directory rather than loaded or deleted. One bad file can never crash the daemon or stop it starting.
• Each window records its working directory. The daemon reads it from the live shell process when it saves. The client never supplies it.
• RestorePTY starts a fresh shell in the saved directory, sets TUIOS_RESTORED=1 in its environment and writes a dimmed one-line banner into the pane.
• tuios kill-session deletes the saved state, so a session you killed on purpose never comes back.

The second did the restore. On start, before it accepts a single client, the daemon recreates every saved session: windows, workspaces, titles, the layout and the BSP trees, with a new shell in each window. The old PTY ids died with the old daemon, so every window is remapped to its new PTY. tuios resurrect lists what is saved or brings one session back by name, and tuios daemon --no-restore leaves restoring to you.

Here is what that looks like today, on macOS, against a scratch state directory. I created a session, opened a second window, and sent the daemon a SIGKILL three seconds later:

An interactive figure goes here. Open the page to use it.

Starting a daemon again brings it back, marked as restored until the first attach, with a new shell in the directory the old one was in:

An interactive figure goes here. Open the page to use it.

That is the version that works. Getting there took another two months, because the moment the file was read, everything wrong with how it was written started to matter.

What reading it exposed

On 12 August four fixes landed within ten minutes of each other. Each one is a case where the saved state described something that did not exist, or failed to describe something that did.

The session you just made was the one you lost

The saver was a ticker started when the session was created. Its first write was 30 seconds later. So a session created 25 seconds before a SIGKILL was not stale after the restart. It had never been written at all, and it was the session you had just made.

The fix sets a flag on every change to a session's structure, and the saver polls that flag every 2 seconds. A poll that finds nothing changed does nothing. The 30 second write stayed exactly as it was, because it is what keeps each window's working directory current: typing cd changes no structure, so no flag is set. A structural change now reaches disk within about two seconds, and an idle session writes no more often than it did before.

Two tests pin both halves of that bargain. One fails if a changed session is not on disk within three polls. The other fails if the faster poll turns into faster writes for a session where nothing changed.

An interactive figure goes here. Open the page to use it.

Drag the kill before 30 seconds and the old ticker has nothing on disk. Drag it to 40 and the old ticker brings back one window and loses the one opened at 35 seconds. The new saver has both by 36 seconds, and by the end it has written three times to the old ticker's two.

A window that was really a hole

If a restored window's shell would not start, the window stayed in the session holding the PTY id of the previous daemon's process. Nothing answers to that id. The pane could not print, could not be typed into and could not be revived, and nothing anywhere draws a pane as dead. It looked like a window and it was a hole in the layout.

The fix drops the window, which is what the daemon already does whenever a PTY goes away, and repairs focus in case the dropped window had it.

Sessions with nothing in them

A session whose windows had all been closed still left a state file, and the next daemon start restored it as a live session with no windows. It listed, it showed in the rail, you could attach to it, and there was nothing in it. The restore now refuses a session with no windows, for the automatic path and for tuios resurrect <name> alike, and the automatic path deletes the file so it stops being offered forever.

Files nothing cleaned up

A save writes <name>.json.tmp and renames it. A daemon killed between those two steps left the temp file forever. The archive, which exists so a file that would not load can still be inspected, had no bound at all. The daemon now sweeps the directory on start, the one moment none of its own saves can be in flight: temp files go unconditionally, and archived files go after 14 days. The sweep is best effort. A directory that cannot be swept must never stop the daemon from starting.

Two days later there was one more: with --no-restore, tuios attach work called a session that was sitting right there on disk unknown, and offered to create it. It now says the daemon has not restored it, how many windows it saved, and to run tuios resurrect work.

The working directory on macOS

The daemon read each shell's working directory with os.Readlink("/proc/<pid>/cwd"). macOS has no /proc. The read failed quietly, the field stayed empty, and every restored macOS pane opened in whatever directory the daemon happened to be started from. No error, no warning, on a platform I use every day.

The same assumption had been written three times in two packages, and it was found three times:

The resurrection fix came in almost by accident, inside a commit that made new windows open in the focused pane's directory. That feature needed the same read, found it empty on macOS, and fixed it for both. On 22 September the two copies of the per-platform reader were merged into one ptyspawn.ProcessCwd, and the resurrection cwd test now runs on macOS too. The darwin path goes through proc_pidinfo without cgo, so the release build stays CGO_ENABLED=0.

The second half: a pane has two sources

A restarted daemon has respawned shells with nothing on screen. A daemon that kept running has live panes, and a client that reattaches to one has to end up drawing the same screen the daemon holds. That is harder than it sounds, and REHYDRATION.md is the document I wrote to keep it straight.

The daemon runs a terminal emulator for every pane, and a client runs a second one. There are two ways to fill the client's:

• The snapshot. The daemon serializes its emulator: the grid, the cursor, the pen, the modes, the scroll region, the character sets, up to 1000 rows of scrollback. It is a picture of now, and it carries the stream position it was taken at.
• The stream. Every PTY keeps a 64 KB ring of the bytes it produced and a counter of every byte it ever produced. Subscribing from a position replays the ring from there and then streams live. It is history: applying the same bytes twice paints them twice.

They are not interchangeable and they are not additive. The rule is that a client lays down the snapshot, then subscribes from exactly the position the snapshot ends at, so nothing overlaps:

An interactive figure goes here. Open the page to use it.

Two bugs broke that rule this summer, in two different ways.

Painting the old picture over the new one

The client used to subscribe to the PTY first and fetch the snapshot second. Live output was already streaming into the pane while cells that were by definition older were painted over it. And the restore wrote into the emulator with no lock held, while the output goroutine wrote the same cell buffer under the I/O lock and the renderer read it under the same lock.

I found it by building with -race and driving the build through the e2e suite. This was the most frequent signature in that run: 112 of 182 race reports, 96 through the window sync path and 16 through the tape playback re-fetch. The fix on 26 July does the restore under the I/O lock, and does it before subscribing, in one function that both call sites now share. The cost of restoring first is whatever the pane prints during a single round trip. The new test drives a restore against live output directly, and against the old unlocked code it reports 46 races.

top, one blank line apart

On 16 August a user on Linux reported issue #123: run top, detach, attach, and top comes back with an empty line between its rows. Quit top and run it again, and it still looks like that.

The fix came from a contributor, SebaWag, in PR #139, and it landed on 1 September. The shape was this. When a pane produces output faster than the daemon's emulator consumes it, the ring can roll past the position the client's snapshot was taken at. The bytes between the snapshot and the ring's start are gone. The old code handled that case by prefixing the replay with a screen clear, ESC[H ESC[2J ESC[3J, and then sending the ring from its first byte.

That was the right call for a client that still held a stale screen from before it left. It was the wrong call for a client that had just laid down an authoritative snapshot, for two reasons:

• The clear threw away the snapshot. A full-screen program like top does not rewrite every row on every tick. Every row that only the snapshot held came back blank.
• The ring's first byte is almost never the first byte of a chunk. It is the middle of whatever the program wrote, sometimes the middle of an escape sequence, replayed against cursor and mode state the client never saw.

The fix has two daemon-side parts. A client that restored a snapshot says so when it subscribes, and a rolled catch-up replays on top of the snapshot instead of clearing it. And the PTY records where each chunk began, so a rolled catch-up starts at the first whole chunk inside the ring rather than at its raw first byte.

An interactive figure goes here. Open the page to use it.

Drag the slider until the ring rolls past the snapshot, then switch between the two paths. Before, the header rows top did not rewrite come back blank between the ones it did. After, the snapshot supplies them and the screen matches the daemon.

I have to be honest about where this ended. I could not reproduce the user's steps. Real top, detach, wait, attach, at the same size and at a smaller size, came back clean every time. To reach the bug in a test, the pane had to repaint far faster than top does. The fix is real and its tests cover the quiet, redrawing, rolled and quit-the-alternate-screen reattach paths, but it does not fully explain the report. I reopened the issue on purpose, and at the time of writing it is still open, waiting on the reporter.

History that only gets longer

One more rule was written into the contract in September. A pane whose emulator survived a workspace switch should be handed only the rows that scrolled off while it was away, and should never have its history replaced by the daemon's bounded window. The ghostty backend broke that: its restore started from a hard reset, which drops the library's history, so the tail the daemon sent became the whole history and deep scrollback vanished on every workspace switch (issue #146). It now starts from a hard reset only when the emulator holds no history, and a wire test with eleven streams covers the second restore on both backends.

What does not come back, on purpose

Resurrection brings back the shape of a session, not what was running in it. The sessions docs spell it out:

• Running programs. Every window gets a fresh shell. A vim, a build or an ssh connection is not restarted.
• Screen contents and scrollback. They live only in the daemon's memory and are never written to disk.
• The last couple of seconds. A window created just before a SIGKILL may be missing, and a cd made less than 30 seconds before it may not be recorded.

A restored shell is marked so neither you nor your tools mistake it for the old one. The banner is for you. TUIOS_RESTORED=1 is for your shell profile and your scripts, which can check it and behave differently in a restored shell. The session shows a restored tag in tuios ls, the rail and the switcher until you first attach.

What I took from it

The resurrection file was the cleanest kind of dead code: it ran, it did its job correctly, it was tested, and its output went nowhere. A round-trip test proves the two halves agree. It does not prove anyone uses either half, and nothing about a green test run would ever have said so.

Everything after that came from reading the state for real. The blind ticker, the dead PTY, the empty sessions and the macOS directories were all present from the day the file was first written. None of them could hurt anyone while nobody read the file, so none of them were found until someone did.

That question turned out to be much harder than drawing the list. This post is about the weeks it took to answer it, and mostly about the one case every version got wrong: an agent sitting on a permission prompt.

Version one: ask the kernel

The first commit, on 9 August, added the state model. Each pane got an agent state (none, working, needs_input, idle, done, errored), held by the daemon so it survives a detach. A pane could report its own state with tuios set-agent-state, and a reference shim mapped Claude Code's lifecycle hooks onto it.

It also added a fallback for agents that report nothing: a pane that said working and then produced no output for 30 seconds was demoted to idle. The commit message calls it conservative. It only ever reads working and only ever writes idle, so an explicit report is never overridden.

The next day I added detection that needs no setup at all. Every two seconds the daemon reads the pane shell's tpgid from /proc, finds the foreground process group leader, and, if that process is a known agent CLI, marks the pane working. When the agent leaves the foreground, the pane clears.

Two things could now set a pane's state: the agent's own report and the detector. One bool per pane recorded whether the detector had claimed it. The detector only promoted a pane with no state, and yielded to any explicit report. With two sources, a bool is enough.

A bool cannot rank

Within days I wanted more sources. Harnesses already emit OSC 9;4, the progress bar sequence, and tuios had been parsing it only to throw it away. Some harnesses change the window title when they need you. Some things can only be read off the screen.

The commit that replaced the bool, on 12 August, says it directly: a bool "cannot say which of three sources should win". So each source got a rank, and one rule: a source may write over a claim ranked at or below its own, and never over one ranked above it.

The numbers are spaced so a tier can go between two others without renumbering. That paid off four days later: the transcript tier slotted in at 35 and nothing else moved.

Two days after the ranking, OSC 9;4 was wired in (a bar means working, clearing it means idle, the error state means errored), and a small hold went in beside it. A harness that clears its progress bar between two steps of one task would blink the pane through idle and straight back. Now a state that lowers how much the pane wants a human has to stand for 700ms before it is published, and anything at or above the current level is published at once. Being slow to say an agent needs you costs the user the thing the feature is for. Being slow to say it went quiet costs nothing.

Silence looks the same on every channel

Then I watched what a pane actually does when Claude Code stops to ask permission. The comment in its manifest records the measurement: a session sitting on a permission prompt emitted nothing at all. No further output, no title change, no progress sequence. The prompt is painted once, and then the pane is silent.

Every channel tuios was listening to carried that silence. So 30 seconds later the stall timer did exactly what it was built to do and called the pane idle. And the alert policy ignores idle, because idle means fine. The one moment a user needed to be told about was the one moment nothing could tell them.

A harness that finished and a harness waiting on a human produce byte for byte the same silence. No timer can separate them. Only looking can.

The manifest schema already had screen rules. They had been parsed and validated since the registry landed, and nothing read them. The fix on 15 August wired them up: read the bottom lines of the pane's active screen, match the manifest's rules, and report through the screen tier. For Claude Code the permission rule needs the question stem and the numbered list together, so a single stray string cannot fire it:

[[screen.rule]]
state    = "needs_input"
priority = 30
message  = "Waits for approval of a tool call."
all      = ["Do you want"]
any      = ["1. Yes", "❯ 1."]

The trigger was the hard part. Scanning on every output chunk is wasteful, so scans are throttled to one per 250ms. But a throttle drops exactly the chunk that matters. The prompt is painted by the last chunk before the pane goes quiet, the throttle swallows it, and then nothing else ever arrives to trigger a scan. So each pane also arms one timer on output, pushed back by every new chunk, that fires 400ms after the last one. There is no ticker, so a silent pane still costs nothing.

Only needs_input rules ship enabled. Working is already carried by OSC 9;4 and by output arriving at all, and a rule keyed on a spinner glyph is the first thing to break when an agent restyles its TUI in a patch release.

An hour and a half later a second commit closed the other half. The stall timer now hands each stalled pane to the screen tier first and leaves alone any pane whose screen answers. Later still, a stalled pane whose screen says nothing got unknown instead of idle: the screen was read and said nothing, and idle would claim nothing needs you when nothing here knows that.

Here is one pane through a permission prompt. The toggle switches between the first version and the ranked one. Add events, or click a column to see why the rail shows what it shows.

An interactive figure goes here. Open the page to use it.

The claim that would not let go

The screen tier fixed agents that report nothing. It broke on agents that report something.

With the hook installed, Claude Code reports working for itself, at rank 40. Then it stops on a permission prompt and says nothing further. The screen rule sees the prompt, reports needs_input at rank 20, and is refused, because rank 20 cannot write over rank 40. So the pane showed working for as long as the user was being waited for. The better integrated the agent was, the worse this case got.

The ranking was not wrong. It was being applied to a claim that had gone stale. The fix on 16 August is one exception to the ranking, and it is narrow:

func blockerOverridesClaim(w *WindowState, r AgentReport, now time.Time) bool {
	if r.Source != AgentSourceScreen || !agentStateBlocks(r.State) || w.AgentState == r.State {
		return false
	}
	if r.paneWroteAt <= w.AgentStateAt {
		return false
	}
	return now.UnixNano()-w.AgentStateAt >= int64(agentBlockerOverrideGrace)
}

A screen rule that matches a blocking state may take a higher claim only if the pane has painted since that claim was stamped, and the claim has stood unrefreshed for two seconds. The two seconds give a hook the chance to describe the new screen first. If it does, the hook wins as before.

The part I almost left out is what happens after. The screen tier only ever asserts needs_input. If the override simply took the pane, nothing would ever move it off again, and the pane would stick on needs_input exactly the way it used to stick on working. So the override records what it displaced. The next look that finds no rule matching, which is the look triggered by the agent painting over its prompt, hands the old claim back as it was. The exception is a loan.

A few weeks later that loan became the rule for every screen claim, not only the ones that jumped a rank. A pane the detector held had stayed on needs_input through the turn after the answer, because its screen claim had displaced nothing worth remembering.

Reading the agent's own record, and nothing else

Screen rules are coupled to how one version of one TUI happened to look. Claude Code also writes a JSON record per line to a file under ~/.claude/projects/, appended at every message boundary. That file says what the agent did rather than what it looked like. It is also the whole conversation, which made me careful about how it is read.

The reader is built so that leaking the conversation is a property of the types, not of care. Records decode into a struct with six scalar fields and one nested stop_reason. encoding/json walks an unknown field to find its end and never builds a Go value for it, so message content, tool results and prompts are never constructed. There is no map[string]any and no json.RawMessage in the package. Nothing logs. A line that fails to parse increments a counter and is dropped, and the error, which would quote it, is not kept. The read buffer is a field on the reader, zeroed after every parse, so a heap dump taken later holds no stale page. It is a field and not a parameter because a panic traceback prints argument words. The only thing that leaves the package is a turn, one of three constants.

Size was the other constraint. These files reach 150MB. A cold read looks at the last 128KB and no further, and after that the reader resumes from where it stopped. Against 20 real transcripts on my machine, a 151MB file read in 652 microseconds with zero unparseable lines. The largest files are among the fastest, because the cold read is a fixed window whatever the size.

Two details give a confident wrong answer if you get them wrong. The last line of a live file is usually half written, so everything past the final newline is dropped and read again when it completes. And subagents write to their parent's file, so sidechain records are skipped. Without that, a subagent finishing would report the whole pane done while the agent that spawned it kept working.

The transcript ranks below the report because of latency, not trust. A hook speaks about now. Records land at message boundaries, and one measurement found a live transcript 44 seconds behind during active work. It also does not claim needs_input at all. The signature for that would be an inference, never checked against a live permission prompt, and it is the one state that raises an alert. The screen tier already reads that prompt, and the visible-blocker exception is what lets it say so over the transcript.

The transcript also exposed a gap the ranking had all along. A claim is held until something ranked at least as high replaces it. That is right while the source can still speak, and wrong once it cannot: a reader whose agent died would hold its pane against every weaker tier forever. So a source can now yield its claim, saying it has nothing further to add without asserting anything in its place. A file that is gone for three consecutive reads ends the join and yields.

Watching is per directory, not per file. inotify on a directory reports writes to the files inside it, so forty panes across twelve projects is twelve watches. Directories are refcounted, so the twelfth pane in a project adds nothing. Idle cost is zero by construction: one goroutine for the whole daemon, blocked in a channel receive, and a burst of appends collapses into one read through a one-shot timer. Two tests assert it. An idle join holds no timer and grows no goroutines, and twenty events produce one read.

Two ways detection lied

All of that assumes the daemon knows which agent is in the pane. That part had its own bugs.

On 14 August the list of known agents moved from a slice of ten strings in the detector into TOML manifests, one per harness, so supporting a new agent no longer meant shipping tuios. A manifest can match a process by comm, argv0, a package path in argv (argv_path), or a glob over the real executable (exe_glob). Five days later I found that two of those four were broken in opposite directions.

exe_glob matched nothing, ever. path.Match's * does not cross /:

path.Match("*/claude", "/home/u/.local/bin/claude") // false
path.Match("*/claude", "bin/claude")                // true

The executable path is always absolute, so all six patterns in the three manifests that had them were dead. Matching is now component-wise: * stays inside a component, ** spans any number, and a pattern without a leading / matches any suffix, which is what */claude was always written to mean.

argv_path matched far too much. The registry asked whether any argument contained a manifest's string, and every manifest carried one: /opencode/, /aider/, @anthropic-ai/claude-code. So tail -f ~/dev/opencode/main.go was opencode. A scan of every process on a real machine found six matches, and five of them were not agents. Now argv is read only when the process is an interpreter, and only the one token it was asked to run, so python3 -m pytest tests/aider/test_x.py is pytest.

Neither bug was caught by tests, and neither could be seen from outside. A pane was an agent or it was not, with nothing said about why. So the same day I added tuios explain-agent-detect. It reads the pane's process at the moment of the call and prints what the daemon saw, which manifest matched and on which predicate, and for every manifest that did not match, what it compared against. After a later round of detection work in September, it also leads with a verdict in plain words. This is its output for an agent behind a wrapper, from the agents reference:

An interactive figure goes here. Open the page to use it.

That September round made identity rest on what a process calls itself, so a directory called claude or codex is no longer an agent, and it reads through wrappers like sh -c, timeout, npx and nix develop. The benchmark in that commit records 14 microseconds per pane when the leader is the agent and 201 microseconds in the bounded worst case behind a wrapper.

What herdr already knew

herdr publishes a curated set of agent detection manifests (Apache-2.0) covering 20 agents. When I compared them with tuios's, the overlap was large and the format was close enough that conversion is mechanical for most rules. So on 22 August I wrote a converter rather than transcribing by hand.

The converter does not pretend the formats are the same. herdr keeps process detection in code, so the [detect] block it emits is a placeholder. herdr's rules nest predicate groups, and tuios rules are flat, so a compound rule is split into one rule per alternative, which keeps the OR exact. A rule that reads a region tuios cannot see, such as the OSC title or prompt-box geometry, is dropped with that reason written into the output. Where the flattening had to bend, it bends toward suppressing a match, never toward a false one. The output is a draft for review, not a manifest to ship.

From it came manifests for the 13 agents herdr detected and tuios did not: amp, antigravity, cline, devin, github-copilot, grok, hermes, kilo, kimi, kiro, maki, qoder and qwen. Each file carries herdr's attribution and version, a [detect] block written for tuios from that agent's real install layouts, and only the blocked-prompt rules enabled. herdr's working and idle rules read spinners and footers, which is exactly what the bundled policy leaves out. Fixtures pin the rules against the prompts as herdr documents them, and detection against the process shapes each agent runs as, node shims included. There are 22 manifests now.

The comparison mostly confirmed the shape of the problem. herdr's manifests, too, find the blocked prompt by reading what the agent drew. That is the one fact no agent reliably tells anything else.

What I keep from this

Every version before the screen tier was correct about every signal it received. The bug was a state that sends no signal. Silence was being read as an answer, and it was the same answer for a finished agent and a blocked one.

The second lesson was about precedence. A ranking says who to trust, but not for how long. The report outranked the screen for good reasons, and a report that had not been refreshed while the pane painted a prompt over it was still outranking the one source that could see the prompt. The fix was not to change the ranks. It was to notice staleness, and to give back what was borrowed.

The details are in the agents docs, the session rail docs, and the changes since v0.7.0.

The focused pane, the one you are typing into, never showed an underline. An unfocused pane showed it. Leave the pane and the underline appeared. Come back and it went away. An unfocused pane in window mode that was dimmed, in copy mode or scrolled back lost more than that: bold, italic, faint, reverse and strikethrough went too, and only the colours survived.

How I found it

Not from a report. It came out of an audit of the renderer, reading two functions in internal/app/render_helpers.go side by side. The first, buildCellStyle, built the style for one cell:

if cell.Style.Attrs != 0 {
    attrs := cell.Style.Attrs
    if attrs&1 != 0 {
        cellStyle = cellStyle.Bold(true)
    }
    if attrs&2 != 0 {
        cellStyle = cellStyle.Faint(true)
    }
    if attrs&4 != 0 {
        cellStyle = cellStyle.Italic(true)
    }
    if attrs&32 != 0 {
        cellStyle = cellStyle.Reverse(true)
    }
    if attrs&128 != 0 {
        cellStyle = cellStyle.Strikethrough(true)
    }
}

return cellStyle

The bits are right. They match ultraviolet's AttrBold, AttrFaint, AttrItalic, AttrReverse and AttrStrikethrough. What is missing is everything that is not a bit. Underline in ultraviolet is its own field, Style.Underline, with a style (single, double, curly, dotted, dashed), and the underline colour is a third field, Style.UnderlineColor. The function never read either. It also skipped bit 8, blink.

The second function was buildOptimizedCellStyle. It set the foreground and the background, and returned. No attributes at all. It was chosen by one line in renderTerminal:

useOptimizedRendering := !isFocused && !inTerminalMode

So there were at least two opinions in the code about what a cell looks like. Reading told me that much. It did not tell me which panes took which path on a real frame, and I have been wrong before about a causal story I only read. So the next step was a probe, not a fix.

The probe

The probe writes one line into a pane's emulator, with a run of each attribute:

const attrsLine = "\x1b[4munder\x1b[0m \x1b[1mbold\x1b[0m \x1b[3mital\x1b[0m " +
	"\x1b[4:3;58;5;196mcurl\x1b[0m \x1b[2mfaint\x1b[0m \x1b[5mblink\x1b[0m " +
	"\x1b[7mrev\x1b[0m \x1b[9mstrike\x1b[0m \x1b[1;3;4mall\x1b[0m"

4:3 is a curly underline, and 58;5;196 sets its colour to xterm red. Then it renders the pane through renderTerminal, parses the frame that comes out back into cells, and compares every cell of that row with the cell the emulator holds. Colours are left out of the comparison, because the dim setting changes them on purpose. Text attributes and the underline are compared exactly.

Run once for each way a pane can be drawn, that probe became the regression test, TestEveryRenderPathKeepsTextAttributes in internal/app/render_attrs_test.go. It has ten cases. Here it is run against the parent of the fix commit:

--- FAIL: TestEveryRenderPathKeepsTextAttributes/focused
--- FAIL: TestEveryRenderPathKeepsTextAttributes/focused_window_mode
--- PASS: TestEveryRenderPathKeepsTextAttributes/unfocused_fast_path
--- PASS: TestEveryRenderPathKeepsTextAttributes/unfocused_fast_path_window_mode
--- FAIL: TestEveryRenderPathKeepsTextAttributes/unfocused_dimmed
--- FAIL: TestEveryRenderPathKeepsTextAttributes/unfocused_dimmed_window_mode
--- FAIL: TestEveryRenderPathKeepsTextAttributes/focused_copy_mode
--- FAIL: TestEveryRenderPathKeepsTextAttributes/unfocused_copy_mode
--- FAIL: TestEveryRenderPathKeepsTextAttributes/focused_scrollback
--- FAIL: TestEveryRenderPathKeepsTextAttributes/unfocused_scrollback

Two of the ten passed. Both were the unfocused fast path. Every failure reported the same first cell:

focused: cell (0,0) rendered as "u" {... Underline:0 Attrs:0},
         emulator holds "u" {... Underline:1 Attrs:0}

The test stops at the first wrong cell of a path, so that says nothing about the rest of the line. To see the rest I ran a copy of it that logs every run instead of stopping. The ten cases fell into three groups:

Five ways to draw a pane

A pane reaches the screen through one of these, decided per frame:

• Unfocused, nothing special. The fast path. renderTerminal hands the grid to the emulator's own Render, which emits every attribute, underline style and colour included.
• Focused. Always the cell loop, because it has to draw a cursor and handle things the fast path cannot. Styles came from buildCellStyle.
• Dimmed. The dim setting blends an unfocused pane's colours, which the fast path cannot do, so a dimmed pane takes the cell loop too.
• Copy mode. The cell loop, for the copy cursor, the selection and the search highlights.
• Scrollback. The cell loop, reading rows out of scrollback instead of the grid. A hovered link also takes a pane off the fast path.

The cell loop took its styles from buildCellStyle, which dropped the underline, except for an unfocused pane while the app was in window mode. That pane got buildOptimizedCellStyle, which dropped the rest as well. Only the fast path was right, and only because it used neither builder.

An interactive figure goes here. Open the page to use it.

Why nobody saw it

Each path, looked at on its own, draws something plausible. A missing underline does not look broken. It looks like text without an underline. A focused pane showing a man page with plain text where the underline should be is not something you stop and question.

What does look broken is the same text changing when nothing about it changed. That only happens when a pane moves from one path to another: focus moves, the dim kicks in, the app switches between terminal mode and window mode, you enter copy mode. Take the focus change, since it happens all day. It is a moment when you are looking at something else. You switched focus to type in the other pane. The pane that changed is the one you just stopped looking at.

A style that depends on which path drew it is a bug nobody sees until they switch focus, and when they switch focus they are not looking.

The fix

fb8735a6 makes buildCellStyle read everything the emulator's renderer reads. The magic numbers became the named ultraviolet constants, blink was added, and the underline style and colour are mapped:

if cell.Style.Underline != uv.UnderlineNone {
    cellStyle = cellStyle.UnderlineStyle(cell.Style.Underline)
}
if isColorSafe(cell.Style.UnderlineColor) {
    cellStyle = cellStyle.UnderlineColor(cell.Style.UnderlineColor)
}

Mapping it on the style was not enough, and this is the part a quick fix would have missed. The underline has to survive four more places on the way out:

• shouldApplyStyle decides whether a cell gets a style at all. It returned true only when the cell had a foreground, a background or an attribute bit set. Underline is not an attribute bit, so a cell with an underline and no colour was never styled. The under in the test line above is exactly that cell: with buildCellStyle and the other three places fixed, it would still have come out plain. It now also checks the underline style and the underline colour.
• styleToANSI, which turns the style into the escape the loop writes, only knew a boolean underline. It now emits the underline style and the underline colour.
• The style cache keys styles by a hash of the cell. The hash had no underline in it, so an underlined cell and a plain one with the same colours would have shared one cache entry, and whichever came first would have won. The key now includes the underline style and colour.
• The loop batches neighbouring cells with the same style into one run. Its comparison checked colours and attribute bits, so an underlined cell next to a plain one would have joined its run. The comparison now checks the underline too.

buildOptimizedCellStyle is gone. It existed to be cheaper, and it was not: both builders sat behind the same style cache, so after the first cell of a given style both cost one hash and one map lookup. The only thing the optimized builder saved was correctness.

After the fix all ten paths pass, and the test stays in the suite. It also checks, before comparing anything, that the emulator really did give the first cell a single underline and the curl cell a coloured curly one. If the emulator ever stopped parsing 4:3 the comparison would pass on two plain lines, and a test that passes on nothing proves nothing.

What it cost, and what I could not measure

Reading three more fields per cell, and comparing them in the batching check, is not free. The cell loop runs once per cell per frame on every focused pane. So I went looking for work in the same loop that did not need to be there.

The third follow-up, 6ca97f96, says the extra compares "measured as a few percent on the focused render benchmarks". Four follow-ups removed work from the same code, and none of them changes the output:

• d1321aa9: in scrollback and copy mode the loop fetched the scrollback line once per cell. Each fetch took a lock, checked a generation and did a map lookup, for a value that only depends on the row. It is now read once per row.
• 1d858a68: the emulator's own Render, the unfocused fast path, compared neighbouring cells with uv.Style.Equal, which converts every colour to RGBA even when both sides are the same value copied from one pen. It now returns early when both colours have the same type and are equal, and compares RGBA otherwise. A test checks the new comparison against ultraviolet's.
• 6ca97f96: the fake cursor test re-read six flags and made an interface call for every cell, though none of them change within a frame. It is decided once per frame. shouldApplyStyle was asked for every cell and only used where a batch starts, so it is only asked there.
• ca2c8edc: the style cache is looked up once per style run, which under a full-screen colour flood is nearly once per cell. Its key wrote the one-byte attribute field into the hash as eight bytes, and ran a reflection-based nil check on a colour before testing whether it was an indexed one, which can never be nil. It now writes one byte, tests for the indexed colour first, and skips an absent underline colour entirely.

I built the render benchmarks at three commits, the one before the fix, the fix, and ca2c8edc after all four follow-ups, and ran them interleaved so drift hit all three alike. It did not work. The machine was busy with other work, and it still was when I came back to rerun them quietly: the load average was 21 on 11 cores. The timings show it.

The clearest sign is two benchmarks that measure the same thing. RenderTerminalUnfocused and the 120x40 unfocused case of RenderTerminalReal render the same size of pane down the same path. In one run benchstat called the first one a quarter faster after the follow-ups and could not tell the second one apart from the code before the fix. The same code cannot be both. CellLoopPaneNoDim, which isolates the cell loop the fix touched, showed no significant change in either run, and its dimmed twin showed none in the run that had it. And the code before the fix, which did not change between the runs, measured 28 to 75% slower in the second run than in the first on three of the four benchmarks both runs shared. Numbers like that describe the load, not the code.

So I am not putting a time on any of this. The one number the runs agree on is allocations: every benchmark allocated exactly as many times per render at all three commits, and within a run the median memory per render matched to within a byte. The fix and the follow-ups did not add a single allocation to the render path. For time I have the commit's own "few percent" for the cost and the list above for what was removed, which is work that was provably repeated for nothing. Whether that nets out ahead or behind needs a quiet machine, and I have not had one.

What I keep from this

Two paths that draw the same thing are two answers to one question, and the question only gets asked once per frame. There is no moment where both answers are on screen next to each other to be compared. The disagreement shows up across time instead, as a pane changing when focus moves, and a change on the pane you just left is the one change you are guaranteed not to watch.

So the test does the comparing I could not do by eye. It takes the expected cell from the emulator, not from the renderer, so no path can agree with itself and pass. It runs every path, not the one I happened to be reading. And the next time someone adds a sixth way to draw a pane, the table in that test is the first place it has to go.

The renderer's cell came from two sources that did not know about each other:

// cellSize derives the grid cell from the primary face: the advance of "M"
// wide, a terminal-ish 1.25 line height tall.

The width was measured, the advance of the letter M in the actual face. The height was fs.size * 1.25, a number I had typed because it looked like a line height. Half measured and half guessed, and the guess set the shape.

Round one: measure the other half

JetBrainsMono's own metrics are a 0.600 em advance in a 1.320 em line box, a ratio of 0.455. kitty on this machine draws that font in a 10 by 22 pixel cell, which is that ratio exactly. The half-guessed cell measured 0.486. Every capture came out about seven percent wider per cell than the screen it pictured, uniformly. That is exactly what "horizontally stretched" looks like to someone holding the picture next to their own terminal.

The fix was satisfying in the way that makes you stop checking. The height now came from the face's own line box, ascent minus descent plus line gap, the same box kitty and ghostty size their cells from. Both halves of the cell were measured from the same font file, with no invented constants anywhere. I shipped it as done.

Round two: the font was the wrong oracle

The next round of reports came from the preview panel, and this time I measured the output against a real kitty instead of against the font. I turned the frame off, so the PNG is exactly the grid, and held it against the host cell for cell. The host's cell was 9 by 20 pixels, a ratio of 0.450. The picture's cell was 5.76 by 14.4, a ratio of 0.400. Every column of the picture was eleven percent narrower than the column it pictured.

An interactive figure goes here. Open the page to use it.

Round one had been wrong because it guessed. Round two measured every number and was still wrong, because it measured the wrong thing. A terminal's cell is not the font's cell. The terminal picks its own: the width from its own rounding of the advance at its pixel size, the height from its own leading rules, both snapped to whole pixels. The face's advance and line box agree with that choice only by luck, and on this machine, at this size, they did not.

Both rounds have the same shape. The obviously correct move, twice, was to read the answer off the font, and the font was never the authority. The terminal is.

Ask the thing that decided

The host already knows its cell in pixels and reports it when asked, and tuios already asks: it is how the client places kitty graphics. So the capture path now carries the host's cell shape to the renderer, and the raster's cell is grown to match it.

Grown, never shrunk. A cell narrower than the advance runs neighbouring glyphs into each other, and a cell shorter than the line box clips them. A cell wider than the advance only means the glyph sits centred in it, which the renderer already did. So the aspect correction only ever grows one axis:

if want := ch * f.CellAspect; want > cw {
    return want, ch
}
return cw, cw / f.CellAspect

Measured again the same way, the capture and the host agree to four decimal places: 0.4545 against 0.4545.

What I keep from this

The tests that came out of this take their expected values from something other than the code under test: the PNG's own header, a cell size forced through the environment, the panel's footer rule as it was actually drawn. Taking the expected value from the same source as the code is the trap this feature fell into twice. Both wrong cells were internally consistent, derived from real data by defensible arithmetic, and would have passed any test built from the same font metrics that produced them.

The claim being made was never about the font. It was "the picture is the shape of the screen", and there is exactly one oracle for that claim: a running terminal, measured. Round one taught me not to guess. It took round two to teach me that measuring the wrong authority is guessing with more steps.

The first two theories are the comfortable ones: a timer left armed, or scrollback being redrawn. It is neither. It is a backlog. The daemon's emulator paces the guest, but nothing paces the client, and the client is the slower side because it also draws. So the queue between daemon and client grows for the length of the flood and is worked through afterwards, frame by frame, on screen.

An interactive figure goes here. Open the page to use it.

The same number twice

What made this worth a post is where the backlog comes from. Measured on the flood above: the client's output writer waited 1117 ms in total for the pane's read lock, and the pane went on painting for 1215 ms after the source process exited.

Those are the same number twice. Composing a frame holds the pane's read lock for the length of a compose, and every millisecond the renderer holds it is a millisecond the writer is not draining the queue. The backlog a client builds is very nearly the time its own renderer took from it. And every frame drawn out of that backlog is overwritten by bytes already queued behind it, so the client is falling behind in order to draw what it is falling behind on. The renderer is not a witness to the backlog. It is a cause of it.

An interactive figure goes here. Open the page to use it.

Pacing by the debt, not only by the cost

The render coalescer already paced a pane by what its frames cost the client. It now also paces by what the pane is behind, tracked in bytes queued for the emulator rather than in channel slots, which vary in size by two orders of magnitude and say nothing about how much work is queued.

if w.queuedBytes.Load() >= catchUpBacklog {
    return catchUpCoalesceInterval
}

The threshold is 4 MiB, roughly a tenth of a second of the client's own parsing, so an ordinary burst (a paste, a large directory listing) stays under it. A pane past it drops to a frame every 250 ms until it catches up: slow enough that the renderer stops taking the read lock out from under the pane's own writer, fast enough to stay visibly alive.

Nothing is discarded. The emulator still sees every byte in order, so the scrollback is exactly what it would have been. The alternative, throwing the queue away and resyncing from a daemon snapshot, would have put a silent hole in it, because the snapshot carries a bounded scrollback window and the queue does not.

The numbers

Same flood, three runs each, measuring how long the pane paints after the source process dies:

An interactive figure goes here. Open the page to use it.

The drain is faster as well as quieter: a 256 KiB batch went from 6.2 ms to 3.95 ms, with the writer's lock wait halved. What remains of the tail is the client emulator's own parse rate, which is the honest floor. The bytes exist and something has to parse them.

The pacing is described with the other render settings on the architecture page. The same morning, a profile of a DOOM-fire flood at nearly the same size found a fifth of each frame in a wrap that changed nothing. That is a separate cost from this one. And the shape is the one from a resize drag that trailed the mouse: a queue that grows while you work, and drains after you stop.

What I keep from this

The instinct with a backlog is to look at the producer: something wrote too much, too fast. Here the producer was innocent and dead, and the consumer was manufacturing its own lateness, spending drain time on frames whose only property was being already obsolete. The fix was not to work faster. It was to notice that once a pane is behind, the frames it is being asked for are already spent, and the cheapest thing a renderer can do with spent work is not do it.

A campaign of 590,000 executions found nothing. In the same stretch, a conformance round done by hand found eleven bugs. The fuzzer found none of them. Not a reduced form of one, not a hint. None.

Not a coverage problem

The first suspect was reach, so I measured it. The generated-input targets cover 55.3% of internal/vt. The unit suite covers 75.4%. That gap looks like the answer, and it is not. The structured generator beats raw random bytes by only 3.4 points, and it demonstrably produces the traffic: DECSED, DECSTBM and the rest, thousands of times each across a campaign.

The sequences were reaching the code. The code was running. Whatever the code did next, nobody looked.

What passing meant

The targets check four structural invariants: the screen has a non-negative size, the scroll region is inside it, the cursor is inside it, and no cell claims more columns than the row has left. Each one is a class of bug that has actually shipped here. A scroll region past the end of the screen was once a daemon-wide panic.

But all four are statements about the shape of the data structure, and an emulator that silently does the wrong thing produces a perfectly well-formed wrong answer. Move the cursor to the wrong row, drop an attribute, lose a column: the grid stays exactly as tidy as before. 590,000 executions were asking "is the screen well formed" of screens that were quietly wrong, and getting the true answer yes.

The fix is not more invariants of the same kind. Writing down what the screen should hold after DECSED with a hostile parameter means implementing DECSED a second time in the test, and the second implementation is no more trustworthy than the first.

Two questions that need no answer key

There are properties that need nobody to write down the right answer, because they relate the emulator to itself:

• a frame the emulator emits must redraw the screen the emulator holds
• the same bytes must draw the same screen however the read boundaries fall

The first matters because Render is what the app sends to the host terminal, and nothing else checks it is faithful: the grid tests read cells, and the frame goes out unread. The second matters because a PTY hands the parser whatever the kernel had ready, so the same program output arrives in different pieces on every run. If the screen depends on the pieces, a pane renders differently depending on machine load.

Neither property knows what any sequence means. Both only ask the emulator to be consistent with itself, so any violation is a bug by construction.

An interactive figure goes here. Open the page to use it.

I wrote both. Both fail, and both failures are real.

An interactive figure goes here. Open the page to use it.

The E that was on screen and not in the frame

The render round trip fails on an input of two sequences. DECALN fills the screen with E, and then one zero-width character with nothing to attach to is enough:

\x1b#8      DECALN, fill every cell with E
\u0301      a combining acute, with no base to combine with

Such a character (a combining mark at the start of a row, a bidi control) was stored as a cell of its own, taken from whatever was there. The row then held one more cell than it had columns. Render emitted the row without it, everything after shifted one column left, and the last column fell off the end. The frame was missing an E the emulator still believed was on screen: a character visible in the pane's own grid that never reaches the host.

The ghostty differential suite reached the same defect from the other side. The library discards a baseless mark rather than storing it, and that divergence was already pinned in its own test. Two oracles, one asking "do you agree with ghostty" and one asking "do you agree with yourself", pointed at the same decision: what to do with a mark that has no base.

Seed 28

Split equivalence fails at seed 28 of the generator, reduced to eight steps ending in a wide base and its combining mark arriving at the right margin with autowrap off. Hand reduction did not get it smaller, so the test records the seed rather than claiming a repro it does not have.

This is the shape of bug that gets reported as a pane that sometimes corrupts. Nothing about the guest's output changed between the good run and the bad one. Only how much of it the kernel had ready on each read.

Two ways the harness could have lied

Both are worth recording, because both would have produced confident nonsense in the other direction: failures that were the harness's own fault.

Render separates rows with a bare LF, the way a host with ONLCR expects. Replay a frame into a fresh emulator without adding the carriage return and every row starts one column further right than the last, the screen scrolls out from under itself, and every input "fails" the round trip. The replay has to feed back \r\n for the property to test the emulator rather than the replay.

And a shrinker whose oracle is "still fails" reduces a script holding two bugs to whichever one survives the cuts, then prints that reduction under a report about the other. The shrinker here requires a candidate to fail the same way, by comparing a signature of the failure with the varying detail stripped, so the reduction stays attached to the failure being reported.

Pinning a bug you have not fixed

Both root causes went in as tests asserting the current broken behaviour, so either one changing fires a test. The full metamorphic sweep sat behind an environment variable, because with the bugs open a default-on sweep is a permanently red test people learn to ignore.

Then the fix. A zero-width character now combines with the cell before the cursor. It is dropped when there is nothing there, or when the code point cannot extend that cell's cluster. The decision is made one rune at a time and is final for each rune. That is what ghostty and xterm do, and the per-rune finality is what makes the outcome independent of write boundaries. That closes the split side too. A cluster left open across a write now records the margins it was drawn under. A widened cluster that no longer fits wraps whole, exactly as it would have if it had arrived unsplit. A refused cluster is never left open to be extended later.

Both properties now hold across the first 10,000 generated seeds. The sweep and both fuzz targets lost their opt-in gates and run on every build.

What I keep from this

Structural invariants are cheap, worth having, and silent on the only question that matters. They catch the emulator being broken. They cannot catch it being wrong, because a wrong screen is still a well-formed one. 590,000 executions of a question the system cannot fail measure the question, not the system.

The properties that worked cost nothing the invariants did not. No expectations, no second implementation, no oracle to maintain. They ask the emulator to agree with itself, in the two places where disagreement is exactly what a user sees: the frame that goes to the host, and the read boundaries the kernel chooses. When a fuzzer finds nothing for that long, the interesting question is not where the bugs are. It is what the fuzzer is actually able to notice.

Later, September 2026

Split equivalence caught something it was never written for. In a performance pass I gave each screen row a bound on where its text ends, and then deleted one line that keeps that bound up to date, to see what would notice. The conformance corpus and the tmux differential passed. The metamorphic sweep reduced the failure to two steps, a run of x and a full reset, where one way of splitting the input left an x on screen. That is in Most of a scroll was spent on blank cells.

That would be fine if the call did anything. The content being wrapped is a terminal pane's body: the output of a terminal emulator whose whole job is to produce a grid of exactly the pane's width and height. Every line is already exactly as wide as the wrap width. The wrapper walks the whole frame, finds nothing to break, and returns its input. The second-hottest call in the client was, on this content, the identity function.

How it got there

Nobody calls Wrap in the tuios codebase. It comes free with the box. A pane is drawn by wrapping the emulator's content in a lipgloss border style, and setting Width on that style is what arms it: inside Style.Render, a width means the content might need wrapping to fit, so it is wrapped. Reasonable for the general case lipgloss serves. Redundant for content that a grid-based emulator has already shaped.

It is also expensive out of proportion to what it does, because Wrap makes two passes. The first pass is the actual word wrap, which finds nothing. The second pass streams the result through a writer whose job is repairing styles across the line breaks wrapping introduces, and it does this one byte at a time: for every byte of the frame, one ANSI parser advance and one one-byte heap-allocating write. On this content it changes nothing, which the tests below prove byte for byte. What it costs is the copy itself, byte by byte, allocation by allocation, twice per pane per frame at flood rates.

The waste has a sibling: before the box is rendered, the content is measured with lipgloss.Size and clamped with MaxWidth and MaxHeight, a full ANSI-aware walk over a string the renderer itself just produced to known dimensions. Measuring what you just made, then wrapping what cannot wrap.

Deleting work is a correctness claim

The fix is obvious, and it is not the point. Do not set Width when the content is already the right shape, and the wrap never runs. The point is the guard, because "already the right shape" is a claim about every frame tuios will ever compose, including the ones where a pane is mid-resize and the grid has not caught up, the ones where the terminal is closed, the ones where a frame came out of a cache. Skip the wrap on one frame where the claim is false and the box misassembles on screen.

The rule I held the change to: nothing on the fast path is allowed to count bytes or runes. Counting is how this kind of guard goes wrong, because byte count, rune count and column count are three different numbers the moment the content stops being ASCII. Instead, the renderer reports the rectangle it produced, from the inside. As the cell loop emits each row, it sums the printed cell widths. It publishes the dimensions only if every row landed exactly on the content width and the row count matched. Any early exit, any row that fell short or overshot, and any path the renderer cannot vouch for publishes zero instead, and zero means the old wrapping path runs. The cached-frame path stores the rectangle alongside the frame, so a cached answer can never be trusted further than the frame it describes.

An interactive figure goes here. Open the page to use it.

The claim then gets attacked from three directions.

First, a 13-case corpus of the content most likely to break width accounting: CJK text, combining marks, ZWJ emoji families, flag pairs from regional indicators, block elements, styled runs, and wide or combining characters placed deliberately at the last column, where a two-column glyph cannot fit and the accounting is most likely to be off by one. Over that corpus, one test asserts the reported rectangle fills the content box, and another asserts the literal proposition being relied on:

if wrapped := lipgloss.Wrap(body, win.ContentWidth(), ""); wrapped != body {
    t.Errorf("wrap must be an identity on a pre-shaped pane body")
}

If the wrap is the identity function, skipping it is unobservable. That test turns the whole optimisation into a checkable statement.

Second, a differential test renders every corpus case through both paths, skip on and skip off, and compares the finished boxes byte for byte. And third, two adversarial tests desynchronise the grid from the pane on purpose, resizing one out from under the other in both directions, and assert the renderer declines to publish a rectangle rather than publishing a wrong one.

Above the unit tests, an end-to-end test runs the real binary in a PTY, fills rows to exactly the terminal width with 你, é and ❤️ (a heart made two columns wide by a variation selector). It compares every screen cell, colors included, against the same binary with the skip turned off through an escape hatch that ships in the build. The negative control is recorded next to the test: inject the fault, report a rectangle one column wide of the truth, and both tests fail on the broken binary. A guard you have never seen fail is a guard you have not tested.

The numbers

Same binary, wrap on against wrap off, 158x40 under flood:

An interactive figure goes here. Open the page to use it.

The allocation numbers say it more plainly than the milliseconds: assembling the box alone went from 107,260 allocations to 660. That is the one-byte writes. A hundred thousand allocations per pane per frame, every one of them in service of returning the input unchanged.

The same morning I found why a flooded pane kept painting after its source died: the renderer holding the pane's lock, which is a separate post. The architecture page has the rest of how the render path is paced.

What I keep from this

The cheapest optimisation is deleting work, and deleted work is the only optimisation that needs a proof rather than a benchmark. A benchmark tells you the fast path is fast. It cannot tell you the fast path is right on the frame where the grid is three columns behind the pane. That is why the guard comes from what the renderer actually produced, not from a prediction of what it should produce. The identity test is the part I would keep even if the performance win evaporated: it converts "this call does nothing here" from a belief the optimisation rests on into a sentence the suite checks against real Unicode on every run. Beliefs rot. Sentences fail loudly.

The suite was green. The first ls I ran in a real build painted every filename on a block of hot pink.

The cache

The backend converts libghostty's cell styles into the style type the rest of tuios consumes. Conversion walks foreground, background, underline and every other attribute, so it is not free, and libghostty hands you a convenient handle to cache on: every distinct style in the terminal is interned and named by a 16-bit style ID. Same style, same ID. The conversion cache wrote itself:

// styleCache maps a libghostty style ID to its uv conversion. Reset
// when the theme changes, since conversion depends on the theme.
styleCache map[uint16]uv.Style

Invalidated when the theme changes, and when OSC 4 rewrites the palette, because the conversion depends on those. Otherwise it lived as long as the terminal did. Within one frame the hit rate is excellent, since a screen full of text is mostly a handful of styles repeated thousands of times.

The part I had not priced in: interning works by reference counting, and a refcount means reuse. The moment a style's last cell disappears, its ID goes back in the pool, and the next new style can be issued the same number. A style ID is not a name for a style. It is a name for a style while that style is on screen, which is a much shorter lease than the cache had assumed.

The sequence

My prompt draws a powerline segment: truecolor background \x1b[48;2;255;105;180m, hot pink, white foreground. So the sequence that produces a pink ls is nothing exotic:

1. The prompt renders. A frame is composed, the cache converts the prompt's styles and files them under their IDs. One of those conversions carries the pink background.
2. clear. The prompt's cells are gone, the refcounts hit zero, the IDs return to the pool.
3. ls prints filenames with foreground-only colors. New styles, interned, and the allocator hands them the freshly freed IDs.
4. The next frame syncs, asks the cache about those IDs, and gets the answer that was true one generation ago: white on hot pink.

The rendered frame in step 1 is load-bearing. Without a compose between the prompt and the clear, nothing caches the doomed conversions and nothing goes wrong. The bug only exists when reads interleave writes, which is exactly how the application behaves and exactly how a test suite does not.

An interactive figure goes here. Open the page to use it.

Why the suite could not see it

Three reasons, and they compound.

The corpus test fed each capture to both emulators in 4096-byte chunks, to exercise chunk-boundary handling, and then compared the screens once, after the last chunk. One comparison means one read, which means one snapshot, one generation of style IDs, and nothing ever recycled. The very structure that makes the bug possible, a read between two writes, was the structure the test avoided.

The hand-written sequence tests each made a single write, so they had no intermediate state to get wrong even in principle.

And every comparison was of internal grid state: walk both grids, compare cells, with an equivalence that knows a color can be spelled more than one way. That is the right comparison for most divergences and it is entirely blind to this one, because the bug does not live in the grid. libghostty's grid was correct the whole time. The corruption happened in the conversion layer on the way out, which means it only exists in what the host terminal is shown, and nothing compared that.

A differential harness is a claim with two parameters: what you compare and when. I had the objects right and both parameters wrong.

The fix is two lines. The tests are the work

The fix: clear the cache at every render snapshot, in both places that take one.

// Style IDs are only stable within one render snapshot: the library
// interns styles and recycles an ID as soon as its last cell is gone.
clear(t.styleCache)

Cleared, not reallocated, so the map's capacity survives. The cache keeps earning its keep within a frame, which is where the hit rate was anyway, and stops asserting anything across frames, which is where it was lying.

Then I closed the holes in the suite. The corpus test now compares after every chunk, not once at the end, because a sync per chunk is what the running application actually does. A new comparison renders both emulators' output, the actual byte stream a host would receive, and when the bytes differ it re-parses each stream through a fresh emulator and compares what displays, so two spellings of the same color stay equal but a wrong color has nowhere to hide. And a churn test runs twelve generations of clear-and-recolor, the minimal loop that forces ID recycling, comparing both representations after every generation. On the pre-fix code it fails immediately.

There is also an end-to-end guard that works at the level the bug was found: paint a pink prompt segment, wait for a rendered frame, clear, print foreground-only filenames, and assert on the final screen that every filename cell has a default background. The waiting step is commented for what it is: the trigger, the frame that caches conversions the clear is about to free.

The differential suite now runs in CI on every change that touches the emulator, in both backends. It runs locally with the ghostty backend command in the contributing guide, and the architecture page covers the two emulators.

The same shape, two panes over

The same week, the kitty graphics passthrough had a bug with a different surface and the identical skeleton. Dragging a pane narrower while a program inside it streamed frames made the image stretch and spill 40 columns into the neighbouring pane. The host had been told the placement was 118 columns wide, once, and kept scaling 780-pixel frames into that rectangle while the pane underneath had long since shrunk to 78 columns.

The re-placement was held back during a resize by a flag that is set on mouse press and cleared on mouse release. Release is precisely the event that goes missing when the pointer leaves the surface mid-drag, so the hold could outlive the gesture indefinitely. The fix keys the hold on the geometry still changing, pass over pass, instead of on how the gesture ends: when the size stops moving, the hold releases, whatever the flag says.

Two caches of a fact, both keyed to an event that was allowed to never arrive. The style cache waited for a theme change that had nothing to do with style lifetime. The placement hold waited for a mouse release that never came. The fix in both cases was the same move: stop trusting the event, derive the lease from the data itself. A converted style is good for one snapshot. A placement rectangle is good while the geometry it describes still holds.

What I keep from this

Caching something keyed by an identifier you do not own means inheriting that identifier's lifetime rules, whether or not you have read them. Nothing in the code was wrong about what a style ID meant at any instant. It was wrong about how long the answer stayed true.

And a differential suite passing is a statement about its comparison points, not about the system. Mine compared internal state at the end of input, and the bug lived in output in the middle. Neither "compare more things" nor "compare more often" would have found it alone. It took both, plus the humility to recheck the suite's blind spots against how the application actually interleaves reads and writes. The suite is better now because the screen was pink. I would rather have learned it the other way around, but I have yet to find where correct-looking screens teach you anything.

Four findings, in order of how much they changed my mind about what a passing test means.

The command that lost its spaces

send-keys takes a token string in the tmux style: key names and text, comma-separated. So the obvious way to run a command is:

tuios send-keys "echo hello,Enter"

The pane receives echohello, followed by Enter. The tokenizer aliases commas to spaces and then splits on whitespace, so echo, hello and Enter are three tokens, each sent as its content, with the separators dropped. That is correct for a key-token language, where the separator is punctuation rather than payload. It is also a trap for exactly the caller this surface exists to serve: a machine that composed a shell command and reached for the verb with "keys" in the name. Nothing warns you. The call exits zero, and the pane runs a command that does not exist.

An interactive figure goes here. Open the page to use it.

This one is not fixed in code, because the tokenizer behaves as specified. The fix is in the surface. The skill documentation now shows the failure inline, annotated with exactly what gets typed, and steers text, including whole shell commands, toward send-text, which sends its argument verbatim. The examples in that documentation are held by a test that parses every one of them against the real command tree, so the docs cannot drift from the binary.

The parameter that fell on the floor

new-window takes a session and a name. I asked it for a workspace too:

{"verb": "new-window", "params": {"session": "dev", "workspace": 2}}

A window came back: created, named, success envelope, everything in order. On workspace 1. Dropping an unknown field is what encoding/json does by default, and it is the worst answer available to a machine caller. The call did less than it was asked, reported success, and left no trace of the difference. A human notices the window opened in the wrong place. An agent reads "type": "window_created" and moves on, wrong about the world from then on.

The fix has two halves. workspace became real, along with cwd and focus. And the protocol layer now checks every incoming parameter against the verb's published schema before the handler runs. Anything unrecognised is refused, with the closest match and the full list of accepted parameters. The refusal is useful beyond typos. A parameter the verb does not take yet is exactly what a caller built against a newer tuios sends to an older one, and a refusal is the only honest answer in that situation. Turning the check on also caught two parameters that were real but missing from their verbs' declared schemas, which would have become unreachable the moment enforcement landed. The declared surface and the implemented surface had already drifted apart. Nothing was comparing them.

This is the same kind of request against the daemon today, with the workspace parameter misspelt:

An interactive figure goes here. Open the page to use it.

The full list of error codes is in the protocol reference.

The option that did not exist

{"verb": "set-option", "params": {"key": "appearance.totally_made_up", "value": "whatever"}}

This came back option_set. Not an error. The response did include "applied": false, which sounds like a signal until you learn it was one bit meaning two things: "no client is attached to apply this right now" and "that key means nothing and never will". A caller cannot tell a setting that will take effect on next attach from a typo. Both were reported as success.

Now the path is resolved against the option registry first, and a miss is an error, option_not_found, carrying the closest match and the complete list of valid paths. The value is validated too, by trying the assignment on a throwaway default config before touching the session, so the check and the apply cannot disagree about what is assignable. And when applied is false, the response says why. Both requests from this section, sent to a headless session today:

An interactive figure goes here. Open the page to use it.

Six paths out of 88

The last one was the quietest and the largest. Applying a setting to a running tuios went through a hand-written switch, and the switch knew six paths, all of them appearance options: border style, dock position, animations, three flavours of window button. The option registry at that commit declared 88 (the full list today is one tuios list-options away). Everything else, most of the sidebar's settings and all but one of the dock's, could be written into the config file, read at startup, listed by the CLI, accepted by set-option, and would do nothing whatsoever to the running program. Not rejected. Accepted, recorded, inert.

So the whole sidebar could be configured over the socket, verb by verb, success by success, without a single visible consequence. The fix deleted the dead end rather than extending the switch. Every path now goes through the same route a config file load uses: one assignment function driven by the registry, then the same live-apply step. The registry itself is held to the config struct by a reflection test that walks every scalar field and fails in both directions: a field without a registry entry, or an entry without a field. The number 88 can never again be quietly larger than the number six, because there is no six.

Why every test was green

The tests were not thin. They were pointed at the wrong layer.

The end-to-end suite really did drive set-option over the socket, and here is the assertion it made: set mouse to on, get mouse back, expect on. mouse is not a tuios option. The test proved the daemon's key-value store round-trips a string, which it does beautifully, and proved nothing about any option existing or taking effect. It now sets a real registry path and asserts the value came back from the session, not from defaults.

The unit tests called verb handlers as functions, with a fake client wired to answer every command with a hardcoded success. A handler tested like that cannot fail for any of the four reasons above. It cannot even fail for not being registered. The commit that replaced them says it plainly: a handler test would pass for a verb that was never registered, and registration is half of what these verbs are.

The replacement suite starts a real daemon, connects over the socket, sends the verb as bytes, and asserts on the response and then on the state the daemon holds afterwards. It is slower, and it is the only kind of test on this surface whose passing means what it appears to mean.

What I keep from this

An API for machines fails differently from a UI for people. A person who asks for a window on workspace 2 and watches it open on workspace 1 has already noticed: the interface reports itself through their eyes. A program has only the response envelope, so every gap between what was reported and what was done becomes a false belief in the caller, compounding silently. For this kind of surface, "accepted and ignored" is strictly worse than any error. The couple of hours I spent driving my own API the way its real callers would found more product defects than the handler-level suite had found in its whole lifetime. The features were not broken. They were never wired to anything, and no green is as untrustworthy as the green of a test that cannot reach the wire.

Note, September 2026: an earlier version of this post showed the option example as tuios set-option .... There was no such CLI command: set-option is the socket verb, and the CLI command at the time was tuios set-config. It also said the skill steered whole commands toward run-command, which runs tuios commands such as ToggleZoom, not shell commands. Both are corrected above.

I closed that report three times. Each closure rested on a different wrong conclusion, and each one failed for a different reason. The bug itself turned out to be one of the least interesting things in the investigation, so this post is mostly about the three failures.

Closure one: it must be the image protocol

I had recently spent a long stretch inside the kitty graphics protocol, where corruption meant image data going wrong somewhere in a pipeline. Sprites, corruption, browser terminal: I pattern-matched straight to image transmission and started investigating there.

pokemon-colorscripts does not transmit images. It prints text. Half blocks with colour codes are the oldest trick for pixel art in a terminal, and one look at the wire would have said so. I did not take that look before forming the theory. That is the whole of the first failure: I assumed what kind of data it was without checking what was on the wire.

Closure two: the sprite really is like that

The second time, I did look at the data, and I did it properly. I rendered ground truth from the escape codes with independent code, no terminal and no browser in the path, and compared it against a screenshot of the browser terminal. It matched. I closed the report as not-a-bug: the artwork is speckled.

The artwork genuinely is speckled. Blastoise carries rgb(255,255,255) 71 times and rgb(189,189,197) 34 times among its 794 colour runs, so there are legitimate near-white dots in the sprite. The reasoning was superficially sound: predict the appearance, compare, match, close.

The failure was in what "match" meant. I asked "does the speckle pattern I predicted appear on screen", and it did. I never put a bound on the difference. The renderer was adding its own speckling on top of the artwork's, one colour step away from correct, and a visual comparison between two speckled images hides that forever. I had confirmed a prediction rather than measured an error. Of the three closures this one was the most defensible, which made it the most dangerous. I walked away sure.

Closure three: it is a browser engine difference

The report survived, so the third round compared browsers. Measured in the page with getImageData over the rendered sprite: Firefox reported 3 distinct colours, Helium reported 44.

Conclusion: an engine rendering difference. Firefox tight, Chromium-derived Helium loose, nothing for me to fix. Closed again.

This is the closure I find most uncomfortable, because it applied the lesson of the previous one. I did not assume. I measured, with a real API, over real pixels, and the numbers were unambiguous. They were also both wrong.

Measuring from outside the box

The measurement that finally held removed the browser's own APIs from the path entirely: capture through the Wayland compositor, so the pixels examined are the pixels the browser handed to the display server, read back by nothing the browser controls.

stock Chromium   95.6% of pixels exactly the specified colour, 12 variants
Helium           41.2% exact, 45 variants
Firefox          identical to stock Chromium, 0 differing pixels of 389,120

Firefox does not render this differently from Chromium. Not approximately the same: zero differing pixels out of 389,120. The 3 distinct colours it reported through getImageData were an artefact of the readback, not a property of the rendering, and I never did establish what produced that number, because by this point I had stopped trusting in-browser readback for anything.

Helium is genuinely different, and now the defect had a precise shape. It renders (123,190,255) more often than the specified (123,189,255): an off-by-one in one channel, spread across the sprite as a roughly 50/50 dither. Only textured WebGL content is affected. A flat WebGL clear comes back 100 percent exact, so the cause is not the swap chain or the page compositor. It is the texture path. Switching the terminal's renderer moves the number:

An interactive figure goes here. Open the page to use it.

The suite that cleared it was blind by construction

Through all of this, a 292-test rendering suite kept passing, and it deserved scrutiny it did not get until the end.

The suite compares rendered pixels against expected colours with a tolerance of 2. The defect is off-by-one and off-by-two. A correct expectation compared under a tolerance wider than the fault cannot see the fault. Not "is unlikely to". Cannot, because every affected pixel sits inside the acceptance band, so no number of tests, runs or scenarios changes the outcome.

The suite also insets its scan region, 20 percent horizontally and 12 percent vertically, to keep cell borders out of the comparison. The dither concentrates at cell edges. The sampling was cropping out the region where the defect is strongest.

Neither decision is foolish on its own. Tolerance absorbs harmless rasterisation differences across platforms and drivers, and the inset avoids asserting on the fuzziest pixels. Together they define exactly the class of defect the suite cannot detect: small in amplitude, concentrated at edges. This bug sat in the middle of that class. That is survivorship, not bad luck: a defect the suite could see would have died long before I looked at it with my own eyes.

An interactive figure goes here. Open the page to use it.

The instrument was corrupting the sample

One question was left over from closure three: why had getImageData painted such a wrong picture of Helium as well?

Because Helium perturbs getImageData with stochastic anti-fingerprinting noise. The demonstration is as small as it gets: fill a canvas with one flat colour and read it back. In one session, 74.8 percent of the pixels came back exact. In another session, the same fill read back 100 percent. Same page, same colour, different answer.

So every in-browser measurement of Helium had been the sum of two signals, the renderer's real dither and the API's deliberate noise, in proportions that changed between sessions. I had been measuring a browser with that browser's own API, and this particular browser treats canvas readback as a fingerprinting surface and salts it on purpose.

I tried to remove the salt. Helium ships two named fingerprinting defences, FingerprintingCanvasMeasureTextNoise and FingerprintingClientRectsNoise. Disabling both changes nothing about pixel readback, and the noise source that actually affected these measurements has no discoverable flag at all.

Where it ended

Not with a fix. The terminal's output is correct: what it asks the browser to draw is what stock Chromium and Firefox draw. The speckling comes from Helium's WebGL texture path, no flag turns it off, and the workaround for anyone it bothers is the DOM renderer, at 95.1 percent even inside Helium. I changed no source anywhere, and none needed changing. The right answer to the report was a paragraph of explanation and a renderer toggle, both available from day one.

Three failures, three shapes

They do not reduce to one moral, which is why I am writing them out separately.

The first conclusion failed because I never looked. I assumed the category from recent experience and never checked the wire. It was the cheapest failure and the least excusable, and the easiest kind to catch, because the evidence was one command away the whole time.

The second failed because I looked and saw my prediction. The ground-truth render was real work and a real comparison, and it matched. Matching a prediction is not the same as bounding an error. My visual comparison had no stated tolerance, which means it had an enormous unstated one, and the 292-test suite had the same disease in mechanised form. Between my eyes and its tolerance of 2, an off-by-one defect fitted under both bars.

The third failed because the instrument was corrupt. This is the one I keep returning to, because it was the round where I did everything the earlier rounds had taught. I measured, compared browsers, held the scene constant. The numbers were precise, repeatable within a session, and partly manufactured by the thing being measured. No amount of discipline inside the browser would have caught it, because the corruption lives in the only readback path the browser offers.

What broke the sequence was not more care of the same kind. It was moving the measurement outside the system under test. The compositor does not care what the browser thinks it drew. It reports what arrived. Once I measured from there, every one of my earlier conclusions fell within an afternoon, including the two I had defended in writing.

Each narrowing looked like one bug slowly coming into focus. It was the opposite. That one gesture had three distinct defects, and each rewording was a different one of them showing through. I spent a while looking for the single cause of a compound symptom, which is a reliable way to fix a third of a bug and be told, correctly, that nothing has changed.

I had also just finished a round of performance work on this exact code path, with benchmarks around it, all green. After the first report I ran them again. Green. I extended them and ran them a third time. Still green. Three rounds of benchmarks vouched for a path that the one person using it kept saying was broken.

An instrument for the actual question

The benchmarks answered "how long does a motion event take". The report was about which windows move. Nothing I had measured that, and re-running a timer was never going to say anything about geometry.

So the fourth round was not a benchmark. I put a trace into the binary itself: run with TUIOS_RESIZE_TRACE=1 and it logs, per input event, the pointer position, which window is grabbed and by which corner, and every window whose geometry changed as a result. No timing, no statistics. A record of who moved.

One real drag settled what three rounds of benchmarks could not. This is the summary of the first captured log, a single grab moving a single divider:

An interactive figure goes here. Open the page to use it.

Everything else in this post fell out of that one log.

Defect one: neighbours found by geometry

adjustTilingNeighborsGeneric decided which windows a drag affects by scanning every window in the workspace for an edge within one cell of the dragged line. Not by asking the layout tree which panes share the divider. By coordinates.

The layout is a binary space partitioning tree, and two dividers in entirely different subtrees land on the same screen line whenever their ratios happen to agree. Fresh splits all start at 0.5, so they agree by default. Drag a divider, and any unrelated divider that happens to be collinear with it gets picked up as a neighbour, along with every window attached to it.

This is also why the bug was intermittent, and why the first report said "sometimes". It fired only when two dividers were collinear, and whether they were depended on the entire resize history of the workspace. It is easier to cause than to describe:

An interactive figure goes here. Open the page to use it.

Here it is on a 2x2 grid, dragging the divider inside the left column:

An interactive figure goes here. Open the page to use it.

The whole right column re-laid itself out in response to a drag it had no part in.

The fix reverses the direction of authority. Resize is now tree-driven: walk up from the dragged leaf to the nearest ancestor that splits on the drag axis, move that node's ratio, and rebuild the geometry from the tree. Only windows under that ancestor can move, because only their geometry depends on its ratio. Coordinates stop being evidence of adjacency.

On the 2x2 grid above, the tree makes the answer plain. The drag is vertical, so the walk stops at the first horizontal split above the dragged pane, and that node owns only the left column:

An interactive figure goes here. Open the page to use it.

Its ratio moves, and the right column's ratio is never touched, so when the tree is laid out again win-3 and win-4 come back exactly where they were. Whether the two dividers happen to share a row no longer enters into it.

Defect two: easing toward a target that already moved

The previous post already told how I found the second defect: with shared borders on, SyncBSPTreeFromGeometry calls ApplyBSPLayout on every composed frame during a drag. What belongs here is what that actually did, counted rather than inferred, at nine windows:

52 snap animations created over 20 drag frames    (0 with plain borders)
205 PTY resizes over 30 frames                    (0 with plain borders)

Each animation cancelled the one before it and started a fresh 300 ms ease toward a target that had already moved by the next frame. The panes lived permanently in the opening milliseconds of an ease and never arrived anywhere. Each of the 205 PTY resizes was a daemon round trip to resize a terminal whose size was about to change again.

The per-frame cost predated the performance work. What the performance work changed was frequency: raising the drag tick from a fixed 30 FPS to NormalFPS made the same per-frame work happen two to eight times more often. The optimisation amplified an existing defect, which is a large part of why the report arrived when it did.

The benchmarks could not see any of this, for three independent reasons. Building an animation object is cheap, and the defect is that panes never arrive, which is felt rather than timed. DaemonResizeFunc is nil under test, so 205 round trips in production cost 205 nil checks in the benchmark. And the benchmark drove motion into idle terminals, where coalescing made almost every frame a cache hit, so ApplyBSPLayout ran once in twenty motion events under measurement and once per frame in real use.

Defect three: the ratio that never comes back

The final form of the report, "the bottom split doesnt maintain its ratio", was a third defect, and the subtlest. Resizing a pane vertically walked the ratio of an untouched split below it.

The mechanism is one line of arithmetic in each direction. applyLayoutRecursive converts a ratio to a divider position with int(ratio*extent), which truncates. SyncRatiosFromGeometry reads the truncated divider back as line/extent. One pass through that pair takes 0.500 to 0.482759, which is exactly 14/29. The fractional row that truncation discarded is now gone from the ratio itself, and nothing ever puts it back.

After that first pass the value is a fixed point: 14/29 of 29 rows truncates to divider 14, which reads back as 14/29. That is why TestSyncRoundTripIsStable never saw the defect. Syncing a pristine layout at a fixed size round-trips perfectly, because the loss happens on the first pass and the test can only observe the passes after it.

A loss of 1.7 points sounds too small to matter. It compounds at the next resize instead: grow the pair from 28 rows to 36 and a stored ratio of 0.482759 hands out 17 rows against 19, where 0.5 would have given 18 against 18. The trace showed the lifetime of one such ratio across a single drag: it starts at 0.500, never returns to it, ranges between 0.455 and 0.500, and settles at 0.480. Across the whole log the same pair ranged from 0.161, five rows against 26, to 0.654, seventeen against 9. This was a split the user had set once and never touched again.

An interactive figure goes here. Open the page to use it.

The theory I got wrong

Partway through, watching the trace, I saw one pane hold at exactly 13 rows across six consecutive growth events while its sibling took every new row. I concluded that growth and shrinkage took different code paths, one respecting the ratio and one ignoring it, and went looking for the second path.

There is no second path. Growth and shrinkage run the same code. The pane held at 13 because the truncation loss had already drifted its ratio to a value whose product with each new extent kept truncating to 13 rows. The observation was real and careful. The explanation I reached for was structural, and the truth was arithmetic. The same trace that produced the wrong theory also killed it, once I followed the ratio value instead of the row counts.

The defect that was not one

One item from the trace still looked wrong after all three fixes: 9838fc53, changing width 631 times with 22 direction reversals. It looked like jitter, some feedback loop hunting between two states, and I had it pencilled in as defect four.

Then I measured it properly, with a drag swept out and then back, before and after the other fixes: 48 height changes each time, zero hysteresis each time. Every pointer position produced exactly the same pane heights on the way out and on the way back. That is not instability. That is a pane correctly tracking a divider one cell at a time as the drag passes over it, and the direction reversals were reversals of my hand.

I had flagged it as a bug because it looked like other bugs, without checking whether the return path differed from the outbound one. It did not, in any run. Zero hysteresis means the system has no memory and is simply following the pointer, which is the specification. Anything I had "fixed" there would have introduced the very lag the rest of this work removed.

What the instrument cost

The trace was an evening: an environment variable, a log line per geometry change, and a script to summarise the output. It found three defects and acquitted a fourth candidate in one drag, after three rounds of benchmarks had found nothing, because it recorded the thing the report was about.

A benchmark compresses behaviour into a duration. None of these defects lived in a duration. Windows moved that should not have, targets were never reached, a ratio drifted one way and stayed there. All of it was invisible to a timer and plain in a log of who moved. The narrowing reports were never one bug coming into focus. They were four phenomena taking turns at the front, and I only saw that once I stopped assuming they were the same thing and looked at the evidence.

Note, September 2026: the trace never landed. TUIOS_RESIZE_TRACE is not in the tuios repository or its history, so setting it on a released binary does nothing.

That last detail is the whole diagnosis, though I nearly skipped past it. A uniformly slow frame feels the same at the start of a gesture as at the end. Something that degrades as you go is a queue you are not draining fast enough.

Backlog, not slow frame

Every mouse motion event composed a full frame. A frame during a tiling resize costs 3.3 to 6.7 ms, which caps the drain rate somewhere around 150 to 300 events per second. A drag emits roughly one event per cell crossed, and a brisk drag across a terminal easily outpaces that.

So the queue grows for the duration of the drag, and what you see is wherever the pointer was some number of events ago. Stop moving and it catches up. That is why it read as lag rather than as slowness. (A flooded pane fell behind the same way later, with PTY output in place of mouse events.)

The fix bounds redraws to one per frame interval. Every event's geometry is still applied before the draw decision, so no input is dropped and the layout still settles exactly where you released the button. Only the redundant intermediate frames go.

An interactive figure goes here. Open the page to use it.

The difference is easier to feel than to read:

An interactive figure goes here. Open the page to use it.

The tooling, and the flags that mattered

Go's benchmark support did most of the work, and the useful flags are not the obvious ones.

go test -run XXX -bench BenchmarkResizeMotion -benchmem -count=5 ./internal/input/

-run XXX matches no test, so nothing but the benchmark runs. -benchmem gives allocations per operation, which mattered more than nanoseconds in two places below. -count=5 runs the whole thing five times, and the spread across those runs is the only thing that tells you whether a 15% improvement is real. Absolute timings on this machine drift with thermal state, so anything I compared had to be interleaved rather than measured an hour apart.

TUIOS wires up net/http/pprof behind a flag:

if pprofAddr != "" {
    runtime.SetBlockProfileRate(10000) // one sample per ~10us blocked
    runtime.SetMutexProfileFraction(100)
    go func() { _ = http.ListenAndServe(pprofAddr, nil) }()
}

Block and mutex profiling are off by default in Go and cost real overhead when on, which is why they sit behind the same flag instead of being always enabled. They are sampled too, because recording every event made a --pprof run feel much slower than a normal one. For a program that spends its life waiting on PTY reads and holding locks, those are the two profiles that explain anything.

Corrected 22 September 2026

An earlier version of this post showed SetBlockProfileRate(1) and SetMutexProfileFraction(1). The code used those values for 25 minutes on the morning of 4 July, before this post was written. The snippet now shows the sampled rates it actually used.

Four hypotheses. Two wrong.

I wrote them down before measuring, mostly so that being wrong would stay visible afterwards instead of quietly evaporating.

One: the whole-tree ratio sync is expensive. It recomputes every split ratio in the BSP tree on every motion event, which sounds bad. Measured at 6 to 86 us against a multi-millisecond frame. Roughly 1%. I would have spent a day there.

Two: damage tracking is doing nothing. Mine, and wrong in a way that still stings. I read a benchmark where the one-dirty case measured slightly slower than all-dirty and concluded the tracking was dead weight. At one window, one-dirty and all-dirty are the same case. Identical timings prove nothing. At four and nine windows it does substantial work: 929 against 1343 us, and 737 against 1221 us. I had compared a thing to itself, written it down as settled, and built the next hour of work on top of it.

Three: the tick throttle is not helping. True. SlowTickCmd governed the periodic tick during a drag while motion events drove their own renders anyway, so it lowered the ceiling without touching the flood. Removed.

Four: PTY resizes are firing per event. Not happening, and worth having checked, because a TIOCSWINSZ per motion event would produce exactly these symptoms. Confirming it was already deferred to drag completion cost one grep.

Two of four wrong is about my usual rate. That is the argument for writing them down.

Fixing the top cost promotes the next one

With renders coalesced, the handler still ran on every event, so it became the new floor. That exposed a gap the old cost had been hiding.

An interactive figure goes here. Open the page to use it.

Allocations said it more bluntly: 10,665 B in 8 allocations against 57,238 B in 34, per event, at nine windows.

The culprit was SyncBSPTreeFromGeometry running on every motion event, rebuilding a geometry map over every window and re-deriving every ratio in the tree. Same medicine as the renders: defer the sync to the frame that actually draws. The drag-completion sync stays unconditional, because a stale tree means the next retile silently discards the user's resize. After the change, 7,215 and 7,278 ns at four and nine windows, flat again, back to 8 allocations.

A performance conclusion has a shelf life. Hypothesis one, correctly measured at 1% of a frame, was the bottleneck an hour later, because I had removed everything that used to dwarf it.

The performance fix caused a rendering bug

Deferring the ratio sync meant the separator overlay could draw from tree state that lagged the real window geometry. The overlay takes divider positions from the tree and the highlight from live geometry, so mid-drag it drew the divider where the drag had already left, in the unfocused colour, because that column was no longer on the focused pane's perimeter.

It looked like a red afterimage trailing the cyan separator.

My first fix put the flush on the paths that change geometry, which is the obvious place and the wrong one. Any frame composed for another reason bypassed it, and PTYDataMsg composes constantly during a real drag, because the terminals in the other panes are still producing output. That is why it showed up all the time in use but needed interleaved PTY output to reproduce in a test.

The flush belongs in View, immediately before composing. Geometry can be applied whenever, but the ratios have to agree with it on any frame that reaches the screen.

Eighteen of twenty-four mid-drag frames wrong before, none after. Removing the flush with the test in place fails twenty-four of twenty-four.

Where micro-optimisation could not help

Two bugs where tuning the hot path would have been wasted effort.

A quadratic behind a passing benchmark

Typing in the browser client was unusable. getLine walked the whole viewport for every row, making a frame O(rows squared times cols).

No amount of constant-factor work closes a quadratic gap. It took three changes, each of which promoted the next bottleneck:

Full repaint stayed at 11.5 to 11.2 ms through that last one, which is correct. That path has no dirty rows to skip. A change that improved it too would have meant I was measuring something else by accident.

An aside from the same stretch of work, because it wasted three runs before I saw it. The browser test harness reused an already-running server when outside CI, and the client assets are compiled into the Go binary. So a server left over from an earlier build kept serving the previous client, and editing the client then rerunning the tests exercised the old build, with nothing logged. Three confident results about code that was never loaded. The harness now rebuilds per run, and I trust "the test passed" a little less than I used to.

A freeze no profiler would find

The multiplexer locked up within seconds of any command producing output.

The render path took a window's I/O read lock, then called a function that took the same lock again. Go's RWMutex is not reentrant for readers. If a writer queues between the two acquisitions, the second read blocks behind the writer, the writer blocks behind the first read, and everything stops.

An interactive figure goes here. Open the page to use it.

There is no hot path to profile here. The program is not slow, it is stopped. What finds it is SIGQUIT with GOTRACEBACK=all, reading the goroutine dump, and spotting two frames of the same stack holding and wanting the same lock. The fix was hoisting the cursor query above the lock.

There was also an optimisation I measured and threw away entirely: porting correct grapheme width tables from ghostty-vt, which changed nothing because cluster advance was never a per-codepoint question in the first place. That one got its own post.

What a benchmark cannot tell you

Later in the same work, shared-border drags still felt wrong. The benchmark disagreed, emphatically. At nine windows a frame had gone from 8,981,357 ns to 326,214 ns across this work, which is 27 times faster, and the shared-border path had improved more than the plain one.

Then I used it, and it felt worse. Not marginally. Worse than before I started.

Three reasons the measurement could not see it.

Building an animation object is cheap. The cost was that panes never arrived at the pointer, because each frame cancelled the previous animation and started a fresh 300 ms ease toward a target that had already moved. Not arriving is felt, never timed.

The daemon resize callback is nil under test. So 205 socket round trips per 30 frames cost a nil check in the benchmark and a real round trip in production.

And the benchmark drove motion into idle terminals, where coalescing made almost every frame a cache hit. The layout reapply ran once in twenty motion events under test. In use, with output flowing, it ran on every frame.

What found it was instrumenting the running program: log the pointer position, the grabbed divider, and every window whose geometry changed, then do the drag and read the log. One drag showed five windows moving where two should have, one of them reversing direction twenty-two times. The benchmarks had been accurately measuring the wrong thing for three rounds. That log turned up three separate defects and one suspect that turned out to be correct behaviour, and they got their own post.

When the numbers and the experience disagree, the experience wins. The benchmark is not wrong. It is answering the question you encoded, precisely, and it has no opinion on whether that was the right question.

So I wrote tuitest. It spawns a program on a PTY, drives it, keeps a terminal emulator's worth of screen state on the other side, and lets you assert on what the program drew. TUIOS's own end-to-end suite runs on it. Once that worked, writing the input by hand started to feel silly.

Fuzzing a TUI

A TUI has a wide input surface and nothing much protecting it. Keystrokes, mouse reports, resizes, bracketed pastes, text whose width the program has to get right. The bugs I ship in this area are almost never logic errors in a handler. They are combinations. A resize during a drag. A paste with a control character in it. A grapheme cluster that one layer measures as one column and another measures as two.

tuitest fuzz -- ./myapp
tuitest fuzz -seed 42 -iterations 200 -corpus testdata/fuzz -- ./myapp

Everything the generator emits is expressible as a tape command. The text corpus is deliberately nasty: ASCII mixed with accented Latin, CJK, ZWJ emoji, regional indicators, combining marks, zero-width and bidi overrides. Width handling is where layout code usually goes wrong.

Ctrl+c is in the key set even though it usually quits the program. That is on purpose. The check for "did this program restore the terminal on the way out" can only run against a program that has exited, and leaving someone's terminal in raw mode with the cursor hidden is the single most common TUI bug there is. A generator that never quits anything can never find it. Ctrl+z is excluded, because suspending a child under a PTY just wedges the run.

When a run fails, the fuzzer shrinks the input and writes a tape. Candidates replay through the same tape player tuitest run uses, so the file you get is not a description of what the fuzzer did. It is the thing itself, and it runs with no fuzz-specific machinery involved.

Then the fuzz suite started failing

Intermittently. Never the same test twice, and only when the whole package ran under -race.

Both failure modes had the same shape. Something fails once, then refuses to reproduce:

fuzz_test.go:314: the corpus entry should still reproduce on replay, got: no failures
fuzz_test.go:188: the minimised reproduction did not reproduce on confirmation

The obvious reading is that the fuzzer's bookkeeping is broken. A failure recorded against the wrong input, or shrinking losing the case somewhere. I spent most of an evening inside the shrinker on that theory.

Before going further I wanted a number, because "sometimes" is not a bug report. Twelve sequential runs, then twenty-four across four parallel lanes:

An interactive figure goes here. Open the page to use it.

No DATA RACE ever fired. Not once, in any of those runs. -race mattered here because it slows the process down and reshuffles scheduling, not because it detected anything. I now reach for it to perturb timing about as often as I reach for it to find races.

Reading the tape I had been ignoring

The shrinker had been reducing failures to two commands: spawn the program, send a burst of ^C. I had written that off as a degenerate case.

Run those two commands twenty times, outside the package, and it fails four times. Not zero, not twenty. On the failures the harness reported bytes=2, dirty=false, status="killed by interrupt", and the captured screen contained this:

An interactive figure goes here. Open the page to use it.

tuitest created a PTY and never configured it. A fresh PTY comes up in the kernel's default cooked mode, so every byte written to the master went through the line discipline before it reached the program. 0x03 raised SIGINT and killed the child. 0x13 stopped output through flow control and could hang a session indefinitely. Everything the harness typed was echoed back down the master and arrived at the screen model as though the program had printed it, which quietly corrupted both the screen state and the byte counter the hang detector reads.

You can reproduce all three by hand:

An interactive figure goes here. Open the page to use it.

Real TUIs hide this, because they call MakeRaw on startup. That is why it survived so long. If your program reaches raw mode before the first byte lands, everything works. It is a race, and any input sent during terminal setup still belongs to the line discipline. Whether a tape drove a program or killed it came down to how fast the child got scheduled, and under -race with the rest of the package running, it got scheduled slower.

The shrinker was helping the bug

The shrinker is delta debugging. Remove chunks in decreasing sizes, simplify what survives, and accept a candidate if it still fails. "Still fails" meant failing on one replay.

That criterion is fine for deterministic failures. For a race it is actively harmful. Every command that takes time gives the program more time to reach raw mode, which makes the race harder to lose, which makes the candidate less likely to fail. So the commands that suppress the bug are precisely the ones the shrinker deletes first.

An interactive figure goes here. Open the page to use it.

The tool was optimising toward its own defect, and the tape it handed me was not noise. It was the shortest path to the bug, written to a file, which I had skimmed and dismissed for two days.

If your minimiser accepts on a single execution, it is a search for flaky inputs. It will find them whether you asked for that or not.

Fixing it, and the line I nearly crossed

Configure the PTY before the child exists, so there is never a moment where a signal-generating line discipline sits in front of a running program.

The real decision is how much to clear. My instinct was MakeRaw, and it was wrong. tuitest tests what programs do under a real terminal, and a real terminal does apply line editing to a program that has not gone raw yet. If the harness strips all of that, it stops reproducing the environment it exists to reproduce.

So it clears only what reinterprets or manufactures bytes: ISIG, IEXTEN, the echo flags, IXON, IXOFF, IXANY. ICANON and CR/NL mapping stay.

I know that boundary is right because I put it in the wrong place first. My first attempt also cleared ICRNL, and the tape roundtrip tests failed immediately. That was the codebase telling me I had crossed from "stop the kernel eating my input" into "change what the program sees", and I was glad something caught it.

The rest was portability. Termios constants live in different places across platforms, and my first version silently broke the Solaris build. Cross-compiling caught it: the constants now split across linux || solaris || aix and the BSD set, verified for darwin, freebsd, netbsd, openbsd, solaris and aix/ppc64.

Forty-four runs after the fix, zero failures. The two-command tape reproduces 20 in 20 with bytes=947, dirty=true, status="exit status 0".

While I was in there

The same audit turned up a second hole, unrelated to the PTY. The emulator generated correct replies to terminal queries, but nothing ever carried them back to the program that asked, which is why tuitest snap -- glow -p file.md captured a blank screen and exited 0. That turned out to be one instance of a bug I had already hit twice in other codebases, so it has its own post. The version relevant here: my own source contained comments describing the component that would carry replies back, and the component did not exist. Fixing it also broke a test that had been green since the day it was written, without once reaching the scenario it was named after.

Odds and ends

--duration was only checked between iterations, so --duration 5s ran 6.47s and --duration 20s ran 54s at default action counts. Now 5.09s and 20.27s.

The closest-line heuristic in failure output scored by shared prefix length, so against /a headless testing framework for TUIs/ it helpfully suggested the four-character fragment a VT. Replaced with normalised edit distance.

A blank capture now exits 5 instead of 0, with a note about --timeout and --wait if the program was still running when the capture was taken.

What I would keep

The fuzzer found almost nothing in the program it was pointed at. It found bugs in the harness, a bug in a test, and a bug in its own shrinker. For a tool whose whole job is to tell the truth about what a program did, I think that is the better outcome, though it did not feel like one at the time. (A later fuzzer, pointed at the terminal emulator itself, found nothing for a different reason, and that got its own post.)

The habit I am keeping is counting. "Sometimes" cost me an evening in the wrong file. Seventeen in thirty-six took two minutes to establish and immediately ruled out half of what I had been considering. And I will read the minimised output even when it looks stupid, especially when it looks stupid, because mine was correct and specific for two days while I ignored it.

What icat is actually doing

The kitty graphics protocol has a capability query. Before sending anything real, icat asks the terminal whether it supports a given transmission medium, and it asks about three of them:

_Ga=q,f=24,s=1,v=1,S=3,i=1        can you take pixels inline
_Ga=q,f=24,t=t,s=1,v=1,S=47,i=2   can you read a temp file
_Ga=q,f=24,t=s,s=1,v=1,S=18,i=3   can you read shared memory

It uses the first medium the terminal answers OK to, which is sensible. A file is much cheaper than inline base64, so if the terminal can read files, everyone wins.

The lie

The multiplexer's passthrough answered OK to all three. Unconditionally.

response := vt.BuildKittyResponse(true, cmd.ImageID, "")
ptyInput(response)

It never asked its own host anything. It just said yes.

So icat picked a file medium, wrote the image to /tmp, and sent a path. The multiplexer forwarded that path out to the browser. The browser cannot read /tmp on the server, so it dropped the transmission and drew nothing.

Captured at the browser, the failure looks like this:

An interactive figure goes here. Open the page to use it.

Why each layer alone was fine

In a real terminal, the host genuinely can read the path, so the lie is true by accident.

In the browser client alone, nothing lies. The client answers ENOTSUPPORTED to the file media, icat falls back to streaming the bytes inline, and everything works.

To hit the bug you need a dishonest middle layer in front of a host that cannot do the thing, and that only happens in the three-layer stack. Each component was correct in the environment its author tested it in. That is why staring at any one of them got me nowhere.

Try the combinations yourself:

An interactive figure goes here. Open the page to use it.

The fix

Probe the host for file transmission alongside the direct probe, record the answer, and pass it on:

if isFileMedium(cmd.Medium) && !kp.hostReadsFiles() {
    ok = false
    errMsg = "ENOTSUPPORTED:host terminal cannot read files from this machine"
}

A guest that never asks is covered too. If a file transmission arrives and the host cannot read files, the multiplexer re-encodes it as direct data rather than forwarding a path that will be silently discarded. A file-capable host still gets the plain path with no extra copy, so the fast case stays fast.

There was a second bug hiding in the same function. Quiet mode has two levels, q=1 to suppress success responses and q=2 to suppress everything, and the code treated both as "say nothing". A guest that asked for q=1 and hit an error got silence, waited out its timeout, and never tried the next medium. An error has to reach a guest that only suppressed successes.

Ask, or say no

A proxy answering capability questions on behalf of something else has two honest options: ask the far end, or say no and let the guest fall back to the medium that always works. Saying yes because yes usually works gets you a failure that only appears when the layers are combined, only for people running unusual stacks, with nothing logged anywhere and every component pointing at its own passing tests.

What annoys me is that the multiplexer already had the machinery to handle this. A function that re-encodes file transmissions as inline data was sitting in the codebase, gated behind a build-specific flag instead of behind "can the host actually read files". I did not so much write the fix as move it.

It is the same protocol as the query bugs in an earlier post, with a new failure: there the answer went missing, and here it was made up. What tuios now tells programs about graphics, in a native terminal and in the browser, is on the architecture page and the web terminal page.

The obvious fix was right there. ghostty has a VT implementation that gets this right, compiled to wasm, with width tables I could extract. Take its answers, use them instead of mine, done.

I built the table offline, wired it in, and measured against a corpus of hard cases.

It changed nothing. Not "small improvement", not "fixed some cases". The output was identical.

Why a table cannot work

Here is the thing I had wrong, and you can check it yourself:

An interactive figure goes here. Open the page to use it.

A per-codepoint table answers "how wide is this codepoint". Correctly, in ghostty's case. But the terminal is not asking that question. It is asking "how far does the cursor advance", and the answer depends on how codepoints combine, not on what each of them is worth alone.

A family emoji is four people joined by zero-width joiners. Seven codepoints, several of which are individually wide. Add up the table's answers and you get eight columns. The terminal advances two, because it is one grapheme cluster and a cluster gets one advance.

No table of per-codepoint values can produce that. The information is not in the codepoints. It is in the boundaries between them, which is UAX 29 segmentation, a completely different algorithm. I had spent a day importing correct answers to a question nobody was asking.

What the corpus said

The corpus is 45 cases measured against real ghostty-vt with mode 2027 clustering enabled, so the expected values come from an implementation known to be right rather than from my reading of a spec.

With the official grapheme addon in place, 39 of 45 agree. The remaining six diverge in documented, degenerate cases, and they are asserted as expected values rather than treated as outstanding bugs. That is a deliberate choice: pretending they are bugs means either carrying a patch forever or having a permanently red test.

The real fix was already available and was not a table at all. Use a segmenter that implements UAX 29, and let the cluster boundaries fall out of it. Everything else follows.

What I kept from the exercise

The width table went in the bin, but the wasm build did not. It is now an oracle.

That distinction turned out to be the useful part. As a runtime dependency, ghostty-vt was a large coupling for a benefit I had not verified. As a test oracle, it tells me whether my output is right, costs nothing at runtime, and can be as heavy as it likes because it only runs in CI.

I would not have found that framing if the table had worked. It only came up because I had to ask what the wasm build was still good for once the reason I built it was gone.

What parity means once pixels are involved

The corpus asserts integers, and integers can be compared for equality: a cluster advances two columns or it does not. The renderer's own tests do not have that luxury, and deciding what "matches" should mean for pixels took longer than wiring up the oracle did.

Pixel parity is a tolerance, not equality. Four golden scenarios pass under a budget of 2 percent differing pixels against the reference, and the emoji scenario under 5 percent, because glyph rasterisation is allowed to differ in its antialiasing without anything being wrong.

The 24-entry torture corpus needed a different comparison entirely. Those rows are mostly background, and that is exactly what makes a per-pixel comparison useless on them: a subpixel shift in glyph position swings the differing-pixel fraction wildly while nothing is actually wrong, so the number measures jitter rather than correctness. Those rows are compared by ink coverage per cell instead. The question is whether the right cell contains the right amount of glyph, not whether the exact pixels are identical.

Side by side, the three comparisons and what each one accepts:

Every threshold in that list is a claim about the size of defect I am prepared not to notice. Choosing them felt like bookkeeping at the time. It was not. The same choice, made carelessly in another suite, later cost me three wrong closures on one bug. A tolerance wider than a defect does not make the defect unlikely to be seen. It makes it invisible.

What the benchmarks refuse to claim

The renderer's benchmarks run under headless chromium, where WebGL executes on SwiftShader and rasterises on the CPU. That is a different machine from the one any user has, so the repo states plainly which columns transfer to real hardware: the CPU timings, the draw-call counts and the allocation figures. Nothing else, and no frame rate is claimed anywhere, because a frame rate measured on a software rasteriser describes the rasteriser.

Writing down what a measurement cannot tell you costs a paragraph in a README. I have since been on the other end of skipping that paragraph, with three rounds of benchmarks vouching for a broken code path, so it no longer reads as pedantry to me.

The bit I would do differently

I should have run the cheap experiment first. Take one family emoji, ask the current implementation how wide it thinks it is, ask ghostty, compare. Fifteen minutes, and if both said eight, the table was never the problem and the bug lived somewhere else entirely.

I skipped that because the fix felt obviously right. Correct data replacing incorrect data is such a clean shape that I did not stop to ask whether the data was the thing that was wrong.

The day was not a total loss. Nobody on this project will spend another day importing width tables, and there is now a corpus and an oracle that would catch anyone who tried.

It took four rounds. Three of them ended with me handing over a build and saying it was fixed. Each of those three fixes was for a real bug, which is exactly what made it so easy to keep doing.

Round one: a lock taken twice

The first thing I found was genuinely bad. The render path took a window's I/O read lock, then called a function that took the same lock again.

Go's RWMutex is not reentrant for readers. Two RLock calls nest fine on their own, but if a writer queues between them, the second read blocks behind the writer and the writer blocks behind the first read. Everything stops.

A profiler is useless on a stopped program, because nothing is hot. What shows a deadlock is the goroutine dump: SIGQUIT with GOTRACEBACK=all. There, in a single goroutine's stack, is the same lock, held at one frame and wanted at a deeper one.

It was a real, reachable freeze, and I fixed it by moving the cursor query above the lock. I also believed it explained the blank pane, because a render that deadlocks partway could plausibly leave a pane unpainted.

It did not explain it. The blank pane came back.

Round two: alt-screen cache invalidation

Next I found that switching focus could leave a stale cached layer for a window in alternate-screen mode. That is exactly the sort of thing that leaves a pane showing nothing.

Fixed it. Shipped it. Hit the blank pane again the same afternoon.

Round three: shared border state

Then I found a discrepancy in how tiled windows tracked border state across a focus change. It was real and worth fixing, and once again I could connect it to a blank pane if I did not look too hard.

The reply I got was, roughly: your stuff is still broken.

That was the moment the approach should have changed. It did not.

What I was doing wrong

The pattern is obvious in hindsight and slightly embarrassing.

Each round I read code, found something genuinely wrong, and stopped. The fix was plausible. "Plausible" is doing a lot of work there, because a multiplexer has many ways to leave a pane unpainted, and I could build a story connecting almost any of them to the symptom.

I never reproduced it deliberately. Not once, across three rounds. I had no way to make a pane go blank on demand, so I had no way to tell whether a fix worked. The loop was: find a bug, fix it, ship it, go back to using the thing and wait to notice. Three rounds of that before I changed approach.

What I had was three real bugs found by auditing, which is a fine activity, and zero evidence about the one I was chasing.

Round four: reproducing it

Different approach. Instead of reading, drive the thing and check the pixels.

I set up the layout I kept hitting it in: one tall pane on the left, two stacked on the right. Focus one, focus another, and after each step compose the frame and check whether any pane's content came out empty.

It reproduced in about a minute. The cause was one line:

windowWidth := ansi.StringWidth(lines[0])

clipWindowContent measured a window's width from its first line, then discarded content falling outside the visible region. A window whose first line is blank measures zero wide. Zero width makes x + windowWidth <= 0 true for anything at or left of the origin, so the entire pane is discarded as off-screen.

An interactive figure goes here. Open the page to use it.

Which panes have a blank first line? The ones running full-screen applications that just repainted, and the ones scrolled so row zero is empty. That is why it was intermittent, and why it followed focus changes: a focus change triggers a repaint.

The fix takes the maximum width across all lines instead of trusting the first. Here is the guard on the same layout. Blank a first line and pick a measurement:

An interactive figure goes here. Open the page to use it.

The lesson

For anything visual, reproduce it in the rendered output before claiming a fix. Not in a unit test of the function you suspect. In the frame that reaches the screen.

That sounds obvious. It was not obvious in the moment, because each of my three wrong fixes had a passing test. I could demonstrate, with a green test, that the lock nesting was gone, the cache invalidation correct, the border state consistent. All true. None of it evidence about the actual symptom, because none of those tests composed a frame and looked at it.

The three bugs were worth fixing, and two of them could have caused a freeze under the right conditions. I would find them again. What I would not do again is call any of them the fix for a bug I had never once seen happen.

Each time, I said "this is fixed" when what I actually knew was "I found a bug and it might be this one". Those are different claims, and I made the stronger one three times running, to someone who then had to spend their afternoon finding out otherwise.

Two checks I now run

The first is the negative control. After a fix lands with its test, revert the fix, confirm the test fails, restore it. A green test with the fix in place only proves the test can pass. A red test without the fix proves the test is actually connected to the change. In a later stretch of work I ran that revert-and-confirm step on every fix, and it caught five cases where a fix had looked settled and was not. Five, from a check that takes a minute each.

It would not have caught anything here. My three wrong fixes all pass the negative control, because each one really did fix the bug its test guarded. The negative control proves the test measures the fix. Only reproducing the symptom proves the fix addresses the bug, which is why both checks exist and neither substitutes for the other.

The second check is on myself, not the code. When I notice that a causal story has been assembled entirely by reading, I stop and execute something before acting on it. Across the same stretch of work, reading produced a confidently wrong root cause at least three times, and in every one of those cases executing the code settled the question in a single command. Reading is how I find candidates. I have stopped letting it also pick between them.

A later bug had the same trigger, a focus change, and a quieter symptom: the pane kept its content and lost its underline. That time reading found the candidate and a probe confirmed it before any fix. It is written up in The underline depended on which code drew it.

A terminal is a request-response protocol. Programs ask it questions constantly, and they block waiting for answers. What is the background colour. Where is the cursor. What are your device attributes. Do you support this image format. If nobody answers, the program does not error. It waits, retries, gives up, and carries on in a degraded state that looks like a completely different bug.

I hit that three times in a month, in three codebases, and each time the reply was going somewhere different. Writing them next to each other is the only reason I noticed they were the same bug.

What a query looks like

Here is what glow sends before it draws anything at all:

An interactive figure goes here. Open the page to use it.

lipgloss uses that background colour to decide whether to render a light or dark palette, which means most Bubble Tea programs do this. It is not an edge case. It is the common path.

One: the replies were generated and then dropped

tuitest snap -- glow -p file.md captured a blank screen and exited 0. (tuitest is my PTY test harness, and the rest of that audit is in a later post.) Not a crash, not a timeout. A confident report of nothing.

My first diagnosis was a nil guard in the OSC handler that looked like it would suppress colour replies. Wrong, and one command proved it. Drive the emulator with exactly the bytes above and read what comes back:

in:  "\x1b]11;?\x1b\\\x1b[6n"
out: "\x1b]11;rgb:0000/0000/0000\a\x1b[1;1R"

Both replies, correctly formed. The emulator had been right the whole time, and the guard was dead code because the colour getters have non-nil fallbacks.

The replies went into an internal pipe, exposed through a Read method, and nothing in the repository ever called it. The interface the rest of the harness was written against had no drain method at all. Bytes in, nothing out.

The evidence was sitting in my own source. bufpipe.go carries comments about "the response drainer" and "the terminal-response forwarder". I had built the producer, documented the consumer, and never written it.

One follow-up change mattered beyond the fix itself, because "a confident report of nothing" is a failure mode in its own right. A capture that succeeds and comes back empty now exits 5 with a diagnostic, a code kept separate from the assertion and harness exit codes. A blank screen can be legitimate, since a program is allowed to draw nothing. But it is rare and suspicious enough that it should never again look the same as success.

Two: the wrong terminal answered

While making a screen recording of a test run, I got a burst of garbage on screen right as the program exited:

An interactive figure goes here. Open the page to use it.

The replay tool mirrors the child's output to its own stdout so you can watch a run. Verbatim. So the queries the program under test emitted travelled straight out of the harness and into the outer tmux, which answered them, because it is a terminal and that is what terminals do.

Those answers landed on the driver script's stdin, where nothing read them, and the pane's line discipline echoed them to the screen. They sat on the normal screen buffer, invisible underneath the program's alternate screen, and appeared the instant it exited.

Same protocol, opposite failure. In the first case a query got no answer. Here it got an answer from a terminal three layers away that had no business replying.

Three: a fake shell echoed them back

Third instance, in a browser terminal demo. Displaying an image and then typing produced this at the prompt:

An interactive figure goes here. Open the page to use it.

The demo's fake shell processes input one character at a time. It drops anything below 0x20, which discards the escape, and everything else is printable, so _Gi=2;OK gets appended to the line buffer and echoed.

The terminal answered correctly. The thing receiving the answer had no idea it was in a protocol.

The shape

Three bugs. In each one, some component treated a bidirectional protocol as unidirectional:

• tuitest generated replies and had no path to send them anywhere.
• The replay tool forwarded queries to a terminal that was not the intended recipient, and had no path to route the answers back.
• The demo shell received replies and had no parser to recognise them.

Producer with no consumer. Consumer that is the wrong process. Consumer with no parser. If you draw the data flow, the missing arrow is in a different place each time, and the visible symptom is different every time too: a blank capture, a burst of garbage after exit, text appearing at a prompt nobody typed.

An interactive figure goes here. Open the page to use it.

None of those symptoms says "terminal query". That is what made it three separate investigations instead of one.

What I do differently now

When something under a PTY behaves oddly and the obvious explanations do not fit, I check the query path before anything else. Three questions, in order.

Does the program send queries? Run it under script or a raw PTY and look at the bytes. glow sending OSC 11 three times in a row is unmistakable once you have seen it.

If it does, does something answer? Drive the emulator directly with those exact bytes and read the output. It is one command, and it narrows the search a lot.

If something answers, does the answer reach the program that asked? This is the one that bit me twice, and the one nobody checks, because generating a correct reply feels like the hard part. Routing it is where people actually fail, me included.

There is a fourth question I have started asking about my own code: if a component produces something, who consumes it? bufpipe.go documented a consumer that did not exist for months. The comment was aspirational, and I read it every time as a description of something real.

Later I found a fourth shape, in TUIOS itself: a component that answered a capability query without asking. The answer arrived. It was made up.

The test that had never run

One more, because it kept bothering me after everything else was fixed.

Fixing the reply path in tuitest broke a test that had been green since the day it was written. Its fixture explains, in a comment, why it queries the terminal: doing so "is what makes replies arrive on the input channel and is the situation that produced the reported corruption". Replies had never arrived. The scenario the test was written to cover had never once executed. It asserted a real property, went green every run, and never reached the code path it was named after.

A test like that is worse than a missing one, because it occupies the place where a working test would go and reports success from it. And when replies finally did arrive, the fixture's own parser turned out to be broken too: it only reset its buffer on a recognised sequence, so a device-attributes reply sat there and prefixed the next mouse report, which then matched nothing. The test and the code it was guarding had the same blind spot, so of course they agreed.