I shipped three fixes for a bug I had not found

It was easy to describe and hard to catch. Focus a different terminal, and sometimes one of the other panes goes blank. Focus it again and the content comes back.

It took four rounds. Three of those ended with me handing over a build and saying it was fixed. Every one of those three fixes was a real bug, which is exactly what made it so easy to keep doing.

Round one: a lock taken twice

The first thing I found was genuinely bad. The render path took a window's I/O read lock, then called a function that took the same lock again.

Go's RWMutex is not reentrant for readers. Two RLock calls nest fine on their own, but if a writer queues between them, the second read blocks behind the writer and the writer blocks behind the first read. Everything stops.

A profiler is useless on a stopped program, because nothing is hot. What shows a deadlock is the goroutine dump: SIGQUIT with GOTRACEBACK=all, and there in a single goroutine's stack is the same lock, held at one frame and wanted at a deeper one.

Real freeze, reachable, fixed by hoisting the cursor query above the lock. I also believed it explained the blank pane, because a render that deadlocks partway could plausibly leave a pane unpainted.

It did not explain it. The blank pane came back.

Round two: alt-screen cache invalidation

Next I found that switching focus could leave a stale cached layer for a window in alternate-screen mode. Which is exactly the sort of thing that leaves a pane showing nothing.

Fixed it. Shipped it. Hit it again the same afternoon.

Round three: shared border state

Then a discrepancy in how tiled windows tracked border state across a focus change. Real, worth fixing, and once again my mental model let me connect it to a blank pane if I did not look too hard.

The reply I got was, roughly: your stuff is still broken.

That was the moment the approach should have changed. It did not.

What I was doing wrong

The pattern is obvious in hindsight and slightly embarrassing.

Each round I read code, found something genuinely wrong, and stopped. The fix was plausible. Plausible is doing a lot of work in that sentence, because a multiplexer has many ways to leave a pane unpainted, and I could construct a story connecting almost any of them to the symptom.

I never reproduced it deliberately. Not once, across three rounds. I had no case where I could make a pane go blank on demand, so I had no way to tell whether a fix worked. The loop was: find bug, fix bug, ship it, go back to using the thing and wait to notice. Three rounds of that before I changed approach.

What I had was three real bugs found by auditing, which is a fine activity, and zero evidence about the one I was chasing.

Round four: reproducing it

Different approach. Instead of reading, drive the thing and check the pixels.

Set up the layout I kept hitting it in. One tall pane on the left, two stacked on the right. Focus one, focus another, and after each step compose the frame and check whether any pane's content came out empty.

It reproduced in about a minute. The cause was one line:

windowWidth := ansi.StringWidth(lines[0])

clipWindowContent measured a window's width from its first line, then discarded content falling outside the visible region. A window whose first line is blank measures zero wide. Zero width makes x + windowWidth <= 0 true for anything at or left of the origin, so the entire pane is discarded as off-screen.

lines[0] = ""             width measured as 0
lines[1] = "~/dev/tuios"  ignored
lines[2] = "$ nvim"       ignored
 
x + 0 <= 0   ->  discard the entire pane
 

Which panes have a blank first line? The ones running full-screen applications that just repainted, and the ones scrolled so row zero is empty. That is why it was intermittent, and why it correlated with focus changes, because a focus change triggers a repaint.

The fix takes the maximum width across all lines instead of trusting the first.

The lesson

For anything visual, reproduce it in the rendered output before claiming a fix. Not in a unit test of the function you suspect. In the frame that reaches the screen.

That reads as obvious. What made it non-obvious in the moment is that each of my three wrong fixes had a passing test. I could demonstrate, with a green test, that the lock nesting was gone, the cache invalidation correct, the border state consistent. All true. None of it evidence about the actual symptom, because none of those tests composed a frame and looked at it.

The three bugs were worth fixing, and two of them could have caused a freeze under the right conditions. I would find them again. What I would not do again is call any of them the fix for a bug I had never once seen happen.

Each time, I said "this is fixed" when what I actually knew was "I found a bug and it might be this one". Those are different claims, and I made the stronger one three times running, to someone who then had to spend their afternoon finding out otherwise.

Two checks I now run

The first is the negative control. After a fix lands with its test, revert the fix, confirm the test fails, restore it. A green test with the fix in place only proves the test can pass. A red test without the fix proves the test is actually connected to the change. In a later stretch of work I ran that revert-and-confirm step on every single fix, and it caught five cases where a fix had looked settled and was not. Five, from a check that takes a minute each.

Note what it would not have caught here. My three wrong fixes all pass the negative control, because each one really did fix the bug its test guarded. The negative control proves the test measures the fix. Only reproducing the symptom proves the fix addresses the bug, which is why both checks exist and neither substitutes for the other.

The second check is on myself, not the code. When I notice that a causal story has been assembled entirely by reading, I stop and execute something before acting on it. Across the same stretch of work, reading produced a confidently wrong root cause at least three times, and in every one of those cases executing the code settled the question in a single command. Reading is how I find candidates. I have stopped letting it also pick between them.