Reading Finds Candidates. Running Finds Facts.

Part 2 of Building Sales Scribe: What Accounting Software Teaches You About Correctness

When the year-end features of Sales Scribe were finished, we commissioned a structured code review before building anything on top of them. Six sessions, each looking at one question: can the period lock be bypassed, are the derived figures right, can a signed-off year change without anyone knowing, is anything leaking between customers, what happens under concurrency, and could an accountant actually prepare accounts from the result.

It found one critical defect and several serious ones. What surprised us wasn't how many. It was where every one of them came from.

Every significant finding came from running the software. None came from reading the code.

What the review was told to do

The brief set some rules, and one of them turned out to matter more than the rest: the specification is a claim, not evidence. The spec described what we intended and what we believed had been built. The review was to check both against the working code, and treat any disagreement as a finding in its own right.

It also said that a passing test isn't evidence until you've read what it actually asserts, and that every report should end with a list of what couldn't be verified.

We expected those rules to catch a few loose ends. They ended up being the review.

Four findings, and how each one surfaced

The reversal that never happened. Reversing a supplier payment is meant to post an accounting entry cancelling the original. The code created that entry, and never posted it. It sat as a draft forever.

Every report excluded drafts, so the ledger still showed the money as paid out, while the payment itself read "reversed". The balance sheet still balanced. And the tests passed, because they fetched the reversal and checked it cancelled the original — which a draft does perfectly well. Nothing asked whether it had been posted.

It was found by looking at what the running system had actually produced: in the test database, every reversed supplier payment had a draft reversal and none had a posted one.

The feature that had never worked. Accountancy practices can act inside their clients' books through a delegated route. Every sensitive action is audited, and the audit record stores the web address of the request, in a column 128 characters wide.

The delegated route's prefix alone is 113 characters long. So almost every sensitive action through that route produced an address too long to store, the audit write failed, and it took the action down with it. Locking a period, signing off a year, reopening one — none of it worked for a practice.

The test suite never noticed, because every delegated test happened to exercise the one action whose address was short enough to fit, at 127 characters. It was found by a throwaway test that simply tried the other actions and looked at what happened.

A problem eleven days away. Sales Scribe generates accounting periods forward from a business's year end. The review set a 30 September year end on 19 September, with the clock fixed at that date, and asked what period 1 October fell in.

None. The period chain stopped at the end of the current year. Eleven days later, that business's balance sheet would have lost track of which year it was in, silently. Nothing in the code read as wrong. The answer only appeared when the calendar was actually moved forward.

The same click, two different results. Issuing several invoices at once, with one of them dated inside a locked month, produced different outcomes depending on the order the invoices happened to be processed in. Put the locked one first and the other two failed with it. Put it last and they went through.

That's the kind of fault users describe as "it sometimes doesn't work". It was found by running the same three invoices through the real endpoint in both orders.

The claims that were wrong

The other half of the story is the confident statements that turned out to be false.

Over the course of the review we kept a table of claims about how the code worked that were made by reading it, and later contradicted by running it. A report said a particular component marked a reversed payment as reversed; it didn't. A report said no user action ever continued working after a refusal; bulk issue did. A report said something couldn't be verified that, it turned out, could. Several of the entries were our own, made while planning the review.

None of these were careless. Each was a reasonable reading of the code. That's precisely the problem: a plausible explanation of how software behaves feels like evidence, and it isn't.

The rule we took from it is simple: a claim about what the code does is unreliable unless it says it was observed. Reading tells you where to look. It doesn't tell you what you'll find.

What "running it" actually means

It isn't just "write more tests". Several of the defects above sat behind passing tests. It's a handful of habits:

Drive the real path. A test that exercises a method directly can pass while the route users actually take goes around it. We had one where a carefully tested method was bypassed entirely by the code that called it in practice.

Break it on purpose. Remove the fix, or the line you think matters, and check the test actually fails. If it doesn't, it was never testing what it claimed. We found checks that passed whether or not the thing they guarded was there.

Say what you expect before you look. Before breaking something deliberately, write down which tests should fail. If the result is different — even if it looks close — something is wrong with your understanding, and that's worth knowing.

Prove the check can fail. A check that finds nothing might be working, or might be looking in the wrong place. Give it a case it must catch, and see that it does.

Separate what you observed from what you inferred. Every report in the review marked its claims one way or the other. It's a small discipline, and it's what made the wrong claims visible at all.

Why this matters more with AI

Sales Scribe is built with AI coding agents, and the review itself was carried out by one, working to a written brief.

That makes this lesson sharper, not weaker. An agent can read a large codebase quickly and describe it fluently. Those descriptions are often right. But fluency isn't observation, and a confident summary of what the code "does" carries exactly the risk this review kept finding — from people and agents alike.

So the process has to ask for evidence rather than explanation: what was run, what it showed, and what couldn't be checked. That's what turned a well-written review into a useful one.

Reading is still where every investigation starts. It's how you decide where to look. But what it gives you are candidates. The facts come from running the software and watching what it actually does.