What ledger correctness means at scale

Most software systems can fail a little. A search query returns a slightly stale result. A recommendation engine misranks two items. The user does not notice; if they do, the cost is small.

A ledger cannot fail a little. A ledger that is wrong by one cent is wrong, and the wrongness is permanent unless someone makes it right with another transaction. That is the constraint we have been building under for the last five years.

These are six lessons from the team that built and runs the Northbank ledger.

1. Causality is the product

You can think of a ledger as a list of balances, but that is the wrong frame. A ledger is a graph of causes — this transfer happened because that approval happened because that policy permitted it. The balances are a derived view of the graph.

We learned, the hard way, that any operation that mutates a balance without recording its cause is a future bug. Every entry in our ledger has a parent and, with rare exceptions, a parent’s parent. We can answer why is this balance what it is by walking the graph backward.

2. Idempotency is not optional

At our scale, every action will, at some point, be retried. Network blips. Client crashes. Internal recovery loops. The ledger has to be built so that the same logical operation, if it arrives twice, has the same effect once.

This sounds obvious. The trick is that the same logical operation is not always the same payload. A retried wire might have a different timestamp than the original. We use stable, client-supplied keys for every mutation. If we ever see a key twice, we return the original result. We do not retry it.

3. Append-only or bust

We do not delete entries. We do not edit entries. If something needs to be reversed, it is reversed by a new entry that points back at the original. The ledger only grows.

This is more disk-expensive than the alternative. We have made our peace with that. The cost is small compared to the cost of losing the audit trail, which is what delete would, structurally, do.

4. Reconciliation is a feature, not a fix

We reconcile the ledger against external systems — the Fed, our partner banks, our card network — every day. For a long time, we treated reconciliation as a checking activity. If the numbers matched, we did nothing. If they did not, we fixed.

The shift was treating reconciliation as a product surface. The reconciliation system runs constantly. It is the source of truth for did this external action happen. We surface its outputs in operator dashboards. The day after we made that shift, three bugs we had not noticed for months became visible.

5. Every value has a unit

I cannot count the number of bugs across the industry caused by a number floating around without a unit attached. A balance is in cents, a fee is in basis points, an interest accrual is in micro-cents per day. The system has to know.

Our ledger types every value at the type-system level. You cannot pass a cents into a function expecting micro-cents without an explicit conversion. The compiler catches it. We have done less arithmetic-debugging in the last two years than in the prior three.

6. The hardest bug is the one that is almost right

A ledger that is grossly wrong is easy to detect. A ledger that is one cent off, on one customer, on one day, is not. We invested heavily in invariants — automatic checks that run on every commit and on every reconciliation pass — that look for almost-right states.

The category of bug we work hardest to prevent is the one that closes a quarter cleanly and resurfaces, with interest, two years later in an audit.

What this gets you

Ledger correctness is mostly invisible. Customers do not write us thank-you notes when their balance is right. They only write us when it is wrong. The point of the work is to keep the second category empty.

We have not, to our knowledge, lost a cent. We expect to keep it that way for the next decade by continuing to invest in the unglamorous parts of the system that make that true.