Skip to content

Repair and crash recovery

What a crash can do

An append is write + flush + fsync of one complete line under an exclusive lock. If the process or the machine dies in the middle of a write, the file can end with a partial line (no final newline). Nothing else is modified: earlier lines are never rewritten.

What you will see

$ protocolith verify-ledger
TRUNCATED_TAIL line 3: last line is not terminated by a newline
FAILED: TRUNCATED_TAIL

Exit code 1. Further appends are refused while the tail is partial (they report VERIFICATION_FAILED, exit 1), because Protocolith never appends after a damaged line.

Repair

protocolith repair --truncate-partial-tail
# removed 13 bytes of partial tail
protocolith verify-ledger

repair removes only the unterminated last line, and only if everything before it verifies. If the rest of the ledger does not verify, it refuses and reports why; if there is no partial tail, it exits with 2 and NO_PARTIAL_TAIL. It is the only command that shortens the file, and it requires the explicit flag.

The entry that was being written is lost; its effect did not happen as far as the ledger is concerned. Run the command that failed again.

What repair cannot fix

Symptom Meaning What to do
BAD_HASH, BAD_PREV, BAD_SEQ in the middle The file was changed or damaged Restore from a backup or from git; do not edit.
NOT_CANONICAL A line was reformatted by an editor or a tool Restore the original bytes.
Mixed line endings after a transfer A tool converted \n to \r\n Restore; keep the file as binary in git (-text).
BAD_CARD The card cannot be rebuilt from the record Restore; check that you did not change the record or the card validator version.
Lock timeout (LOCK_TIMEOUT, exit 3) Another process holds the lock Wait or stop that process. A stale lock file is harmless: the lock is released when the holder dies.

Do not repair by editing the ledger or recomputing hashes. A repaired ledger that you re-hash yourself verifies, but it is no longer the history you want to show.

Backups

Copy the ledger file after important steps and compare the head against an anchored value (anchoring). Keep .gitattributes rule *.jsonl -text if you store the ledger in git on Windows.