Protocolith Core Specification¶
Status: draft 0.1 (target: Protocolith 0.6.0). This document defines the core. The reference
implementation is the protocolith.core package; any other implementation must pass the
conformance vectors in spec/vectors/ to claim compatibility.
1. Scope¶
Protocolith does exactly one thing: it records, verifies and issues trial records.
- Freeze a protocol before a run.
- Record the outcome of a run against a frozen protocol.
- Verify that the history is intact and every record follows its frozen protocol.
- Issue a human-readable card from a verified record.
Non-goals: running experiments, fetching data, scheduling, storing raw payloads, judging whether a protocol is scientifically good, answering questionnaires. The core uses only the Python standard library.
2. Canonical JSON (PCJ-1)¶
All hashes are computed over canonical bytes.
- UTF-8, no byte-order mark, no insignificant whitespace, no trailing newline in the hashed bytes.
- Objects: keys unique, sorted by Unicode code point; duplicate keys in input are an error.
- Allowed values: object, array, string, boolean, null, integer in
[-(2^53-1), 2^53-1]. - Floats are not allowed. Write decimals as strings (
"0.001"). This keeps hashes identical in every language. - Strings: any Unicode except lone surrogates; escapes are
\"\\\b\f\n\r\tand\u00XXfor other controls below U+0020; everything else is written literally. NaN,Infinity, and non-finite numbers are errors.
sha256(x) is the lowercase 64-hex SHA-256 of the PCJ-1 bytes of x.
3. Ledger¶
A ledger is a UTF-8 text file with one entry per line (\n terminated). Each line is the PCJ-1 form of:
{"body":{...},"hash":"<64 hex>","prev":"<64 hex>","seq":0,"ts":"2026-11-14T16:00:00Z","type":"freeze","v":1}
v: format version, integer,1for this document.seq: integer, starts at 0, increases by exactly 1.ts: UTC timeYYYY-MM-DDTHH:MM:SSZ, whole seconds.prev:hashof the previous entry; 64 zeros forseq0.hash:sha256of the entry object without thehashfield.type:freeze,record,deviationorissue.
3.1 Bodies¶
| type | body fields |
|---|---|
freeze |
protocol_id, version, protocol (object), protocol_sha256 (= sha256(protocol)), supersedes (hash or null) |
record |
protocol_sha256, run_id, status, outcome (object), sources (list of {id, sha256}), reproduction_level, claim_boundary, deviations (list of seq), optional block_reason, drift_reason |
deviation |
protocol_sha256, kind, description |
issue |
record_seq, format, card_sha256 |
status is one of the closed set of result statuses of the evidence-card format: PASSED_UNDER_PROTOCOL,
NEGATIVE_RESULT_UNDER_PROTOCOL, BLOCKED_SOURCE, INSUFFICIENT_COVERAGE, SOURCE_DRIFT, REPRODUCED_OUTCOME.
BLOCKED_SOURCE requires block_reason; SOURCE_DRIFT requires drift_reason. PASSED_UNDER_PROTOCOL,
NEGATIVE_RESULT_UNDER_PROTOCOL and REPRODUCED_OUTCOME require a non-empty sources list (code MISSING_FIELD).
Absence of evidence is never recorded as NEGATIVE_RESULT_UNDER_PROTOCOL; use BLOCKED_SOURCE or INSUFFICIENT_COVERAGE.
3.2 Rules¶
| Rule | Statement | Code on violation |
|---|---|---|
| R1 | seq is contiguous from 0 |
BAD_SEQ |
| R2 | prev equals the previous entry's hash |
BAD_PREV |
| R3 | hash matches the recomputed value; the line is exactly canonical |
BAD_HASH, NOT_CANONICAL |
| R4 | ts never decreases |
TS_REGRESSION |
| R5 | a record names a protocol_sha256 frozen at an earlier seq |
UNKNOWN_PROTOCOL |
| R6 | one (protocol_id, version) is frozen once; its hash never changes |
DUPLICATE_FREEZE |
| R7 | status is in the closed set; conditional fields present |
BAD_STATUS, MISSING_FIELD |
| R8 | every deviations seq and issue.record_seq points to an earlier entry of the right type |
DANGLING_REF |
| R9 | issue.card_sha256 equals the hash of the card re-rendered from the record |
BAD_CARD |
| R10 | v is known |
UNKNOWN_VERSION |
| R11 | the last line is complete (ends with \n, parses) |
TRUNCATED_TAIL |
A changed protocol is a new version: freeze again with supersedes set to the old hash. Records always keep
pointing at the hash they were run under.
3.3 What the ledger does and does not prove¶
A valid ledger proves internal consistency and ordering. It does not prove wall-clock time and cannot detect that a
whole ledger was replaced or that its tail was removed. head (the last hash plus seq) may be published or
anchored elsewhere (a signed git tag, a timestamp service); a later check against that head detects removal (verify-ledger --expect-head SEQ:HASH; a mismatch yields HEAD_MISMATCH, exit 1).
4. Local store¶
Default directory: .protocolith/ in the working directory (override: --store or PROTOCOLITH_STORE).
ledger.jsonl: the ledger.lock: advisory lock file; appends take an exclusive lock (fcntlon POSIX,msvcrton Windows).- Append is
write+flush+fsyncof one complete line; a crash leaves at most one partial last line, whichverifyreports asTRUNCATED_TAILand which only an explicitrepair --truncate-partial-tailremoves. - All paths are handled with
pathlib; newlines are always\n(files opened in binary mode). - The clock is injectable (
PROTOCOLITH_NOWfor tests); timestamps are UTC.
5. Interfaces¶
All three expose the same operations with the same semantics.
| Operation | CLI | Python | MCP tool |
|---|---|---|---|
| freeze | protocolith freeze FILE |
core.freeze(store, protocol) |
protocolith_freeze |
| record | protocolith record FILE |
core.record(store, body) |
protocolith_record |
| deviation | protocolith deviation FILE |
core.deviation(store, body) |
protocolith_deviation |
| verify | protocolith verify-ledger |
core.verify(store) |
protocolith_verify |
| issue | protocolith issue SEQ |
core.issue(store, seq) |
protocolith_issue |
| log / head | protocolith log, protocolith head |
core.entries(store), core.log(store, ...), core.head(store) |
protocolith_log |
| show | protocolith show SEQ |
core.show(store, seq) |
protocolith_show |
| summary | protocolith summary [--protocol ID] |
core.summary(store, protocol=None) |
protocolith_summary |
| capture | protocolith capture [--file PATH ...] [--env] |
core.capture(files, env=False) |
protocolith_capture |
| schema | protocolith schema [NAME] |
core.schema_for(name) |
protocolith_schema |
| atlas | protocolith atlas [--store DIR] [--cards DIR ...] [--out FILE_OR_DIR] |
core.build_atlas(...) |
protocolith_atlas |
| init | protocolith init |
core.init(store) |
CLI exit codes: 0 ok; 1 the stored ledger does not verify (codes on stderr and, with --json, on stdout);
2 the operation was rejected (bad input, or the new entry would break a rule, including TS_REGRESSION) and nothing was written;
3 I/O or environment error (missing ledger, lock timeout). --json output is PCJ-1. Reading a missing ledger is NO_LEDGER
(exit 3), never an empty pass.
The MCP server speaks stdio JSON-RPC with initialize, tools/list, tools/call, resources/list and
resources/read, uses no network and no third-party package, and never writes outside the store, except the one atlas page of protocolith_atlas inside the working directory.
5.1 Read commands¶
logaccepts the filtersprotocol(aprotocol_id; an unknown one isUNKNOWN_PROTOCOL),type(an entry type),status(only records with that result status; a value outside the closed set isBAD_STATUS) andtail(an integer of at least 1: keep the last N matches). Filters combine with AND and keep ledger order.show SEQreturns the entry, its protocol (protocol_id,version,protocol_sha256,freeze_seq; anissueentry has the protocol of its record),points_at(thedeviationsof a record, or therecord_seqof an issue) andpointed_by(the entries that point at it). An unknownseqisDANGLING_REF.summaryreturns, as one object:ok,head(count,hash,seq),entries_by_type,records_by_status,protocols(one row perprotocol_id:versions,latest_version,records,latest_status,deviations,cards) andproblems(the violations, eachcode,seq,message). Unlike other reads it does not fail when the ledger does not verify:okis false andproblemssays why (CLI exit 1, MCPisError). Entries whose protocol was never frozen are not tabulated.--protocollimits the table and the counts to one protocol.initcreates the store with an empty ledger; repeating it changes nothing. An empty ledger verifies.
5.2 Capture¶
capture returns {"environment": {"machine", "platform", "protocolith", "python"}, "inputs": [{"path", "sha256", "size"}]}
to be pasted into the free-form outcome of a record. inputs lists each named file with its path relative to the
working directory (/ separators), size in bytes and sha256. environment is included with --env and whenever no
file is named. It executes nothing and reads no environment variable values. A file outside the working directory (after
resolving links), a missing file or a non-regular file is BAD_PATH. The MCP tool takes files and env.
5.3 Schemas and MCP resources¶
schema prints JSON Schema (draft 2020-12) for freeze (the protocol object), record, deviation, issue (the input
{"seq": n}) and entry (the ledger line; $defs hold the stored body of each type), or all. The schemas are
generated from the same field rules the verifier uses, so they cannot drift from it; conditional requirements
(block_reason, drift_reason, non-empty sources) are included. A schema cannot express every rule (references
between entries, hashes, ordering); the verifier is authoritative.
The MCP server lists and reads these read-only resources (mimeType application/json): protocolith://summary,
protocolith://ledger (all entries and the head) and protocolith://schema/<name> for each schema name above except
all. An unknown URI is JSON-RPC error -32002; a failure such as NO_LEDGER is -32000 with the code in the message.
5.4 Error hints¶
An error may carry a one-line hint: printed as hint (CODE): ... on stderr and, in JSON, as error.hint. Hints are
advice, not part of the contract; codes and exit codes are.
5.5 Atlas¶
atlas writes one self-contained HTML file (default protocolith-atlas.html; --out is a .html file or a directory).
Sources: a ledger (--store, verified like verify-ledger; only record entries become rows, with the card derived as for
issue) and/or evidence card JSON files found recursively under each --cards DIR. With neither option a local
ledger is used, else docs/evidence_cards. A file that is not valid UTF-8 JSON, not an object, larger than 2 MiB, or lacking
a string evidence_id and result_status is skipped and counted (issues); unknown fields are kept. Equal
evidence_id values keep the first record. Cards read from files are not verified against any ledger.
- Failure semantics. A ledger that does not verify still produces the atlas, with a red integrity banner and the
problems; the command exits
1. Other errors keep the usual codes (2input,3environment); nothing is written then. - Page. Inline CSS and JavaScript, the data as
<script type="application/json" id="atlas-data">, no network request, no external font or library, a Content-Security-Policy meta element that forbids connections. The JSON is ASCII with<,>and&written as\uescapes, so no content can end the script block. The script writes data withtextContentand DOM calls only. The same input gives the same bytes; the page holds no wall-clock value. - Data size. A row holds the evidence id, protocol, status, domain, reproduction level, date, claim boundary, hash and
a source text; every other field travels deflate-compressed in blocks of 128 records and is decoded when a record is
opened (a browser without
DecompressionStreamshows the row fields only). Strings are clipped at 2000 characters, lists at 100 items and nesting at 8 levels.--max-rows Nembeds only the newest N records; totals stay complete. - Statuses. The six statuses of this specification have a letter and a symbol in addition to a colour; any other status is displayed as found and never hidden or reclassified.
- Languages.
atlas_i18n/en.jsonis the English catalog (every visible string has a key).--i18n DIRadds every<code>.json({"_name": "Native name", "key": "text"}); unknown keys and non-string values are ignored with a warning, empty values and missing keys fall back to English, and the language switcher appears only when one is present.--default-langmust name an available language (BAD_LANG). No translation ships in the package.
6. Conformance vectors¶
spec/vectors/ holds language-neutral JSON files: canonicalization cases (input text, expected bytes or error),
hash cases, and ledgers (each with the expected verdict and codes). Every rule R1 to R11 has at least one failing vector.
A change to the vectors is a change to the specification and needs a version bump.
7. Versioning¶
The ledger format version v changes only on incompatible changes. The core API follows semantic versioning from 1.0.
Version 1.0.0 additionally requires an independent external rerun and a second implementation that passes the vectors.
8. Relation to evidence cards¶
An issue entry renders the existing evidence-card JSON and SVG from a verified record, so the registry, validator and
static views keep working. Cards created before this specification stay valid under the card validator and are mapped to
their current ids by docs/MIGRATION_ID_MAP.json.
9. Limits¶
- One ledger line is at most
MAX_ENTRY_BYTES = 1,048,576bytes (without the newline). A longer line is not parsed (ENTRY_TOO_LARGE). - JSON nesting depth is at most 64, counting the top-level container as 1 (
TOO_DEEP). protocol_id(at most 128 characters) andversion(at most 64 characters) are filename-safe:[A-Za-z0-9][A-Za-z0-9._+-]*(+is allowed inversiononly).- Lock waits time out after 30 seconds (
LOCK_TIMEOUT).
10. Issuing cards¶
record.body may hold an optional card object with extra evidence-card fields (operator, source name and URL, domain,
claim type, and so on). issue builds the card from the record, the frozen protocol and card, validates it with the
evidence-card validator and renders JSON and SVG.
evidence_idisPLITH-<protocol_id>-<date of the record>-<seq>, whereseqis the ledgerseqof the record as six digits (more if needed), so two records of one protocol on one day each get their own card;registry_sequencedefaults to the recordseq+ 1.- A
cardkey that contradicts a value derived from the record isCARD_CONFLICT; missing required card fields areMISSING_FIELDand are named; any other validator failure isCARD_INVALID. - The card file is the PCJ-1 bytes of the card with no trailing newline, so the file's SHA-256 equals
card_sha256. - Issuing twice writes the same bytes; if the file exists with different content the result is
CARD_EXISTS. raw_payload_committedis always false.
11. Error codes¶
Verification: BAD_SEQ, BAD_PREV, BAD_HASH, NOT_CANONICAL, TS_REGRESSION, UNKNOWN_PROTOCOL, DUPLICATE_FREEZE,
BAD_STATUS, MISSING_FIELD, DANGLING_REF, BAD_CARD, UNKNOWN_VERSION, TRUNCATED_TAIL, ENTRY_TOO_LARGE, TOO_DEEP,
BAD_ENTRY (structure or type problem; bodies are closed, so unknown keys are rejected), HEAD_MISMATCH (--expect-head check failed).
Canonicalization and input: BAD_JSON (including a byte-order mark), BAD_UTF8, BAD_TYPE, FLOAT_NOT_ALLOWED, NON_FINITE,
INT_RANGE, DUPLICATE_KEY, LONE_SURROGATE.
Operations and environment: BAD_PATH, NO_LEDGER, NO_PARTIAL_TAIL, LOCK_TIMEOUT, IO_ERROR, BAD_CLOCK, CARD_EXISTS, CARD_CONFLICT,
CARD_INVALID, VERIFICATION_FAILED.
Chain checks: prev is compared with the stored hash of the previous line, so tampering shows as BAD_HASH on that line only;
after an unparseable line the next prev is not checked, to avoid cascades. supersedes must name an earlier freeze of the same
protocol_id; a referenced deviation must belong to the same protocol and be listed in the record's deviations.
repair --truncate-partial-tail removes only an unterminated last line and refuses if the rest does not verify.
12. Coverage note¶
The reference implementation is held at 100% line and branch coverage of protocolith.core. The two platform-specific lock
modules are marked # pragma: no cover as a whole because each runs only on its own platform; both are exercised in CI.