NWFJ (Neurowire Feed Journal)

nwfj is Neurowire's append-only sibling of NWF. Where an nwf document is a snapshot of a feed, a journal is a log: entries are appended over time, every entry carries a sequence number, and a reader can resume from a cursor instead of re-reading the whole file. The encoder, decoder, and query helpers live in packages/core/src/journal.ts; the segmented on-disk store lives in packages/ingest/src/journal-store.ts.

Media type	application/x-nwf-journal
Extension	nwfj
Version	1
Core functions	createJournalEncoder, resumeJournalEncoder, parseJournal, readJournalSince, journalHead, verifyJournal, queryJournal, journalToFeed
Store	openJournalStore({ dir })

A journal is deliberately not an output format: it is not a way to render a feed, it is a way to store its history. serialize() and the FORMATS registry are untouched.

Why NWF1 cannot simply be appended to

An nwf document puts its complete A / T / S dictionaries near the top and stores each entry's date as a delta from the feed-level updated. Appending one entry would mean rewriting the dictionaries and the header timestamp, which breaks append-only semantics and invalidates every cursor pointing into the file. NWFJ keeps the same cell grammar and escaping, and changes only what append-only forces it to change.

Layout

Lines are LF-separated, cells within a line are TAB-separated, and text cells use exactly the same escaping as nwf (backslash, TAB, CR, LF; sub-fields joined by the ASCII Unit Separator, 0x1f).

J   1  <journalId>  <createdEpoch>                     journal header, one per segment
F   <feedId>  <title>  <home>  <self>                  feed identity, re-emitted when it changes
A+  <index>  <author>                                  dictionary growth, interned on first use
T+  <index>  <tag>
S+  <index>  <source>
B   <baseUrl>                                          link prefix, may be re-declared
E   <seq>  <updated>  <published>  <id>  <link>  <authorRefs>  <tagRefs>  <title>  <summary>  <sourceRef>
C   <seq>  <hash>                                      checkpoint, chain value at seq

What changes relative to NWF1, and why:

• Dictionaries grow incrementally. A+, T+, and S+ each declare one interned value with an explicit index, emitted immediately before the first E line that references it. A reader builds the same lookup tables nwf has, just lazily. Indices are per segment and always dense, starting at 0.
• Timestamps are absolute epoch seconds, not deltas from a header that would go stale as the log grows. Both dates are kept in their own cell (updated, published), and either may be - when absent, so a journal round-trips dates losslessly rather than collapsing them into one field.
• Every E line carries a sequence number. seq starts at 1 and increases by one per entry for the life of the journal, across segment rotations. It is the primary index.
• Checkpoints are optional. A C line records the running chain value at a given seq. Readers may verify or ignore it.

Unknown line kinds are reported as issues by the validator and skipped by the parser, so a future version can add record types without breaking older readers.

Cursors

A cursor is a sequence number plus an optional chain hash, written "42" or "42.9f1c0f0b8ad0f0e3". journalHead(text) returns the cursor at the end of a journal, and readJournalSince(text, cursor) returns exactly the records after it. Because seq is global to the journal and never reused, a cursor survives segment rotation: the store maps a seq back to a segment through its manifest.

import { journalHead, readJournalSince } from '@neurowire/core'

const head = journalHead(text)              // { seq: 128, hash: '...' }
const { records } = readJournalSince(text, head)  // [] until something new is appended

The chain

Each record line after the header folds into a running chain value: chain = hashHex(chain + "\n" + line), seeded from the journal id. C lines record that value and are themselves excluded from the fold. verifyJournal(text) recomputes the chain and reports any checkpoint that disagrees, along with the line number.

The chain uses core's FNV-1a hashHex, the same 64-bit hash behind stableId. It detects corruption, truncation, and accidental reordering. It is a checksum, not a cryptographic digest, and it does not prove authorship: anyone who edits a journal can recompute it. Keeping it here preserves core's portability promise (no node:crypto, no dependencies beyond zod). The J line carries a version cell so a future revision can introduce a cryptographic digest and signed checkpoints where a stronger guarantee is needed.

The format is its own index

Two properties give journals indexed access with no index files beside them:

• seq is the primary index. It is monotonic and dense, and each segment covers a contiguous seq range, so finding a cursor means picking a segment and scanning it, never scanning the whole journal.
• Dictionaries are per-segment skip filters. A segment's A+ / T+ / S+ lines declare every author, tag, and source any entry in that segment can reference. A query filtering on tag:rust can therefore skip a segment whose tag dictionary holds no match without decoding a single E line. The vocabulary is written anyway to make the segment self-contained, so using it for pruning is free.

The store keeps a <id>.manifest.json sidecar recording, per segment, its seq range, its date range, its dictionaries, and its entry keys. The manifest is fully derivable by scanning the segments, so a missing or corrupt manifest is rebuilt rather than fatal. It is a cache, never a second source of truth.

Querying

No query language is invented. The existing filter engine is pointed at journal records, so an archive answers exactly what a live feed answers:

import { queryJournal } from '@neurowire/core'

const entries = queryJournal(records, {
  filter: { include: [{ field: 'tag', pattern: 'rust' }] },
  from: Date.now() - 30 * 86_400_000,
  sort: 'date',
  limit: 20,
})

queryJournal composes filterEntries and selectEntries from core, so journal results and live-feed results are identical for the same spec by construction. The store's query(id, spec) adds planning on top: it prunes segments by date range and dictionary vocabulary, then decodes only the survivors.

From the CLI:

neurowire journal query ai --filter tag:rust --since 30d -f md

Because records are one line each and TAB-separated, crude research also works with ordinary text tools:

grep -c '^E' ~/.config/neurowire/journal/ai.00001.nwfj

When research outgrows the built-ins, neurowire journal cat ai -f json hands the whole archive to duckdb, sqlite, or pandas. The journal stays the source of truth and those databases are disposable views.

Segments and rotation

The store writes <id>.<nnnnn>.nwfj segments and rotates to a new one past a size cap (5 MB by default). Each segment repeats the J header and starts its dictionaries fresh, so any segment can be read on its own, and rotation naturally bounds dictionary growth. Compaction drops whole old segments; a cursor pointing into a dropped segment is reported as too old, so the caller can fall back to reading from the oldest retained segment instead of silently missing entries.

Annotated sample

J	1	ai	1782640800
F	https://example.com/feed	Example Blog	https://example.com/	
B	https://example.com/
T+	0	release
E	1	-	1782640800	post-1	~posts/one		0	First post		
S+	0	Example
E	2	-	1782727200	post-2	~posts/two			Second post	A summary	0
C	2	9573a6ab74a4ba94

Line by line (cells are TAB-separated, and consecutive tabs mean an empty cell):

Line	Reading
J	Version 1, journal id ai, created 2026-06-28.
F	The feed identity. The trailing empty cell is self, which this feed did not declare.
B	The link prefix, taken from the feed's home.
T+	The tag release is interned as index 0, right before the entry that first uses it.
E (seq 1)	No updated (-), published at 1782640800, id post-1, link ~posts/one (relative to B), no authors, tagRefs = 0, title First post, no summary, no source.
S+	The source Example is interned as index 0, again just before its first use.
E (seq 2)	Same shape, with a summary and sourceRef = 0. It declares no tags, so its tagRefs cell is empty.
C	A checkpoint: the chain value after sequence 2.

Notice that the second entry reuses nothing from a dictionary it does not need, and that neither entry repeats the feed identity: F and B were emitted once and stay in effect until something changes.

Round-trip guarantees

A journal round-trips the same list essentials nwf does (feed id / title / home / self, and entry id / title / link / updated / summary / authors / tags / source), plus published as its own cell. It does not carry generator, and it does not store feed-level authors: a journal spans many feed identities over its lifetime, so per-entry authorship is the only attribution that stays meaningful after a merge.