Tail

A feed is a snapshot: whatever a source is showing right now. Everything in Neurowire can be used that way, one fetch at a time, and for most jobs that is the right shape. Tail is the other posture: keep the source open and be told what is new, the way tail -f follows a file.

Nothing about the data changes. A tail is the same canonical model, the same taps, the same meshes and constructs. What changes is who is waiting for whom.

One engine, three surfaces

The polling loop lives in one place, pollFeed in @neurowire/ingest. Everything that follows a source consumes it, so none of them can drift on cadence, dedupe, or what happens when a fetch fails.

One loop per source, whatever is watching it. On the API that sharing is literal: fifty clients following the same target cost one upstream fetch per tick, not fifty.

Surface	What it emits	Where
neurowire tail	one terminal line per entry as it arrives, or raw NWFJ with -f nwf	CLI tail mode
neurowire --watch	one feed per tick, in --format, containing only that tick's new entries	CLI watch mode
GET /tail	server-sent events, one per entry, fanned out to every connected client	HTTP API

The engine itself is an async generator and owns no I/O. You hand it a load function (a fetchFeed, a fetchMesh, a whole filter pipeline) and it hands back the entries that are new:

import { fetchMesh, pollFeed } from '@neurowire/ingest'

for await (const { fresh } of pollFeed(() => fetchMesh(mesh), { intervalMs: 60_000 })) {
  for (const entry of fresh) console.log(entry.title, entry.link)
}

What "new" means

New means "not in the seen-set", and the seen-set is keyed by entryKey: the entry's id, or its link when it has none. That is the same key watch state and journal dedupe use, so the three agree by construction.

Two consequences worth knowing:

• Order does not matter. A source that reorders its front page, or backfills an old post, produces no noise. Only a key the loop has not reported before is new.
• The first tick is not special. It runs immediately, with an empty seen-set, so a fresh tail reports the whole front page and then goes quiet. Seed the set (--state on the CLI, a journal on the server) when you want a restart to pick up where it left off instead.

Shaping happens before dedupe

On the CLI, each tick fetches, then applies your --filter, --since, --sort, and --limit flags, and only then diffs against the seen-set. An entry that a filter excluded was never reported, so it is still new if a later tick lets it through. That is what makes tail --mesh ai.json --filter tag:release behave the way you would expect over hours.

Politeness is built in, not left to you

Following a source means asking for it repeatedly, so the loop is deliberately conservative:

Guard	Value	Why
Interval floor	30s on the CLI, 60s on the API	A client cannot talk the loop into hammering an upstream.
Jitter	up to +10% of the interval, added, never subtracted	Many tails on one host stop arriving in lockstep. The interval stays a floor.
Conditional requests	ETag / Last-Modified per source	An unchanged source costs a 304 rather than a body. See Fetching.
Shared loops	one poll per distinct target on the API	Fifty browsers following ai-news cost one upstream fetch per tick, not fifty.

Pick an interval that suits the source rather than the floor. A blog that posts weekly does not need a 30 second tail, and the floor is a safety rail, not a recommendation.

A failed tick is not a failed tail

A long-running loop that dies on the first flaky response is not much use. load throwing is isolated to its tick: the error is reported, the loop waits, and the next tick tries again. That sits on top of the retry and backoff fetching already does inside a single tick, so a transient upstream problem is usually handled before the loop ever sees it.

[tail] error: Upstream responded 503 Service Unavailable for https://example.com/feed.xml

Cancellation is explicit and immediate: an AbortSignal ends the generator between ticks and mid-sleep, which is how the API tears a loop down the moment its last subscriber disconnects.

Composing with journals

A tail and a journal answer the two halves of the same question. The tail says what is arriving; the journal says what arrived. Put them together and a reader can drop off the wire without losing anything.

A tail stream is a journal being written. tail -f nwf does not invent a second streaming encoding: it emits NWFJ, the same append-only format the on-disk store writes, because that format is already designed to be appended to one entry at a time. The output is a complete document, header, dictionary growth, and checkpoints included, so anything that reads an archive reads the stream.

neurowire tail --mesh ai-news.json -f nwf > live.nwfj

Cursors make resume lossless. Give the API a journal that already exists and the route writes through it, so each event's SSE id is a real cursor. A client that reconnects with Last-Event-ID is replayed everything after that cursor and then put back on the live stream:

curl -N "http://localhost:8787/tail?src=ai-news&journal=ai"
# ...connection drops at id 812, reconnect...
curl -N -H 'Last-Event-ID: 812' "http://localhost:8787/tail?src=ai-news&journal=ai"

Without a journal the tail is live-only: event ids are stream-local, resume has nothing to replay from, and the init event says so rather than pretending otherwise. The route never creates a journal of its own, so replay is something an operator turns on, not something a client can ask for.

Tail, watch, or a plain fetch

You want	Use	Because
A feed, a file, a page, right now	a one-shot fetch	Nothing needs to stay running. Put it on cron if you want it repeated.
To see posts appear while you work	neurowire tail	Line-per-entry output, readable as it scrolls.
To pipe new entries into another tool	neurowire tail -f nwf	A stream of parseable records with no terminal formatting in it.
A batch of new entries per interval, serialized	neurowire --watch -f json	One feed document per tick is what a downstream script wants to parse.
To notify a channel	--watch or tail with --sink	Delivery of only the new entries. See Sinks.
To fan one source out to many readers	GET /tail	One upstream poll, many subscribers, resume for free with a journal.

Cron is often the right answer

A tail holds a process open. If you only need a page rebuilt every morning or an archive topped up every hour, a one-shot fetch on a timer is simpler, survives reboots, and cannot leak a process. Reach for a tail when something is actually waiting on the output.

What tail is not

Not a delivery guarantee

The loop reports what a source is showing when it looks. A post that appears and is deleted between two ticks is never seen, and there is no acknowledgement, no redelivery, and no ordering promise beyond the order the source lists. Journal-backed resume closes the gap for a client that disconnects; it does not turn polling into a message queue.

It is also polling by design. There is no WebSub or webhook ingestion, so nothing pushes to Neurowire, and there is no WebSocket transport: SSE covers the fan-out case over plain HTTP, through proxies, with reconnection already specified. Exchanging journal deltas between machines, so a second node pulls from a peer instead of re-fetching every upstream site, is planned separately.

See CLI tail mode for the flags, GET /tail for the event stream, and the poll engine reference for the library API.