Fetching

Every source Neurowire reads, a feed, a website, a mesh member, goes through one HTTP client: fetchDocument, in packages/ingest/src/fetch.ts. It is built for fetching feeds from untrusted URLs on a schedule, so it handles timeouts, retries, conditional requests, and redirect-based SSRF in one place.

import { fetchDocument, createMemoryCache } from '@neurowire/ingest'

const doc = await fetchDocument('https://example.com/feed.xml', {
  timeoutMs: 15000,
  retries: 2,
  cache: createMemoryCache(),
})

fetchDocument returns a RawDocument: the final url (after redirects), the contentType, the body, optional etag / lastModified validators, and a notModified flag set when the body came from cache via a 304.

Document detection

Once a document is fetched, detectKind(contentType, body) classifies it before any parser runs: the Content-Type first, then a body sniff when the type is unhelpful (many hosts serve feeds as text/plain or application/octet-stream).

Kind	Matched by
nwf	text/x-neurowire, or an NWF1 first line
atom	application/atom+xml, or a <feed root
rss	application/rss+xml, or an <rss root
rdf	An rdf:RDF root
jsonfeed	application/feed+json, or a JSON body naming jsonfeed.org or items
html	Everything else, which then goes to a tap or auto-detect

Anything but html is parsed directly. nwf is in that list because Neurowire reads its own format back: a published .nwf file is a source like any feed, including as a mesh member. See the NWF format.

FetchOptions

Option	Type	Default	Purpose
timeoutMs	number	15000	Per-attempt deadline. Set 0 to disable.
retries	number	2	Max additional attempts after the first.
backoffMs	number	500	Base delay for exponential backoff.
signal	AbortSignal	none	Caller-driven cancellation.
cache	ConditionalCache	none	Store for ETag / Last-Modified conditional requests.
validate	(url) => void | Promise<void>	none	Per-hop guard (throw to block). See SSRF below.
headers	Record<string, string>	none	Extra request headers, e.g. authorization for a private feed. See caller headers below.
delay	(ms, signal) => Promise<void>	setTimeout-based	Injectable sleep (tests drive it with fake timers).

Retry policy

Each attempt runs the full manual redirect loop. On failure the request is retried up to retries times. The classification is deliberate:

Retried (a fresh attempt might recover):

• Network-level failures (DNS, connection reset), which fetch surfaces as a TypeError.
• Internal timeout aborts (the per-attempt deadline).
• Upstream 5xx responses.
• 429 Too Many Requests.

Not retried (a retry cannot help):

• 4xx other than 429.
• An invalid or unsupported-protocol URL.
• An SSRF rejection from the validate guard.
• Too many redirects (more than 5 hops).
• A caller-driven abort via signal (rejects immediately, no further attempts).

Honoring Retry-After

A 429 carries its Retry-After header, parsed (delta-seconds or an HTTP date) into milliseconds. When present, the retry loop waits that long instead of the computed backoff, capped at 30 seconds so a hostile or huge Retry-After cannot stall the process forever.

Jittered backoff

When there is no Retry-After, the wait is full-jitter exponential backoff: base * 2 ** attempt, multiplied by a random factor in [0.5, 1.0), capped at 30 seconds. Jitter spreads out retries so many sources failing at once do not all retry in lockstep.

Conditional cache (304 handling)

Pass a ConditionalCache and fetchDocument will make conditional requests. After a successful 200, it stores the body together with the response's ETag and Last-Modified. On the next fetch of that URL it sends:

• If-None-Match from the stored ETag.
• If-Modified-Since from the stored Last-Modified.

If the server replies 304 Not Modified, the cached body is returned with notModified: true. A 304 is a success and is never retried. The cache is written only on a final 200.

Conditional headers apply only to the originally requested URL (the first hop), not to redirect targets, and they are re-sent on every retried attempt.

createMemoryCache() returns a simple Map-backed ConditionalCache. The cache is always injected by the caller; the library keeps no global state, so you control its lifetime and scope.

const cache = createMemoryCache()
await fetchDocument(url, { cache }) // 200, body cached
await fetchDocument(url, { cache }) // may return { notModified: true }

Redirects and the per-hop SSRF guard

Redirects are followed manually, one hop at a time, up to 5 hops (Location is resolved against the current URL). This exists so the optional validate guard runs against every target, not just the URL the caller passed in.

This closes the classic SSRF bypass where a public URL 302-redirects to an internal address (http://169.254.169.254/, http://localhost/, a private RFC 1918 range). validate is called for the initial URL and for each redirect target before the request is made. Throwing from validate blocks that hop, and the rejection is not retried.

WARNING

If you fetch user-supplied URLs, supply a validate that rejects private, loopback, and link-local addresses. Because redirects are re-validated per hop, a public URL cannot redirect its way into your internal network.

Only http: and https: are allowed; any other protocol throws an unsupported-protocol error.

Request headers

Every request sends a stable identifying User-Agent:

Neurowire/0.1 (+https://github.com/neurowire/neurowire)

and an Accept header that prefers feed formats, then JSON, then HTML:

application/atom+xml, application/rss+xml, application/feed+json,
application/json;q=0.9, text/html;q=0.8, */*;q=0.5

Caller headers and redirects

FetchOptions.headers adds request headers, which is how a private feed gets its Authorization. Header names are lowercased. A caller header overrides the default User-Agent and Accept, but never the conditional If-None-Match / If-Modified-Since headers: those belong to the cache, and a caller value for them is ignored.

Because redirects are followed manually, each hop builds its headers afresh, and credential headers get the same treatment browsers give them: authorization, proxy-authorization, and cookie are sent only while the hop's origin (scheme, host, and port) matches the origin of the URL you asked for. A 302 to another host, a scheme change, or a port change drops them; every other caller header still goes along. Without this, a token meant for api.example.com would ride along to whatever host the redirect names.

fetchFeed applies the same rule to a discovered feed link: a page on one origin that advertises its feed on another gets that feed fetched without credentials.

Headers are not part of the conditional-cache key (the key is the URL), so do not share one ConditionalCache between callers with different credentials for the same URL.

Cancellation

The caller's signal is composed with the internal per-attempt timeout signal, so either can cancel an in-flight request. A caller abort is distinguished from a timeout: the timeout is retryable, the caller abort is not.