dat-thang/docs/primitives.md

# datgpt primitives

This is the working map of the old `dat://` implementation. Names follow the
project: SLEEP owns feeds, cesspool owns Hyperswarm-style discovery, and dat
joins the feeds into an archive.

The standalone [libcess package](libcess.md) owns Cesspool discovery and UDP
transport. Both the Dat (`ddat`) and Cable (`cable`) applications link it.

## SLEEP storage

- Feed header, tree nodes, signatures, bitfield pages, keys, and data files.
- Flat-tree addressing, spans, roots, leaf and parent hashing.
- Feed open, length, indexed read, append, flush, and close.
- Public-key-only in-memory read-only feeds accept verified remote blocks and
  retain their proofs without pretending the local node has signing keys. A
  `.remote` cache reopens sparse blocks only after verifying each stored proof.
- Data/tree bit access, index regeneration, scan, encode, and decode.
- Key generation, public/secret persistence, hashes, signatures, root signing.
- Merkle proof nodes, proof encode/decode, hash folding, and root signature check.

## Archive

- Paired metadata and content feeds.
- Create, open, close, public keys, writable keypairs.
- Separate secret-key save and writable reopen.
- Header pointing from metadata to content.
- File, directory, symlink, and deletion history entries.
- Entry enumeration, reverse path lookup, and content block lookup.
- Recursive filesystem import for regular files, directories, and symlinks.
- Files are chunked across content blocks and can be exported recursively.
- Snapshot sync compares content, mode, target, and millisecond mtime before
  appending; paths removed locally receive tombstones, while unchanged paths
  do not extend the feed.
- `dat status`, `dat files`, `dat verify`, `dat resolve`, historical `cat`,
  `checkout`, and versioned URL commands are available from the CLI.

## Peer protocol

- The root `NETWORK_PROTOCOL` selects legacy V6 framing (`hypercore`) for the
  old Dat target; `cesspool` selects V7 framing for the in-tree network mode.
- Protobuf feed-open messages, handshake id/live/userData, request/range/data
  messages, extension payloads, and standard DATA node/signature fields.
- Have/want ranges, block requests, cancellation.
- Legacy WANT messages with omitted length run to the peer's current feed end;
  responses are bounded by the local feed length and representable bitfield.
- Data plus Merkle proof, and named extensions.
- Per-channel replication state, remote bitfield, and request queue.
- `dat_session` joins one archive key to its discovery topic, carries discovered
  peer addresses, opens the archive feeds, gates the wire handshake on an
  externally authenticated transport, exchanges HAVE ranges, requests missing
  blocks, and verifies signed proofs before caching received blocks.
- A peer callback exposes newly discovered or rediscovered peers so a transport
  adapter can dial them and call `dat_session_peer_secure` after authentication.
- `dat_session_poll` runs an optional transport poll callback alongside the
  Cesspool UDP poll, refreshes topic membership, and reconciles discovered peers.
- `dat_session_tick` is the nonblocking form of that poll cycle.
- `dat_session_receive` accepts arbitrary byte-stream chunk boundaries, buffers
  partial frames, decrypts the V6 stream at byte offsets, and dispatches multiple
  coalesced frames from one read.
- A read-only replica keeps received proofs and can serve them onward. The
  caller still supplies the authenticated byte transport and poll loop.

## cesspool

- UDP socket transport, bootstrap nodes, listen, poll, and destroy.
- Legacy lookup, announce, unannounce, ping, and holepunch vocabulary.
- Topic join/leave and periodic refresh.
- Peer table, failure/last-seen state, topic table, discovery callbacks.
- Legacy protobuf requests/responses and compact public/local peer parsing.
- XOR-distance peer selection and dht-rpc transaction tracking.

## uTP

- Header encode/decode and connection identifiers.
- SYN/connect/accept, state acknowledgements, data, FIN, and reset states.
- Send segmentation and timer boundary.
- Receive reordering, cumulative and selective acknowledgements, congestion
  window, retransmission timers, and in-flight packet retirement. DATA packets
  are capped at the 1200-byte payload size, and malformed control packets are
  rejected before entering the reorder buffer.

## Name resolution

- `dat-dns` resolution through HTTPS `/.well-known/dat`, plus the local
  `~/.dat/dat-dns` name map and TTL-aware cache. The TXT packet decoder is a
  compatibility helper and is not used for runtime name resolution.
- Canonical `dat://<key>` output; the domain remains a name for the resolved
  archive key rather than a separate archive identity.

## Later layer

- S7-facing records use one tagged, named-field form:
  `(dat:wire-message (channel . N) (type . N) (payload . #u8(...)))`.
  `dat:record-ref` and `dat:record-set` provide the shared field access layer;
  protocol payload bytes remain bytevectors.
- `dat-wire->sexp` and `dat-sexp->wire` adapt the existing C frame codec. This
  keeps the C codec authoritative for peer bytes while Scheme is the common
  application representation; a Scheme record is never serialized directly
  onto the network.
- Initial S7 bindings expose archive create, open, close, entry count, and the
  wire-record adapter. The Scheme adapters and JSON reader are part of the default install.
- Hyperdrive's JS `Index` and `Stat` records should use the same named-field
  convention as their read/write adapters are added; they must retain their
  original protocol field numbers and binary fields.
- The local CLI sits above the archive API; peer transport polling and the
  authenticated stream layer remain separate callers of `dat_session`.