dat-thang/docs/primitives.md

datgpt primitives

This is the working map of the old dat:// implementation. Names follow the project: SLEEP owns feeds, cesspool owns Hyperswarm-style discovery, and dat joins the feeds into an archive.

The standalone libcess package owns Cesspool discovery and UDP transport. Both the Dat (ddat) and Cable (cable) applications link it.

SLEEP storage

• Feed header, tree nodes, signatures, bitfield pages, keys, and data files.
• Flat-tree addressing, spans, roots, leaf and parent hashing.
• Feed open, length, indexed read, append, flush, and close.
• Public-key-only in-memory read-only feeds accept verified remote blocks and retain their proofs without pretending the local node has signing keys. A .remote cache reopens sparse blocks only after verifying each stored proof.
• Data/tree bit access, index regeneration, scan, encode, and decode.
• Key generation, public/secret persistence, hashes, signatures, root signing.
• Merkle proof nodes, proof encode/decode, hash folding, and root signature check.

Archive

• Paired metadata and content feeds.
• Create, open, close, public keys, writable keypairs.
• Separate secret-key save and writable reopen.
• Header pointing from metadata to content.
• File, directory, symlink, and deletion history entries.
• Entry enumeration, reverse path lookup, and content block lookup.
• Recursive filesystem import for regular files, directories, and symlinks.
• Files are chunked across content blocks and can be exported recursively.
• Snapshot sync compares content, mode, target, and millisecond mtime before appending; paths removed locally receive tombstones, while unchanged paths do not extend the feed.
• dat status, dat files, dat verify, dat resolve, historical cat, checkout, and versioned URL commands are available from the CLI.

Peer protocol

• The root NETWORK_PROTOCOL selects legacy V6 framing (hypercore) for the old Dat target; cesspool selects V7 framing for the in-tree network mode.
• Protobuf feed-open messages, handshake id/live/userData, request/range/data messages, extension payloads, and standard DATA node/signature fields.
• Have/want ranges, block requests, cancellation.
• Legacy WANT messages with omitted length run to the peer's current feed end; responses are bounded by the local feed length and representable bitfield.
• Data plus Merkle proof, and named extensions.
• Per-channel replication state, remote bitfield, and request queue.
• dat_session joins one archive key to its discovery topic, carries discovered peer addresses, opens the archive feeds, gates the wire handshake on an externally authenticated transport, exchanges HAVE ranges, requests missing blocks, and verifies signed proofs before caching received blocks.
• A peer callback exposes newly discovered or rediscovered peers so a transport adapter can dial them and call dat_session_peer_secure after authentication.
• dat_session_poll runs an optional transport poll callback alongside the Cesspool UDP poll, refreshes topic membership, and reconciles discovered peers.
• dat_session_tick is the nonblocking form of that poll cycle.
• dat_session_receive accepts arbitrary byte-stream chunk boundaries, buffers partial frames, decrypts the V6 stream at byte offsets, and dispatches multiple coalesced frames from one read.
• A read-only replica keeps received proofs and can serve them onward. The caller still supplies the authenticated byte transport and poll loop.

cesspool

• UDP socket transport, bootstrap nodes, listen, poll, and destroy.
• Legacy lookup, announce, unannounce, ping, and holepunch vocabulary.
• Topic join/leave and periodic refresh.
• Peer table, failure/last-seen state, topic table, discovery callbacks.
• Legacy protobuf requests/responses and compact public/local peer parsing.
• XOR-distance peer selection and dht-rpc transaction tracking.

uTP

• Header encode/decode and connection identifiers.
• SYN/connect/accept, state acknowledgements, data, FIN, and reset states.
• Send segmentation and timer boundary.
• Receive reordering, cumulative and selective acknowledgements, congestion window, retransmission timers, and in-flight packet retirement. DATA packets are capped at the 1200-byte payload size, and malformed control packets are rejected before entering the reorder buffer.

Name resolution

• dat-dns resolution through HTTPS /.well-known/dat, plus the local ~/.dat/dat-dns name map and TTL-aware cache. The TXT packet decoder is a compatibility helper and is not used for runtime name resolution.
• Canonical dat://<key> output; the domain remains a name for the resolved archive key rather than a separate archive identity.

Later layer

• S7-facing records use one tagged, named-field form: (dat:wire-message (channel . N) (type . N) (payload . #u8(...))). dat:record-ref and dat:record-set provide the shared field access layer; protocol payload bytes remain bytevectors.
• dat-wire->sexp and dat-sexp->wire adapt the existing C frame codec. This keeps the C codec authoritative for peer bytes while Scheme is the common application representation; a Scheme record is never serialized directly onto the network.
• Initial S7 bindings expose archive create, open, close, entry count, and the wire-record adapter. The Scheme adapters and JSON reader are part of the default install.
• Hyperdrive's JS Index and Stat records should use the same named-field convention as their read/write adapters are added; they must retain their original protocol field numbers and binary fields.
• The local CLI sits above the archive API; peer transport polling and the authenticated stream layer remain separate callers of dat_session.