dat-thang/docs/beaker-networking.md

Beaker networking notes for dat-gpt

These notes use the archived beakerbrowser/beaker 1.1.0 source and the included old dillo-dat plugin as references. Beaker was archived in 2022. Its later Hyperdrive network stack is useful for subsystem boundaries, but it is not proof of compatibility with the old Dat protocol or a replacement for the Dillo plugin's hyperdrive + hyperdiscovery path.

What Beaker separates

1. Name resolution. URLs are parsed first. A 64-character key is used directly; a domain requires a resolver. Beaker has a dat-dns based resolver and local name/key maps. Its v1.1 HyperDNS implementation leaves the remote DNS logic disabled, and its dat:// handler calls a separate dat-dns resolver. Name resolution therefore belongs above transport and should return the archive key before opening feeds.
2. Drive sessions and persistence. Beaker's later stack talks to a long-lived Hyperspace process through hyperdrive-daemon-client. Sessions are keyed by archive key plus version and read/write mode. The daemon owns the storage and network handles; Beaker expires idle sessions and reconnects them if the daemon restarts.
3. Announce versus lookup. Beaker explicitly configures a drive with announce: true, lookup: true when it should be hosted and discover peers. Merely opening a drive and resolving its key are separate operations.
4. Archive reads. The newer hyper:// handler resolves the host to a key, opens/loads the drive (which starts network lookup), checks out an optional version, reads its manifest, resolves the path, and streams the selected file from the drive. It supports GET/HEAD, byte ranges, redirects for directory URLs without a trailing slash, manifest MIME overrides, and content sniffing. A drive with version zero becomes a clear “no peers found yet” response, so lookup-in-progress is distinct from a missing path. The separate legacy dat:// handler resolves through dat-dns, runs the legacy conversion/download tool into temporary storage, and then serves that filesystem through the same path resolver. That conversion route is materially different from the native Hyperdrive read path.
5. Operational visibility. Beaker has separate debug categories for the application, Dat, DHT discovery, DNS discovery, and the Hypercore protocol. That separation makes it possible to tell whether a failure is lookup, connection, replication, or file resolution.

What applies to this C implementation

• Preserve the existing NETWORK_PROTOCOL switch and cesspool spelling. The default Cesspool build uses the V7 message format. hypercore selects the old Dillo dependency's V6 channels, nonce-bearing feed handshake, and XSalsa20 stream encryption. The CSP0 UDP discovery packet format is still separate from the old JS hyperdiscovery format.
• Keep hostname resolution, discovery, authenticated peer sessions, feed replication, and browser path handling as separate layers. The Dillo DPI helper already consumes the URL/path and local archive layers; its cache lookup is not yet a remote archive loader.
• The archive session joins a key's topic, consumes topic-tagged Cesspool announcements, carries peer feed channels, exchanges HAVE/WANT state, requests blocks, and verifies/forwards signed proofs. The UDP discovery packet format still needs old JS hyperdiscovery interoperability work; Noise authentication and socket-to-session transport integration are separate networking layers.
• Beaker's daemon restart/session handling suggests a lifecycle boundary for the C version: one archive session owns the discovery membership, peer set, replication state, pending block requests, and close/retry behavior. Keep this below both the CLI and Dillo DPI callers so they can share it.
• dat-dns names should be resolved as a mapping to a key, not treated as a second archive identity. Cache the mapping separately from archive content; keep the resolved key visible for debugging and canonical dat://<key> links.
• dat_dns_resolve follows the old dat-dns URL-to-key contract. The bundled Dillo plugin disables DNS-over-HTTPS and resolves through HTTPS /.well-known/dat; the C resolver follows that same path after checking its local alias map. It does not call the operating system's DNS TXT resolver. The record contains a dat://<64-hex-key> line and optional TTL=<seconds>. This lookup returns a key only; archive discovery and transfer remain separate Cesspool/session operations.

Suggested implementation order

1. Define an archive network-session API with explicit open, announce, lookup, peer-up, peer-down, request-block, block-arrived, and close transitions. Keep transport details behind the existing cesspool boundary.
2. Make one peer connection carry a real authenticated protocol handshake and identify the metadata and content feeds. Do not start with browser DNS or directory listing behavior.
3. Connect remote metadata block requests to feed verification and archive reads. A cold dat://<key>/index.html should be able to wait for and serve the requested bytes, while earlier verified blocks remain cached.
4. Add retry, peer removal, and session cleanup. Expose peer and missing-block state through a diagnostic command before adding a dashboard.
5. Add name resolution and any Beaker-style convenience behaviors only after key-addressed remote reads work.

Source references

• Beaker 1.1.0 app/package.json: dat-dns, discovery-swarm, hyperspace, hyperdrive-daemon-client, and pauls-dat-api2 dependencies.
• app/bg/hyper/daemon.js: daemon setup/reconnect, per-drive sessions, peer addresses, and session cleanup.
• app/bg/hyper/drives.js: key/domain resolution, network configuration, drive loading, manifest reads, and version checkouts.
• app/bg/hyper/dns.js: local name/key map; remote DNS implementation is commented out in this archived revision.
• app/bg/protocols/dat.js: legacy dat:// URL host resolution, local legacy download/conversion to temporary storage, path resolution, MIME selection, and response handling.
• app/bg/protocols/hyper.js: native Hyperdrive network open, historic checkout, path resolution, no-peer state, range reads, and browser response behavior.
• Beaker's repository README identifies version 1.1.0 as the last release and the repository as archived; its debug namespaces include dat, bittorrent-dht, dns-discovery, and hypercore-protocol.