dat-thang/docs/scheme-interop.md

Scheme records at protocol boundaries

Representation contract

The old Dat/Cabal implementations use JavaScript objects and JSON because they run on Node. This implementation uses S7 Scheme values and tagged S-expressions as its native application representation. Keep records, configuration, and the C-to-client boundary in that form; translate to JSON only at a compatibility edge whose existing peer format requires JSON. Dat wire bytes and Cabal's canonical Hypercore JSON remain their established protocol formats.

S7 is the application view for structured records that would otherwise be assembled as JavaScript objects or JSON-shaped maps. New archive descriptions are stored as dat.scm association lists. Existing wire codecs, feed files, and peer protocol bytes retain their established formats; dat.json is not written for newly created archives.

Record form

Records are tagged lists with named association fields:

(dat:wire-message
  (channel . 1)
  (type . 3)
  (payload . #u8(...)))

dat:record-ref reads a named field and accepts an optional fallback. dat:record-set returns a record with one field replaced or added. Names are symbols; strings stay strings; protocol integers stay exact integers; binary data stays an S7 byte-vector. Absence is represented by no field, not by conflating an absent value with #f or an empty byte-vector.

Wire boundary

dat-wire->sexp and dat-sexp->wire adapt the C dat_wire frame codec. The frame record keeps the channel, type, and opaque payload bytes named and inspectable. The active C codec remains authoritative for varints, framing, message payloads, channel assignments, protocol identity, and encryption. S-expressions are not sent to Dat peers and do not alter feed blocks or signatures. Cable's local UI and room-event boundary uses S7 and S-expressions inside the native C application; it does not start a Node adapter.

Protocol-specific adapters can add named records such as dat:feed-open or dat:hyperdrive-stat above this frame boundary. Each adapter should map explicitly to the existing C type or legacy field numbers, preserve unknown fields when forwarding is required, and keep byte strings as byte-vectors. This allows Cesspool and test-only legacy Hypercore views to share Scheme-level record operations while retaining their separate codecs and wire identities.

Build and use

S7 is bundled and required by the default build. The build produces ddat-scheme and links the runtime into the application library. With no arguments it starts the REPL; -e evaluates an expression, -l loads a file, -I adds a module directory, and --repl enters the REPL after scripts run. Dat's host runtime registers the C bindings, assembles the module search path, and loads dat-init.scm, dat-record.scm, dat-wire.scm, and dat-json.scm before user code. dat:json-parse accepts strict JSON and returns tagged Scheme objects/arrays while preserving JSON strings, booleans, numbers, and null. dat:json-stringify maps those same tagged Scheme values back to JSON at compatibility boundaries. Scheme/S-expression records remain the application representation; JSON is emitted only where an existing peer format requires it. The test-only Cabal harness owns Cabal's canonical JSON Hypercore encoding.

The runtime searches DAT_SCHEME_PATH (a platform-separated list), DAT_SCHEME_DIR, the source-tree scheme/ directory, and the configured installed directory. Application modules map feature names such as dat:archive to dat/archive.scm, load through (dat:require 'dat:archive), and finish with (provide 'dat:archive). dat:with-archive closes an archive even when its body exits through a Scheme condition. Archive values are S7 foreign objects: explicit close is idempotent, and garbage collection closes an archive that the caller abandons.

This is the embedding boundary S7 expects: the host owns startup, load paths, native types, resource lifetime, CLI policy, and application modules. The C wire/feed implementations still own protocol bytes and on-disk formats.