Meta overview

What audiosilo-meta is

audiosilo-meta is the community audiobook metadata database behind meta.audiosilo.app. Its defining idea: the GitHub repository is the database. There is no application database and no server-side accounts - metadata lives as plain JSON files under data/, one file per entity, edited by pull request or issue form, validated by Go tooling in CI, and compiled into a single SQLite artifact that consumers download and serve read-only.

The reason it exists is a gap no other open database fills: none is both audiobook-specific and community-editable. Open Library, Wikidata, and BookBrainz are open but carry essentially no narrator, recording, or chapter structure; Audnexus is a read-only cache over Audible's own catalogue. So narrators, specific recordings, and chapters are first-class data here - the fields every other database lacks.

Module path: github.com/kodestar/audiosilo-meta. The code is AGPL-3.0; the data is split across two licence layers (below).

  contributor
      │  pull request  /  issue form
      ▼
  data/*.json  ──►  CI validation  ──►  meta.sqlite  ──►  metaserve (API + site)
  (the database)    (metacheck +        (release          read-only, hot-swaps
                     metafmt, Go)        artifact)         new releases

The two licence layers

Data lives in two layers, and the boundary is enforced structurally by the JSON Schema, not by convention (see LICENSING.md in the repo): the license field of a core record accepts only CC0-1.0, and a sidecar's only CC-BY-SA-4.0 (common.schema.json $defs/license vs $defs/license_content).

Layer	Entities	Licence
Factual core	works, recordings, people, series	CC0-1.0 public-domain dedication
Expressive layer	per-work characters + recaps sidecars	CC BY-SA 4.0

Publisher blurbs and cover art are referenced, never copied: descriptions are community-written and covers are URLs. Every entity also carries a sources[] provenance array so any source can be audited or retracted wholesale. Full detail is in the data model.

Package layout

The tooling follows the same discipline as the rest of the workspace: thin CLIs under cmd/, all logic in reusable packages, and a clean split between the public pkg/* (consumed as ordinary dependencies by the sibling audiosilo-sidecars module and, for pkg/query and pkg/release, by the AudioSilo server; see Public Go packages) and the private internal/*.

data/ carries its own data/go.mod - a nested Go module with no code in it, so the go command leaves the whole ~1.6 GB JSON tree out of this module's source zip. Without it, that zip passed Go's 500 MiB module-source limit and go get of any tag after v0.8.0 failed for a pkg/* consumer such as audiosilo-sidecars; with it the zip is about 3.4 MiB. The nested module holds no data of its own, so a pull request that only touches it runs the Go gate (not the data-only path) and triggers neither a data release nor ai-verify.

data/          the CC0 core, range-packed (PACK-SPEC.md): works/ (composites), people/, series/, redirects.json - the CC BY-SA works-community/ family lives in audiosilo-meta-community
schema/        JSON Schemas (one per entity) - the public contract, embedded via schema.go
cmd/           thin CLIs: metacheck, metafmt, metabuild, metaserve, metascan,
               metaimport, metaissue, metaextract (flag wiring only)
pkg/model      PUBLIC entity structs, slug rules, reserved slugs, pack addressing
pkg/pack       PUBLIC pack-file storage: bounds, splits, and the read-through Store every writer uses
pkg/canonical  PUBLIC canonical JSON (sorted keys, 2-space indent, trailing LF)
pkg/check      PUBLIC schema validation + pack-storage invariants + integrity/uniqueness/chapter/series rules
pkg/extract    PUBLIC epub split + the word-shingle near-verbatim check
pkg/scan       PUBLIC local folder scanner (tags + path/filename heuristics + ffprobe)
pkg/query      PUBLIC read side of the artifact: Open, every query, and the JSON API handler (NewHandler)
pkg/query/querytest  PUBLIC test support: a small real fixture artifact + constants naming its facts
pkg/release    PUBLIC data-release fetcher: newest data release, verified download
pkg/release/releasetest  PUBLIC test support: one fake of GitHub's releases API + checksum/gzip helpers
internal/importer   OpenAudible / Libation export -> canonical records (ASIN dedup)
internal/issueform  issue-form body -> canonical records + an ok/duplicate/needs-human/invalid verdict
internal/build      the deterministic SQLite builder (FTS5, ASIN/ISBN indexes, added_at)
internal/serve      metaserve: serves pkg/query's handler (behind CORS + gzip), the hot swap,
                    the GitHub-release poller (over pkg/release), webhook, OpenAPI, HTML pages, sitemaps
internal/httpx      the response helpers pkg/query and internal/serve share
Dockerfile     image: the site build + the metaserve binary - no baked data; the catalogue is fetched from the newest data release at boot (see below)
.github/       issue forms + CI workflows (check, release, image, intake, ai-verify)

Dependency direction mirrors the server's "transport is logic-free" rule: cmd/metaserve is flag wiring only. The queries and the JSON API handler live in pkg/query; internal/serve adds what is metaserve's own.

Public Go packages

These packages are a public API for programs outside this repo. Mirror-mode AudioSilo servers (metadata.mode: mirror, see the server's configuration) use them to download the data release once a day and answer the same API in-process:

• pkg/query - Open(path, tag) opens an artifact read-only (a safe DSN, so a ? or # in the path can't drop read-only mode) and runs the load-time integrity checks; DB.Info() reports SchemaVersion, BuiltAt, Tag, Version and the catalogue's counts; NewHandler(current, HandlerOptions) serves the JSON API (/healthz, /api/v1/... and the /abs provider routes; Routes() lists them) over whichever *DB current returns at request time, answering 503 with Retry-After while it returns nil. It is the very handler metaserve serves, without CORS or gzip (the caller adds its own). HandlerOptions carries the logger, the 503s' Retry-After, SiteURL (the origin the watch feeds link to), the feeds' clock, and the works/match budget and concurrency; the zero value suits an in-process consumer. MaxSchemaVersion is the newest artifact schema the code understands (pinned to the builder's SchemaVersion by a test).
• pkg/query/querytest - Build(tb, dir) writes a small, deterministic, real artifact (well under 1 MiB, built by the real builder from a fixed fixture) and constants naming its facts: an ASIN and an ISBN that look up (with their work and recording ids), an ASIN that doesn't, a numbered series, a chaptered recording, a retired slug and its target, and a work carrying the community layer. Downstream tests run against true metaserve behaviour.
• pkg/release - the one implementation of the release asset contract (below): New(repo, token, opts...), LatestData(ctx, etag) (the newest data release, conditional on an ETag) and DownloadData(ctx, rel, dst, progress) (the gz streamed through gunzip into a temp .meta-*.tmp file beside dst, verified against meta.sqlite.gz.sha256, fsynced and renamed; bounded, stall-guarded and only from allowlisted GitHub hosts). metaserve's poller uses it too, adding its delta and cache paths on top.
• pkg/release/releasetest - the one fake of GitHub's releases API (ETag and 304, hit counts, failing, throttled and hung releases, a redirect route, the User-Agent of each request) and the checksum and gzip helpers its assets are built with. pkg/release's tests, metaserve's refresh tests and the AudioSilo server's mirror tests all publish their releases there, so they can't drift apart on what a release looks like on the wire.

Being public, they follow the cross-repo contract's rules: the handler's responses change additively, and so does the artifact schema (how its versions are gated).

The CLI tool set at a glance

Each command is a thin cmd/* wrapper over a package; run any of them with go run ./cmd/<name>.

Command	What it does
metacheck	Validates the whole data/ tree - schema, pack placement and caps, referential integrity, uniqueness, chapter ordering, series positions. Prints one line per problem and exits 1 if any are found.
metafmt	Enforces canonical JSON for data/**/*.json (sorted keys, 2-space indent, single trailing LF) and the pack-storage invariants: --write also relocates misplaced entries, performs due pack splits, and rebinds, so placement self-heals and nobody computes it by hand. --check reports and exits 1.
metabuild	Compiles data/ into the SQLite artifact (-o meta.sqlite). Runs the full validation first and refuses to build invalid data. Deterministic: identical data produces an identical artifact; each record is dated by its own added_at, falling back to its newest sources[].imported_at.
metaserve	Serves the compiled artifact read-only over HTTP (and optionally the static site at /), hot-swapping newer GitHub releases. See the HTTP API.
metascan	Scans a local audiobook folder into an import JSON - see contributing data.
metaimport	Ingests an OpenAudible/Libation library export into data/ - see contributing data.
metaissue	Issue-form body to canonical records + verdict, for the intake bot - see contributing data.
metaextract	Supports the source-to-sidecar extraction pipeline: split (epub -> chapter text + manifest) and ngram (near-verbatim overlap check against the source text). split refuses a crafted epub rather than exhausting memory: any archive member over 64 MiB decompressed, or a spine over 256 MiB in total, rejects the whole book before anything is written.

Build, validate, and serve locally

Requires Go 1.26+ (pure Go, no cgo, no external services). CI reads the Go minor from the one-line .go-version file at the repo root and resolves it to the newest patch, so a local govulncheck should run under the newest 1.26.x too, or it reports standard-library advisories CI does not. The full gate, matching CI (.github/workflows/check.yml):

go build ./... && go vet ./... && go test -race ./... && golangci-lint run
go run ./cmd/metacheck --profile core       # validate the data tree
go run ./cmd/metafmt --check --profile core # canonical formatting (--write to fix)

--profile core is what this repo's data/ tree now is (works, people, series, redirects.json) since the CC BY-SA sidecar layer moved to audiosilo-meta-community - metacheck/metafmt default to --profile all (the whole database in one tree) for backward compatibility, so CI and this snippet pass --profile core explicitly rather than relying on that default.

The -race suite takes under a minute: every test that walks the real data tree skips under the race detector, since the fixture suites cover the concurrent code and metacheck/metafmt validate the real tree anyway. Those real-data tests still run in CI: the real-data.yml workflow runs the whole suite without -race every day, on demand, and on every push to main that touches anything the Go gate would run on (a paths: filter skipping data/** other than data/go.mod and *.md, so the bots' data-only commits never trigger it). The one deliberate difference from the Go gate is site/**, which no real-data test reads, so site-only pushes do not trigger it either. It is never run on pull requests. A newer run cancels an older one, so the newest commit on main is what reports, and a cancelled run reports nothing. A failure opens or updates a single issue labelled ci-real-data, which the next green run closes. Run them locally with go test -count=1 ./.... The linter set is pinned by the committed .golangci.yml (errcheck, govet, ineffassign, staticcheck, unused), so a new golangci-lint release cannot change which linters run; adding one is an edit there, with its findings fixed in the same change.

CI runs that same build/vet/go test -race/golangci-lint gate, plus a govulncheck scan, in check.yml's check/lint/vuln jobs - but only when a changes job classifies the diff as touching Go code. A data-and-prose-only pull request (most of them here, from the intake and sync bots) skips those three jobs straight through; metacheck and metafmt are never gated, since they are the data's own check regardless of what else changed.

Build the artifact and serve it:

go run ./cmd/metabuild -o meta.sqlite
go run ./cmd/metaserve --db meta.sqlite --addr :8080

Release artifacts

On merge to main, .github/workflows/release.yml publishes a dated data release tagged data-vYYYY.MM.DD-<core7>-<community7> - the short SHAs of both this repo's core tree and the composed audiosilo-meta-community checkout, since a community-only merge can trigger a release too and the core SHA alone stopped being unique - when data or schema changes land, when the builder itself changes (internal/build/**, cmd/metabuild/**), or when the cross-repo compose code changes (pkg/check/**, pkg/pack/**), so an artifact-shaping change such as a new index actually reaches a published artifact.

The release is created as a draft, every asset is then verified against the GitHub API's own reported size and sha256 digest for the bytes it actually stored (never re-downloaded and re-hashed locally), and only once every asset checks out is the draft published - so a consumer can never select a release whose assets are still uploading, and a verification failure deletes the draft instead of leaving a broken release visible. A manual workflow_dispatch run is refused unless it is checked out on main (the push trigger is already main-only; dispatch is the one door that takes any ref, and a release cut from a branch would become the catalogue the moment it lands). The asset contract:

• meta.sqlite.gz + meta.sqlite.gz.sha256 - the universal anchor every consumer verifies against.
• meta.sqlite.sha256 - the raw-file digest, used to verify a patched artifact.
• meta.sqlite.patch.from-<PREV_TAG>.zst - a best-effort zstd --patch-from binary delta against the previous data release (--long=31; a stock zstd CLI consumer must also pass --long=31 at decompression time).

The repo also cuts code/image v* releases with no data assets, so consumers select the newest data release by asset presence: the non-draft, non-prerelease release carrying meta.sqlite.gz with the maximum published_at (GitHub's release list order is not publish-chronological, and its "latest" can be either kind). The metaserve refresh loop and mirror-mode AudioSilo servers apply the same rule, through pkg/release.

The published image

ghcr.io/kodestar/audiosilo-meta is the site build plus the metaserve binary, and no data at all. The catalogue is not baked in: the container fetches the newest data release at boot and hot-swaps every release after that, which keeps image size and image build time independent of how large the catalogue grows.

The practical consequences for a deployment:

• Give it a cache volume. The image runs with --poll --cache /data/cache and declares /data as a volume. A restart that finds the current release's artifact already in the cache verifies it against the release's meta.sqlite.sha256 and adopts it without downloading - so restarts stay cheap. Without a persistent volume every restart re-downloads the catalogue. Budget for two artifacts at peak, one in steady state.
• A boot with no data is a valid state, not a crash. The server stays up and degrades visibly if GitHub is unreachable - full detail in boot and degraded start.
• Probe /healthz for readiness only. It reports readiness, not liveness; a liveness probe on it will restart-loop a server that is correctly waiting out an outage. For the same reason, the image deliberately ships no Dockerfile HEALTHCHECK of its own - an orchestrator's own "unhealthy" verdict is exactly what triggers an automatic restart, which would turn the degraded boot above into the crash loop it exists to avoid. Wire /healthz as readiness/startup in the orchestrator instead.

How it connects to the rest of AudioSilo

audiosilo-meta is the upstream of a three-repo metadata seam. metaserve serves the community data (or, for a server in mirror mode, the same handler answers in-process over a downloaded copy of the artifact); the AudioSilo server composes a book's enrichment from it (and re-exposes a single work document, unchanged, for the earlier books of a series the listener may not own); the player renders both. Covers are referenced by URL, so the server also fetches the images behind those cover_urls from their own hosts (in mirror mode too) and hands the player thumbnails (meta_covers): a listener's device never contacts a cover host.

The seam of record is the cross-repo contract's community-metadata section - what couples, which files sit on each side, and what a metaserve response-shape change ripples into. The consumer sides are documented on server configuration, the server's HTTP API reference, and the frontend's state and data.

One more service sits upstream of the data rather than downstream of it: the series-completion bot, a daily job that keeps the series the catalogue already holds complete and is the only automation allowed to merge its own pull requests.

metaserve additionally doubles as an Audiobookshelf custom metadata provider (GET /abs/search, plus GET /abs/{lang}/search, which ranks one language's matches first) for that competitor's users - see the HTTP API.

The site's watchlist is client-side only

The static site (site/, Astro + React islands) carries one feature that looks like an account and deliberately isn't: the series watchlist behind /watching. Which series a reader follows, which volumes they have marked as owned, which entries they have already been shown, and which series they have dismissed all live in a single localStorage key in the reader's browser (site/src/lib/watchlist.ts), never in a database and never in a request body - the API is read-only and there is nothing to attach a reader to. The rules are pure functions over a store value, the library-import path shares its matching code with /import, and the only network traffic the page generates is the same public series lookups the series page makes. The consequences are the honest ones: clearing site data clears the list (hence the page's download/merge backup), nothing syncs between devices by itself, and notifications are served by the stateless watch feed, whose subscription URL carries the series list rather than pointing at stored state.