Skip to main content

Media pipeline

Everything that turns a (library_id, rel_path) into bytes a player can render lives in two places: internal/media (pure logic, no HTTP routing) and the two media handlers in internal/api/handlers_library.go (handleStream, handleCover). This page follows a request end to end. It explains the design decisions that are easy to break by accident. Most of them exist because of one strict client, iOS AVPlayer.

For the invariants behind this page ("path is the identity", "stream the file, not the book"), see Architecture invariants. For the request and response shapes, see the API reference.

The endpoints​

RouteHandlerPurpose
GET /api/v1/libraries/{id}/stream?path=handleStreamServe one audio file with Range streaming. ?download=1 forces a download and ?transcode=1&t= transcodes. An MP3-in-Matroska .mka is served as its cached remux to MP3 unless ?original=1. ?rendition=watch serves the file's cached watch rendition
POST/GET /api/v1/libraries/{id}/renditions?path=handlePrepareRenditions / handleRenditionStatusQueue (POST) or report (GET) the watch renditions of every audio file of a book
GET /api/v1/libraries/{id}/cover?path=handleCoverServe a book's cover: the custom cover, else a sidecar image, else embedded art. With ?size=96|160|320|640, a JPEG thumbnail of it (handleCoverThumbnail). Size 96 is accepted from 2.5.1
POST /api/v1/admin/covershandleAdminCoversAdmin only: JPEG thumbnails of many books' covers as data: URLs, for the admin console
GET /api/v1/libraries/{id}/chapters?path=handleChaptersThe normalized playable-units envelope {chapters, files, duration, …}. Warms the book's remuxes
GET /api/v1/libraries/{id}/item?path=handleItemBook detail. Carries direct_playable and warms the book's remuxes

There is no separate download endpoint. A download is ?download=1 on /stream. It makes media.ServeFile add a Content-Disposition: attachment header with the quoted base filename, so browsers save the file instead of playing it. A remuxed .mka downloads as <stem>.mp3, since the bytes are MP3.

On every media request, two things happen before any bytes are read:

  1. Authorization. authorizedPath resolves {id} and ?path= and checks the caller's share scope (Scope.Allows). The cover and stream routes are wired through requireBookMediaAuth. Like requireMediaAuth on /meta/cover, it also accepts the session token as a ?token= query parameter, because browser <img> and <audio> elements cannot set an Authorization header. On these two routes it also accepts a cast token. Every other route rejects query tokens, with two exceptions:
    • the Upcoming cover (/me/upcoming/cover), which also uses requireMediaAuth;
    • the calendar feed (releases.ics), which takes a feed token only (see Auth & security).
  2. Path safety. library.SafeJoin joins the relative path to the library root and rejects any traversal outside the root.

Direct streaming: media.ServeFile​

ServeFile opens the file and hands it to Go's http.ServeContent, which provides HTTP Range support with no custom code. Seeking and scrubbing in browsers, native players' byte-range fetches and resumable downloads all rely on it. ServeFile sets Accept-Ranges: bytes explicitly.

The one thing ServeContent gets wrong for audiobooks is the Content-Type, so ServeFile sets it explicitly first.

Why Content-Type is byte-sniffed (the iOS -12847 story)​

Go's mime table and http.DetectContentType do not recognize .m4b or .aax, so ServeContent would fall back to application/octet-stream. The server also sends X-Content-Type-Options: nosniff globally (see the secureHeaders middleware), which forbids the client from second-guessing that type. Strict players, iOS AVPlayer in particular, then refuse the stream outright (MediaToolbox error -12847). The fix has two layers:

  1. sniffAudioType reads the first 16 bytes of the actual file, then rewinds, so the type is correct even for mislabeled files:

    Leading bytesResultNotes
    fLaCaudio/flac
    OggSaudio/oggOgg container (Vorbis/Opus)
    RIFF + WAVE at offset 8audio/wav
    ID3audio/mpegID3-tagged MP3
    0xFF with b[1] & 0xF6 == 0xF0audio/aacRaw ADTS AAC (e.g. per-chapter .aac files). Checked before the MP3 case because the ADTS sync word also matches the MPEG mask
    0xFF with b[1] & 0xE0 == 0xE0audio/mpegBare MPEG audio frame sync (untagged MP3)
    ftyp at offset 4audio/mp4ISO base media: .m4a/.m4b/.mp4, AAC or ALAC inside
    1A 45 DF A3audio/x-matroskaEBML header: Matroska audio (.mka), MP3 or another codec inside. Only the stored bytes are served with this type. While ffmpeg is configured, an MP3-in-Matroska file goes out as its remux, audio/mpeg

  2. audioContentType falls back to the extension when the sniff comes up empty:

    ExtensionType
    .m4b, .m4a, .aax, .mp4audio/mp4
    .mp3audio/mpeg
    .aacaudio/aac
    .flacaudio/flac
    .ogg, .ogaaudio/ogg
    .opusaudio/opus
    .wav, .waveaudio/wav
    .mkaaudio/x-matroska

    For any other extension, ServeContent decides.

warning

If you add a new audio format, extend both sniffAudioType and audioContentType, and the recognized-extension set below. A format that streams with the wrong Content-Type looks fine in Chrome and silently fails on iOS.

Recognized audio: metadata.AudioExtensions​

The scanner and the /fs browse view treat a file as an audiobook when its extension is in metadata.AudioExtensions (checked via metadata.IsAudio):

.m4b  .m4a  .mp4  .mp3  .flac  .ogg  .opus  .mka

.mp4 is included because audiobooks are sometimes delivered as AAC-in-MP4. media serves it as audio/mp4.

.mka is Matroska audio, often MP3 with its chapters, as Audiobookshelf reads it. The scanner indexes it and reads its chapters with ffprobe. No browser and no iOS player opens the container, so how it plays depends on what is inside and on ffmpeg:

  • MP3 inside, ffmpeg configured. The stream URL serves a lossless copy of the audio as plain MP3 (Remuxing Matroska). Every client streams it, old native builds included, and /item and /chapters say direct_playable: true.
  • Another codec inside (AAC, Opus, FLAC), or no ffmpeg. The server serves the stored bytes as audio/x-matroska, and the book is not direct-playable. The web player streams it through ?transcode=1, which needs ffmpeg. The iOS app can't play it. Android's Media3 opens Matroska itself, so the Android app plays it where the device decodes the codec inside.

An .mka beside a file of the same name in another audio format is left out of the book (folderAudio). Like an .aax beside its .m4b, it is that file's source or copy, and reading both would give the book its chapters twice. The /fs listing applies the same rule (see the filesystem view).

Audible's DRM formats (.aax, .aaxc) are deliberately excluded. The server can never stream them, because they need per-account decryption. Indexing an .aax that sits next to its converted .m4b would also lump both into one book, with duplicated chapters and an unplayable track.

audioContentType knows a slightly broader set (for example .aax and .wav), so the server sends a correct type even for files the scanner would not index.

Covers: handleCover​

Cover resolution has three tiers. An admin's custom cover wins, then a sidecar image found by the scanner, and embedded art is the fallback. For all three, the handler first authorizes the path against the caller's share scope.

  1. Custom (catalog.Cover, the book_covers table). An image uploaded through PUT /admin/libraries/{id}/cover and stored in the database, never in the library folder.
    • It is served only while a book is indexed at the path (CoverInfo and Cover join books), so a pruned book's cover row stays dormant until the book returns.
    • The handler tries the requested path first. When that path is a part inside a folder book, it tries the book's own path next, after bookForPath resolves it.
    • The cover is served with its stored MIME type through http.ServeContent, with Cache-Control: private, no-cache and an ETag (coverETag: "cover-<base-36 Unix nanoseconds of updated_at>"). An admin can replace it at any moment, so clients revalidate rather than cache it for a day.
    • A matching If-None-Match is answered 304 from the cover's row alone (catalog.CoverInfo), without reading the image blob.
    • The validator is deliberately an ETag and not Last-Modified. With a date validator, a client revalidating after the custom cover was deleted could get a 304 from the older sidecar file and keep showing the removed cover.
  2. Sidecar (books.cover_path, set at scan time by findCover in internal/library/scanner.go). A conventionally named file (cover.jpg, cover.jpeg, cover.png, folder.jpg, folder.png) in the book's own folder for a folder book, or next to the file for a loose single-file book.
    • Inside a folder book a stray image is almost certainly the cover, so any image (.jpg .jpeg .png .webp .gif) is accepted as a fallback. The scanner prefers a filename containing "cover", else the first image alphabetically.
    • Multi-CD subfolders (CD1, Disc 2, …) look one level up for the parent book's art.
    • A sidecar cover is served through ServeFile, so it gets Range and correct headers, with Cache-Control: private, max-age=86400. The header is set only when the file is there, so a 404 is never cached. Without an explicit lifetime, a browser keeps a sidecar image fresh by heuristic (a tenth of the file's age), and a custom cover uploaded later could go unseen for weeks.
  3. Embedded (media.EmbeddedCover). The handler reads the book's primary audio file with dhowden/tag and returns its embedded picture. The primary file is the first files entry for a folder book, and the file itself otherwise.
    • The Content-Type comes from the picture's bytes (coverMIME, http.DetectContentType against an allow-list: JPEG, PNG, GIF, WebP, BMP), never from the MIME type the tag declares. Data that sniffs as anything else is treated as no art. The response comes from this origin, so trusting the tag would let a crafted audio file declare text/html and have the server serve a page.
    • It is served with Cache-Control: private, max-age=86400.

When no tier has a cover, the answer is 404 {"error":"no cover"}.

Thumbnails for the admin console: POST /admin/covers​

The admin console can't show covers as plain <img src> URLs, for two reasons. Its session is a full-privilege admin token that must never ride in a URL, and its CSP allows images only from 'self' and data:. So the console asks for covers in batches through POST /api/v1/admin/covers (handleAdminCovers, internal/api/handlers_covers.go). One request takes up to 60 books, and the answer gives each a small JPEG thumbnail as a data: URL. A grid of hundreds of covers is then a handful of requests and about 20 KB a cover, instead of full-size art.

  • Resolving the art. coverArt resolves the art in the same order as handleCover: the custom cover by the requested path, then by the book the path resolves to, then the sidecar through SafeJoin, then embedded art. It never indexes on demand.
  • Scaling. media.Thumbnail scales to fit within 96, 160, 320 or 640 pixels (thumbSizes) and encodes a JPEG at quality 80. 96 is a table row's cover. It never scales up.
  • Small JPEGs. A JPEG that is already that small is re-encoded at quality 80, and Thumbnail returns the smaller of that and the original bytes. A cover.jpg is often saved near quality 95, about twice the bytes for a 180 px tile. One saved at a lower quality comes out larger from the re-encode, so the original is kept.
  • Returned as is. Thumbnail returns the source unchanged in these cases:
    • it is CMYK, or carries an EXIF orientation other than 1 or an ICC colour profile (ownRendering). Go's encoder writes neither, so a re-encode would show it turned or in other colours.
    • it doesn't decode fully;
    • its segments can't be read.
  • Size limit. Thumbnail reads the image header first and refuses anything over MaxThumbnailSourcePixels (40 megapixels) before decoding, so a small file that claims huge dimensions can't exhaust memory.
  • Cache. media.ThumbCache is a byte-bounded LRU (48 MiB). Its key is the library, the path, the size and the art's version (a custom cover's updated_at, or a file's size and modification time), so a changed cover misses the cache. It holds the raw JPEG, and the batch base64-encodes it into its data: URL. A read failure (an unreachable mount) is not cached. An undecodable image is cached as no art.
  • Concurrency. Two semaphores bound the work across all requests, the batch and the player's ?size= alike. coverReads (8) bounds the art being read or waiting to be decoded, so a slow mount never has a grid's 60 reads in flight. thumbSem (4) bounds the decodes themselves.
  • Version. Each thumbnail carries a version: catalog.CoverVersion of the colour record's Version, else of its Art, else of the source's Art. A first thumbnail moves the row's version through RecordCoverColors. POST /admin/covers explains why a client files a thumbnail under this version rather than the row's. The console's side is its thumbnail cache.

Thumbnails for the player: ?size=​

The player asks for the same thumbnails one cover at a time, as an image URL: GET /libraries/{id}/cover?path=&size=160|320|640 (handleCoverThumbnail, capability cover_sizes). The player never sends 96, because an older server refuses it. The route shares everything with the batch: coverArt, coverThumbnail, the cache and both semaphores.

A path with no book indexed at it (a part, a disc folder, a book the scan hasn't reached) is resolved like /item, through bookForPath. Unlike the batch, bookForPath may index on demand. Any size outside thumbSizes, including an empty one, is a 400.

  • ETag. The ETag is "thumb-<size>-<rev>-<hash of the art's own version>" (thumbETag). It is matched with the same If-None-Match list matcher as the server's other validators.
  • rev. rev is media.ThumbnailRev. Change it whenever Thumbnail would return other bytes for the same source. Every validator then changes, and clients fetch each cover once more rather than keep other bytes or resume a range of them.
  • 304 and the colour. The handler answers a match with 304 before reading any art only once the book's colour is recorded for this art (src.Colored, with its identity equal to the art's version). Until then it makes the thumbnail or takes it from the cache, records the colour, and still sends the 304. Clients that only revalidate therefore don't leave a book without its colour.
  • Cache-Control. private, no-cache for a custom cover and private, max-age=86400 for sidecar or embedded art, mirroring the full-art tiers above. Both headers are set only once there is art to answer with, so a 404 is never cached.
  • Errors. A read failure (an unreachable mount) is a 500 could not load cover, and is not cached. Art that can't be decoded, or is over 40 megapixels, is a 404 no cover. The full-art URL may still serve it, and clients fall back to it.
  • Body. The body is served through http.ServeContent as image/jpeg.

Cover identity, colours and version​

Two derived columns on books (migration 0023) carry what the player's Book JSON says about the cover:

  • cover_art, the cover's art identity. It is never sent itself.
    • From index data alone it is c plus the custom cover's updated_at, else f plus the book's mtime, size and sidecar path (catalog.coverArtSQL, also the migration's backfill). Every writer of those inputs recomputes it (refreshCoverArt): UpsertBook, SetCover, DeleteCover, and a move or a disc join that carries a custom cover.
    • The wire's cover_version is its hash (catalog.CoverVersion), present on every indexed book. The hash is the first 10 hex characters of its SHA-256, so the wire carries no stamp, size or path.
    • The scan records file art by the art file's own version (catalog.FileArtVersion, the scan's Book.ArtVersion): the sidecar or the first audio file, with its kind, size and mtime. This is the same name the cover handler gives its thumbnail. A re-index of unchanged art therefore keeps its version and colour, and art replaced in place gets a new version at the next scan. A scan that finds the audio unchanged still records a replaced sidecar (SetArtVersions).
    • The index form above is left only for a row with no file version. The value changes, not its shape.
  • cover_color, the palette read from a thumbnail, stored as version bg or version bg accent on_accent. It is tagged with the cover_version it was read for, and decoded onto the book only while that tag is current. New art needs nothing cleared, because the old colour stops counting.

The palette (media.CoverPalette, internal/media/covercolor.go) is read from the thumbnail JPEG when a book lacks a colour for the art:

  • It samples at most 64 x 64 pixels, with transparent areas counted as white.
  • It quantizes each sample to 16 levels per channel, and takes the most common bucket's average as bg.
  • accent is the most common vibrant bucket: HSL saturation at least 0.3, lightness 0.2-0.8, and at least 1/200 of the samples. Its lightness is nudged until it reaches a WCAG contrast of 4.5:1 against bg.
  • on_accent is white or black, whichever contrasts more.

A near-greyscale cover, or one whose vibrant colour cannot be made to read against bg, has bg only.

catalog.RecordCoverColors stores the colour together with the art's own version as the new cover_art. For file art that is the sidecar's or audio file's size and modification time. A custom cover's stamp is unchanged. The write is a compare-and-set on the identity the source was read under, so a slow thumbnail of old art never overwrites what a cover upload just set.

The identity therefore moves once, at a book's first thumbnail of file art. From then on it follows a sidecar overwritten in place (which leaves the index as it was), and it equals the thumbnail ETag's hash. A re-index of a changed book recomputes the identity from index data, and the next thumbnail moves it again.

The write is detached from the request and bounded to 250 ms (coverColorWriteTimeout). A busy writer costs only the colour, until the next thumbnail.

The background colour pass​

If colours were read only from the thumbnails people ask for, a library nobody has browsed would have none. So internal/covercolors reads them in the background.

When it runs. Its Runner waits until the start, or a burst of book changes, has been quiet for 30 s (at most 10 minutes), then runs a pass. A book change is anything Catalog.OnBookChange reports: an index write, an edit, or a custom cover set or removed. The runner also runs a pass every hour, which retries art that failed to read. A scan that indexes books changes one every moment, so the pass waits for the scan to finish rather than compete with it for the disk. A rescan that finds nothing new changes nothing, so it doesn't hold the pass back.

What a pass reads. A pass walks catalog.CoverColorsDue in pages of 100, in the index's own order. It lists the books that may have art (a custom cover, or has_cover set or not yet checked by a scan) and hold no colour for their current art, each with its CoverSource. The pass reads one book at a time, the way a thumbnail would (api.colorCover). It uses coverArt and the same coverReads and thumbSem bounds as the cover endpoints, so it never holds more than one of their slots.

  • A thumbnail of the art already in the cache is used as it is. Otherwise the pass makes a 160 px thumbnail and does not cache it, so a pass over a library never pushes out the thumbnails people are looking at.
  • A book with no art, or none that decodes, is recorded as having no colour (cover_color holds only the version). No pass reads it again until its art changes, across restarts too. A thumbnail that does decode still records its colour over that, since the pass's read may have failed in passing.
  • A book whose art files aren't there is covercolors.ErrArtMissing. This covers an unmounted share, whose books the scan keeps indexed, and a book not pruned yet. Nothing is recorded, so the next pass reads it again once the share is back, and it doesn't count as a failure.

Writing and stopping. Each page's colours are recorded in one RecordCoverColors transaction (bounded to 30 s), compare-and-set like any other. Ten books in a row whose art fails to read (an unreadable mount) end the pass until the next hour. A scan doesn't re-index unchanged books, so after its first full pass the runner only reads books whose art is new.

DirectPlayable: when a client should transcode​

The scanner records each book's audio codec in books.codec: ffprobe's codec_name verbatim, or the in-process probe's identical answer. media.DirectPlayable(format, codec) reports whether mainstream browsers can play the stored file natively.

A container that no browser plays (containerFormats: mka, by the file extension in books.format) is never direct-playable, whatever the codec inside. For a folder book every file's container counts, not only the first file's. Two places apply this rule:

  • Book.PlaysInBrowsers, part of what /item and /chapters say (below);
  • directPlayableExpr, built from DirectPlayableSQL, for the admin console, which stays as stored.

Otherwise DirectPlayable checks the codec against this allow-list (browserCodecs):

aac  mp3  mp2  flac  opus  vorbis  pcm_s16le

These are codec names, not containers. AAC-in-MP4 probes as aac (not mp4a), and WAV as pcm_s16le. An empty codec (ffprobe unavailable, or the book not yet probed) is treated as playable, and the client streams it directly. There is no automatic retry through the transcoder on a playback failure.

The flag is surfaced as direct_playable on the item and chapters responses, so a web client knows up front that, for example, an AC-3 or ALAC book needs the transcoder. What those two responses say is "plays in browsers as the server serves it" (API.directPlayable). That is Book.PlaysInBrowsers, or Book.PlaysRemuxed while the remux is on (API.remuxOn).

PlaysRemuxed holds when the book's codec is mp3 and every file is either in a browser container or an .mka holding MP3. Each .mka part is judged by its own codec (Book.FileRemuxable). A part indexed before per-file codecs were recorded is judged by the book's codec. An .mka book with AAC inside stays false.

Web transcode negotiation describes how the web player acts on the flag. The native apps always stream directly.

Transcoding: media.Transcode​

GET /libraries/{id}/stream?path=…&transcode=1[&t=<seconds>] pipes the file through ffmpeg to MP3. The handler returns 503 when no ffmpeg is configured, either because of --ffmpeg "" or because the binary was never found. The transcode capability flag in GET /server reflects this (see Configuration). The exact invocation:

ffmpeg -nostdin -loglevel error [-ss <t>] -i <abs path> \
       -vn -c:a libmp3lame -b:a 128k -f mp3 pipe:1

A podcast episode is transcoded from the descriptor the server already verified (media.TranscodeOpened), never by reopening its path. The server rewinds the descriptor and hands it to ffmpeg as its stdin. When that ffmpeg has the fd: protocol (ffmpeg 6.1 and later), ffmpeg reads it seekably through fd::

ffmpeg -nostdin -loglevel error -protocol_whitelist fd [-ss <t>] -i fd: \
       -vn -c:a libmp3lame -b:a 128k -f mp3 pipe:1

With fd:, -ss is a fast input seek, and a non-faststart m4a (its moov index after the audio) demuxes.

inputFor asks each ffmpeg binary once per process whether its Input protocols list fd (-hide_banner -protocols, 5 s at most, cached per path). A probe that couldn't run is asked again on the next request. An older ffmpeg gets the previous non-seekable form, -protocol_whitelist pipe -i pipe:0 -ss <t>, an output seek that decodes up to t. Either way, the descriptor is the only input ffmpeg may open. transcodeArgs builds both shapes and is unit-tested.

Design points worth knowing before touching it:

  • -ss is input-side (before -i) for a path and for fd:. ffmpeg seeks near the start position before decoding, which is fast even deep into a long m4b.
  • -vn drops any embedded cover-art or video stream, so the output is pure audio.
  • The response is Content-Type: audio/mpeg, Accept-Ranges: none, Cache-Control: no-store, and has no Content-Length, because the output is produced on the fly. A HEAD gets these headers without an ffmpeg run.
  • Why it isn't byte-seekable. The total encoded size is unknown while encoding, and a byte offset into the MP3 output has no computable mapping back to a source timestamp. So the handler refuses Range requests, and a client seeks by re-requesting with a new t= value. This is exactly the ?transcode=1&t= contract the frontend's stream URLs implement.
  • The handler creates the ffmpeg process with exec.CommandContext(r.Context(), …), so a client disconnect (pause, seek, navigate away) kills it. A broken pipe is logged at debug level. Only a genuine ffmpeg failure (a non-nil exit while the request context is still live) logs a warning, with ffmpeg's stderr attached.

Direct serving with Range stays the default. Transcoding is strictly the fallback, for codecs DirectPlayable rejects and for an MP3-in-Matroska file whose remux isn't available right now (below).

Remuxing Matroska: media.NewRemuxer​

The remux and the watch rendition are two caches of the same type, media.FileCache (filecache.go). Each holds copies that ffmpeg makes of library files, in the data directory. A cacheSpec describes each cache: the directory, extension, ffmpeg arguments, size cap, how many runs go at once, the timeout, the ETag, and an optional check of the finished copy. Both caches share the LRU, single-flight, .part-and-rename, disk guard and refusal memory described below. The methods are:

  • Open waits for a copy.
  • Status reports where a copy stands.
  • Queue starts a copy without waiting. It does this only for a cache with a queue limit, which is the watch rendition's. The remux's Queue starts nothing and leaves the file missing.
  • WarmAll warms a book's copies.

NewRemuxer (remux.go) is the remux's spec.

An .mka holding MP3 is served at the same stream URL as a lossless copy of its audio in a plain MP3 file. Every client then plays it with Range seeks: browsers, iOS AVPlayer (which types by Content-Type when streaming) and shipped native builds that only ever ask for the plain URL. No capability flag announces it, and no client branches on it. A shipped iOS build saves its download under .mka, which AVFoundation won't open offline. The updated app renames it (see Offline).

When. The remux is on (API.remuxOn) while ffmpeg is configured and the cache is open, which needs <data> set. A cache that can't be created logs a warning, and the server serves Matroska as stored. A podcast library's stream never remuxes, because servePodcastStream answers it first. Otherwise handleStream remuxes a path ending in .mka (any case), with or without download=1, except in two cases:

  • transcode=1 gets the live transcode, as before;
  • original=1 gets the stored bytes, for API tools and backups.

--ffmpeg "" restores the 2.4.x behaviour exactly.

The ffmpeg run. The server opens the source in Go after SafeJoin, as ServeFile does, and hands it over as stdin (fd: or pipe:0, as inputFor decides above):

ffmpeg -nostdin -loglevel error -xerror -protocol_whitelist fd -f matroska -i fd: \
       -map 0:a:0 -c:a copy -map_chapters 0 -fflags +bitexact -id3v2_version 3 \
       -f mp3 -y file:<cache>/<key>.<rand>.part
  • The forced demuxer and the descriptor-only whitelist mean crafted bytes can't make ffmpeg open other files.
  • -c:a copy is a stream copy, not an encode: lossless and I/O-bound.
  • The chapters become ID3 CHAP frames, and ffmpeg writes a Xing/Info header, so a player gets a real duration.
  • +bitexact makes the output deterministic, so a resumed download's If-Range still matches after the copy was evicted and made again.
  • -xerror fails the run on a read error rather than leaving a short copy.

Is it MP3? The server decides this by trying. ffmpeg's mp3 muxer refuses anything but exactly one MP3 audio stream, which also covers folder parts whose codec the index doesn't hold. That refusal is ErrRefused. The server answers it with the stored bytes and remembers it in memory per cache key for an hour (cacheRefusalTTL, up to 1,024 keys). The memory expires because what looks like the file's fault may be the environment's, such as an ffmpeg missing an encoder. A changed file is a new key, and the server tries it again at once.

These are not refusals:

  • an exit after ffmpeg wrote output (killed partway, a full disk, a read error);
  • an exit whose log names an I/O error before any output (readFailed, for example a flaky share). The next request tries again.
  • the server shutting down mid-run.

A run that its own timeout killed is a refusal, in either cache (timedOut), so a copy that can't finish in time isn't started from zero on every request.

The cache. The cache lives in <data>/cache/remux/ (0700, made again if it went). It holds one file per source, named by the first 32 hex digits of a SHA-256 of the absolute path, size and mtime. A changed source is therefore a new key, and its old copy ages out.

  • A remux writes <key>.<rand>.part, syncs it and renames it into place. It never renames it under the key when the source changed while it was read. The server sweeps .part leftovers at start.
  • The cache is an LRU by file mtime. A hit bumps the mtime at most once every 10 minutes, so a player's run of Range requests counts as one use.
  • After each new remux, and at start, the server removes the oldest copies while the total is over 8 GiB. It keeps the new copy even when that copy alone is bigger.
  • A remux is skipped (ErrUnavailable) when the disk would be left with under 1 GiB free after it and the other runs still writing (internal/diskspace). The server logs one warning per low-disk spell. Both caches live under <data>/cache, so they share one reservation (FileCache.ShareDiskGuard), and a remux and a rendition encode running together count each other.

Concurrency. One remux runs per key at a time. This is single-flight: every request for the key waits on the same run. At most 2 remuxes run at once across the server, a limit separate from the 4 transcode slots. The run belongs to the server's base context, never to a request, with a 30-minute timeout from when it gets a slot. A waiter that leaves (a seek, an AVPlayer probe) therefore doesn't stop it. At shutdown the launcher waits for the runs of both caches (API.WaitMediaCaches), which kills their ffmpeg and removes their .part files.

A cold request. serveRemuxed waits at most remuxColdWait, 2 s, for the copy. That is under the player's 3 s stall timer and Android's 8 s read timeout. After that, the request gets the live transcode instead when any of these holds:

  • the remux is still running;
  • the disk guard skipped it;
  • ffmpeg failed some other way.

The transcode is needed because direct_playable told the web player not to ask for one. With download=1 the file is still named <stem>.mp3. The remux goes on, and the next request gets the copy. The transcode answer starts from the beginning (the request carries no t), is not byte-seekable, and is a lossy 128 kb/s encode. A seek made in that window gets those bytes, not the copy's.

Two kinds of request wait longer instead, up to remuxDownloadWait, 45 s. That is under the 60 s an idle response gets from Android's downloader and from a reverse proxy's default read timeout.

  • A download=1 request, whose saved copy would otherwise be the lossy transcode.
  • A stream that finds every transcode slot busy (takeTranscodeSlot returns none), which would otherwise be a 503.

Past that wait, they get the transcode after all. While the server is stopping, the answer is 503.

Warming. /item and /chapters start the remuxes of the book's MP3-in-Matroska files in the background (API.warmRemux calls FileCache.WarmAll), so the first stream is usually a hit. The answer never waits for them.

  • Warms run one file and one book at a time across the server, so they hold at most one of the 2 slots.
  • A queued book is warmed once; at most 16 books are queued, and more are dropped.
  • Cached copies and known refusals are skipped. A warm doesn't mark a copy used, since a book looked at is not one played.
  • A podcast library is never warmed.

The response. media.ServeOpenedAs serves the copy with Content-Type: audio/mpeg, Accept-Ranges: bytes and Range, named <stem>.mp3 (the download=1 filename). Its validators are its own: a strong ETag: "remux-<key>" and the copy's mtime as Last-Modified, never the source's. A 2.4.x client may have kept an If-Range or If-Modified-Since from the stored Matroska. That request gets the whole MP3, never a 304 or MP3 bytes spliced onto Matroska. A remuxed stream is not a "transcoded" listening session.

What stays as stored. These describe the stored files, so an MP3-in-Matroska book still reads as needing transcoding there:

  • the admin console's Playback column and its direct_playable filter;
  • the book page's playback line and Files card;
  • the Health transcode issue.

Watch renditions: media.NewRenditionCache​

Capability renditions. A watch downloads a book to play it away from the phone, and the library's files are often too big for it. A 20-hour book as 128 kb/s stereo MP3 is over a gigabyte. So the server makes a small copy of each audio file once, a rendition, and serves it like any file, with Range. The library stays read-only, because the copies live in the data directory.

The encode (renditionArgs, media/rendition.go):

ffmpeg -nostdin -loglevel error -protocol_whitelist fd -i fd: \
       -map 0:a:0 -vn -sn -dn -map_metadata -1 -map_chapters -1 \
       -c:a aac -b:a 48k -ac 1 -ar 44100 -fflags +bitexact -flags:a +bitexact \
       -movflags +faststart -f mp4 -y file:<cache>/<key>.<rand>.part
  • AAC-LC, mono, 48 kb/s, 44.1 kHz, MP4 (.m4a), about 430 MB for 20 hours. It uses ffmpeg's own aac encoder. HE-AAC would be smaller, but needs the non-free libfdk_aac.
  • The same duration as the source (within 0.1 s on MP3 and M4B), so whole-book positions, chapters and the file places of listening sync hold on the copy unchanged. The copy has no cover, tags or chapters, because the watch reads those from the API.
  • +faststart puts the index first, so a player can start before the end arrives. +bitexact makes the same ffmpeg write the same bytes every run.
  • The only input is the descriptor (fd:, or pipe:0 for an older ffmpeg, as for the transcode). The demuxer is ffmpeg's guess, since the library holds every audio format.
  • Unlike the remux, the encode has no -xerror, because it would fail the whole encode on one damaged MP3 frame that playback would skip. A read error partway still exits 0, so the server checks the finished copy instead (renditionCheck). A copy fails the check when it has no audio track, or when it is more than 1 s shorter than the source's exact duration (the in-process probe, metadata.ExactDuration). The length check is skipped when the source's length is only an estimate, such as a VBR MP3 with no Xing header. A copy that fails is not cached, and the server remembers it as a refusal for an hour.

The cache. <data>/cache/renditions/watch/<key>.m4a, a FileCache like the remux's (above) with its own bounds:

Watch renditionsRemux (for comparison)
KeySHA-256 of the absolute path, size, mtime and watch-v1the absolute path, size and mtime
Size cap (LRU by mtime)10 GiB8 GiB
At once1 encode (an encoder pins a core)2 stream copies
Timeout from its slot2 hours30 minutes
Waiting or runningat most 256 files, server-widewarms: 16 books queued
Disk guard1 GiB left free, one reservation shared with the remuxthe same

The encode version (renditionVersion) is part of the key and the ETag, so a change to the encode is a new key, and old copies age out of the LRU. A source that changes while it is read is never cached under the old key.

A refusal is remembered for an hour, or until the file changes. A refusal is ffmpeg's exit before writing anything, a trackless or short copy, or an encode that its 2-hour timeout killed. A failed run's log carries the first and last 2 KiB of ffmpeg's output.

Without ffmpeg (--ffmpeg "") or without a data directory, the cache is off:

  • the renditions flag is false;
  • the top-level renditions list is [];
  • a watch rendition request outside a podcast library answers 404 rendition_unavailable. The kind and podcast checks come first, so those still answer 400 bad_rendition.

Serving (serveRendition, api/handlers_renditions.go). handleStream checks rendition= straight after the scope check, before the podcast and remux branches. It checks, in this order:

  1. the kind: 400 bad_rendition for anything but watch, and for a podcast library;
  2. the flag: 404 rendition_unavailable;
  3. recognised audio and a regular file: 404.

Then FileCache.Open waits up to renditionWait for the copy. That is 45 s, the same as the remux's download wait. The outcomes are:

  • Ready. serveCopy answers through media.ServeOpenedAs, with Range and If-Range. The response has Content-Type: audio/mp4, a strong ETag: "rw1-<key>-<size>", and the copy's own mtime as Last-Modified. download=1 adds Content-Disposition: attachment named <stem>.m4a. The size in the ETag tells two encodes apart, should a re-encode after an eviction ever differ, so a resumed download never splices bytes of two encodes.
  • Still encoding. 503 rendition_pending with Retry-After: 15. The encode goes on under the server's base context, never the request's. The server never falls back to a live transcode here, because its bytes aren't rangeable, so a watch couldn't resume it.
  • Refused. 422 rendition_failed (remembered, as above). Retry-After gives the seconds the remembered refusal has left (FileCache.RefusedFor, 1 to 3600).
  • Failed otherwise. 503 rendition_error with Retry-After: 60. This covers the disk guard, a run killed partway by a shutdown, a source that changed, and a full 256-file queue.

Preparing a book. POST /libraries/{id}/renditions?path=<book> with {"kind":"watch"} resolves the book as /item does, so a part path names its book. It then calls FileCache.Queue for each of the book's audio files (Book.AudioFiles, in play order). A single-file book is its own file. GET with ?kind=watch calls Status only. Both answer {kind, files: [{path, state, bytes?, retry_after_seconds?}]}, with state ready, pending, failed or missing. retry_after_seconds appears only on a failed file whose refusal is remembered (see POST /libraries/{id}/renditions in the reference). Both need a header credential (requireAuth, no ?token=).

A rendition request is not a listen: it opens no listening session and never marks one as transcoded.

The chapters envelope: {chapters, files, duration}​

GET /libraries/{id}/chapters?path= is what makes a single chaptered m4b and a folder of fifty mp3 parts render identically in a player. The response:

{
  "library_id": 1,
  "path": "Brandon Sanderson/Mistborn/01 - The Final Empire",
  "duration": 88453.2,
  "is_folder": true,
  "codec": "mp3",
  "direct_playable": true,
  "files": [
    { "rel_path": ".../Part 01.mp3", "seq": 0, "duration": 3520.1, "format": "mp3", "size": 84480000 },
    { "rel_path": ".../Part 02.mp3", "seq": 1, "duration": 3498.7, "format": "mp3", "size": 83968000 }
  ],
  "chapters": [
    { "index": 0, "title": "Chapter 1", "file_index": 0,
      "file_path": ".../Part 01.mp3", "start": 0,      "end": 3520.1, "book_offset": 0 },
    { "index": 1, "title": "Chapter 2", "file_index": 1,
      "file_path": ".../Part 02.mp3", "start": 0,      "end": 3498.7, "book_offset": 3520.1 }
  ]
}

Each chapter (metadata.Chapter) is one playable unit, normalized across book shapes:

FieldMeaning
file_pathLibrary-relative path of the audio file to stream. This is what goes into /stream?path=. Never a folder or book path
file_index0-based ordinal of the containing file (matches BookFile.seq; 0 for single-file books). Used for ordering only
start / endOffsets in seconds within that file: seek to start after loading file_path
book_offsetThe chapter's start on the whole-book timeline (the sum of earlier files' durations), so progress is one continuous position

Two relationships must hold:

  • A chapter's book_offset equals the sum of the durations of all files before file_index, plus its own start.
  • The last chapter's book_offset + (end − start) reaches duration.

For folder books the scanner probes each part. A part with its own embedded chapters (a chaptered m4b inside a book folder) is expanded into those chapters. Otherwise the whole part becomes one chapter (see Scanner).

The files array is the track list a client actually queues. The rule is stream the file, not the book: a folder path handed to /stream cannot work, and in the frontend this once surfaced as MediaToolbox -12864. Frontend playback covers how the player maps between (trackIndex, position) and the whole-book position from this envelope.