RAG media ingestion

mediaRagIngestion bridges a media library into the RAG stack: when a file lands, it fetches the bytes, extracts text by content type, chunks and embeds it, and upserts the chunks into the same vector store your retriever searches — tagging every chunk with { mediaId, ownerType, ownerId, collection, tenantRef } so retrieval can be scoped per tenant/owner/collection. It lives on the @adonis-agora/agent/rag-media subpath.

Wire it

import { mediaRagIngestion, defaultTextExtractor } from '@adonis-agora/agent/rag-media'

const ingestion = mediaRagIngestion({
  media: await app.container.make('media.manager'), // structural MediaManager handle
  embedder: myEmbedder, // the same EmbeddingProvider your retriever uses
  store: myVectorStore, // the same VectorStore the retriever searches
  contentTypes: ['text/plain', 'text/markdown', 'application/pdf'],
  extractor: defaultTextExtractor().register('application/pdf', myPdfExtractor),
  resolve: async ({ id, disk, key }) => lookupMediaRecord(id), // upload.complete payload → MediaRef
})

const off = ingestion.subscribe() // auto-ingest on every upload.complete

Two triggers

• Explicit — await ingestion.ingestMedia(ref) with a MediaRef ({ id, disk, key, contentType, ownerType?, ownerId?, collection?, tenantRef?, size? }), e.g. from your own upload flow.
• Subscribe — ingestion.subscribe() listens on the media library's agora:media:upload.complete diagnostics channel and auto-ingests each finished upload. It needs a resolve seam to turn the event payload ({ id, disk, key }) into a full MediaRef (the event carries no content-type or owner metadata); without one it throws MediaRagResolveRequiredError.

subscribe() is idempotent, returns an unsubscribe function, and catches ingestion errors (publishing them on agora:rag:media.failed) so a bad file never breaks the channel. unsubscribe() and settle() await in-flight work for a graceful shutdown.

The pipeline

Per file: content-type filter → size gate (maxBytes) → read bytes → extract text → remove-then-chunk (so a re-upload that shrinks doesn't leave a stale tail) → embed → upsert. Unsupported, oversized, or empty-text files are skipped, not errored — MediaIngestResult is { status: 'ingested', chunks } or { status: 'skipped', reason } where reason is unsupported-type / too-large / empty-text.

Config reference

removeMedia(mediaId) drops a document's chunks — the delete half of keeping RAG in sync. Deterministic in-memory doubles (FakeMediaManager, fakePdfExtractor, inMemoryMediaRagIngestion) ship in the testing kit.