Attachments & multimodal

Let users attach images and PDFs to a message so a vision-capable model sees them natively — the MessageAttachment shape, the attachment-staging SPI, and the optional POST /agent/attachments upload route.

A user can attach an image or PDF to a chat message so a vision-capable model sees it natively. The library stays provider-agnostic: it passes a URL straight through as the model's image/file part and never fetches bytes itself — turning an uploaded file into a model-reachable URL is the app's job, behind the attachment-staging seam.

The attachment shape

Every attachment is a MessageAttachment:

interface MessageAttachment {
  mediaId: string      // stable id in your media store — provenance + replay key
  url: string          // a URL the model provider can fetch at turn time
  contentType: string  // image/* → image part, otherwise → file part
  name: string         // original filename
}

A client sends attachments on the POST /agent/chat body:

{ "message": "What's wrong with this diagram?", "attachments": [ { "mediaId": "…", "url": "…", "contentType": "image/png", "name": "arch.png" } ] }

They're persisted with the message and replayed as-is; the AI SDK adapter renders each as a native model content part (image/* → image part, otherwise → file part).

The URL must be reachable by the model provider

The lib never downloads bytes — url has to be something the provider can fetch: a presigned URL, a proxy, or a data: URI. That upload-time work is the staging seam below.

The staging seam & upload route

AttachmentStagingStore is a one-method seam — stage(input) → MessageAttachment — that persists uploaded bytes somewhere fetchable and returns the reference. Wiring it via attachmentStaging mounts an optional POST /agent/attachments upload route:

config/agent.ts

import { defineConfig, attachmentStores } from '@adonis-agora/agent'

export default defineConfig({
  model: () => aiSdkModel(openai('gpt-4o')),
  attachmentStaging: attachmentStores.memory(), // encodes bytes into a data: URL (tests/dev)
  attachmentMaxBytes: 20 * 1024 * 1024, // default 20 MiB
  // attachmentAllowedContentTypes: [...] // replaces the 7-entry default below
})

The route buffers the multipart file field, validates it against the size cap (413 on overflow) and the content-type allow-list (415 when disallowed), resolves the actor (the same fail-closed resolver as chat), stages it, and returns the MessageAttachment. The client then sends that reference on its next chat call.

The content-type allow-list

The default is an exact-match list of seven types — what multimodal providers commonly accept as native image or file parts:

['image/png', 'image/jpeg', 'image/gif', 'image/webp', 'application/pdf', 'text/plain', 'text/csv']

There is no wildcard and no prefix rule: the uploaded type must be one of those strings. text/markdown, image/svg+xml, image/heic, and application/json are all 415.

Setting attachmentAllowedContentTypes replaces the list rather than extending it, so spread what you want to keep:

attachmentAllowedContentTypes: ['image/png', 'image/jpeg', 'application/pdf', 'text/markdown'],

Widen it only to types your model provider actually renders. An accepted upload that the provider cannot read fails later, in the model call, where the error is far less obvious than a 415 at the door.

The route is opt-in

Omit attachmentStaging and no upload surface is exposed — a client sends already-staged MessageAttachment references directly on chat. attachmentStores.memory() covers tests and dev (it data:-encodes the bytes); a real deployment binds its own store presigning against S3/GCS or wrapping the host's media pipeline.

Non-public object stores — the download bypass

When the model provider can consume the attachment url natively (most hosted vision models can fetch a public HTTPS URL), the URL passes straight through and nothing downloads it. But when it can't — or when your url points at a non-public object store the provider can't reach (local MinIO in dev, a VPC-only S3 bucket) — the Vercel AI SDK's experimental_download step fetches the bytes and inlines them for the model.

The catch: the AI SDK's default downloader refuses localhost and private hostnames as an SSRF guard, so a staged-against-MinIO attachment dies with AI_DownloadError: URL with hostname localhost is not allowed. The package ships a ready-made replacement, attachmentFetchDownloader, on the @adonis-agora/agent/ai-sdk subpath — a plain-fetch downloader with no hostname policy — which you pass as the model's experimental_download:

config/agent.ts

import { defineConfig } from '@adonis-agora/agent'
import { aiSdkModel, attachmentFetchDownloader } from '@adonis-agora/agent/ai-sdk'
import { openai } from '@ai-sdk/openai'

export default defineConfig({
  model: () =>
    aiSdkModel(openai('gpt-4o'), {
      // Relax the SDK's SSRF guard so a MinIO/VPC-staged attachment can be fetched.
      experimental_download: attachmentFetchDownloader(),
    }),
})

It mirrors the default's routing — a URL the model supports natively is left to the provider (null), everything else is fetched and inlined as bytes.

Only safe because staging owns the URLs

attachmentFetchDownloader is safe only because agent attachment URLs come exclusively from your own attachment-staging SPI presigner — never from user input. Do not reuse it for URLs a user can influence: without the hostname guard it will happily fetch an internal address. Pass your own fetch implementation (attachmentFetchDownloader(myFetch)) if you need to constrain it.

The flow

Client POSTs a file to /agent/attachments → receives a MessageAttachment.
Client POSTs to /agent/chat with { message, attachments: [ …that reference… ] }.
The loop persists the attachment with the user message and passes each url to the model as a native image/file part.

To index uploaded files into RAG instead of (or as well as) attaching them to a turn, see RAG media ingestion.