Generative UI per channel

The same agent answers on a web page, in a mobile app, on WhatsApp and on Telegram. A web page draws any component of your catalog, nested, plus a sandbox. WhatsApp draws text, up to three reply buttons, a list, an image. AgentGenuiModule.forRoot({ channels }) tells the agent what each channel can draw:

import { AgentGenuiModule } from '@dudousxd/nestjs-agent/genui';
import { chartImages } from '@dudousxd/nestjs-agent-core/genui/chart-image';

AgentGenuiModule.forRoot({
  catalog,
  streaming: 'partial', // top-level options: the default (and what an unlisted channel gets)
  sandbox: true,
  channels: {
    web: { mode: 'tree', streaming: 'partial', sandbox: { kit: true } },
    mobile: { mode: 'tree', sandbox: false },
    whatsapp: { mode: 'per-component', render: 'native', chartImages: chartImages() },
    telegram: { mode: 'per-component' },
    email: { mode: 'per-component', render: 'html' },
    default: { mode: 'text' },
  },
});

Each entry may set:

A channel without an entry uses default, and without default the top-level options. Without channels, nothing changes: text channels keep answering with each component's text summary, as before. A per-channel sandbox with kit is found the same way as the top-level one (Your design system in the sandbox).

Which channel a turn is onCopy Anchor Link

One rule, turnChannel(pageContext), decides it:

• AG-UI, the HTTP chat (POST /agent/chat) and A2UI are web (they stamp nothing: no channel means web; an A2UI client is web-like) — unless the client's page context names a channel;
• the text channels stamp their adapter's kind — whatsapp for whatsappCloud(), evolutionApi() and whatsmiau(), telegram for telegram() — whatever the adapter is named;
• a client may name its own (pageContext: { channel: 'mobile' }), and server code may too (agentService.send({ …, channel: 'mobile' }));
• tools read it as ctx.channel, describe / previewInput as scope.channel, a prompt builder as ctx.channel, a GenuiCatalogResolver as scope.channel.

Only what the channel can drawCopy Anchor Link

Each turn is offered only the tools of its channel's mode, and their descriptions describe only what it can draw. On WhatsApp in per-component mode, a component gets a tool when it can be drawn there: it has a WhatsApp conversion, a chart renderer draws it, or it has a text summary (fallbackText). A component with none of these would arrive as a JSON dump, so it is not offered; a sandbox never is; and channels: { whatsapp: false } on a component opts it out.

Native conversionsCopy Anchor Link

Say what a component is on a channel, next to its definition:

export const OrderList = defineComponent(
  { name: 'OrderList', title: 'Orders', description: 'Orders with their status.', props, fallbackText },
  {
    channels: {
      whatsapp: (props) => ({
        text: `You have ${props.orders.length} orders.`,
        buttons: props.orders.map((order) => ({
          label: `Refund ${order.id}`,
          action: 'refund',
          value: { orderId: order.id },
        })),
      }),
      telegram: (props) => ({
        text: 'Your orders',
        list: {
          button: 'Pick one',
          items: props.orders.map((o) => ({ label: o.id, description: o.status, value: { orderId: o.id } })),
        },
      }),
      email: (props) => ({ html: `<ul>${props.orders.map((o) => `<li>${escape(o.id)}</li>`).join('')}</ul>` }),
    },
  },
);

A conversion returns { text?, buttons?, list?, image?, html? } (or several). The channel handler maps it onto the adapter, within the provider's real limits:

• buttons → reply buttons (WhatsApp: at most 3, 20-character titles; Telegram: an inline keyboard). More choices than the channel's buttons → a list;
• list → a WhatsApp list message (at most 10 rows, 24-character titles, 72-character descriptions, a 20-character button) — on Evolution/Whatsmiau where interactive messages render — or a Telegram keyboard of one row per entry; a channel with neither gets the choices as numbered text;
• image → a media message with the text as its caption;
• without a conversion, the component's text summary, as before. A conversion that throws falls back to the summary too.

A tree (ui__render) is flattened in order: each node's conversion or summary, consecutive text joined. Components arrive where they came in the text of the turn.

A conversion may also be { convert, describe }: describe says, in words for the model, how the component reaches the user on that channel (see below).

channels: {
  whatsapp: {
    describe: 'a message listing the orders, with a pick-list of them. No Refund buttons: to refund, the user picks an order and replies.',
    convert: (props) => ({ text: '*Your orders*', list: { button: 'Order details', items: … } }),
  },
}

convert(props, context) also gets the frame it converts:

• context.version is the schema version the props were written for: the frame's own version, or, for a node of a tree, the tree's componentVersions entry. It defaults to 1. A frame persisted before the component moved to v2 still says 1, so convert its props by it, as a renderer would;
• context.component and context.id name the component and the frame;
• context.frame is the whole frame as given.

describe may be a function too, ({ channel, component, version }) => string, where version is the component's current one.

channels: {
  whatsapp: (props, { version }) => ({ text: `Total ${version === 1 ? props.amount : props.total}` }),
}

What the model is toldCopy Anchor Link

Tool and component descriptions are written once, usually for the web ("shows the orders as a table with a Refund button per row"). On a channel the server draws for (WhatsApp, Telegram, email), two things keep the model from describing that view there:

• The prompt. The turn's system prompt gets a ## Channel: WhatsApp block: the user is not on a web page, what the channel can show (text, up to 3 reply buttons, a pick-list, images), and how each component of the catalog reaches the user there. That comes from the component's conversion: its describe if it has one, otherwise what a conversion can send; a chart the channel draws as an image is "a picture of the chart"; everything else is "plain text, no table, no buttons, nothing to click"; layouts are their contents. A sandbox is never listed. channelPrompt(catalog, channel) returns that block.
• The tool result. When a tool pushes UI on that channel (ctx.emitUi, or a ui__show_* tool), its result gets a delivered field that says what the user actually received, rendered the way the channel renders it, for example: Shown on WhatsApp as a message "*Your orders*" with a pick-list (opened with "Order details") of 4 entries: "Order #1001", … — picking one comes back as the user's next message. That is all the user got: no table, no web view, and no buttons other than the ones listed. Choices are described the way the channel handler sends them under the adapter's limits, which it stamps on the turn as pageContext.channel.choices: reply buttons when they fit, otherwise a list of the rows that fit, otherwise numbered text. An object output gets delivered added; any other output becomes { result, delivered }. describeChannelDelivery(messages, channel) gives the same words for messages you render yourself.

Web-like channels (web, mobile, A2UI) get neither: the page draws the components the descriptions talk about.

Wiring the text channelsCopy Anchor Link

Nothing to wire: AgentChannelsModule's handlers get the app's genui setup injected (the AGENT_GENUI token AgentGenuiModule binds — an AgentGenui with the catalog, the channels and the top-level options). A channel whose adapter kind (or default) has an entry draws natively; the turn is sent with no uiCapabilities narrowing, so the model may push whatever that channel can draw. Per channel, genui in its options overrides it — another { catalog, channels, base }, or false to keep text summaries only:

AgentChannelsModule.forRoot({
  channels: [{ adapter: telegram({ … }), actor, thread, genui: false }],
});

A ChannelHandler you build yourself draws natively only with genui passed.

Clicks come back as turnsCopy Anchor Link

A pressed button (a WhatsApp button_reply / list_reply, a Telegram callback_query) becomes the user's next turn through the same path as a sandbox's agent.send and an A2UI userAction: a UI action (source: 'component', the button's action, its value as context, its label as what the user said). Buttons carry short ids (ui:<ref>, within Telegram's 64-byte callback data) kept in the channel store for 7 days; a provider that forwards only the label (Whatsmiau) is matched by label against the conversation's recent buttons. A press is only accepted from the conversation the button was sent to.

Charts as imagesCopy Anchor Link

import { chartImages } from '@dudousxd/nestjs-agent-core/genui/chart-image';

channels: { whatsapp: { chartImages: chartImages({ width: 800, height: 450 }) } }

The builtin Chart (bar or line) is drawn as SVG on the server and converted to PNG with @resvg/resvg-js — an optional peer dependency (npm i @resvg/resvg-js), loaded only when a chart is drawn — and sent as an image with the chart's title as caption. components names other components taking the same props; fontFiles gives resvg fonts on a server without system fonts. Without resvg (or for a chart it cannot draw) the text summary is sent. chartSvg(props) gives you the SVG alone.

History follows the conversationCopy Anchor Link

On a configured channel the turn pushes the structured component (validated, versioned) and history stores it like on the web; each channel draws it its own way. A thread started on the web and continued on WhatsApp keeps its components; the web shows them as components, WhatsApp got them as messages, and the model sees the same history on both.