Flags 1.x is a release candidate. npm's latest tag still installs 0.x, so install these docs' version from the rc tag:
npm i @adonis-agora/flags@rc
node ace configure @adonis-agora/flagsThe same goes for every @adonis-agora/* package these docs install: add @rc, and where a page says node ace add, install with @rc first and then run node ace configure. On 0.x? Read the 0.x docs.
Experiments
Run an A/B test — assign, record exposures and conversions, define metrics and read results.
1. Declare the flag and the metrics
Metrics and experiments can live in config/flags.ts or be created in the console. Both kinds work together.
In the console
- On Metrics, click New metric. Name it, pick a kind, and type the event your code sends with
track(). - In the flag editor, under Experiment, mark the flag as an experiment. Tick the metrics that decide it and the ones that act as guardrails. You can also pick the control variant and the confidence there.
Console metrics belong to one environment, like segments and layers. Promote or import carries them to the others. Changes go through approvals and show in the history.
In config
import { defineConfig, flag, metric } from '@adonis-agora/flags'
export default defineConfig({
flags: {
checkout: flag.string({ default: 'control', variants: ['control', 'one_page'] }),
},
metrics: {
purchase: metric.conversion('purchase'),
revenue: metric.sum('purchase', { cupedMs: 14 * 24 * 3600 * 1000 }),
refunds: metric.count('refund', { direction: 'decrease' }),
},
experiments: {
checkout: { metrics: ['purchase', 'revenue'], guardrails: ['refunds'] },
},
})| Metric | Per subject |
|---|---|
metric.conversion(event) | 1 if the subject did event after exposure, else 0 |
metric.count(event) | how many times |
metric.sum(event) | the sum of the events' value (revenue per user) |
Options: windowMs (only count events within this long after exposure), cupedMs (see below), direction (increase is good by default).
When the config and the console define the same metric or experiment, the config wins. The console shows it read-only. A flag's experiment settings can only use metrics that exist, either in the config or on the Metrics screen. A metric an experiment still uses cannot be deleted.
2. Turn it into an experiment
In the console, give the flag a split (for example 50/50) and mark it as an experiment. If you did not configure it in config/flags.ts, pick its metrics there too. Split assignments of experiment flags are recorded as exposures.
3. Read the variant and track conversions
async show({ flags, inertia }: HttpContext) {
const checkout = await flags.variant('checkout') // records the exposure
return inertia.render(`checkout/${checkout}`)
}
async store({ flags, request }: HttpContext) {
await db.transaction(async (trx) => {
const order = await Order.create(request.all(), { client: trx })
// Commits or rolls back with the order.
await flags.track('purchase', { value: order.total, client: trx })
})
}An exposure is recorded when the code uses the variant, not when a page merely receives it. Each subject has one exposure per experiment, however many times they visit.
ctx.flags.all() (the values sent to the browser) does not record exposures. When a variant only takes effect client-side, read it with ctx.flags.variant() in the controller that renders the page.
4. Read the results
node ace flags:experiments:show checkout
node ace flags:experiments:recompute checkout # recompute nowThe worker recomputes results every 10 minutes (resultsIntervalMs); the console shows the latest. For each metric and variant you get the lift, a confidence interval, a p-value and a verdict.
- Sample ratio mismatch (SRM). If a 50/50 experiment shows 54/46, assignment or logging is broken and the results are not trustworthy. It is checked automatically.
- Sequential p-value. The fixed-horizon p-value is only valid if you look once, at the planned end. The sequential one stays valid however often you look; stop when it says so.
- Guardrails alert when they get significantly worse (on the sequential test, so daily checks do not cry wolf).
Every event carries the environment that assigned it, so staging traffic never mixes into production results. The worker computes results for the process's environment; list more with resultsEnvironments: ['production', 'staging'], and switch between them in the console.
CUPED
metric.sum('purchase', { cupedMs: 14 days }) uses each subject's same metric in the 14 days before exposure to remove noise. Same estimate, much tighter interval when behaviour is stable, so experiments decide with fewer users.
Sticky assignment
Changing the weights of a running experiment moves some subjects to another variant. With sticky: true in the config, a subject keeps the variant of their first exposure. It reads the subject's exposures once per request from the lucid sink's table.
Mutual exclusion and the global holdout
- Layers. Experiments in the same layer never share a subject: give each one a bucket range of the layer in the console.
- Global holdout. A fixed percentage of subjects sees no experiment at all, so you can measure the summed effect of everything you shipped. An experiment can opt out.