“Which numbers should this agent report when it finishes?”
Nobody told it. Each run reports what it thinks matters, in its own words, or nothing.
AI agent metrics: Tell your agents which numbers to report.
In ConvOps, define a metric, bind it where the work runs and say when it is due. Each due run is asked for it, and a required one holds the step.
define · bind · when it is due · the gate · record
- definemissing_documents · whole number · count · lower is betterhow: count the documents the claim still needs
- bindproject · Claims intake · every runrequired · enforce · evidence
- guidancemetrics_due · missing_documentsstatus: missing · the step is heldstatus: recorded · the step can advance
- recordmetrics_record · missing_documents = 0evidence: the claim file link · who and where set by the server
in short
How do you tell AI agents which metrics to report?
ConvOps Metrics lets you define the numbers your agents must report, bind each one to a workspace, project, workflow or step, and hold that step until the number is recorded.
ConvOps is the operations layer for AI agents. It holds your team's process as workflows with gates, a shared memory, scheduled and unattended runs, programs, a plan board, metrics and a full audit trail. It does not sell or host models.
key facts · october 2026
- You define each metric once: its type (a number, yes or no, a choice, a date or text), its unit, which direction is better, and a hint telling agents how to collect it. An agent can propose a new metric; it stays a draft and takes no values until it is approved by whoever holds metrics_manage, so keep that tool out of agents' environments.
- A binding attaches a metric to a scope: the workspace, a project, a goal, a program, an initiative, a schedule, a task, a workflow or one step. The narrowest binding wins, and the target lives on the binding: at least, at most, between, a share or a yes rate, over a window.
- Conditions decide when a metric is due: always, on an outcome, on one run in n, when a step is slower than usual, once earlier attempts cost more than a set amount, or after a retry.
- In enforce mode, a required metric holds its step: the advance is refused and the response lists what is missing. There is no override; the ways past are observe mode, making the metric optional, an exclude binding, or cancelling. An observe-mode binding never holds a step and is never shown to the agent; ConvOps records whether each step would have met it.
- Agents read what is due on their step from ConvOps guidance and record it in one call: a single value, up to 500 items at once (all accepted or none) or a "not measurable" mark. The response lists what is still due.
- ConvOps stamps every value with who recorded it, in which run and at which step. The agent cannot set that; it supplies the value and, when asked, evidence as a link or a note.
- Each change to a workflow template leaves a marker on the trend of each metric bound to that workflow or one of its steps, and on the trend of each program whose schedules run it, so the runs after a change show whether it helped.
- ConvOps also measures 17 numbers itself, such as run cost, duration, tokens and outcome. Agents cannot write them.
best for
- Operations leads who want every agent run to leave its numbers behind: tickets resolved, accounts reconciled, files checked, cost.
- Banking, insurance and healthcare teams that need a check recorded, with evidence, before work moves on.
- Teams that change their workflows and agents and want to see whether each change made runs better.
not for
- Checking that a reported value is true. ConvOps records who reported it and where, and keeps the evidence link; it does not verify the value. Pair agent numbers with the ones ConvOps measures itself.
- Monitoring your own applications and servers. These metrics describe agent runs and the work they do.
updated
the problem
Nobody told the agent what counts.
Agent work is judged by feel: a prompt is edited, the next run looks better, and no number was asked for. ConvOps asks every run for the numbers you define.
“Did that prompt change make the work better, or does it only feel better?”
Without the same number recorded on every run, each change is judged by whoever read the last one.
“Who checked the figures before the batch went out?”
A checklist in the prompt is a request. Nothing holds the step when the count is missing.
define a metric
Define the number once.
Name a metric, pick its kind of value, its unit and which way is better, and write how an agent measures it. ConvOps shows that hint wherever it is due.
- key
- kyc_file_complete
- name
- KYC file complete
- kind
- yes or no
- one value per
- customer file
- unit
- none
- better
- up
- breakdown
- risk tier
- how agents measure it
- Yes only when every required document is on file and in date.
Evidence is asked for where you bind the metric: a link or a note for whoever reviews the value. ConvOps stores the evidence and never fetches it.
- Any kind of valueA number, a whole number, yes or no, one choice or several, a date or text.
- Per run or per itemOne value per run, or one per item, such as each invoice, ticket or customer file in a batch.
- Up to three breakdownsSplit a metric by region, locale or risk tier, and read it grouped later.
- Derived numbersCompute a metric from others, such as cost per 1,000 words or the share of runs that needed a person.
payload
{
"action": "define",
"key": "cost_per_1000_words",
"name": "Cost per 1,000 words",
"direction": "down",
"formula": "m('run_cost_usd', 'sum') / m('words_translated', 'sum') * 1000"
}bind it
Bind AI agent KPIs where the work runs.
In ConvOps, bind a metric to a workspace, project, goal, program, initiative, schedule, task, workflow or a single step. The runs below that point inherit it.
- workspaceRetail bankexceptions_raised · observe
- projectKYC operations
- goalClean customer files
- programKYC refreshkyc_file_complete · enforce
- initiativeQ4 refresh wave
- scheduleNightly batch
- taskBatch of 24 October
- workflowKYC reviewsanctions_hits · observe
- stepSubmit batchqa_score · enforce
widest at the top · a narrower binding overrides a wider one
- The narrowest binding winsBind at the workspace, override on one step, and exclude a sandbox project.
- The target sits on the bindingAt least, at most, between, a share or a rate, over a window of 1, 7, 30 or 90 days.
- Evidence when you ask for itRequire a link or a note with each value, for whoever reviews it.
- Observe, then enforceTry a new metric in observe mode first. The agent is not asked and no step is held, and ConvOps records whether each step would have met it.
example · sales
Replies per outreach batch
Bound to the outreach project, so every batch records the replies it earned, with the thread as evidence.
project binding · evidence
example · content
Claims checked per article
Bound to the fact-check step, so each article records how many claims were checked, with the source link for the editor.
step binding · evidence required
example · finance opsenterprise
Mismatches per nightly reconciliation
Bound to the close program with a target of at most 0, so a night with any mismatch reads as a missed target.
program binding · target: at most 0
example · bankingenterprise
Every refreshed KYC file, complete or not
Bound to the KYC refresh program: one value per customer file, up to 500 files in one batch.
one value per item
when it is due
Say when it is due.
Each binding says when its metric is due. ConvOps checks that condition on every step response and again when the agent tries to advance.
- alwaysEvery runThe default: no condition, so it is due on every run in scope.e.g. replies per outreach batch
- on_outcomeOn an outcomeWhen the latest ended attempt had a given outcome, such as a model failure or a time-limit stop.e.g. a failure category after a failed run
- sample · nOne run in nAbout one run in n. A run is in the sample or not, every time it is checked.e.g. a QA score on one claim in ten
- after_retryAfter a retryFrom the second attempt on.e.g. the cause of the retry
- cost_over · usdWhen earlier attempts cost moreOnce the attempts before this one cost more than a set amount.e.g. a cost reason once retries pass $3
- slower_than_usualSlower than usualWhen time on the step is above its usual range, once the step has enough history.e.g. a delay reason on a slow close step
A condition that does not hold leaves the metric off that run, and the preview says why.
example · insuranceenterprise
A QA score on one claim in ten
About one settle run in ten is asked for a QA score with evidence. The other nine move on.
when: a sample of 1 in 10
example · procuremententerprise
A cost reason when retries add up
Once earlier attempts on a quote cost more than $3, the agent is asked to pick a cost reason from a short list.
when: earlier attempts cost over $3
the gate
A step waits for its numbers.
Make a metric required on a step. In enforce mode ConvOps does not advance the workflow until the metric is recorded, with evidence when the binding asks for it.
Draftwriter agent
Check claimsfact-check agent
Required metricrequired metric
Publishthe step moves
There is no override on the gate. The step moves once the metric is recorded, or marked not measurable where the binding allows it, or when someone changes the binding.
A parent task waits on its step too. A task without a workflow cannot be completed while a required number is missing. Cancelling a task is never held.
example · content
Publishing waits for the fact-check count
The publish step holds until the agent records how many claims it checked, each with its source.
required · enforce · evidence
example · supportenterprise
Triage closes once the categories are in
The agent records one category per ticket, up to 500 in one batch. The close step waits for that batch.
required · one value per ticket
example · healthcareenterprise
Every prior-auth packet carries its count
The binding does not allow not measurable, so each packet records its missing fields before submit.
not measurable: refused
what the agent sees
The agent is told, then it records.
Each step response lists the metrics due on that step, how to measure each one and its target. The agent records with metrics_record, and the server stamps who and where.
- kyc_file_completeenforce
- qa_scoreenforce
- sanctions_hitsobserve
- exceptions_raisedobserve
before Submit batch runs, the agent is asked for
- kyc_file_complete
- qa_score
qa_score: this run is in the sample. Observe-mode items are not asked and never block.
payload
{
"current_step": {
"guidance": {
"metrics_due": [
{
"key": "kyc_file_complete",
"value_type": "boolean",
"how": "Yes only when every required document is on file and in date.",
"target": {
"kind": "rate_of_yes",
"value": 0.98,
"window": "7d"
},
"required": true,
"status": "missing"
},
{
"key": "qa_score",
"value_type": "integer",
"how": "Score the batch from 1 to 5 against the QA checklist.",
"required": true,
"status": "missing"
}
]
}
}
}- One value or a whole batchRecord one value, or up to 500 items in one call, all or nothing.
- Not measurable, with a reasonWhere the binding allows it, the agent marks a number not measurable and says why.
- Who and where, set by the serverEvery value carries who recorded it, in which run and at which step. The agent cannot fill that in.
- Preview before it runsSee which metrics a run will be asked for before it runs, with a warning for required items that can never apply.
change markers
See whether a change made runs better.
In ConvOps, each change to a workflow template leaves a marker, with who made it, on the trend of each metric bound to that workflow or one of its steps, so you can see whether runs got better after it.
W1W2W3W4W5W6W7W8W9
Agents and people read every recorded number: compare with the previous period, group by a breakdown, filter by program, project, workflow or step.
- A marker on every changeEach workflow save leaves a marker, with the new version and who saved it, on the metrics bound to that workflow or its steps.
- Notes of your ownAdd a note of up to 500 characters to a metric or a program. It shows on the trend as a marker.
- Any window, any grainFrom 24 hours to a year, by hour, day, week or month.
Save and share metric dashboards.Planned
plus run numbers
Plus the numbers every run records itself.
ConvOps records the cost, duration, queue wait, tokens, turns and outcome of every run on its own. Agents cannot write these numbers.
- outcome
- succeeded
- started by
- schedule
- cost
- $1.71
- duration
- 13m 05s
- queue wait
- 42s
- tokens in · out
- 176,000 · 22,000
- turns
- 38
- Cost, time, tokens, turns, outcomeWritten when each run ends, with queue wait, what started the run and its error class.
- Lead time, cycle time, step timeTask lead and cycle time on completion. Time on a step, and whether each due metric and report was there, on every passing advance.
- Out of the agent's reachAn agent cannot record the numbers it is judged on. A write to a system metric is refused.
- Read beside your ownGroup, filter and compare them with the numbers your agents report, by program, project, workflow or step.
proposed metrics
Agents can propose a metric. You decide what becomes active.
In ConvOps an agent can propose a metric. It stays a draft and takes no values until it is approved by whoever holds metrics_manage, so keep that tool out of agents' environments.
qa_disagreement
QA disagreement
Share of settled claims where the QA reviewer disagrees with the settle decision.
- draft · no values
- approved
- recording
- A draft, not a metricA proposed metric records no values until it is approved. The app lists drafts on a proposed tab.
- Approve or rejectWhoever holds metrics_manage approves it, and it becomes active and starts recording. Reject it and it never records.
- Open lists grow, people tidy themOn an open choice list an agent can add a new option. People relabel, promote or merge options later.
recursive self-improvement
Recursive self-improvement, with the bar held by people
Recursive self-improvement is a loop where agents edit their own workflows, record metrics against targets people set, and read the results to decide the next edit.
Self-improving on recorded numbers, not on how a run felt.
In ConvOps the loop is a workflow you build: continuous improvement for AI agents, measured on the numbers you define. Every number a run records can be read by agents and people. A scheduled agent reads the trend, changes a workflow, an agent's instructions, a model tier or a schedule, and the next runs show whether it worked. People set the metrics and the targets. Add an approval gate to hold a change until an explicit approval is given, recorded with the account that gave it.
A run edits only what you grant it. Without a grant, a run changes no configuration: it works its task, records metrics and files reports. The model's weights never change. The loop changes the system around the model: workflows, instructions, model tiers and schedules.
Readthe trend
Proposea report or note
Holdoptional gate
Applygranted edits
Markchange marker
Measurethe next runs
| Step | What happens | What it needs |
|---|---|---|
| Trigger | A schedule starts the improvement workflow, every Monday for example. | A schedule |
| Read | The agent reads the last 30 days against the previous period, step by step. | A grant to read metrics |
| Decide | It compares the numbers with the target a person set on the binding. | A target on the binding |
| Propose | It writes the change it wants in a report or a task note. | Nothing extra |
| Hold | An approval gate holds the step until an explicit approval is given, recorded with the account that gave it. | Optional, your choice |
| Apply | It edits a step, an agent's instructions, a model tier or a schedule. | A grant for each edit |
| Mark | The workflow save leaves a marker, with who made it, on the trend of each metric bound to that workflow or its steps. | Automatic |
| Measure | The next runs record the same metrics. The improver is a workflow too, under metrics of its own. | The metric bound and required |
| What could drift | What protects it | What you keep |
|---|---|---|
| Who recorded a value | The server sets who recorded it and at which step. The agent cannot fill that in. | Nothing to set: it is always on |
| What a metric means | A proposed metric stays a draft and records no values until it is approved. | metrics_manage, which defines and approves metrics, out of the improver's grants |
| The bar | The target sits on the binding, not on the metric and not with the agent. | Bindings out of the improver's grants |
| The numbers judged | Cost, time, tokens, turns and outcome are recorded by ConvOps. Agents cannot write them. | Targets bound to these where one fits |
The server records who and where. The agent supplies the value and the evidence, which is a link or a note for a reviewer, not a check.
Give the improver one environment with metric reads and the edit tools it needs. Keep metric definitions, bindings and environment grants in a second environment that people hold.
Today each change to a workflow template leaves a marker, with who made it, on the trend of each metric bound to that workflow or one of its steps, and on the trend of each program whose schedules run it. Roll back a workflow change in one step.Planned
Agents record the numbers. People set the bar.
example · supportenterprise
Every Monday, fix the slowest step
An agent compares step time with the previous period, edits the instructions of the slowest triage step and files a report. Next week shows whether it held.
schedule · metric reads · step edit
example · operations
A model tier change, judged on what agents report
Agents report fields extracted per document. An agent points a tier at another model and compares that count with its target, with run cost and duration as guards. Every agent on the tier follows on its next run.
model tier edit · reported metric · target
example · finance opsenterprise
Month-end close, instructions revised
An agent saves a new revision of the reconciler's instructions. If mismatches rise, a person sets the earlier revision active again.
agent revisions
for engineers
The calls behind each section.
The ConvOps actions an agent or a script uses to record, bind and read metrics. Every payload here is an example.
Record a value with evidence
An agent records against a metric bound to its step. Provenance comes from the run, and the response lists what is still due.
payload
{
"action": "record",
"key": "claims_checked",
"value": 23,
"evidence": {
"url": "https://docs.example.com/fact-check/42",
"note": "23 of 23 claims sourced"
}
}Bind a metric to a step
Required and enforced, with a target, evidence and no not-measurable escape. A step binding names the workflow and the step.
payload
{
"action": "bind",
"kind": "metric",
"ref_id": "met_claims_checked",
"level": "step",
"target_id": "tpl_publish_article",
"step_id": "check_claims",
"mode": "enforce",
"config": {
"required": true,
"evidence_required": true,
"allow_not_measurable": false,
"target": {
"kind": "at_least",
"value": 10,
"window": "7d"
}
}
}Read a trend against the previous period
The 90th percentile of time on one step, by week, for 30 days, beside the 30 days before.
payload
{
"action": "query",
"key": "step_dwell_ms",
"agg": "p",
"agg_arg": 90,
"window": "30d",
"grain": "week",
"filters": {
"template": "ticket-triage",
"step": "classify"
},
"compare": "previous_period"
}Self-hosted
Recording, bindings and the gate run on PostgreSQL. Trends need OpenSearch.
questions
AI agent metrics questions, answered.
What is ConvOps Metrics?
ConvOps Metrics lets you define the numbers your agents must report, bind each one to a workspace, project, workflow or step, and hold that step until the number is recorded. Every value carries who recorded it and at which step. ConvOps also writes the cost, duration, tokens, turns and outcome of every run itself, and marks each workflow change on the trends of the metrics bound to it.
How do I tell an AI agent which metrics to report?
Define the metric once in ConvOps, bind it to the workspace, project, workflow or step where the work runs, and say when it is due. Each due run finds the metric in its step guidance, with how to measure it and its target, and records it with metrics_record. Make the binding required and set it to enforce, and the step waits until the number is recorded.
How do you measure AI agent performance?
Measure it with numbers the platform records and numbers the agent reports, read side by side over time. ConvOps records the cost, duration, queue wait, tokens, turns and outcome of every run by itself. Your team defines the rest, such as replies per batch or claims checked, binds each one to the work and sets a target on the binding. Then compare any window with the previous period, by program, project, workflow or step.
How do you stop an agent gaming its metric?
Keep the parts that decide the score out of the agent's hands. In ConvOps the cost, time, tokens, turns and outcome of a run are written by the platform, so an agent cannot write those numbers. The server sets who recorded each value and at which step. A metric an agent proposes stays a draft until it is approved, and the target sits on the binding. Keep the metric and binding tools out of the agent's grants.
Who is ConvOps Metrics for?
ConvOps Metrics is for teams that run AI agents on repeatable work and need every run to report the numbers that show whether the work was done. That includes operations leads who want a step held until a count is in, finance and compliance teams that want evidence with each value, and team leads who read the same reported numbers by project, workflow or step. It fits research, sales, content, support and back-office workflows alike.
Can a workflow step wait until a metric is recorded?
Yes. In ConvOps you make a metric required on a step and set its binding to enforce, and the workflow does not advance until the metric is recorded. The refusal names what is missing. There is no override on the gate: the step moves once the metric is recorded, or marked not measurable where the binding allows it, or when someone changes the binding. Observe mode never blocks and records whether each step would have met it.
What if a number cannot be measured on a run?
The agent marks it not measurable with a reason, and ConvOps accepts that in place of a value unless the binding forbids it. A binding that does not allow it keeps the step held until a real value is recorded. That suits checks with no valid excuse, such as the missing-fields count on a prior-authorisation packet.
Can AI agents improve their own workflows with ConvOps?
Yes, when you build the workflow for it and grant the tools it needs. In ConvOps a scheduled agent can read the recorded numbers, change a workflow, an agent's instructions, a model tier or a schedule, and the next runs show whether it worked. A run edits only what you grant it. People set the metrics and the targets. Add an approval gate to hold a change until an explicit approval is given, recorded with the account that gave it.
Does ConvOps retrain the model when agents improve a workflow?
No. ConvOps never changes model weights. What changes is the system around the model: the steps of a workflow, an agent's instructions, the model a tier points to and the schedules that start the work. Each workflow change leaves a marker, with who made it, on the trend of each metric bound to that workflow or its steps, so you can see whether runs got better after it.