Agents Autonomous
PE portfolio companies / A practical workflow

Evaluate an assistant change before copying it elsewhere

One company wants to update its internal assistant. A reusable evaluation method is helpful, but the other companies ask different questions.

Plan this workflow ↗See how it works ↓
Two small wooden test benches compare an old and proposed assistant mechanism using the same sample cards, while a second company's separate sample box sits unopened. Evaluation evidence before copying a change elsewhere.
From input to handoff

The source stays
with the work.

One possible workflow, with the preparation and review responsibilities made clear.

  1. 01 / The starting point

    A sanitized, company-specific question set, approved source versions, known failures, and baseline and candidate model configurations.

  2. 02 / The right context

    Compare the same tasks with source support, refusal behavior, review effort, latency, and cost recorded as relevant. Keep failure categories and sampling limits visible.

  3. 03 / Prepared for review

    A change-review record and failure queue, plus a checklist of what another company would have to validate independently.

  4. 04 / A person decides

    The local AI workflow owner judges regressions; the authorized technology owner separately decides whether to approve a rollout.

Tangible work

What the next person receives.

A versioned test folio displays paired result cards with a copper-tabbed regression, a separate question set and a closed transfer box. A human approval lever remains down until local evidence is reviewed.
01

A prepared working record

A change-review record and failure queue, plus a checklist of what another company would have to validate independently.

02

A clear review route

The local AI workflow owner judges regressions; the authorized technology owner separately decides whether to approve a rollout.

03

An agreed scope

A strong result in one company is not portfolio-wide evidence. Production changes require a separate decision.

Agree what good looks like

A useful result
can be checked.

The acceptance decision

A candidate’s use of an obsolete policy in an exception remains visible as a separate failure for the local owner to assess.

Include the difficult case

Try a candidate that answers routine questions well but quotes an obsolete policy in a sensitive exception. Keep that failure separate from the average and ask the local owner whether it blocks the change.

Keep the scope clear

A strong result in one company is not portfolio-wide evidence. Production changes require a separate decision.

How we validate the work ↗
A first conversation

Bring the job.
We’ll shape the brief.

Start with a general outline of this workflow and the team that owns it.

Plan this workflow ↗

Three useful questions

  • Which company-specific questions and known failures should baseline and candidate both face?

  • Which source versions and acceptance criteria will the local workflow owner use?

  • Who assesses regressions, and who separately decides whether to approve a rollout?

A few useful answers

Before we begin.

What does the reviewer receive?

A change-review record, failure queue, and checklist for separate company validation.

Who reviews the result before anything changes?

Local AI workflow owner.

What stays outside this scope?

A strong result in one company is not portfolio-wide evidence. Production changes require a separate decision.

Which difficult case do we test first?

Try a candidate that answers routine questions well but quotes an obsolete policy in a sensitive exception. Keep that failure separate from the average and ask the local owner whether it blocks the change.

How do we know it works?

A candidate’s use of an obsolete policy in an exception remains visible as a separate failure for the local owner to assess. Agree this acceptance check before work starts.