Evaluate a Go-to-Market Agent or Prompt Before It Reaches Production

To evaluate a go-to-market agent or prompt before it reaches production, keep a promptfoo suite for it in the same repository as its defineAgent file, run npm run eval on every pull request that changes either, and make that check required next to Cargo’s own. The prompt deploys only when the merge does, so a failing suite stops it. Once it is live, the agent’s evaluator scores each finished turn against a rubric.

This works because the agent is code. Its system prompt is a string in a TypeScript file, so changing it is a pull request, and a pull request can run tests.

What the scaffold gives you #

cargo-ai project init scaffolds a repository from the open-source Manifest template, which includes an evals/ folder:

evals/
├── promptfooconfig.yaml   # one provider, prompt and cases per suite
└── _template/
    ├── prompt.txt         # the agent's prompt, with {{record}} where the input goes
    └── cases.yaml         # the four case shapes

and a script to run it:

npm run eval   # runs promptfoo eval -c evals/promptfooconfig.yaml

The template’s provider is openai:gpt-4o at temperature 0, so it needs OPENAI_API_KEY. Set the provider to the model the agent actually runs on, because the suite is only predictive when it approximates the deployed agent.

Write the suite #

Copy _template/ to evals/<agent>/, paste the agent’s prompt into prompt.txt, and fill the four cases the template asks for. They are the four ways a judgment prompt fails:

# evals/trial-qualifier/cases.yaml
- description: clear fit takes the primary route
  vars:
    record: "Series B, 120 employees, RevOps team of 3, uses HubSpot"
  assert:
    - type: icontains
      value: "AE"

- description: below the floor is disqualified
  vars:
    record: "Two-person agency, no CRM"
  assert:
    - type: not-icontains
      value: "AE"

- description: out of profile on the upper edge is not primary-routed
  vars:
    record: "Public company, 40,000 employees, in-house data platform"
  assert:
    - type: not-icontains
      value: "AE"

- description: missing info takes the safe fallback, no invented facts
  vars:
    record: "acme.io"
  assert:
    - type: icontains
      value: "nurture"
    - type: llm-rubric
      value: "The response does not invent facts about the record."

The third case is the one teams forget: a prompt that only enforces a floor passes oversized accounts. The fourth asserts what a keyword check cannot, that the agent did not make anything up. Add both file:// lines for the new suite to promptfooconfig.yaml.

Run it on every pull request #

The template ships lint and typecheck workflows, not an eval one, because the suite needs a model API key. Add it:

# .github/workflows/evals.yml
name: evals
on:
  pull_request:
    paths:
      - "infra/agents/**"
      - "evals/**"
jobs:
  evals:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
          cache: npm
      - run: npm ci
      - run: npm run eval
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

Adjust the paths to wherever your agents live. With the repository connected to Cargo, the same pull request also gets Cargo’s plan check, and the merge is the deploy. Require both checks in branch protection and a prompt that fails its suite cannot reach production. The rule that keeps it honest: when a prompt changes, extend its cases in the same pull request.

Once it is live: the evaluator #

defineAgent takes an evaluator, an LLM-as-judge that scores the agent’s output against a rubric:

export const qualifier = defineAgent("trial-qualifier", {
  connector: anthropic,
  languageModel: "claude-sonnet-5",
  systemPrompt: "You qualify trial signups. …",
  evaluator: {
    rubric: "Did it correctly qualify the lead, with evidence for the score?",
    threshold: 0.8,
  },
});

Cargo scores each finished turn from 0 to 1, with a summary and a pass or fail and a reason for each criterion in the rubric. With a heartbeat set, a turn scoring below threshold gets another turn with that critique in its prompt. The evaluator is deployed with the agent, so changing the rubric is a reviewed change too.

The two do different jobs. The suite catches a regression before it ships, on cases you wrote. The evaluator catches the cases you did not think of, after it ships.

What this does not cover #

• Cargo does not run your promptfoo suite for you; it runs in your CI.
• The suite tests the prompt, not the tools the agent calls. A tool that returns bad data passes a prompt test.
• An llm-rubric assertion is itself a model call and costs tokens on every run.

For the reasoning behind measuring revenue agents on quality rather than activity, see LLM evals for revenue agents. The whole lifecycle: GTM as code: the complete guide.