Evaluation cases for procurement agents
The bid changed.
Did your agent?
Test how your agent responds when a solicitation, amendment or clarification changes the work. Commission cases with staged evidence and explicit acceptance criteria.
For GovCon product teams and contractors evaluating AI workflows.
Solicitation · section L.4
Offerors must include an on-site staffing plan.
- Expected action
- Include the staffing plan in the compliance checklist.
- Acceptance criterion
- The checklist cites L.4 and includes the staffing-plan deliverable.
A document answer can be right.
The next action can still be wrong.
Your agent needs to notice which requirement changed, retire the old one, update its work and cite the evidence. These are the behaviors a pilot can test.
A case pack your team can inspect.
Use it with your current evaluation process. We agree on the workflow, sources and scoring rules with your reviewer before assembling the pack.
- Evidence
- Document snapshots grouped by release stage, with source identifiers.
- Tasks
- Concrete actions such as revising a compliance checklist after an amendment.
- Scoring
- Expected actions, acceptable alternatives and source-linked criteria.
- Delivery
- JSON case files, readable case notes and one reviewer calibration session.
Start with one failure mode.
$1,500 fixed-scope pilot
12 cases around one agreed procurement task, with up to three document-release stages per case. One scoring review call of up to 60 minutes and one factual revision pass.
Public or permissioned documents only. We confirm source availability, case-use rights, review responsibilities and delivery dates before you commit. The case structure and annotations are delivered for internal evaluation use; source rights, redistribution and exclusivity are agreed in the engagement terms. Specialist GovCon review can be scoped separately.
What about our existing internal evaluations?
Keep them. The question is whether an additional set of amendment-driven cases exposes failures your current tests miss. We’ll check that fit before proposing work.
Does this prevent benchmark contamination?
No case pack alone guarantees that. We can scope held-out documents and counterfactual variants, and record the limitations of each case.