Evidence, not assurances
Controlled experiments that show how AI models perform on your task
I design bounded evaluations to investigate what selected models get right on your workflow, how they fail, and where human verification remains necessary — including alternative and lower-cost models, not only frontier systems.
You receive an evidence-backed report of tested capabilities, failure patterns, costs, and limitations — not a guarantee of production performance.
How an engagement works
You bring the workflow and the question. I design a bounded evaluation to investigate it. An engagement begins only after we agree in writing on the question, evidence standard, deliverables, and exclusions.
Scoping in writing. You supply a sanitized description of the workflow — what starts it, what it must produce, and what makes a result acceptable, unacceptable, or uncertain — together with the decision you need to make.
Test design. I turn your requirements into representative tasks, acceptance checks, and trial conditions. Domain questions are resolved by your subject-matter expert; a model's answer is never treated as authority for your business rules.
Controlled trials. Repeated runs of the selected models, with usage, cost, and elapsed time recorded where available and failures preserved as evidence.
Findings and handoff. A written report covering what was tested, what failed and how, what still required human verification, what it cost, and what remains uncertain. A reusable benchmark your team can rerun is a separate, commissioned deliverable.
See the full services and client boundaries for required inputs, written communication terms, deliverables, and exclusions.
Questions an evaluation can answer
- How do selected models perform on this task?
- What mistakes do they make, and how consequential are those mistakes?
- Which outputs or decisions still require human checking?
- Do specified prompt, context, or input changes improve the observed results?
- What tradeoffs appear among accuracy, coverage, uncertainty, cost, and elapsed time?
- Can the evaluation become a repeatable benchmark for future trials?
Proof of work
The methods and capabilities described here are supported by a published, inspectable portfolio: four completed evaluations and one observational study. The completed evaluations publish their inputs or runnable tasks, recorded outputs, verification method, and bounded findings. The observational study publishes a product-abstracted evidence narrative because the underlying application remains private.
| Evaluation | Capability demonstrated | Recorded result |
|---|---|---|
| Broadcast Conform | Turning detailed deliverable requirements into objective acceptance checks | 20 trials; routes scored 0/5 to 5/5; failures isolated by decoded-signal checks |
| Linker Performance Harness | Detecting misleading measurements and verifying the evidence behind a performance claim | 15 balanced trials; every failed submission broke the controlled protocol |
| Clip Provenance Comparison | Comparing accuracy, uncertainty handling, coverage, and remaining review work | Three isolated workflows produced materially different pipelines against withheld ground truth |
| AI Conversation Review | Assessing the reliability of AI-assisted professional review | Three model reports checked against a fixed answer key and reviewed for evaluator error |
| Agent Engineering Loops | Examining whether completion claims are supported by actual acceptance evidence | Three reconstructed episodes, with agent claims separated from verified outcomes |
How to read the results
A note on evidence
A passing artifact is not automatically a good process, and a polished report is not automatically correct. Small trial sets support findings about these tasks and runs; they do not establish universal model rankings.
Contact
To discuss an evaluation, email a brief, non-confidential description of your workflow and the decision you need to make. All project communication is written — email or another agreed channel — so requirements, questions, decisions, and scope changes stay documented throughout.