How to Evaluate an AI Proposal Before Hiring

An AI proposal can include a convincing demo and still leave the important questions unanswered: what happens when information is missing, who authorizes an action, and how much work remains for your team? Before hiring, I would look for answers you can test, not a list of models and tools.

You do not need to master the architecture. You do need to know what you are buying, how you will check that it works, and when to stop.

First: do you need an agent?

Ask the supplier to describe the current process: inputs, decisions, owners, and expected output. Then ask it to compare alternatives.

If the steps and rules are stable, conventional automation may be enough. If you only need to classify messages or extract fields, a model call inside a fixed workflow might suffice. An agent makes more sense when it needs to choose steps or tools based on what it finds. That flexibility also requires more controls.

My question would be: what decision must the system make that explicit rules cannot reasonably handle? Without a concrete answer, I would ask for a simpler option as a reference for cost and complexity.

Test the actual work, not just the demo

A demo shows that something is possible. An evaluation should reveal where it works and where it fails.

Agree on representative tasks before tuning the system: common requests, ambiguous cases, incomplete information, and situations it should reject. Use anonymized or authorized data, and hold back some cases for the final evaluation.

Compare against the current process or a simpler alternative using the same cases. Measure more than speed: include correctness, human review time, critical errors, and cost per accepted task. Define who reviews the answers and what rubric they use. If your team rewrites almost everything, that effort belongs in the comparison.

Ask about failure, recovery, and approval

I would ask the supplier to walk through a specific failure from start to finish:

• If a tool stops responding, does the system retry, pause, or hand off to a person?
• If an operation repeats, how does it avoid sending two messages or creating duplicate records?
• If evidence is missing or sources disagree, does it ask for clarification or invent an answer?
• Which actions need human approval, and what information does the reviewer see?
• How can someone stop execution, inspect what happened, and resume without repeating completed actions?

“There is a human in the loop” is not enough. The proposal should identify the approval point and what remains blocked until approval arrives. An irreversible action needs different safeguards from an editable draft.

Set boundaries around data access

Ask for a simple map: which sources the system reads, where it writes, and what information goes to external providers. Separate read and write permissions; do not grant administrative access for convenience.

The proposal should also explain conversation and log retention, deletion, supplier access, and separation between customers or teams. A document’s contents should not be able to expand the system’s permissions. Ask how that boundary is tested, not just what the prompt says.

Expect an explainable architecture, not a fashionable one

In Ketzai Agent, the system my team is building at Ketzai, a modular architecture supports different educational tasks through shared coordination. I lead the team’s work on orchestration, routing, and the core runtime connecting context, generation, and review, along with tools for testing and evaluation.

That design serves different educational tasks that share orchestration. It is not a prescription for every project. What I would expect in any proposal is the same clarity: how the work is organized, who controls each step, and how changes are checked.

Define the pilot before signing the contract

The pilot should deliver more than a demo: documented scope, test cases, results including failures, agreed code and configuration, operating instructions, and a recommendation to proceed or stop.

This checklist is illustrative, not a reported result or a universal threshold:

• Evaluate 30 agreed requests, including incomplete inputs and tool failures.
• Accept at least 27 under the agreed rubric; record corrections and review time.
• Block all unauthorized writes and hand off out-of-scope cases.
• Keep cost per task and latency within limits agreed before work begins.
• Do not launch if a critical control fails, even when average output quality is good.

Also clarify who owns the code, configurations, and accounts; what another supplier would receive to maintain the system; and who handles incidents. The operating budget should include models, infrastructure, observability, human review, and maintenance, with explicit volume assumptions.

To assess AI opportunities in your business, explore my AI consulting and implementation services or tell me about your processes and needs. You do not need a defined solution.