How to evaluate an AI employee: eight questions before you hire

A demo runs on material the vendor chose, guided by a person who knows where the edges are. Your work has gaps, old versions and exceptions that no demo includes. NIST's AI Risk Management Framework says accuracy measurements should be paired with realistic test sets that represent the conditions of expected use. For a small business, that means your documents, your role and your reviewer, not the sample in the sales deck. A demo can tell you whether a trial is worth running. Only the trial tells you whether the product does your job.

Ask them in this order. The early questions decide whether a trial is worth running at all, and the later ones decide whether you can live with the product after it starts. Put the same questions to every vendor, in writing, and keep their answers next to your scorecard. Answers that change between the sales call and the trial are evidence too.

• What is the job, in one sentence, and what is outside it?
• What does it work from, and can you see the source behind each output?
• What can it touch: which systems, which data and which actions?
• Who approves its work before anyone else sees it?
• What happens when it is wrong, and how do you find out?
• Which capabilities work today, which are in pilots and which are only planned?
• How much of your team's time does it take each week?
• What do you keep if you stop: your content, your records and your exports?

Write the job sentence before the trial starts, and hold the product to it. Then give it one real source, such as a procedure your team actually uses, and ask for one output. Check whether every step in that output can be traced back to the source. Ask the vendor to show you the permission screen instead of describing it, and grant only what the trial needs. Name your approver before the first draft arrives, so nothing reaches your people unchecked. Then try to break the rule once: ask the product to send or publish something without approval, and note what it does.

NIST's Generative AI Profile lists confabulation among the risks of generative AI: content stated with confidence that is wrong or false. So plant a test. Remove one step from a copy of your procedure and see whether the draft invents the missing step or flags the gap. A product that flags it has earned some trust. Then test the vendor. Ask them to label every capability they show as available now, confirmed in pilots or planned. A vendor who cannot do that leaves you to find the difference after you sign.

An AI employee is not free to run just because the subscription is paid. Someone gives it sources, answers its questions, reviews its drafts and fixes what is wrong. During the trial, log every minute your people spend on it, by person and by task. Count the time to fix a confident mistake separately, because that is the cost a demo never shows. At the end, compare those minutes with what the same job took your team before. If the product saved drafting time but your expert now spends longer checking, write that down. It is the number you will live with every week.

Ask where your sources are stored, who at the vendor can see them and how you delete them when you leave. Ask whether your material is used to train models for other customers, and get the answer in writing. Then check the exit row with a real export. Training content should leave in formats other tools can read, such as SCORM or xAPI packages for courses and MP4 files for video. Progress records should come out as a file you can open. If an export only works with the vendor's help, score it as locked.

Use one row per question and write one line of evidence next to each score. Zero means you saw no evidence, or the evidence showed the problem. One means the vendor described it or showed it on their own material, but you have not seen it on yours. Two means you saw it in your own trial, on your own material, and wrote down what you saw. Some results end the trial whatever the total says. Compare totals only between products that passed every stop condition below.

• Stop if the product sends, publishes or changes anything without the named approver
• Stop if it needs access beyond what the job sentence requires and the vendor cannot narrow it
• Stop if a planted gap comes back filled with an invented step that nothing in the output marks
• Stop if the vendor cannot say whether a capability you need is available now, confirmed in pilots or planned

A fictional locksmith company tests an AI employee on one job: a first-week course for new dispatchers who book lockout calls. The owner keeps the trial small on purpose, so every row of the scorecard can be filled with something he saw.

• The owner writes the job sentence and leaves price quotes out of it, since those stay with him
• He gives it the call script and the van coverage map, with one step removed from a copy of the script
• The draft lesson flags the missing step and asks for it, which earns a two on the errors row
• The office manager approves the lesson and logs her review time on the scorecard
• The vendor labels a call summary feature from the demo as planned, so the owner scores it as not available

We hold ourselves to the same scorecard. Elvina's job is training. Drafting lessons, job aids, quizzes and video lessons in Creation Studio from your sources in Company Knowledge is available now. Approval by a person before anything is assigned on the Learning Platform is available now. Translation, SCORM and xAPI export and progress in the Web workspace are available now as well. Your materials are used to build your training. Model-training terms depend on your plan and applicable agreement. See the ContentBuilder Privacy Policy and your workspace agreement for data use, service providers and available opt-out choices. Data is encrypted in transit.

A pre-hire skills quiz and a first-week readiness check are available now. Five things on our list are planned, not available today: reminders, progress digests, answers to employees in Chat, scheduled reports to managers and Elvina working on her own. She does not act in your other systems, recruit or decide who is ready. Today our team runs the build with you on the platform; Elvina takes on each capability as it is confirmed.

Score the trial on evidence you could show to someone who was not in the room.

• Outputs traced to a named source, out of all outputs
• Planted gaps flagged versus gaps filled with invented steps
• Reviewer minutes per output, and how they change across the trial
• Actions taken that you did not approve, which should be none
• Capabilities shown in the demo that turned out to be planned

How do I evaluate an AI employee? Give it one real job with your own source, a named approver and a written job sentence. Score what comes back on sources, access, approval, errors, status, your team's time and exit. Judge it by evidence from that trial, not by the demo.

• How long should a trial run? Long enough to see one complete job: source in, draft out, review, approval and use by at least one real person. For a training role, that usually means one first-week course for one role, reviewed by the person who knows the work.
• Is an AI employee the same as an AI agent? The terms overlap, and there is no single agreed meaning for AI employee or AI coworker. An agent is usually built to finish a task. In Elvina's terminology, an AI employee holds a defined job over time, works from approved knowledge and hands its work to a person who approves it. You can evaluate any of them with the same questions.

Public sources, checked on the date shown.

• NIST's AI Risk Management Framework says accuracy measurements should be paired with realistic test sets representative of expected use. Source: https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-1.pdf (checked 2026-10-02).
• NIST's Generative AI Profile defines confabulation as confidently stated but erroneous or false content that may mislead users. Source: https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf (checked 2026-10-02).