AI Receptionists
How to Test an AI Receptionist Before It Answers Real Calls (2026)
Short answer: Test an AI receptionist in four layers before it answers a real customer. First, chat with it in text until the basics work. Second, save every important caller type as a graded test case and run the whole suite after every change. Third, call it in a browser to hear the voice and the interruptions. Fourth, place real phone calls to check transfers and texts. Mock anything that books, charges or writes to your systems, require each case to pass more than once, and keep the old version live until the new one passes.
That's the routine we run at Wrk Less on every voice line we manage. A cleaning company's line has a 26-case suite, and 6 of those cases are replays of real calls. An accounting office's line has 21 cases, and each fence line has 12. Nothing gets published to a phone number until its suite passes. Below is how we build those suites, what Retell's own testing tools do (checked November 15, 2026), and what each layer costs.
The four ways to test, and what each one catches
Retell's testing overview splits testing into four methods. We use all four, in this order:
| Method | What it catches | What it can't do | What it costs on Retell |
|---|---|---|---|
| Text playground (chat by hand or with an AI caller) | Wrong answers, missed questions, wrong branch in a flow, tool calls with bad inputs | Grade anything, or show how it sounds | Billed per message, about $0.001 to $0.05 per message depending on the model |
| Simulation and batch tests | Regressions: a fix for one caller type that breaks another | Hear audio, latency or interruptions | Per message for both the agent and the simulated caller, plus one grading unit per case |
| Web call in the browser | Voice, pacing, talk-over, turn-taking | Transfer to a phone number or send a text (both fail in a browser call) | Billed per minute like a real call, roughly $0.07 to $0.31 per minute |
| Real phone call | Transfers, keypad input, texts, carrier audio | Run 20 scenarios cheaply | Voice rate plus about $0.015 per minute; Retell numbers cost $2.00 per month |
Retell's testing pricing page is blunt about it: testing bills at the same rates as production, and there's no free sandbox. That's why we do most of the work in text and save phone minutes for the final checks.
What goes in the test suite
A suite is a list of callers, written down once and rerun forever. Ours start from the call types the line actually gets, which we pull from call history before we write a single case. Most of our lines end up with some version of this list:
- The main job: a new customer who wants a quote, an appointment or a callback
- An existing customer changing or canceling something
- A caller who asks for a person by name, and one who names someone who isn't on staff
- A caller who insists on a human, or asks "Are you an AI?"
- A caller who gives a name and address but never a callback number
- A caller who wants the owner's personal cell
- A robocall, a live sales pitch, and a silent caller
- An hours question, and a question asked in the same breath as "goodbye"
- A replay of every real call that went wrong, and of the calls that went right and must stay that way
The last item is the one that pays off. When a real call exposes a gap, we copy what the caller said into a new case before we touch the prompt. Retell now does this for you: its Conductor copilot can turn a call ID from call history into a test case, with a persona and success criteria drafted from the transcript.
Write checks for things the caller never hears
Retell's simulation testing grades each run with success criteria you write in plain language, and all criteria are judged together into one pass or fail. We add checks that a listener would miss:
- If it says "connecting you," a transfer must fire. In a Retell conversation flow, a talking step waits for the caller's next turn before it moves on. A step that says "One moment, please" and then waits can leave a quiet caller on hold with no transfer. On the accounting office line we route a named person straight from an extraction step to the transfer, and every case carries a global rule: any "I'll connect you" or "please hold" must be followed by a transfer node.
- The callback number is read back before the call ends. Not "not provided," and not skipped because caller ID exists. Our message-taking guide explains why.
- One question per turn, and no vendor names. The agent never mentions the software it runs on.
On the fence lines, the November 12 version of the agent asks for name and number in one question, confirms caller ID, and requires a callback number before the read-back. It passed 7 of 7 new cases on both lines before it went live. The fence company guide covers that build.
Mock everything that touches production
This is the setting that bites people. Retell's docs warn that an unmocked custom function, code tool, calendar tool or MCP tool calls its real production endpoint during a test, so a test can create a real booking or record. A mock left blank is skipped, so the real function runs. MCP tools ignore mocks entirely. Transfers and texts are the exception: in a simulation they're always faked.
On our own front-desk line, every tool is mocked in the batch suite (check availability, book appointment, transfer and end call), so nothing is booked and no call is placed. Mock a transfer as failed in one case and as successful in another. A receptionist that handles a failed transfer badly is one you want to find in text, not on a Saturday.
Run every case twice and read the errors
The simulated caller and the grader are both AI, so a single result can be noise. Retell's own advice is to rerun a batch before you trust a single failure. Our harness requires each case to pass both runs. On November 12 the accounting office's new version passed 21/21 as a draft and 21/21 again after publishing.
Know the difference between a fail and an error. Retell marks a run as Error when it stops before grading: a test times out after 10 minutes, or the conversation passes 400 utterances, or the simulated user starts repeating itself. When our suite shows an error, the first thing we check is whether the fake caller looped on the same question. Then we tighten the persona and run it again.
Draft, test, publish, keep the rollback
Retell's versions make this safe. A new version starts as an editable draft. Publishing locks it. Environment tags such as prod and staging let a phone number follow whichever version the tag points to, and each tag can carry its own dynamic variables. Retell's example: point the transfer number at your own phone under staging so test transfers don't ring your team.
When the accounting office removed a staff member on November 11, the change touched four places in the flow. The draft ran unpublished and the phone number stayed on the old version until the suite passed 16/16. The previous version stayed available as the rollback. The accounting firm guide tells that story, and the transfer-to-human guide covers the routing itself.
Then pick up a real phone
Text can't test telephony. Retell's web call docs say transfers to a phone number and texts fail in a browser call, and a web call uses one of your concurrent call slots. So the last layer is a handful of real calls from a cell phone:
- Every transfer route connects to the right person, including after hours
- Texts the agent sends actually arrive
- The greeting sounds right on a bad connection (greeting script tips)
- The AI disclosure and recording notice play (disclosure rules)
- The fallback number rings when the AI line can't answer (backup plan)
If you're on an all-in-one product instead of Retell, look for the same layers. Jobber's Receptionist setup, for example, includes a test step where it calls you with example questions. That covers the phone layer, but you still need your own list of callers. Our Jobber Receptionist alternatives guide covers what to test before you switch.
After launch: keep feeding the suite
Testing doesn't end at go-live. We review new calls every day, separate spam from real callers (spam call handling), and flag anyone who calls back within 14 days. Any call that surprises us becomes a new case. For scoring real calls at scale, Retell's AI QA is free for the first 100 minutes of analyzed call time per workspace, then $0.10 per minute.
Want a receptionist that's tested before it ever picks up?
That's how we build every line: a suite written from your real calls, a draft that passes before it touches your number, and a rollback if anything looks off. See how we set up Retell voice agents and the post-call hand-off to your CRM.
Book an AI systems assessment and we'll walk through your call types and what your test suite should cover. Rather hear one first? Call our AI receptionist on +1 407-259-4842.
Questions people ask
How do you test an AI receptionist before it goes live?
Test it in four layers. Chat with it in text until the basics work, save each important caller type as a graded test case and rerun the whole suite after every change, make a browser call to hear the voice and interruptions, then place real phone calls to check transfers and texts. Keep the current version on the phone number until the new one passes.
How many test cases does an AI receptionist need?
Enough to cover every call type the line really gets, plus the awkward ones: a caller who asks for someone by name, someone who insists on a human, a caller who never gives a callback number, a robocall and a silent caller. The lines we run have suites of 12 to 26 cases, and every real call that goes wrong becomes a new case.
Can testing an AI receptionist create real bookings?
Yes, on Retell it can. Retell's docs warn that an unmocked custom function, code tool, calendar tool or MCP tool calls its real endpoint during a test, and a mock left blank is skipped. Mock every tool that books, charges or writes a record before you run tests. Transfers and texts are always faked in Retell's text simulations.
Does testing an AI receptionist cost money?
On Retell, yes. Testing bills at production rates with no free sandbox: text tests are billed per message, roughly $0.001 to $0.05 per message depending on the model, and browser or phone test calls bill per minute like real calls. That's why most testing should happen in text, with real calls saved for transfers, texts and the final listen.
What's the difference between a failed test and an errored test in Retell?
A failed test was graded and missed at least one success criterion. An errored test stopped before grading: it timed out after 10 minutes, ran past 400 utterances, or the simulated caller started repeating itself. Check the transcript of an errored run before changing the agent, because the fake caller may be the one stuck in a loop.