How to Vet an AI Automation Agency

There are over 12,000 self-described AI automation agencies as of mid-2026. The majority launched in 2024 or 2025 after watching YouTube tutorials about ChatGPT and Zapier. Many have never shipped a system to a real client. Some have shipped one or two small projects. A small minority have the depth to handle production systems for growing service businesses.

The 10-point checklist below will identify which category any agency falls into within one discovery call. Ask these questions and listen carefully to how they answer. Evasion, jargon, and credential-dropping without specifics are disqualifiers.

We apply this same checklist to ourselves. If you asked us these questions, we could answer every one with specifics. That is the bar.

Red Flags That Disqualify an Agency Immediately

Before the 10-point checklist, these are automatic disqualifiers. Any one of these should end the conversation.

They cannot explain what they built without jargon: Ask "Tell me about the last system you shipped. What did it do? What was the result?" If the answer is full of terms like "synergistic AI ecosystem" without concrete specifics, they have not shipped anything real.

They charge for a "discovery call": A sales call is free. An actual analysis of your business is worth charging for. If they call it discovery but it is really a 30-minute pitch, that is a misrepresentation.

They promise results in weeks for complex systems: Production-grade AI systems with real integrations take 2 to 8 weeks to build properly. Anyone promising a complete lead machine in 72 hours is delivering a template, not a custom system.

They cannot name the actual tools they will use: Ask "What automation platform do you build on? What AI model? What telephony provider?" Legitimate agencies have specific answers. Non-answers mean they will figure it out after you pay them.

They have no client references: Not testimonials on their website -- actual references you can call. A one-year-old agency with no referenceable clients has not delivered anything worth referencing.

Their pricing is "custom" for everything: Agencies with real productized services know what things cost because they have built them before. Agencies that quote everything as "custom" are figuring out pricing as they go.

The 10-Point Vetting Checklist

Work through these questions in your first call. Take notes on the answers.

1. Show me a live system you built for a client in my industry. Acceptable: A live link, a screen share, a recorded demo. Not acceptable: "We have done similar work" without specifics.

2. What was the measurable business outcome for that client? Acceptable: "They recovered 12 leads per month, which at their close rate and average job value equals $X in additional revenue." Not acceptable: "The client was happy with it."

3. What tech stack do you build on? Acceptable: Specific tools -- n8n, Vapi, Twilio, Retell AI, Claude API, Resend, Supabase. Not acceptable: "Whatever is best for your situation."

4. Who will actually build my project? Acceptable: Named people with verifiable experience. Not acceptable: "Our team" without being able to identify who that team is.

5. What does post-launch support look like? Acceptable: Defined window (30-day included, then retainer options). Not acceptable: Vague reassurances about "being there for you."

6. What is your process when something breaks after launch? Acceptable: SLA commitments, escalation paths, monitoring tools they use. Not acceptable: "We will fix it" without specifics.

7. What does your discovery process look like? Acceptable: A defined output (roadmap, ROI projections, prioritized recommendations). Not acceptable: "We will get to know your business."

8. Can you provide two or three client references in my industry? Acceptable: Names and contact information for clients who will take your call. Not acceptable: Names with no contact information.

9. Have you personally run a business in my industry or a similar one? Acceptable: Direct operator experience. Not acceptable: "I consulted for clients in that industry."

10. What will you not do for me? Acceptable: Honest scope limitations. Not acceptable: "We can do anything."

What Should Good Answers Look Like?

For each of the 10 questions, here is the shape of a credible answer versus a non-answer. These are illustrations of the pattern, not quotes from any specific vendor.

Question 1 (Show me a live system): A credible answer shares a screen recording of a system currently in production and names the platform it writes into, so you can verify the integration exists. A non-answer says they have done work in your space and would be happy to build something similar.

Question 4 (Who will build it): A credible answer names the specific person doing the strategy and the system design, and points you to their profile so you can check their background yourself. A non-answer refers to "a team of experienced developers" you never meet.

Question 8 (References): A credible answer hands you a named client and their direct contact information, and tells you how long that client has been running the system. A non-answer offers a page of testimonials or a link to a Google listing. If a vendor will not put you on the phone with a current client, treat that as the answer.

Question 9 (Operator experience): A credible answer describes a specific operational role the vendor has actually held and what they learned about where work falls through. A non-answer says they have worked with a lot of clients in your industry.

The pattern: credible answers point to something you can independently check. Non-answers are vague, jargon-heavy, and built to avoid commitment.

How Do You Evaluate Pricing Proposals?

Once you have narrowed to two or three agencies using the checklist, you will receive proposals. Here is how to evaluate them.

Scope clarity: Does the proposal define exactly what is being built, what integrations are included, and what is explicitly not included? Vague scope is the source of most scope creep and cost overruns.

Deliverable specificity: "AI automation system" is not a deliverable. "Scheduled job that polls the scanner folder every 20 minutes, splits multi-document scans, classifies each one against your seven document types, extracts date, amount, vendor and entity, renames to your convention, files to the matching folder, and writes one ledger row per document" is a deliverable.

Timeline realism: Two-week timelines for complex systems are unrealistic. Three to six weeks for a properly built, tested, production-grade system is normal.

Payment structure: 50% at kickoff, 50% at delivery is standard and fair. 100% upfront is a red flag.

Post-launch definition: What specifically is included in post-launch support? How long? What is the escalation path for issues?

References and guarantees: Will they let you talk to references before you commit? Compare proposals on scope specificity, not total price. The cheapest proposal often wins on paper and loses in delivery.

How to Think About Agency vs. Freelancer vs. DIY

Your three options for AI automation are a full-service agency, a freelancer from a platform like Upwork or Fiverr, or building it yourself with tools like Zapier or n8n. Each has a legitimate use case.

Freelancers work for simple, well-defined projects with low stakes. Need a Zapier workflow that moves data from one place to another? A $200 Upwork hire might be perfect. Need a production AI voice system handling your after-hours emergency calls? A freelancer is the wrong call. No monitoring, no support, no one to call at 9 PM when the system starts failing calls.

DIY tools (Zapier, Make, n8n) work if you have the technical patience to learn them and the time to maintain them. The true cost calculation needs to include your time. If you spend 15 hours per month on automation maintenance at a $200 effective hourly rate, that is $3,000 per month in opportunity cost.

Full-service agencies make sense when the automation is revenue-critical, complex, or when your time is better spent running your business than maintaining software. The right agency delivers a system that works, documents it, and maintains it so you do not have to think about it.

The red flag scenario: business owner hires cheap freelancer, gets a barely-working system, spends months troubleshooting it, finally gives up, then pays an agency to build the right system from scratch. Total cost: 2x. This is extremely common.