NICKE STUDIO / Choosing and evaluating an AI agency
AI agency red flags: what to check before you sign
Spot AI agency red flags before you sign: unsupported promises, data access, hidden operating costs, weak testing and difficult handovers.
Read in another language ↗
An AI agency should be able to explain the problem it is solving, the decisions its system can make and the evidence that will justify keeping it. A polished chatbot or a dramatic savings claim answers none of those questions. The strongest warning sign is a proposal that becomes less specific when you ask about actual operations.
These red flags are reasons to investigate, rather than proof of misconduct. A small agency can be excellent without a long client list. A large one can still deliver a workflow nobody can maintain. Look for verifiable commitments instead of judging the presentation.
Promises without a starting point
Claims such as “we will cut your costs by half” need a denominator, a measurement period and a description of what is included. Ask how the agency will measure the current process before changing it. Does the promised saving include human review, exceptions, software subscriptions and support?
An illustrative example: a task falls from ten minutes to four, but checking and correcting the output takes another three. The net capacity gain is three minutes, not six. Whether that creates cash savings is a separate question. Our guide to measuring AI agency results explains that distinction.
A credible proposal makes its assumptions visible before it makes its promises impressive. — Nicke Studio editorial note
What the warning signs look like
| Warning sign | Evidence to request | A more useful response |
|---|---|---|
| A universal demo is presented as your solution | A walkthrough using representative, sanitized cases | A documented pilot with exceptions |
| The agency promises zero errors | Failure categories and escalation rules | Defined limits and human approval |
| One subscription is described as the entire cost | A complete operating budget | Usage, maintenance and review separated |
| Everything must stay in agency-owned accounts | Account ownership and export documentation | Agreed ownership and a tested handover |
| Success means calls made or content produced | Baseline and business acceptance criteria | Quality, cost and completed outcomes |
The table is a starting point for a conversation. A supplier may have a reasonable explanation for a managed account or an initial discovery phase. What matters is whether that explanation includes access, accountability and an exit route.
Access to data needs a specific purpose
Be cautious when a supplier asks for administrator access to every system before mapping the workflow. Ask which data each component reads, what it can change, where information is stored and how credentials are revoked. A password sent through a casual chat is not an access policy.
OWASP's 2026 guidance on excessive agency describes the risks of excessive functionality, permissions and autonomy. For a sales assistant, permission to prepare a draft may be appropriate while permission to change prices or issue refunds requires a different decision.
Also ask how the system handles instructions hidden in retrieved documents or tool responses. The current OWASP prompt-injection guidance explains why outside content can influence model behavior. A stronger system uses permission boundaries and validation; it does not rely on a clever prompt alone.
A demonstration is not an evaluation
A prepared example can hide missing information, awkward inputs or manual intervention. Ask the agency to show both a successful case and a case where the system should stop. Include ambiguous requests, outdated documents and failures of the connected service.
- Agree on the workflow and an accountable owner.
- Select representative cases and define an acceptable outcome.
- Reserve some cases for a later evaluation.
- Record errors, review effort and operating cost.
- Define when to pause the system and return to the previous process.
OpenAI Evals is one example of a framework for repeatable model and system evaluations. The important deliverable is your test set and acceptance criteria, not a particular brand of testing software.
Check the handover before the project ends
Ask for a written list of deliverables before paying for a full implementation:
- Accounts, repositories and credentials your organization will control.
- Workflow diagrams, dependencies and the purpose of each integration.
- Test cases, known limitations and a version history.
- Logging rules that protect sensitive information.
- Backup, recovery, maintenance and termination responsibilities.
A refusal to discuss these items deserves more attention than an agency's use of any one tool. Compare the proposed stack with our AI agency tools guide and check the licenses in our open-source guide.
Decide with evidence, not pressure
A useful next step is a bounded pilot with a clear price, data scope and review date. A supplier should be able to explain what happens if the pilot does not meet the agreed criteria. That makes continuing, adjusting or stopping a practical decision.
Bring the workflow you want to improve to Nicke Studio. We can discuss the expected deliverables and the questions a proposal needs to answer before you commit to a larger project. You can also send your project context without treating a first conversation as a performance guarantee.