The Demo Works. Production Is a Different Story.
You see a ChatGPT demo that summarises customer emails in three seconds. Someone builds a quick prototype using the OpenAI API. It works perfectly in testing. Then you connect it to real data, real users, and real volume — and three new problems appear that nobody mentioned in the pitch.
LLM integration is genuinely useful. But the gap between "this works in a demo" and "this runs reliably in production" is wider than most teams anticipate. Before you sign off on building AI into your business workflow, you need to understand what you are actually buying into — the full picture, not just the API pricing page.
What You're Actually Paying Per Token
The API call itself is not expensive. It becomes expensive at scale, and the pricing model catches teams off guard more often than it should.
LLM APIs charge by token — roughly 0.75 words per token. You pay separately for input tokens (everything you send: the system prompt, conversation history, context documents) and output tokens (everything the model returns). Input and output rates are different, and cached input tokens are often cheaper still.
Here is where it compounds: most business workflows require context. A document summarisation system might send 2,000 tokens of document text plus 500 tokens of system prompt per request. At GPT-4o pricing (approximately $2.50 per million input tokens, $10 per million output tokens as of early 2025), a single request costs fractions of a cent. At 100,000 requests per month — not unrealistic for an active internal workflow — you are looking at meaningful spend before you have accounted for your own infrastructure costs.
The real trap is this: teams prototype with the most capable model available and budget accordingly. Then they discover that the capable model is more expensive than their business case supports at scale. They switch to a smaller model. The smaller model behaves differently. Two weeks disappear rewriting prompts and re-validating outputs.
Run the volume numbers before you build. Not after the system is in staging.
Latency Is a UX Problem Before It's a Technical One
A standard REST API call in a well-architected system returns in under 100 milliseconds. An LLM API call — depending on output length, model size, and current provider load — takes between 1 and 6 seconds. Streaming (where the model sends tokens as they are generated rather than waiting to return the full response) improves perceived speed but adds implementation complexity.
For asynchronous workflows — nightly report generation, batch email classification, background document processing — latency is manageable. Users are not waiting in real time, so a 3-second call is invisible to them.
For synchronous, user-facing features — an AI assistant embedded in your CRM, a live chatbot on your support portal, an inline suggestion in your billing tool — that 2 to 3 second pause is a genuine problem. Users interpret latency as the tool being broken, not thinking. You will need streaming output, skeleton loading states, optimistic UI patterns, or some combination. None of those are trivial to build correctly, and none of them are the AI work your team scoped.
If your integration touches a user interface in real time, budget UX engineering time that has nothing to do with the model itself.
Hallucinations Don't Throw Errors
This is the part that surprises engineering teams most. When a traditional API call fails, you get an error code. You catch it, log it, alert on it, retry it. When an LLM hallucinates, it returns HTTP 200 with a confident, well-formatted, factually wrong answer.
Hallucination — the model generating plausible but incorrect output — is not a fringe failure mode. It is a baseline characteristic of every current LLM. The frequency varies by task: closed-domain extraction from structured data hallucinates far less than open-ended generation from vague prompts. But it never reaches zero.
This means you need a validation layer. What that looks like depends on the task:
- For structured outputs (JSON, specific fields), enforce output schemas with strict parsing and reject non-conforming responses automatically
- For factual claims, cross-reference against a known data source before surfacing the result to any user
- For high-stakes decisions — anything touching credit, legal documents, or compliance — human review is not an optional feature, it is the architecture
The question is not whether your LLM will produce a wrong answer. It is whether your system will catch it before it causes a problem downstream.
Building that validation layer takes real time. It often takes more time than the initial LLM integration itself. Teams that skip it find out why the hard way.
The Prompt Maintenance Nobody Budgets For
Your prompt is not a one-time artifact you write once and forget. It is a living piece of configuration with its own maintenance burden.
LLM providers update their models on rolling schedules. GPT-4o in January behaves differently from GPT-4o in August — sometimes subtly, sometimes in ways that break your output format or change your model's tendency to follow specific instructions. Anthropic, OpenAI, and Google all push model updates without guaranteeing behavior parity across versions. Your prompts need version control. Your outputs need production monitoring. You need a regression test suite that tells you when a model update has changed how your workflow behaves.
This is engineering work. Not heavy engineering — but ongoing, unglamorous, necessary engineering. Most teams scope the initial integration and forget to scope the maintenance. Six months later, someone notices that output quality has silently degraded, and nobody can say when or why it started.
Treat your prompts the way you treat environment configuration files: version them in source control, write tests against expected outputs, and monitor production results on a dashboard someone actually looks at.
A Framework for Deciding Whether to Integrate
Not every workflow benefits from LLM integration. Before you commit engineering time and budget, answer these four questions honestly:
Is the task async or real-time?
Async workflows absorb LLM latency cleanly. Real-time, user-facing features need additional UX engineering investment to feel acceptable.
What is the cost of a wrong answer?
Summarizing internal meeting notes: low stakes, an occasional hallucination is annoying but recoverable. Generating customer-facing financial guidance: high stakes, requires a human review step or should not use an LLM at all.
What is your request volume?
Below 10,000 requests per month, API costs are rarely the binding constraint. Above 100,000, your choice of model and pricing tier becomes a meaningful line item in your operating costs.
Can you validate the output programmatically?
If you cannot write a test that reliably catches a bad output, you are depending on human spot-checks to catch errors. That is a process risk, not just a technical one, and it does not scale.
If your answers are: async, low stakes, moderate volume, and validatable output — LLM integration is likely a good fit for your workflow. If you are building a real-time, high-stakes feature at scale with no clear validation path, you are taking on significantly more risk than the demo suggested.
The technology is capable. The question is whether your workflow, your data quality, and your team's engineering bandwidth are ready for what production actually requires. Answer that honestly before you start building.
