Prompting Is Easy. Engineering Reliable AI Systems Isn't.

Ishita Tyagi — Sat, 11 Jul 2026 21:12:12 GMT

TL;DR

I thought building an AI application would mostly be about prompt engineering. It wasn't. By the end, prompting occupied less than 5% of the codebase. Everything else was software engineering.

When I started this project, I thought the hard part would be getting Google Gemini to say the right thing. By the end, prompting was maybe 5% of the codebase. Everything else - the part nobody warns you about - was ordinary, unglamorous software engineering.

Here's how that happened.

The idea

The project was simple on paper: build an AI-powered research pipeline for SaaS applications. Point it at a product, and it figures out:

Authentication method
Category
API surface
Buildability (can you realistically build an integration against it?)
Documentation URLs

Store the results. Generate a report, move on to the next one. The final goal was never just JSON - it was a report someone could actually open, inspect, and trust. That part matters later.

My mental model of the architecture was almost embarrassingly simple:

Gemini → JSON

Ask the model, parse the response, done. I figured the real work would be iterating on the prompt until the outputs were clean. That assumption lasted about a day.

Where it fell apart

The first thing I noticed was authentication values. I'd ask for "auth type" and get back, across different runs and different apps:

OAuth
OAuth2
OAuth 2
OAuth 2.0
Bearer OAuth

All the same concept. Five different strings. Any analytics I tried to run on top of this - "how many apps use OAuth?" - were immediately useless, because the answer depended on how I grouped five spellings of the same word.

My first instinct was to fix it the way you'd fix a bad prompt: I kept rewriting the instructions. Be more specific about the format. Give it an enum. Add examples. Every time OAuth came back differently, I tweaked the wording again. It never really solved the problem - the next batch would produce a sixth variant I hadn't anticipated. Eventually I realized what I was actually doing: trying to fix a deterministic problem with a probabilistic tool. That realization is what changed the rest of the architecture.

Categories had the same problem, just quieter. Documentation URLs were worse: sometimes I'd get a marketing page instead of the actual developer docs, and there was no way to tell the difference just by looking at the JSON.

Then there was the free tier. Gemini's quota limits turned into a constant background hum of RESOURCE_EXHAUSTED, 429s, and the occasional 503. At first these felt like an annoyance I'd engineer around with retries. It took a while to realize they were actually telling me something about how the whole system should be shaped.

Version 1: the naive pipeline

Gemini → JSON

Problems: inconsistent outputs, unusable analytics, no real way to verify anything. This is the version most people ship first, and it's the version that makes "AI engineering" look like it's just prompting. It isn't - you just haven't hit the wall yet.

Version 2: normalization

Gemini → Normalization → JSON

I stopped trying to solve the OAuth-spelling problem in the prompt. Prompts are probabilistic; a mapping table isn't. I could have kept tweaking the wording forever, chasing every new variant the model invented, and never actually closed the problem - or I could write ten lines of deterministic code that map OAuth2, OAuth 2, OAuth 2.0, and Bearer OAuth to one canonical value, permanently. That's the trade-off in miniature: probabilistic effort scales with how many edge cases you're willing to keep discovering; a normalization layer just ends the discovery.

So I added one, between the model's output and everything downstream. The effect was immediate. Categories became stable. Auth types became groupable. Analytics that had been meaningless a day earlier suddenly worked. None of this involved touching the prompt.

Version 3: verification

Gemini → Normalization → Verification → Confidence Score

The obvious next move - the one every tutorial suggests - is to ask another LLM whether the first LLM's answer looks good. I didn't do that.

Instead I wrote a verifier as plain code. It checks structure, completeness, whether there's actual evidence backing a claim, whether the documentation URL points somewhere official, whether required fields are populated. Out of that comes a confidence score - not a vibe from a second model, but a number you can trace back to specific, inspectable checks.

This mattered more than it sounds. "Does this look good?" asked to an LLM gives you an opinion. A deterministic verifier gives you a reason. When something scored low, I could point at exactly which check failed, instead of shrugging and re-running the prompt.

Version 4: the actual system

Gemini → Normalization → Verification → Analytics → HTML Dashboard

By the final version, there's exactly one network call to a model per application researched. Everything after that - normalization, verification, aggregation, report generation - runs offline, deterministically, for free, as fast as the CPU allows.

That one-call constraint wasn't an optimization I added later. It was forced on me by the quota limits, and it turned out to be the right design anyway - more on that below.

Why not MCP, and why no agent

At some point I considered wiring in MCP, or restructuring the whole thing as a more agentic loop - let the model decide what to fetch, when to retry, how to explore. I looked at it seriously and then didn't do it.

The problem didn't call for it. Research pipeline in, structured record out - that's a linear job, not one that benefits from a model making its own control-flow decisions. Adding MCP would have meant more moving parts, more failure modes, more surface area to debug, for a task that was already well served by a fixed pipeline. Simpler architecture was just better architecture here. That's not a universal law, but it was true for this problem, and I had to actually build the complicated version in my head to be sure.

Why deterministic verification - and what the quota taught me

It's common to see AI projects reach for a second LLM, or an eval framework like RAGAS, to judge the first model's output. I avoided that path deliberately, and the free-tier quota is a big part of why.

The Gemini free tier constantly reminded me that every request has a cost - not an abstract one, an immediate RESOURCE_EXHAUSTED one. If I'd gone the "second LLM judges the first" route, I'd have doubled my request volume for a resource that was already the tightest constraint in the system. That wasn't an option, so I was forced to ask what a judge actually needs to do - check structure, check evidence, check required fields - and realized none of that requires a model at all. It's just code.

So the verifier became normal software, not another AI system. That gave me fewer API calls, lower latency, lower cost, and - the part I cared about most - outputs I can explain. A verifier written as code gives you the same answer every time on the same input. An LLM judging an LLM gives you another probability distribution to reason about, stacked on top of the first one. Ironically, the constraint I resented most in week one is the reason the architecture ended up sound: treat tokens like any other expensive, rate-limited resource, and the right design tends to fall out on its own.

Partial failures and retries

Once the pipeline became a batch processor, another problem appeared: partial failures. A single API timeout shouldn't stop research for the remaining applications. Instead of treating failures as fatal, I stored partial results, marked them for manual review, and let the batch continue. Reliability isn't about preventing failures - it's about handling them predictably. That's backend engineering, not AI engineering, and it's most of what "productionizing" actually meant here.

I also kept verbose logs throughout development - not because they looked nice, but because watching every stage of the pipeline made debugging significantly easier than staring at final JSON outputs and guessing which stage had gone wrong.

What the project actually was

If you'd asked me at the start what percentage of this project would be prompt engineering, I'd have guessed most of it. By the end, prompting was a small, stable piece - a few carefully worded requests that don't change much once they're right. The rest was normalization logic, a verification layer, retry and fault-tolerance handling, observability into what the pipeline was actually doing, analytics code, and a reporting layer to make the results legible.

That's not a knock on the model. Gemini did its job. The job just turned out to be smaller than I expected, and everything around it turned out to be bigger.

Where I landed

The biggest lesson wasn't how to use Gemini. It was learning when not to ask Gemini.

Prompting gives you an answer. Engineering determines whether you can trust it.

What surprised you the most the last time you built an AI application - the model, or everything around it?

If this article focused on the "why," the repository contains the "how."

Behind the Build