Alongside my enterprise work, I've spent the past few years on a deliberately ambitious piece of personal R&D: a complete multi-tenant agentic platform, built end to end with my own hands. Three cooperating AI agents run inside it - a reception agent that handles customer communication and booking, an accounts agent that deals with invoicing and tax compliance, and a marketing agent that plans campaigns. I built every layer, from the infrastructure and data model to the knowledge pipeline, tools, and prompts, and hardened the whole thing to production grade - because the point of the exercise was to learn what survives contact with real-world constraints, not what demos well.
That vantage point - owner of the whole system rather than author of one prompt - taught me things I don't see emphasised enough in the agent discourse. None of them are about model selection.
Agents fail on context, not intelligence
Almost every bad agent behaviour I've debugged traced back to the same root cause: the model didn't know something it needed, or was drowning in things it didn't. In testing, it booked an appointment type the business doesn't offer because nobody told it the service list had changed. It answered a billing question with generic tax advice because the jurisdiction-specific rule wasn't in its context. The model was plenty smart; it was under-informed or over-fed.
The fix is what Anthropic calls context engineering: treating the model's attention as a scarce budget and curating the smallest set of high-signal tokens that gets the job done. In practice this meant building a proper knowledge layer - structured, validated, versioned domain knowledge that agents retrieve at the moment of need - instead of dumping raw records and documents into the prompt. The knowledge layer took longer to build than the agents themselves. It was worth more, and I now believe this holds at every scale: the organisations that get value from agents will be the ones that invest in what agents know, not just what agents do.
Tools are API contracts - design them like it
An agent's tools are the interface between probabilistic reasoning and deterministic systems, and they deserve the same care as a public API. My working rules, all learned the hard way:
- Narrow beats general. A
book_appointmenttool with typed, validated parameters outperforms a generalrun_queryevery time. Every degree of freedom you give the model is a way for it to be creatively wrong. - Errors are prompts. The model reads error messages and acts on them. "Slot unavailable - the next three openings are…" recovers gracefully; a stack trace produces improvisation. Write errors for the model the way you'd write them for a junior colleague.
- Idempotency is sanity. Models retry, sometimes enthusiastically. Any tool with side effects needs to make double-execution harmless, or someone will eventually be explaining a duplicate invoice.
- Names do heavy lifting. The model chooses tools by reading their names and descriptions. Ambiguous naming is a routing bug you shipped.
Standards like MCP are making tool ecosystems portable, which raises the stakes: a well-designed tool is now reusable infrastructure, and a sloppy one is a liability you'll integrate everywhere.
Guardrails are architecture, not an afterthought
The question that shapes agent design isn't "how capable can it be?" but "what is it allowed to do, and who finds out when it does?" In my platform that's structural: agents operate under per-tenant permissions, destructive or financially meaningful actions sit behind allowlists and confirmation steps, and everything an agent does lands in an audit trail. When anyone asks "why did the system send that message?", the full chain is there - context in, decision made, action taken.
Bolting this on later is miserable, because guardrails cut across every layer: the data model (who owns this record?), the tools (what does this role permit?), the prompts (when must the agent escalate to a human?). Design them in from the first commit. The discipline also pays a commercial dividend - trust is a feature people can feel, and in regulated or money-adjacent domains it's the difference between a pilot and a contract.
RAG is a data pipeline problem wearing an AI costume
Retrieval-augmented generation gets marketed as "connect a vector database." The vector search is the trivial 10%. The other 90% is unglamorous data engineering: what gets ingested and how it's chunked, how updates propagate when reality changes, how stale or contradictory knowledge gets caught before an agent states it confidently, which version of a document the agent saw when it answered. My retrieval quality improved more from fixing ingestion and validation than from any embedding or reranking upgrade. If your RAG "hallucinates," audit your pipeline before you blame your model.
Cost and latency are product features
A chat demo can take eight seconds and cost whatever it costs. A production-grade agent answering someone mid-booking cannot. Model tiering became core architecture: cheap fast models for classification and routing, capable models reserved for the steps that genuinely need reasoning, caching for everything repeatable. Token spend is a unit economic - it scales with usage, and in a multi-tenant system an inefficiency is multiplied by every tenant. I learned to track cost per conversation the way a business tracks any COGS line. Nobody on a demo stage mentions this; your CFO will.
Multi-tenancy makes context a security boundary
Running many tenants on one platform adds a rule with no forgiving failure mode: tenant A's knowledge must never surface in tenant B's conversation. Retrieval scoping, prompt assembly, caching, logging - every layer that touches context has to enforce isolation, because a cache key that ignores tenancy or an over-broad vector query is a data breach with a friendly conversational interface. If you're building multi-tenant AI, treat context isolation with the seriousness you'd give row-level security, and test it adversarially.
The agent is 20% of the system
If there's one meta-lesson, it's proportions. The prompt-and-model loop everyone demos is maybe a fifth of what runs in production. The rest is the knowledge layer, the tool contracts, the permission and audit architecture, the data pipelines, the cost engineering, and the monitoring that tells you it's all still working. That's not a disappointment - it's good news. It means the discipline our industry already has is exactly what the agentic era rewards. The teams that ship durable agents won't be the ones with the cleverest prompts. They'll be the ones that did the boring parts properly.