Field notes
An agent is six layers. Most people build one.
The first question everybody asks is which model to use. It is close to the least important decision you will make. The model is one layer out of six, and the layer people skip is the one that decides whether your agent does something once or keeps working for a year.
Almost every conversation I have about building an agent opens the same way. Claude or GPT? Which one is better right now?
I understand the instinct. It is the part of the system with a leaderboard. But if you build an agent that is only a model, what you have is a chat window with extra steps. It answers when you ask, forgets you existed, and does nothing while you sleep.
An agent that actually does work has six layers. Here they are, and then a real one I run.
The six layers
Claude
Notion
GitHub
The LLMthe brain. it reasons.
MCPthe connectors. your apps and data.
01The LLM, the brain
The reasoning. It reads the situation and decides what to do next. This is the layer with the leaderboard, which is exactly why it gets all the attention, and it is also the layer you can swap in an afternoon. If you want the argument about which one and what it costs, I wrote that up separately in open source AI is not free.
02MCP, the connectors
A brain with no senses is useless. MCP is how the agent reaches your actual apps and databases: your Notion, your calendar, your analytics, your repo. Without this layer the agent knows a great deal about the world in general and nothing about your business. I did a full walkthrough of how data moves through it in what is MCP.
03The CLI, the hands
Reading is not doing. The command line is where the agent edits files, runs code, and executes real work on a real machine. This is the difference between an assistant that describes a fix and one that applies it, runs the tests, and tells you the result. More on why the terminal is the whole game in learn Claude Code or Codex.
04Memory, the rules
How the agent remembers your preferences, your constraints, and what has worked before. Without memory you re-explain yourself every session and the agent makes the same mistake in March that it made in January.
Skills live here too. A skill is a written method the agent can reuse, which is really memory with a job attached. Mine picks them up from videos people send me, which I gave away as a free file in stop saving reels.
05Permissions, the brakes
Where the agent acts on its own and where it has to stop and ask you. Get this wrong in the loose direction and it publishes something at 2am that you would not have signed off. Get it wrong in the tight direction and you are approving every trivial step, which means you have built yourself a slower way to do your own job.
This layer is genuinely hard to tune and it is worth more thought than it usually gets. I ran into the physical version of the same problem when OpenAI put an approval button on a keyboard, which I wrote about in copying Codex Micro.
06Loops, the engine
This is the one most people miss, and it is the one that separates a demo from a system.
Asking once is the wrong shape. You prompt, you get output, you are done, and tomorrow you do the whole thing again by hand. A loop is a system that keeps going without you: check the repo, pick up a task, write the code, run the tests, hand it to another agent for review, fix what comes back, report in.
That is the actual reason people stopped obsessing over prompt wording and started building skills, connectors, sub-agents and scheduled runs. Not because prompting stopped mattering, but because a better prompt gets you one better answer, and a loop gets you a thousand.
A better prompt gets you a better answer once. A loop gets you a system that is still working next month while you are asleep.
A real one: notes into a script
Here is one I actually run. It turns my raw notes into a finished video script, and every layer does a specific job.
- The LLM reads the brief and decides what it needs.layer 1, the brain
- MCP pulls my raw notes out of Notion. A second connector fetches trending topics and hook patterns from another source.layer 2, the connectors
- The CLI writes the script to an actual file on my laptop.layer 3, the hands
- Skills shape the writing: hook structure, pacing, how long it should run.layer 4, memory with a job attached
- Memory keeps it in my voice and inside my rules, so it does not hand me something that sounds like a stranger.layer 4, the rules
- Permissions stop it publishing anything without me looking first.layer 5, the brakes
- The loop reruns the whole thing until every check passes, instead of handing me a draft and giving up.layer 6, the engine
Six layers, one agent. Notice that only the first step involves the model at all, and that the last step is what makes the other six worth building. I set that up once. It runs whenever I have notes.
That is the mental shift worth making. You are not writing a prompt. You are building infrastructure, and infrastructure is judged on whether it is still standing in six months.
Better questions than "which model"
When somebody tells me they want to build an agent, these are the four things I ask, and none of them are about the model.
What do you want it to do? Not "help with marketing." One job, specific enough that you would recognize a good result.
What does it need to remember? Your rules, your tone, the mistakes you do not want repeated.
Where must it ask you first? Decide this before you build, not after something goes out that should not have.
What tools does it need to reach that goal? Which apps, which files, which systems. This is your connector list.
Answer those four and the model choice mostly answers itself. Get them wrong and the best model on the leaderboard will not save you.
If you want to go deeper on a layer
Most of these layers already have their own post on this site. This one is the map.
The six-layer version of this started as a short, if you would rather watch it: the important layers of an agent.