Experiments with Agents

While working at companies, I avoided using LLM agents for security reasons, and I feel like I've fallen quite far behind developments in this space. Now that I'm back to working on my own project, I'm trying to catch up: explore the tools, try different models, and figure out where agents genuinely help—and where they still demand more attention than they save in time.

These are my first impressions. Not a market overview or a guide to building the perfect agent system, just notes from an ongoing experiment.

Choosing a Tool and Setting Up Access

I narrowed my choice of tools down to Aider and PI Agent. I went with PI because its minimalist approach and flexibility appealed to me.

Another important requirement was independence from any particular model provider. I wanted to be able to switch between different LLMs, including local ones, rather than build the entire workflow around a single company.

I also didn't want to give agents direct access to the host system. Ideally, each agent should see only what it needs for its work and have write access only to its own part of the project.

So I set everything up using Docker and Docker Compose, mounting only the necessary directories and explicitly defining access permissions. For example:

This lets the architect refer to the business requirements without rewriting them, while saving its own work in a separate section.

Docker and Docker Compose have worked well for this: the configuration makes it immediately clear which resources each agent can access and what it can do with them. The project itself is also built in containers, so this approach fits naturally into the development workflow. It has been a convenient starting point for explicitly restricting file access. There may be better solutions—I'm still new to agent systems, and this is the setup I'm experimenting with for now.

The Current Agent Lineup

So far, I've created eight roles:

• CEO
• CTO
• Architect
• Analyst
• Audit
• Developer
• Reviewer
• Tester

The lineup itself is an experiment, not a settled structure. I'm trying this split of responsibilities to see which roles are actually useful, which overlap, and where interactions between agents only make things more complicated. As I go, I may merge or remove some roles and add others.

Only the first three are fully configured so far, including testing and refining their system prompts. The others are still waiting their turn.

The workflow is still partly manual: I watch the results, step in, adjust instructions, and try to understand which tasks the agents handle well and where they start making mistakes or doing unnecessary work.

I don't yet know what will come of this. For now, understanding how individual agents behave matters more to me than rushing to build a fully automated team.

Models and First Impressions

Here's how the models are currently assigned:

Initially, every role used Claude Sonnet 5.5. Yet it's also the model I have the most reservations about so far.

I keep hearing that Claude is the best model for this kind of work. Friends and acquaintances speak almost exclusively in glowing terms about it. But in my experiments, its responses have contained the most unnecessary noise, with frequent mistakes. A few times, it even got stuck in what looked like endless generation.

I've also tried Opus. In my experience, it consumes a lot of tokens and costs several times more to use, but I haven't yet seen a corresponding improvement in quality on my tasks.

The difference is particularly noticeable when dealing with uncertainty. In my limited experience, when GPT spots an error or lacks information, it tends to ask questions and clarify details without producing a lot of unnecessary text. Claude is more likely to fill in the gaps with assumptions, make a bunch of changes or produce a long response, and then apologize for getting things wrong.

I'm not trying to turn this into a universal rule. It may come down to my tasks, configuration, or system prompts. Perhaps I simply haven't learned how to work with it yet. For now, my personal experience differs noticeably from what I hear from people around me.

All Models Make Mistakes, but the Costs Differ

One of my early takeaways from these experiments is that a model's ability to solve a task isn't the only thing that matters—the cost of correcting its mistakes matters too.

Every model can make mistakes. But with DeepSeek, revisions and retries are inexpensive enough that experimenting feels easier. With Claude and GPT models, rework has a more noticeable impact on the bill.

That doesn't automatically make DeepSeek the best choice for every role. But the cost of a failed attempt has already become a separate criterion for me, alongside the quality of the result.

For now, I'm continuing to configure the agents, try different models in different roles, and observe the results. I'm still a long way from drawing firm conclusions. First, I want to understand where this system genuinely helps move the project forward—and where I'm simply spending time managing the system itself.