In such systems, producing code is the easy part, verification is hard. Verification via test cases just feels like the same agent validating its own assumptions.
Wondering what the actual verification loop looks like once you start taking these systems to production.
We’re actually working on this problem. It’s an interesting space to be in. Initially started with static code verification, but now we’re expanding into the area of behavioral verification (does design intent match behavior) across the application lifecycle. Currently working on getting some open source stuff ready. Check us out (https://bohmian.ai - don’t mean for those to be a plug, it’s just relevant)
I'm building an orchestrator just for the "build" part of building.
The only thing we've truly got left realistically is deciding what to build and making all the product decisions on how it should work.
So that's what I'm trying to enjoy the most now and then when I'm done planning and writing the spec I just delegate to my orchestration engine and walk away while it builds.
It spawns fresh agents to implement, then verifies mechanically (test/lint/build etc.), then fans out multiple reviewers across different providers/models, all those findings go into a triage step with an agent who is given standardized context of previous triage and review rounds and who deduplicates issues and writes guidance for fixing issues fully, then a fix agent gets spawned , then a post-fix verification agent and then back to review until no more issues are found.
For a large feature it can work for a day or more uninterrupted and contrary to just using something like /goal or having an agent orchestrate the whole process, the code stays on track and gets delivered as close to the spec as possible.
All this repetition is necessary if the goal is to be able to generate tons of code and maintain quality while not needing to review everything yourself.
(soft plug: in case this is really exciting to you but you don't want to own all of this infra for your company, check out our "software factory" factory over at noriagentic.com)
So - apparently it's not fully self-hosted, since I don't see a GPU.
I'm interested in hearing from folks who are hosting their own GPU to run coding models. So far my own results are... not great. Seems like frontier models are needed via the big providers?
My favorite real-world example that I worked on: I created a YouTube documentary series about corruption in a small town in Illinois. This required downloading thousands of hours of government meeting videos from YouTube, transcribing + chunking + embedding, summarizing meeting segments, running a couple passes of validation, then searching for interesting storylines. I also scraped thousands of public documents which exposed campaign finance violations and some truly nefarious stuff going on behind the scenes.
I did 90-95% of the work on this with a local GPU. I still ran Claude and Codex, but they were just writing code to orchestrate API calls against local models, local Whisper, etc. Doing the same with the public cloud would have cost thousands of dollars, instead it cost a few hundred dollars for a frontier lab subscription and tens of dollars for electricity.
I recently got into the DGX Spark and just a month ago, got an AMD Ryzen developer platform. Both are incredibly useful tools for some upcoming projects I'm working on (also related to government corruption), but in my limited experience of trying to run opencode on them, the "good models" are still completely unusable because the system prompt alone requires a minute of thinking.
1) I have knowledge DB that I query, spend a lot of time to get "query N models, stream to me all N results, let me pick which one" - only to find out that qwen beat all other contenders (those were picked before I got R9700, so the rest is =<8B parameters)
2) Setup OpenCode to use it and gave it a few tasks:
- first task (new feature to my MCP) took 40 minutes to complete with zero input from me, while it took 20 for sonnet-5 and sonnet-5 kept bugging me. Results are near identical.
- second task (port a specific version of a package to my flake) it got stuck in a hilarious loop where model already been told what hash to use by nix itself, but it wanted to figure our how to get hash another way for some reason.
- third task (another task, but much harder than first one with most of the discovery already done), kept doing discovery and running out of context, went through 3 compactions (256k context is what I can fit on R9700). Room got too hot, so I stopped it.
3) virtual assistant like Hermes but my own: no notes, works great.
I'm pretty sure codding issues are just harness and lack of memory that Claude Code already had. Pretty nice setup, similar to mine but I built my own lightweight PaaS that is highly specific to what I run.
I have a similar setup to the OP (although I'm not sure if I'd call it a "software factory"), and I utilize a local model for some aspects of the setup. Specifically, I have a single RTX 3090 Ti with 24GB VRAM on my (main) home server, and I've been primarily running Qwen 3.6 35B A3B out of it via Ollama (I just switched to Qwen 3.8 27B, though, and I've also tried other Qwen models as well as Gemma models).
The Qwen models are decent, but they don't come close to the full Claude experience I've come to expect. As such, I only use the local models for specific tasks where it makes sense to do so. Really the setup is that my Claude-powered agents are able to incorporate my local model into work it builds out. The agents can perform inference against the Ollama API as they see fit, and I encourage them to do so for tasks where (a) the low-level capacity of the local models make sense and/or (b) where costs can become a concern.
It seems to work well when it comes into play (like having Claude drive a web browsing session but letting Qwen handle much of the actual browser interactions, image analysis, etc.). Still, Qwen just isn't smart enough (or fast enough on my machine) to handle anything agentic that isn't non-trivial.
Using a RTX 5090 ($3~4k), I can run Qwen 3.8 27B at ~180 TPS with ninfer [0]. With its thinking maxed out, I can confirm that the quality of output is roughly on par with Opus 4.5~4.6 - that is, this 20GB file really can write software by itself, but the amount of thinking required makes it strictly slower than larger models, even at 180 TPS.
Still, it's largely replaced the cheap tier of the frontiers that I would otherwise be using. It can be run with older GPUs (a 3090 is ~1k), but the time spent thinking will become a fairly noticeable impediment for staying in the flow.
The next step up would be to run DeepSeek V4 Flash 0731 on two DGX Sparks (~$10k), which serve at 60 TPS and sit somewhere around Opus 4.7 level without 3.8-tier thinking.
However, it is worth noting that, if you are buying this hardware just to serve LLMs, it is not cost-effective. It would take over a decade of continuous use to make back the cost of the DGX Spark setup in 0731 tokens. I'm running this setup because I happen to have a 5090, and the two 3090s in my server were cheap enough when amortised over several years.
Check out some NIXOS communities. People they are doing fun crazy stuff, as NixOS is immutable, thus creates separate sandboxes (building/taking down) per agent, etc.
I don't know most of the stuff there, but at least you can dive there.
I was recently inspired by another article here to start my own. My skills are written and tested, the factory has built the first test project, and I’m setting up the final machine to run it. Here’s my initial post about my motivation and early plans plus some follow-ups, and there are more to come.
People who build software factory are in for a disappointing future. Spoken as someone who uses AI day to day for job and personal projects, I've came across a few super gnarly bugs already, ranging from frontend React apps (yes, believe it or not, frontend is far from solved) to embedded that Claude can't solve. Those bugs are feature breaker. My primary skillset is fullstack/frontend leaning, not embedded. Granted, I might be out of depth in embedded/RTOS but I am very qualified to do frontend. Yet I found bugs in those areas that Claude Opus high can't even solve for hours.
I pointed Claude at those for hours and hours, didn't fucking work, ended up creating more and more slop. Until I stepped in and fixed it in a few lines of code. Thankfully with all my AI assisted work, I took time to understand everything so I was able to debug and fix.
The problem for software factory is, that up to a certain point, you'll encounter some nasty bugs that are really important for your features, but you can't ship it because while the AI deals with the 90%, turns out that last remaining 10% are the one that make or break your project. And you need to deal with that remaining 10%. But how can you deal with the remaining 10% if you don't understand that 90% or majority of it?
Also, don't listen to influencers like Kent Dodds or Uncle Bob or Steve Yegge, who advocated for software factory walk away from the code kinda thing. They are already rich, and they sell teaching content, but most importantly, their time in this industry has passed. They are not in the weeds doing enterprise software development anymore.
Their first and primary concern is trying to be relevant to sell courses. They don't care about you. Yes I used to listen to them (though not often), but now not anymore. The world doesn't get better or worse whether they choose to sip margarita on a beach or spin yet another "so many words to say absolutely nothing" video/blogs.
Can anyone that has OpenClaw/Hermex experience tell me what it's like working on this over OpenCode served?
On bigger tasks OpenCode sometimes hangs. I'm not sure if that's on the provider side or on my network side, but i sometimes have to stop a subagent because its done but doesnt clean up. I wanted to try Pi, but now I'm wondering if Hermis is a better fit.
I can relate to this feeling, having encountered it many times in academia. But the inverse would also be true, would you stop reading yet another article that explains for the umpteenth time what library Y does? Not arguing for the author, but there’s certainly something about knowing and writing for your audience.
Neat stuff. I use Hermes with the $20/mo ChatGPT Codex integration, myself, as well as Qwen running locally on a 64GB M1 Max Macbook Pro to help save my ever-quickly shrinking Codex credits. It's been wonderful for stuff like, say, giving Hermes an account on my Home Assistant server (after taking a backup) and having it make a bunch of configurations, or draw conclusions based on historical sensor data (trying to determine if my A/C is undersized by feeding Hermes all the capabilities of my system, size of house, and letting it pull room temp sensor data from HA is one project I did recently), is absolutely fantastic. It also spins me up little webapps for things like a very me-specific RSS reader, a "to do" checklist that references my email (and archives said emails when I mark an item off the list), and various little internal IT tools and report generators.
The Hermes subreddit is a curious place where every second person has quick their $300k a year job and is making a living off of Hermes doing... something? They treat themselves as the CEO of a bunch of agentic employees and have AI generated infographics of their "Stack" (all hail the mighty Stack) and everybody stands in a circle and applauds the most convoluted Hermes setups you've ever seen with not a single word how any of this is supposedly making anybody any money.
Are people developing like this? Are people asking money for something they one-shot instructions to an orchestration agent which delegated to fifteen other agents in Kanban and then spat out something that "works"? Every project I've ever had an LLM do a majority of the work for me has been strictly for my personal use; I'd never let anyone else use this stuff because it doesn't pass the vibe check. When a new frontier model comes out I'll pass the previous frontier model's work past the new model and let it tear it to shreds and see what improvements could be made for shits, giggles, and to waste a week's worth of tokens in the course of thirty minutes, but Reddit is overflowing with seeming non-coders who are passing this 100% organic slop off as sellable product?
In my experience agentic sw development delivers if you are not too picky with the details or if you don't notice what's not done according to your instructions. If you want exactly what you had in mind, the last 10% might last a very long time and cost many tokens. More or less like working with humans.
That’s a great use of Hermes. Had it concluded if your A/C is undersized?
Before this little experiment I generally used it as a very specific search engine for houses, ‘Find me a house in the country but close to amenities with a workshop barn but close enough on the train to Manchester’. It’s great for this!
I’ve never been on the Hermes subreddit I’m gonna go check it out. I agree with the commenter below, it’s fine when I need no record my reps but don’t really care how. It writes tests but I don’t verify. It’s a manual ‘works or it doesn’t’ Thing. I’d never sell the code to anyone.
It seems at this stage hermes has the upper hand over OpenClaw in people building multi-agent systems. What about hermes do people like above OpenClaw. I'm building a completely different kind of "software factory" called Abject and I'm curious what people are valuing in Hermes above other systems.
People assemble projects from a collection of buzzwords these days without understanding the underlying technology. Did he need to use tailscale to interact with the box remotely? No. But, it's what a google search will tell you to do.
In such systems, producing code is the easy part, verification is hard. Verification via test cases just feels like the same agent validating its own assumptions.
Wondering what the actual verification loop looks like once you start taking these systems to production.
We’re actually working on this problem. It’s an interesting space to be in. Initially started with static code verification, but now we’re expanding into the area of behavioral verification (does design intent match behavior) across the application lifecycle. Currently working on getting some open source stuff ready. Check us out (https://bohmian.ai - don’t mean for those to be a plug, it’s just relevant)
It's really funny. I would read HN headlines about PhD research in the area of https://en.wikipedia.org/wiki/Correctness_(computer_science) and it always seemed really boring
Now it is obvious to me how important this work is
Some people have alternative models review each other's output.
The future might be parallel attempts and selecting the best-of-N solutions.
I'm building an orchestrator just for the "build" part of building.
The only thing we've truly got left realistically is deciding what to build and making all the product decisions on how it should work.
So that's what I'm trying to enjoy the most now and then when I'm done planning and writing the spec I just delegate to my orchestration engine and walk away while it builds.
It spawns fresh agents to implement, then verifies mechanically (test/lint/build etc.), then fans out multiple reviewers across different providers/models, all those findings go into a triage step with an agent who is given standardized context of previous triage and review rounds and who deduplicates issues and writes guidance for fixing issues fully, then a fix agent gets spawned , then a post-fix verification agent and then back to review until no more issues are found.
For a large feature it can work for a day or more uninterrupted and contrary to just using something like /goal or having an agent orchestrate the whole process, the code stays on track and gets delivered as close to the spec as possible.
All this repetition is necessary if the goal is to be able to generate tons of code and maintain quality while not needing to review everything yourself.
(soft plug: in case this is really exciting to you but you don't want to own all of this infra for your company, check out our "software factory" factory over at noriagentic.com)
So - apparently it's not fully self-hosted, since I don't see a GPU.
I'm interested in hearing from folks who are hosting their own GPU to run coding models. So far my own results are... not great. Seems like frontier models are needed via the big providers?
My favorite real-world example that I worked on: I created a YouTube documentary series about corruption in a small town in Illinois. This required downloading thousands of hours of government meeting videos from YouTube, transcribing + chunking + embedding, summarizing meeting segments, running a couple passes of validation, then searching for interesting storylines. I also scraped thousands of public documents which exposed campaign finance violations and some truly nefarious stuff going on behind the scenes.
I did 90-95% of the work on this with a local GPU. I still ran Claude and Codex, but they were just writing code to orchestrate API calls against local models, local Whisper, etc. Doing the same with the public cloud would have cost thousands of dollars, instead it cost a few hundred dollars for a frontier lab subscription and tens of dollars for electricity.
I recently got into the DGX Spark and just a month ago, got an AMD Ryzen developer platform. Both are incredibly useful tools for some upcoming projects I'm working on (also related to government corruption), but in my limited experience of trying to run opencode on them, the "good models" are still completely unusable because the system prompt alone requires a minute of thinking.
Sounds amazing! What’s the name of the series if we wanted to find it on YouTube?
I recently picked up R9700 to run Qwen 3.8-27B.
1) I have knowledge DB that I query, spend a lot of time to get "query N models, stream to me all N results, let me pick which one" - only to find out that qwen beat all other contenders (those were picked before I got R9700, so the rest is =<8B parameters)
2) Setup OpenCode to use it and gave it a few tasks:
I'm pretty sure codding issues are just harness and lack of memory that Claude Code already had. Pretty nice setup, similar to mine but I built my own lightweight PaaS that is highly specific to what I run.I have a similar setup to the OP (although I'm not sure if I'd call it a "software factory"), and I utilize a local model for some aspects of the setup. Specifically, I have a single RTX 3090 Ti with 24GB VRAM on my (main) home server, and I've been primarily running Qwen 3.6 35B A3B out of it via Ollama (I just switched to Qwen 3.8 27B, though, and I've also tried other Qwen models as well as Gemma models).
The Qwen models are decent, but they don't come close to the full Claude experience I've come to expect. As such, I only use the local models for specific tasks where it makes sense to do so. Really the setup is that my Claude-powered agents are able to incorporate my local model into work it builds out. The agents can perform inference against the Ollama API as they see fit, and I encourage them to do so for tasks where (a) the low-level capacity of the local models make sense and/or (b) where costs can become a concern.
It seems to work well when it comes into play (like having Claude drive a web browsing session but letting Qwen handle much of the actual browser interactions, image analysis, etc.). Still, Qwen just isn't smart enough (or fast enough on my machine) to handle anything agentic that isn't non-trivial.
Using a RTX 5090 ($3~4k), I can run Qwen 3.8 27B at ~180 TPS with ninfer [0]. With its thinking maxed out, I can confirm that the quality of output is roughly on par with Opus 4.5~4.6 - that is, this 20GB file really can write software by itself, but the amount of thinking required makes it strictly slower than larger models, even at 180 TPS.
Still, it's largely replaced the cheap tier of the frontiers that I would otherwise be using. It can be run with older GPUs (a 3090 is ~1k), but the time spent thinking will become a fairly noticeable impediment for staying in the flow.
The next step up would be to run DeepSeek V4 Flash 0731 on two DGX Sparks (~$10k), which serve at 60 TPS and sit somewhere around Opus 4.7 level without 3.8-tier thinking.
However, it is worth noting that, if you are buying this hardware just to serve LLMs, it is not cost-effective. It would take over a decade of continuous use to make back the cost of the DGX Spark setup in 0731 tokens. I'm running this setup because I happen to have a 5090, and the two 3090s in my server were cheap enough when amortised over several years.
[0]: https://github.com/Neroued/ninfer
"using Codex for inference" says the article
We've put almost back in the title above.
(Submitted title was "A self hosted AI software factory")
Apologies, I was trying to choose between a snappy title and the whole story.
Check out some NIXOS communities. People they are doing fun crazy stuff, as NixOS is immutable, thus creates separate sandboxes (building/taking down) per agent, etc.
I don't know most of the stuff there, but at least you can dive there.
I was recently inspired by another article here to start my own. My skills are written and tested, the factory has built the first test project, and I’m setting up the final machine to run it. Here’s my initial post about my motivation and early plans plus some follow-ups, and there are more to come.
https://joeldare.com/creating-a-minimal-dark-factory
Just read the intro post, it’s a good read! I’ll read the rest when I get home from work. It’s nice to see others have had the same idea.
People who build software factory are in for a disappointing future. Spoken as someone who uses AI day to day for job and personal projects, I've came across a few super gnarly bugs already, ranging from frontend React apps (yes, believe it or not, frontend is far from solved) to embedded that Claude can't solve. Those bugs are feature breaker. My primary skillset is fullstack/frontend leaning, not embedded. Granted, I might be out of depth in embedded/RTOS but I am very qualified to do frontend. Yet I found bugs in those areas that Claude Opus high can't even solve for hours.
I pointed Claude at those for hours and hours, didn't fucking work, ended up creating more and more slop. Until I stepped in and fixed it in a few lines of code. Thankfully with all my AI assisted work, I took time to understand everything so I was able to debug and fix.
The problem for software factory is, that up to a certain point, you'll encounter some nasty bugs that are really important for your features, but you can't ship it because while the AI deals with the 90%, turns out that last remaining 10% are the one that make or break your project. And you need to deal with that remaining 10%. But how can you deal with the remaining 10% if you don't understand that 90% or majority of it?
Also, don't listen to influencers like Kent Dodds or Uncle Bob or Steve Yegge, who advocated for software factory walk away from the code kinda thing. They are already rich, and they sell teaching content, but most importantly, their time in this industry has passed. They are not in the weeds doing enterprise software development anymore.
Their first and primary concern is trying to be relevant to sell courses. They don't care about you. Yes I used to listen to them (though not often), but now not anymore. The world doesn't get better or worse whether they choose to sip margarita on a beach or spin yet another "so many words to say absolutely nothing" video/blogs.
What most people leave out of the conversation is the element of time. We are early. The technology continues to improve.
Is it perfect now? Absolutely not. Will it continue to get better at a surprising pace? Absolutely yes.
So, just adjust your framing from "we're there" to "we're getting there" and all of your concerns go away.
I skipped straight to the end to find exactly what I expected the tutorial would cover: security and isolation.
Can anyone that has OpenClaw/Hermex experience tell me what it's like working on this over OpenCode served?
On bigger tasks OpenCode sometimes hangs. I'm not sure if that's on the provider side or on my network side, but i sometimes have to stop a subagent because its done but doesnt clean up. I wanted to try Pi, but now I'm wondering if Hermis is a better fit.
Why is it always assumed the reader knows the function of 15 obscure libraries and/or frameworks when posting projects like this?
I don't know most of these tools either, but that is why I find posts like this useful. Don't you end up learning something new in cases like this?
I can relate to this feeling, having encountered it many times in academia. But the inverse would also be true, would you stop reading yet another article that explains for the umpteenth time what library Y does? Not arguing for the author, but there’s certainly something about knowing and writing for your audience.
Maybe it's assumed you're capable of Googling?
Neat stuff. I use Hermes with the $20/mo ChatGPT Codex integration, myself, as well as Qwen running locally on a 64GB M1 Max Macbook Pro to help save my ever-quickly shrinking Codex credits. It's been wonderful for stuff like, say, giving Hermes an account on my Home Assistant server (after taking a backup) and having it make a bunch of configurations, or draw conclusions based on historical sensor data (trying to determine if my A/C is undersized by feeding Hermes all the capabilities of my system, size of house, and letting it pull room temp sensor data from HA is one project I did recently), is absolutely fantastic. It also spins me up little webapps for things like a very me-specific RSS reader, a "to do" checklist that references my email (and archives said emails when I mark an item off the list), and various little internal IT tools and report generators.
The Hermes subreddit is a curious place where every second person has quick their $300k a year job and is making a living off of Hermes doing... something? They treat themselves as the CEO of a bunch of agentic employees and have AI generated infographics of their "Stack" (all hail the mighty Stack) and everybody stands in a circle and applauds the most convoluted Hermes setups you've ever seen with not a single word how any of this is supposedly making anybody any money.
Are people developing like this? Are people asking money for something they one-shot instructions to an orchestration agent which delegated to fifteen other agents in Kanban and then spat out something that "works"? Every project I've ever had an LLM do a majority of the work for me has been strictly for my personal use; I'd never let anyone else use this stuff because it doesn't pass the vibe check. When a new frontier model comes out I'll pass the previous frontier model's work past the new model and let it tear it to shreds and see what improvements could be made for shits, giggles, and to waste a week's worth of tokens in the course of thirty minutes, but Reddit is overflowing with seeming non-coders who are passing this 100% organic slop off as sellable product?
I need fewer morals.
In my experience agentic sw development delivers if you are not too picky with the details or if you don't notice what's not done according to your instructions. If you want exactly what you had in mind, the last 10% might last a very long time and cost many tokens. More or less like working with humans.
That’s a great use of Hermes. Had it concluded if your A/C is undersized?
Before this little experiment I generally used it as a very specific search engine for houses, ‘Find me a house in the country but close to amenities with a workshop barn but close enough on the train to Manchester’. It’s great for this!
I’ve never been on the Hermes subreddit I’m gonna go check it out. I agree with the commenter below, it’s fine when I need no record my reps but don’t really care how. It writes tests but I don’t verify. It’s a manual ‘works or it doesn’t’ Thing. I’d never sell the code to anyone.
Quite a lot of people are automating the SDLC now and are looking at dark factory solutions.
Maybe I missed it but what AI model did you use for this?
> All on my home server, without another cloud infrastructure bill. The only ongoing cost specific to this experiment is a £20 Codex sub.
It seems at this stage hermes has the upper hand over OpenClaw in people building multi-agent systems. What about hermes do people like above OpenClaw. I'm building a completely different kind of "software factory" called Abject and I'm curious what people are valuing in Hermes above other systems.
People assemble projects from a collection of buzzwords these days without understanding the underlying technology. Did he need to use tailscale to interact with the box remotely? No. But, it's what a google search will tell you to do.
Please share a link if its on the internet.