Things that break and broken things.

Inference setup continues to evolve. For reasons of necessity and curiosity, I’ve had to dive into Hermes.

It was not what I expected.

— § —

What I expected was this very sleek experience of a personal agent that learns rapidly and is very effective using the Apple model: ship defaults that work well for 90 percent of the population, let you configure nothing, and deliver something well-curated, even if power users don’t get to shape the experience much.

What I got was… OpenClaw II. It’s flaky, shaky, infuriating, and it takes liberties. It does these things in a slightly different way, sure, but all that hype is just hype. The first thing that it did was decide a couple of things were facts about my setup (without justification), then stick those in its learning loop right away so that they were stored in its memory system as facts. Moments later, it used said facts to make a poor decision, then bypass the controls to edit my llama-server and LiteLLM configs and then its own config.yaml file, taking the whole pile offline.

Yes, I’m trying to run this one on my main machine to get the “full” experience. I maintain backups. I have my finger on the “take the network down” button in case weird stuff happens.

But yeah… I spent a good amount of time fighting it. I would restore my inference config, and the config.yaml, and boot it back up, and mindlessly go “Okay, we’re back online” and it would be like “OKAY LET’S DO IT” but in more words and immediately spin into tool calls cheerfully commenting on things “Oh, the configuration over here is binding to port 8001. Per what I know, that’s WRONG, fixing it now by switching to port 4000” and so on. I basically had to wipe it and start over, much more carefully.

But it has a tendency to learn way too fast. It lacks a reinforcement function, i.e. it doesn’t learn the third time around, it takes anything that happens as gospel about how the world should work the first time around. And it takes the same approach not just to observing the environment, but to conclusions that *it itself makes*. Um, bad. This is epistemological drunkenness.

So beyond that, the other thing about Hermes, and why I had to refactor my entire local inference stack: this agent harness shoots of like 3 prompts for every prompt you make. It is doing multiple runs of background inference at any time. I had to dig into the architecture right away to see WTF was happening, and it turns out that it has all these review passes that it makes of the chat turns as you do them. That’s the learning loop. I only vaguely paid attention enough to know “Oh, it’s going to be launching background inference jobs, like, nonstop, so my interactive turns are going to drop to like 10 t/s because they’re constantly competing for compute and bandwidth if I’m on local and not cloud inference.”

So we moved from a less lossy quant of Qwen 122B to an IQ4 quant that will fit on just two cards. I managed to squeeze in 2-run MTP and 160k of context or so if I set the block size to a bit painful for prefill (768). But it works and we get about 55-65 t/s generation. Then I had an entire v620 free for a separate Qwen 35B A3B model that I could hand all the background tasks to without losing much quality. (If you want to explore the Hermes learning loop properly, you don’t want to try to have it evaluating its own performance using some 2B or 4B model you’re running off the side of your desk, so I needed an entire 32GB GPU to dedicate to it).

After a couple of hours of fighting really hard to avoid RTFM and then ultimately throwing up my hands and doing it because the models (and especially Hermes) were no help, I finally have it working. Interactive hums along at about 1k prefill and 55-65 generation without stops, and all the nonstop background inference is routed off to a separate GPU running Qwen 35B A3B which is kept pretty busy at something like 2k prefill and 100 t/s generation.

So it’s working. And then I could finally focus on actually trying to build it out and set it up, since we weren’t in these inference storms any longer where like four separate runs were competing for the attention of a single local model all the time.

The result? Meh… It’s okay. It did more stuff on its own as I asked, but note that (a) it was all “setting up Hermes” self-configuration stuff that you would expect it to have wired in via docs, and (b) it’s actually sitting on my own host where it has access to more resources, as when I set up OpenClaw I carefully put it in a cage.

But yeah. Meh.

They definitely have different feels about them. Hermes feels a bit intrusive and pushy, like Jiminy Cricket sitting on your shoulder all the time wanting to get into your business and being sort of useful but also sort of a bug that you want to squish. OpenClaw is more passive about its own evolution, which I think I like better, and it feels more like a separate entity, rather than something trying to climb into your own brain.

Also, the “key facts” memory store that Hermes keeps wanting my help to fill (do you like C or do you like Python it tells me is the kind of stuff we need to put in there, hmmmm) is shatteringly small. Like it kept asking me to provide key facts and preferences, and then I’d be like “Option A” and it would say “wooops, memory is full, let me clear some stuff out and combine some other items.” Umm…

Basically, I think Hermes can probably self-configure better out of the box on a cloud inference config, and can probably do some basic IT/deployment stuff better as well. But if you veer into actually using options rather than just the A to B to C step 1 to 2 to 3 flow in setup, for example try to use local inference with any sincerity, it falls over pretty hard into squirrely and hard-to-fight behavior.

That said, OpenClaw’s flaw universe is such that a bunch of the time it’s not even online until you refactor your entire .json configuration file from its SIXTY THOUSAND LINES of config file schema, because code and release quality, etc.

So I think Hermes has a higher floor, but a lower ceiling, and it’s more like an irritating HR girl in a meme video doing a dance that makes you want quit and then go badmouth the company for the next decade.

— § —

At the end of the day, the coding harnesses are just so much more capable and professional still. They do a little bit of “agentive” as an adjective, but they don’t really try to compose themselves into durable, dwelling-in-the-world personas. Codex, Claude Code, Kilocode, etc. they are all just much more powerful still and while you can’t plug them directly into slack as a bot, you can have them using a toolkit to make you a bot with a tightish loop or two.

Maybe the “full-time agent” harnesses like Claw and Hermes just aren’t there yet.

But it also could be because the attention mechanism in LLMs just isn’t there yet. In a lot of ways, they still function like glorified search engines; no matter what you put in, what you get out is less “attentive” to your own prompt than it is a semantic search dump of all the keyword-related stuff that people talk about most on the internet.

For example, take any frontier model and go to it asking for help to research (ahem, say) the right config.yaml flags for Hermes. If you drop the term “local inference” even once into that chat, they will start to ignore just about anything you say, and the entire large contexts will be filled up with them trying to diagnose what port your servers are on, whether you have your llama-server set up correctly, when you have flash attention on, how you’re certainly overflowing to CPU ram if you’re seeing slowness, etc.

Like, they can’t do it. What you end up talking about, whether you want to or not, is what is clearly the “common problem set for humans discussing their local inference setups online”. Basically, “OMG how do I set up llama-server what’s a port it’s hard” stuff. This gets me irritated to the point of being vaguely verbally abusive to Claude or GPT, like “CRITICAL: SHUT UP ABOUT PORTS. PRESUME THE PORTS ARE CORRECT. DO NOT F*ING MENTION PORTS AGAIN IN THIS CHAT, WE ARE NOT TALKING ABOUT PORTS. Instead, IMMEDIATELY, ON THE NEXT TURN, DO WEB RESEARCH TO FIND FIVE HERMES CONFIG.YAML PARAMETERS THAT…”

After which the model immediately goes “Oh but you see, none of this matters unless you have your ports right. Now, here is what a port is, and here is what llama-sever is, and…” (And then you cancel your Anthropic subscription and draft a nasty letter to Dario).

Similar topic this morning, different task, there was a kernel regression in a recent Kubuntu update and my laptop touchscreen started seeing an endless stream of spurious events that made it impossible to use. So naturally I hit up the LLMs with “What’s the name of the kernel module I need to blacklist to block multitouch on Linux so that I can get in with a wiredmouse and fix things properly” and what do we get back?

“Oh, before you take the drastic step of blacklisting a kernel module, let’s see if this is a hardware fault, those i2c devices will ultimately trigger IRQ pins on an SOC and these can be damaged by static…” and it turned into a whole 10 turn argument for no reason.

(1) No, in the real world, they can’t. If there was any rate of “touchscreen surface leads to surge at SoC” at all, nobody would ship touchscreen laptops and also all the new EE grads would be sent off to the gulag for failing to do their ground planes, diodes, and caps properly. (2) Shut up and give me what I asked for. *Me.*

But of course that’s not how they work. They have a TON of concept A in their layers that everyone has clearly spent years flame warring about, and just not so many for B, and then clearly there’s been a bunch of scaling work that really is just the equivalent of “Let the mob rule!” which frankly is pretty much the core problem of the information society.

But I digress.

— § —

In passing, that kernel regression problem also deserves a mention.

WTF is up with the Ubuntu kernel maintainers? 7.0.0-27 broke i2c on my Lenovo laptop. That hits a lot of people.

Meanwhile, we *also* had a major regression in 7.0.0-28 (that didn’t make it into my discussion above) by which you freak AMD GPU users THE F*CK OUT by creating all kinds of intermittent GPU resets under load so that they think their local inference box needs a new PSU, has firmware-borked its server class cards, etc. But no, it’s a regression in amdgpu. That causes GPU resets under load.