Fixing ROCm instability on Radeon Pro v620s in Linux… finally.

So I made a post a couple days ago about “the monster” but following that post, a few things had become clear:

• ROCm was unstable, maybe even very unstable

• Vulkan was much slower, about half as fast, but far more stable

• I didn’t really know what I was doing (okay, I knew this going in)

I was happy to know that there was a path to booting into Mac OS without having to tear the system apart again, and that it only required editing and recompiling Radeon driver kexts 😛 but meanwhile, inference was crashing more often than I’d like. Sometimes very often. I won’t even tell you all of the things I’ve tried, but among the many tactics tried to identify the error and/or solve it, I tried:

• Multiple models, at different quants, and from different providers

• Multiple versions of ROCm, including 6.2, 6.3, 6.3.4, 6.4, 6.4.4, 7.0, 7.1.1, and 7.2

• Multiple versions and forks of llama.cpp

• Aphrodite and vLLM and even for a minute MLCLLM

• Rearranging card order

• An entire library of environment variables, llama command line options, and kernel command line arguments

• Excluding the RX 6700XT card as the odd-man-out and just running the v620s in tandem

• Many different context sizes and quantizations

Sometimes I would think I had it fixed because it had been 5 or maybe even 10 turns without a crash on ROCm!

…and then it would crash on turn 6 or 11.

Some things worked better than others. For example, llama.cpp pre-b8353 with a Q8 model from Bartowski was much more stable than the Q8 from Unsloth or llama-current. And running it on ROCm 6.2 or 6.3 was more stable than running it on ROCm 7.1.1 or 7.2. Also, using the software SMU via kernel parameter seemed to help some things. But we’d always end up in the same place: ROCm crash, back to Vulkan.

I must have decided to “just use Vulkan” 100 times, but the thing is, when ROCm is giving you 45-50 tokens/second, it’s hard to settle for 21-25 even if it’s stable. And stable it was…throughout it all, Vulkan never crashed.

— § —

The two key problems were as far as I could tell:

• PCIe bus errors which I had sort of grumblingly admitted were just the result of trying to run data center hardware on consumer mainboards

• Page faults that were by far the most common thing to take down llama when using an ROCm build of it

The solutions in my case were different for each of the three items above.

1. PCIe Bus Errors

It turns out that the, um, less expensive (though very highly reviewed) Montech Century 1200w power supply I’d acquired from Amazon left something to be desired. I’d been so wrapped up in the exotic-lack-of-understanding that I felt around LLMs, and the “it’s not going to be quite right, it’s not consumer PC hardware” mentality that I hadn’t been monitoring voltage. When I did, I found that this PSU, which claimed to have 100A on the 12V rail, was running at 11.5V-11.6V at idle when the cards were, according to rocm-smi, only drawing about 5 watts each.

It should not be a surprise to anyone that when we started to push into load, this decreased further.

I’ve replaced that unit with a Corsair unit and now we’re at around 12.05V at idle and around 11.9V under full load. I can live with that. And the PCIe bus errors have gone away.

The lesson here is that “gamer power supplies” rated at 1200W are mostly marketing. They don’t really expect you to draw the 1200W, they expect you to run one really fat graphics card and want the “1200” number to be showing through the clear sides of your gamer case so that people can be impressed. When you actually try to pull 80-90 amps, they just can’t do it.

I keep having to re-learn this lesson build after build… Don’t try to get off cheaply on the power supply. Especially if you’re going to be running 3x RDNA2 cards, including two server cards that want 25 amps each on 12V. Be sure also, for those of you that are new to this, not to put both PCIe power connectors on a fat card like this on the same cable. Run two cables from the PSU or you’ll fry your PSU-side connector on the single cable.

2. (THE BIGGIE) Page faults

These were the bane of my existence, and I’ve spent multiple nights this week up until the wee hours trying to find *anything* that would reduce the “crashiness” of ROCm. So . many . environment . variables. And llama arguments. And kernel arguments. And new builds of ROCm, of llama.cpp, and of half the libraries that they rely on.

The solution: credit where it’s due, I stumbled across this guy’s posts: https://medium.com/@agentz/how-to-fix-rocm-pytorch-memory-faults-on-amd-gpus-segmentation-fault-page-not-present-544b9f62f627

I almost didn’t try this suggestion that he makes:

ttm.pages_limit=25165824

It looks sketch and random and not really related to amdgpu or amdgpu-dkms or really even anything related to anything that I’d been working on. But I decided to look up what it did and once I did, it made a sort of sense, so I tried it. About three hours ago now. After battling this all week.

And voila… no more crashes. At least not yet. And we’re running inference on the fast Lemonade version of llama.cpp that is based on its own fork of ROCm and is pretty exotic. Previously this would page fault on the first turn. Maybe the second. But we’ve been up for several hours now and all appears well.

The above increases the size of the address space that the kernel is allowed to open up. I don’t know what the default is, but at this point, I believe I know that it’s too small for 76GB VRAM + 128GB DRAM, which is what I’ve been trying to make work.

Note that there is also:

ttm.page_pool_size

This pre-allocates the GTT address space in case you expect memory pressure. I’m not using it right now, but if I was planning to run a really big model, I’d probably set both of these values. Right now I’ve just got ttm.pages_limit set to 30000000, which is just a nudge up from where he had it.

— § —

The amazing thing is that in a week of talking to every frontier hyperscaler LLM (Claude, ChatGPT, Gemini, etc.) and extensive Googling, I didn’t step across anyone discussing ttm.* parameters in relation to ROCm. Until tonight.

I think maybe this doesn’t come up all that often since it’s basically going to impact people who:

• Are using multiple AMD GPUs on a consumer system

• In a configuration (i.e. v620s) where you can end up with quite a bit of VRAM (more than triple the typical just 8-24GB)

But many thanks to AgentZ on Medium and hopefully this helps someone else that’s banging their head against a wall.

And, just in case any of them matter (I am *not* messing about with this now that we have ROCm whirrled up and stable), here are my kernel command line args:

pci=realloc=off amdgpu.gpu_recovery=1 amdgpu.mcbp=0 amd_iommu=off intel_iommu=off pcie_aspm=off pcie_port_pm=off amdgpu.swSMU=1 amdgpu.cswr_enable=0 ttm.pages_limit=25165824

With this I am running 2x Radeon Pro v620s and 1x RX 6700XT (total 76GB VRAM, 3x Radeon RDNA2 GPUs) and, finally, they appear to be… stable.

— § —

Important addendum: https://leapdragon.net/2026/03/23/more-was-needed/