The state of AI:
-
The $180 of memory I bought when I built the machine I’m now using for inference would now cost $700
-
The <$1,000 of GPUs (4x32GB) I bought in Q1 of this year would now cost $4,500
-
But as a result everyone is moving to models that don’t need so much compute or bandwidth, e.g. Qwen 3.8 Flash Next with its n-gram sidecar
-
Almost nothing is supported and nothing can keep up with the change
-
All the AI stuff is so buggy that it barely works
-
But you can have your AI empirically profile your hardware and then write or port drivers, applications, and inference engines (vLLM with all custom kernels took maybe 5 hours)
-
And you can just pull the Github repo that’s buggy and tell your AI to fix all the bugs and submit PRs and then the application is perfect
-
AI has “enabled” us and many other orgs to let a lot of people go who may never work in their field again
-
But we’re desperate to hire more AI engineers and are actively giving young people, more conversant with AI, paid internships to do AI
-
The AI world means that I sit here often for hours on end working with AIs
-
But in the past I’d be sitting here for hours on end thinking alone in C or PHP or Python or JS, rather than chatting with another party in English
Is it the best of times or the worst of times?
It is the best of times. it is the worst of times.
— § —
In case you’re one of the many that acquired a happy family of Radeon Pro v620 hardware and now finds yourself wishing vLLM supported it, it does.
My related repos: