Does this translate to older generation 16GB GPUs, e.g. RTX 3080 / RX 6950? (Assuming BF16 is available, or possibly even FP16?)

I have 2 HP DL380s with 2 Xeon Gold 6252 CPU @ 2.10GHz and 256gb of DDR4 memory that I bought for $700 a piece off ebay. I run 4 gpt-oss:120b models on CPU using 120GB memory for each VM. I get about 10-15 tokens/second for each virtual machine running ollama in cpu mode. I run these over night for complicated tasks. I have a multi agent system that will kick out the hard tasks the RTX 5070ti 16 and macbook pro m1 max 32gb can't do to the CPU farm. Works pretty well.

A small update: I've also managed to port the engine to iOS.

I'm testing it on my old iPhone 13 Pro (A15 Bionic, 6GB RAM) with Ling-mini-2.0, a 16B MoE model. The first tests are showing around 13 tokens/sec, which is pretty encouraging for a phone from 2021.

It's still early, and I need to test memory usage, longer generations, and how performance changes as the phone heats up.

I'll keep posting updates here as I run more tests and see how far I can push it.

Cool, I guess.

I guess I'm not really sure what the point of this (100% AI generated) article is.

You have a magic engine with zero details, that you're comparing against a (to me at least) obscure github project.

I guess I'm not really sure what the takeaway for me is supposed to be. The benchmark setup is not detailed enough to make meaningful conclusions from it, and I guess the big takeway seems to be that a bounded cache that an engine process owns behaves better than relying on the system cache?

Paste this into the claude session: "Add a section explaining the architecture of this engine, and how it compares to TUFF, EdgeMOE, etc SOTA implementations. Explain the novelty of this implementation. Show how experts are imported / evicted, and how that decision works. Also add more detail in a benchmark section about the exact testing you ran. Give test runs, upload to a git repo that links to this article. Include some charts showing test to test variance. "

I'm running Deepseek 4.1 Flash in 4 gigabytes, but I'm not going to tell you how.