I just recently installed Strata which runs on qwen 3.8 flash next. I’m getting over 50 toks. Now a deepseek v4 version is out along with glm 5.3 flash. It’s slower but useable and very capable. All this on a 5080 16GB and 128GB ddr5.
The future for me is going to be open weights and on your own hardware.
That’s how it needs to be, and how it should be. Otherwise, we are all permanently screwed by big tech, subscribed and licensed to even own a computer by 2035; MicroSlop says open-source AI is good somehow, but we will see through their actions if they lock down their new line of NVIDIA Spark laptops to an approved number of models. I heard rumours that Deekseek might be banned in America; I can’t see how that would work though.
Deepseek is open weights. You can just download and use it locally. I would never run deepseek using their servers. Or z.ai servers. That I agree with China is still pretty bad.
While I have no problem with most of China’s principles regarding business. I don’t know about all their politics and laws, nor have I given them much thought, but they deserve more respect than they get for essentially being the whole world’s industrial estate and being able to pull off the amazing things they do while having virtually no copyright laws. Open-sourcing so many things we would rather make proprietary, thus very expensive for ourselves. I’m grateful to them for deepseek! Even if they want to spy on me a little, at least I know to expect it lol =)
Nice setup. I’m running Qwen3.8 27B at 17GB with a 32k context window on a 3090. It does most of what I need for everyday use and its all local, private, no internet connection needed.
The gap between running an LLM locally and cloud API is closing fast.
I mean, besides oil prices, the only reason why hardware is more expensive than it was before is because people are hoarding it, fearing that it won’t be available in the future. But there’s no reason why new and better hardware shouldn’t be engineered and made available unless there is another world war.
The electronic hardware cartel has stopped designing new hardware, but it was a deliberate choice to inflate prices. They even backpedaled on 5090s
I started looking into it finally after making that comment and came across strata come back here and everyone is saying strata. I guess im trying strata?
need 64gb of system ram. qwen next architecture uses smaller experts than others, and it can keep up with ram swap. I believe it uses a file on disk (ngram) to predict what experts will be needed.
Running on your hardware is always going to be less efficient than running in a data center. You can run small models on green data centers - best of both worlds (unless you need privacy then run local).
Ever since I switched to Cortecs I’m spending like 20 cents / month for my LLM usage cuz I mostly use small models and try using them responsibly.
I just recently installed Strata which runs on qwen 3.8 flash next. I’m getting over 50 toks. Now a deepseek v4 version is out along with glm 5.3 flash. It’s slower but useable and very capable. All this on a 5080 16GB and 128GB ddr5.
The future for me is going to be open weights and on your own hardware.
That’s how it needs to be, and how it should be. Otherwise, we are all permanently screwed by big tech, subscribed and licensed to even own a computer by 2035; MicroSlop says open-source AI is good somehow, but we will see through their actions if they lock down their new line of NVIDIA Spark laptops to an approved number of models. I heard rumours that Deekseek might be banned in America; I can’t see how that would work though.
Deepseek is open weights. You can just download and use it locally. I would never run deepseek using their servers. Or z.ai servers. That I agree with China is still pretty bad.
While I have no problem with most of China’s principles regarding business. I don’t know about all their politics and laws, nor have I given them much thought, but they deserve more respect than they get for essentially being the whole world’s industrial estate and being able to pull off the amazing things they do while having virtually no copyright laws. Open-sourcing so many things we would rather make proprietary, thus very expensive for ourselves. I’m grateful to them for deepseek! Even if they want to spy on me a little, at least I know to expect it lol =)
Nice setup. I’m running Qwen3.8 27B at 17GB with a 32k context window on a 3090. It does most of what I need for everyday use and its all local, private, no internet connection needed.
The gap between running an LLM locally and cloud API is closing fast.
Yeah but getting your own hardware is getting expensive too.
Only because there is artificial scarcity in play
There’s plenty of actual scarcity to go around too
I mean, besides oil prices, the only reason why hardware is more expensive than it was before is because people are hoarding it, fearing that it won’t be available in the future. But there’s no reason why new and better hardware shouldn’t be engineered and made available unless there is another world war.
The electronic hardware cartel has stopped designing new hardware, but it was a deliberate choice to inflate prices. They even backpedaled on 5090s
How do you run qwen 3.8 flash next on 16gb vram, if you don’t mind me asking ?
At 3 tokens a second and calling it useble I would guess?
I see people claiming that shit all the time and it’s baffling.
They said 50- i looked up strata, and it uses smaller quantizations/optimizations of that model depending on your system.
I have 48g vram and 192g system ram and get ~15t/s on qwen 3.8 flash next at Q4. Off the top of my head. I think Q6 is 12.
I started looking into it finally after making that comment and came across strata come back here and everyone is saying strata. I guess im trying strata?
https://sh.itjust.works/comment/28041846
Try it out and let me know if you can get higher.
https://sh.itjust.works/comment/28041846
I didn’t believed it until I started using it.
https://github.com/niko1221/strata
The model I’m using is a Q4. Unsloth UD-IQ4_XS
need 64gb of system ram. qwen next architecture uses smaller experts than others, and it can keep up with ram swap. I believe it uses a file on disk (ngram) to predict what experts will be needed.
Running on your hardware is always going to be less efficient than running in a data center. You can run small models on green data centers - best of both worlds (unless you need privacy then run local).
Ever since I switched to Cortecs I’m spending like 20 cents / month for my LLM usage cuz I mostly use small models and try using them responsibly.
That’s why you do it at work on a secured laptop you own that’s on their network. Modern problems. They pay electricity and all the jazzzzzzzz.