I have avoided AI (LLMS) because it didn’t seem nearly useful enough to warrant supporting these tech assholes. Is there anyway to use it locally on a normal device (16 Gb ram, 8 year old Geforce graphics, etc?). Can I just slow it down and get it to work locally?
99% of AI models available are based on stolen work, so even running it locally is immoral…
“Copyright” is immoral and exists to protect capitalists. Intellectual property as a concept needs to die, and knowledge needs to be accessable to everybody.
This is what a person who has never developed anything would say.
Like, Stallman?
I’ve developed and written loads. Everything licensed CC BY-SA.
I will now reuse your work without attribution and claim it is the AI’s fault. Easy.
Everything you’ve made is now CC BY-SA too, awesome. Thanks for contributing.
No problem, you’re welcome. I also paid a small car’s worth of money to a fascist oligarch while borrowing the fruits of your work. I, too, am very happy with this.
But look at the bright side, you too can now pay a small car’s worth of money to a fascist oligarch and then get some source code that may or may not be close to my proprietary stuff. You know, the one made from your stuff without attributing. This is fine.
I think you may have lost the plot there.
There are models that will run on a smartphone. They are not as capable and have smaller context windows, but they work.
Can I just slow it down
It will be slow because it is slow. And no, you cannot unless you like reading one word per second.
One word per second is already quite fast actually.
Man you can absolutely run something on ollama yourself. Right now I have ollama running qwen 3:4b-instruct on 6vCPU and no GPU and for very simple conversations it’s quick enough. TTFT is slow when prompt is uncached but otherwise TTFT is like 2s and I get maybe 5 words/s out.
If I gave it my 1660 I could probably get ~100tps but my card won’t pass through to my VM properly…
I’ll just leave this here. https://sleepingrobots.com/dreams/stop-using-ollama/
Reading through it I see some good points but… What’s the alternative? Isn’t Ollama still the least of evils (by far)?
So… I’m with you/them about the vendor lock in but I think they’re slightly underestimating what ollama brings to the table. For example, ollama bundles CUDA libraries so I don’t need to worry about that as I wrangle with my GPU. It’s also well supported by other ecosystems like home assistant in a way that raw openai compatible API server isn’t.
Overall I don’t know if I find these arguments compelling at this time.
ollama
Shittiest local inference engine. Use llama.cpp instead.
qwen 3:4b-instruct
Old version 3 model from Alibaba. 4 billion parameters total. It will struggle with anything other than natural language generation.
on 6vCPU and no GPU
Running on a 6 vcore VM with no GPU pass through.
TTFT
Time.To.First.Token. How long it takes the model to start responding after you hit enter. Dependent on not only processing speed but also context length.
prompt is uncached
Cold start = no cache = slow. But faster when mid conversation and the key-value cache is hot.
100tps
100 tokens per second. 1 token is roughly 1/4 word.
You can, but it’s not worth the effort. The technology itself is set up for people of power, such as governments and corporations. That’s why they steal our hardware and make it unaffordable. The only genuine use is accessibility, but it’s purely audio/text recognition. Everything else is for targeting people with drones and surveillance state.
Edit: how could I forget deepfakes? You practically can’t live without them nowadays /s
The only genuine use is accessibility, but it’s purely audio/text recognition.
On that note:
LocalTranslate is an offline translation application that uses Firefox’s neural translation models (from mozilla/firefox-translations-models) to perform high-quality translations locally on your device.
Wild conclusion
While I respect your cynicism and it’s fully justified, in fact they can do useful and interesting things for individuals too. They must be used cautiously, and with understanding of their limitations and a realistic appreciation for their capabilities. Large neural networks like this are in a fundamental sense statistical models. I have pointed out in the past that many of the large general-purpose “assistant” models that have been trained on what amounts to very close to the entire corpus of knowledge and information contained in libraries and on the internet. Regardless of why they were trained, and whether that was done ethically (it wasn’t) the fact is these models now exist, and we have access to many of them, and they contain statistical models of what I will refer to as the vast majority of human knowledge to-date.
If you can’t imagine that there are ways that statistical information could be valuable, you are either incredibly lacking in creativity or simply being ignorant and myopic as far as I’m concerned.
As an analogy, national and sociological demographics and census data is abbreviated and summarized statistics much like neural networks. It cannot in many cases correctly and confidently predict any particular outcome. But that doesn’t mean it’s not useful or meaningful. It is not a complete and accurate listing of every human’s thoughts, feelings, and preferences, and most honest people would not imagine it is. If you ask it for that, you are doing it wrong, or you are selling something. Same with generative AI models. The fact that some people (including huge companies) misuse it does not mean that such statistics are evil and we should throw it away.
Using it for intelligent, thoughtful purposes is a form of rebellion and I believe it is important that we continue to have access to these technologies. They are not intelligent, and they are not even artificial. They are loosely curated distillations of human knowledge. That is very powerful, it is a very fascinating and practically useful tool. It is not magic, it is not a real boy, it does not have feelings or purpose, it is a tool. It is being used for purposes, but you have the power to use it for purposes too, and those should be your own purposes, not what purposes companies dictate to you. Engage your brain and learn how and when to use it, don’t use it to replace your brain, it’s not what it’s being sold as, but it is a very clever and interesting technology.
It is so refreshing to see a well-formed opinion on the character, quality, and appropriate use of AI. I would like to thank you for your service.
I appreciate your support. Trying to be nuanced and realistic and find common ground on a topic that is so polarizing, and frankly potentially existential, is difficult. We live in a scary world right now for a lot of different reasons, and people are scared (of being left behind, of having their work stolen, of having their humanity stolen), and I don’t blame them. These are valid and justifiable concerns that deserve serious consideration and serious discussion. Which, in a lot of cases, they’re unfortunately not getting.
If I were running the show, I would do things a lot differently. But I’m not running the show, and I don’t think the people running this show have even the vaguest interest in listening to me or probably pretty much anyone, so all I can do is try to educate and provide advice, which is free, while I too worry if AI is going to take my job.
I think I’m (unfortunately) starting to really understand what Frank Herbert meant when he wrote, “Fear is the mind killer.” It’s very disheartening to see legitimate concerns snowball into lemming-like behavior.
When in the throws of despair or grieving for a society in decline I’d typically appeal to the classic MiB quote:
A person is smart. People are dumb, panicky dangerous animals and you know it.
But as the years go by it’s becoming increasingly difficult to find solace there — mostly because my experience is showing me that it’s simply no longer true. I feel like I’m living in a world where everyone read Bonhoeffer’s Theory of Stupidity and most people responded with, “Where do I sign?”
I’ve observed this behavior from both doomers and AI evangelists.
However, in the effort of full-disclosure, I must admit that I am a staunch accelerationist. That’s because, when viewed from a cosmological perspective, the only rational way to describe the last few hundred years of human development would be as an explosion, and I firmly believe the only path out is forward. Standing against the tide of technological progress at this point would be akin to standing against a blast wave.
That’s really damn scary.
The real sticking point seems to be that increased knowledge does not alleviate the existential dread. On the contrary, in many ways it exacerbates the mortal fear. Knowledge without power or prescience is impotent. Not that it would make a difference anyway.
If we had developed the concept of The Great Filter before the steam engine, would the world have responded to Watt’s machine in the same way it’s responding to Dario’s now? And would that have even made a lick of difference?
Very interesting commentary. I get what you’re saying, and I agree. I think there’s something similarly paradoxical to Dunning-Kruger going on with these kind of existential concerns, too, and I think that is probably reflected in your accelerationist position. We can also see it in climate change for example, the people who actually understand it and study it are less likely to find or see anything that can actually stop or reverse it, besides the one thing that it’s increasingly clear we won’t do: drastically cut emissions. And the deeper you understand the actual threat and the realistic psychology around it the more overwhelming and hopeless it seems, and the less it feels worthwhile to even try to stop it.
By the time all the less-understanding, less-pessimistic people start to actually see the unavoidable long-term consequences and trends of climate change, we’re going to be so far beyond having any sort of handle on it to control it, the only remaining hope is for a miracle, no matter how improbable, and if you have no hope of stopping it, then pinning your hopes on the miracle and trying to get as far ahead as you can until and unless that happens is (sadly) actually the rational choice.
No
You can run very small models with Ollama or Llama.cpp. It really depends on the type of work you want to do, but these small models won’t do anything super complex. In my opinion the models don’t start getting really good until around the 30GB size like Qwen 3.8 27b. You could probably still use a small model for some chat, search, rag, maybe simple script writing.
Note that you would need to use Llama.cpp for the Mix of Expert type models like the Qwen mentioned, Ollama doesn’t have the flexibility of commands.
Ollama can run MoE. It just decides for you where to place things. Also the Qwen model mentioned is a sense model.
That wasn’t my experience, but maybe I just didn’t run across the right instructions, or it was the particular model. Llama lets you adjust parameters far more. Two sides of the same coin, really.
Ollama is just llamacpp under the hood. Ollama is just an opinionated version that automatically sets things up that will work for most people. Llamacpp definitely has more of those “nerd knobs” that you can configure for better performance. The flipside is that it is far more complicated. Granted using another LLM to help you makes it not so bad.
Ollama lags behind the cutting edge of Llama.cpp but not for that long. It can do all that stuff. Maybe not well, and certainly much less configurable, but for casual use it’s fine and it’s much easier and more convenient until you start needing more control.
Not bashing Ollama, it’s great for getting started, and finding new models. I just had to switch over when I couldn’t set up the MoE model. The good thing was Llama can use Ollama blobs, so no loss of the older stuff.
You might’ve been running an outdated version of Ollama or something, or maybe just a model that wasn’t set up in a way Ollama liked. It sometimes has some compatibility problems with some particular models or the way they’re configured/implemented, it’s sort of stupid how it happily pulls models from huggingface but without actually configuring them correctly in many cases, but generally it fully supports MoE and all that stuff. Generally you can trust that the ollama-hosted models will pull and run correctly, but huggingface (where the better and more interesting models live) can be a crapshoot, it’s just the way it pulls and configures them though, there’s not actually broken.
You can get the TernaryQuench version of Qwen3.8:27b to run great on a GPU with just 16GB of VRAM. I started using it last week and it’s been amazing.
I have a 4060 Ti 16GB and was using the 14b version for a long while but that ternary conversion tech is just enough to squeeze the full 27b version into my card with plenty of room for context 👍
Is that by penkia on hugging face? (How safe is running random models?)
You should try it out. Ollama is what you’re looking for and it’s easy to set up.
Unsloth Studio is one of the easiest and most approachable methods of doing what you want. There are some smaller models that will run fine on the hardware you described, but they won’t be particularly “smart”. Still, they might work for what you need.
It’s important to bear in mind that as we move forward, these open weight models will just get smaller, better, and more efficient. Data centers are absolutely the wrong approach for this tech, and the only reason it exists that way now is due to capital interests trying to steal all the value. It won’t last as the technology matures, so go ahead and at least familiarize yourself with some ways to run local because that is the future.
I run a “server” (old 9-year gamer laptop) with an 8 GB Geforce 1070 Max-Q. It’s currently running full GPU-offloaded stock llama.cpp, with Unsloth’s Qwen3.5 9B at UD Q4_K_XL (good enough quantization for coding assistance and short agentic tasks) and a 90k context at around 300 t/s prefill and 25 t/s decode, which are good enough speeds for daily use.
It can’t vibecode GTA6, but it can help with quick scripting, research and things like that.
No. Because knowledge is power, and LLMs teach you not to teach yourself to acquire knowledge.
The size and token rates mentioned in this thread for local inference are all going to be exceptionally limited compared to the frontier
Forget about local inference with your hardware, it’s too far behind. 3 years behind in this technology is basically stone age
What you’re looking for is “ZDR” (zero data retention) policies from online providers. There are many options. One powerful model with ZDR you can use is gpt-6 Luna on duck.ai
Even if you’re trying to code, you’ll get better mileage copy pasting to and from duck than ollama with your hardware
trying to code
copy pasting to and from duck
That’s, uh… The worst way to do any of this. And GPT-6 Luna is the weakest GPT-6, if you’re going to run something that’s on someone else’s hardware anyway, you can just use openrouter and choose ZDR hosters of open-weight models that are somewhere between GPT-6.1 Sol and Terra in accuracy.
OP said they didn’t want to support billionaires and data centers
The money used to make zdr requests on openrouter to good models like GLM 5.3 flows into large inference providers like fireworks and baseten
Yes, copy pasting is a terrible way to vibe code. But my point is it’s still going to be better than trying to ollama with a 1060 and 16 GB RAM
For those who wish to avoid big tech, and don’t mind potentially being on a data center: openrouter lets you use w/e model you wish (even free), paying per token, from whatever provider you wish (in particular, smaller companies, folks self hosting, etc.)
Some datacenters are built in terrible places, but not all.
OmniRoute is also pretty nice for that.
No, and why do you want to rot your brain and/or produce more slop anyway?
Because LLMs now work better for searches than commercial search engines.
Ha! No they don’t! It gives you a hallucinated answer that’s not even on the webpage that it claims it got the answer from.
At producing misinformation.
Why would you want misinformation? It’s going to lower your productivity, because you then have to fact check everything it tells you
