What tools and projects do you use for your local LLM chats and agents?
I’m trying to sift through the mess of LLM harness and tools to find how to really get the most out of local LLMs.
You must log in or register to comment.
What’s stuck for me after trying too many things:
- Ollama (or llama.cpp server if you want control over flags) as the model server, bound to localhost only
- Open WebUI as the front-end; it also handles OpenAI-compatible APIs, so local and hosted models live in the same history
- LiteLLM in between once you have more than one backend: one endpoint, per-key budgets, and logs
- SearXNG as the search tool for anything agent-ish; keep concurrency low or upstream engines start blocking it
Biggest lesson: RAM decides what you can run, and document upload/embeddings in Open WebUI eat more memory than chat does. On small boxes, point embeddings at an API or give it 4 GB.
The anti-AI sentiment here is ridiculous. They were asking a legitimate question!
I just use ollama locally but I keep meaning to setup OpenRouter because I like to play with larger models that my hardware can’t handle and I’d like to be able to pick the best deals 🤷♀️
AI effectively raped and pillaged all human information for the benefit of a handful of corporations. It’s use has been shown to reduce the skills of school children and even deskill those with established knowledge. Then there’s the environmental considerations in a time where the world is decades behind appropriate climate change management… Id say it’s pretty understandable that it evokes passionate responses.
