I figured it would be cool to have a thread where everyone can share their setup, could be a good resource if someone is looking for assistance along similar lines. Mac development certainly has its caveats in this space versus PC dev/GPU landscape. I’ll share mine:
Hardware: I’m on an M1 Max Macbook Pro with 32GB RAM, running with ~7B models fully GPU loaded.
Backend: I run ollama locally, I do appreciate LM Studio’s native GUI and its ability to specify context limit/context override methods upfront as presets.
Models I’m working with: Focused on qwen2.5-coder:7B with 64k and 128k context windows. It seems that the speed of 64k versus thoroughness of 128k are both beneficial in different use cases. For vision, I have found that llava-llama3 has been more effective even than the recent llama3.2-vision models. FYI, Cole pointed out in a recent video that Ollama defaults to a 2048 context limit by default. A quick fix is to use a Modelfile to explicitly specify the context limit while cloning an existing model.
Frontend/Chat: Open-webui, along with various tools to seed/investigate RAG knowledge graphs. Tons of features to take advantage of, including tool/function use and Python-based pipelines. Web search is being handled by a local docker container with a self-hosted multi platform search API. Docker is a life & timesaver.
Coding/IDE: I’ve used continue.dev with good results in VSCode, but recently moved to the Zed editor simply due to its built in Assistant/Inline AI tools, and collaboration that allows for project sharing, audio comms and hi-res screenshares directly in the editor. I’ll still probably go back to VSCode at some point once these tools are more ubiquitous, but I’ve been really impressed so far. Zed will also allow you to use your existing Copilot access to GPT models for AI assistance, which was a complete surprise.
I’d love to hear more about what systems folks are running and any promising results from individual testing. Cheers
I will keep it short.
MacBook M3 due to work, would have been fine with Nvidia RTX Windows laptop too if not better.
I am long time WebStorm users with habbits that are hard to break, but am flirting with Cursor these days but sturggling with bad UX is for me as not a VS Code but WebStorm user.
I also have LM Studio and play with models but usually am disappointed by speed/quality. So I use hosted models more for speed and quality.
Am heavy ChatGPT users and have my own project I called Server Commander that allows me to use ChatGPT for local computer manipulation and quick read/write code instead of copy pasting.
Kinda like Cursor just worse
Otherwise for frontend only experiments I am heavy user of WebSim currently but hope that oTToDev/Bolt.New will get to a level where it can take its place.
Thanks for explaining why you are using Zed. On a PC, so will have to probably use Continue.dev or Tabby as I want project-wide context awareness.
I have not tested, but one thing Tabby does is support multiple data connectors to expand context like Repos, docs and config files.
My main machine is a m1 macbook air from 2020, thinking of upgrading to m4 (maybe next year not sure),
I am a VSCode person
I have a home server with following setup
proxmox (with 3060)
|-Tailscale in LXC
|-Portainer in LXC
|- Ollama
|- OpenWebUI
| ...
|- Other stuff in container
I have setup a site 2 site networking with tailscale so I can access my personal server from any locations from local network. and when I’m outside the network I can connect to tail-scale to access the server, and run inference from that server
most of the time i use cloud AI model but uses ollama with llama3.2 sometimes and some 7B models from server
