Hi All,
I am very interested in building self-hosted AI power for help me build a complete code and large project like bolt.new does. I like what bolt.new did for me. Prompt, Run, Edit and Deploy on the go without spending so many effort.
I am using this hardware as my own Personal PC and I would like to know if this is going to be good for self-hosted as server and also using it at the same time?
Or do you have any hardware recommendation that can run this project and support me development app with prompt. My project are not so big, it should be targeting small-medium business side.
one more question, is there any way to integrate this with local IDE that can help me build desktop application?
if my questions does not make sense, please ignore. Thanks for understanding.
1 Like
Great questions @somvannda!
12 GB of VRAM should be enough to run 14b parameter models and smaller. I’m thinking Qwen 2.5 Coder 14b would be your best bet.
It won’t always produce the most fantastic results, but you’ll certainly get good speeds.
Also you can always use OpenRouter if you want to use open source models for dirt cheap without having to run them yourself. Not free but still a lot cheaper than using Claude 3.5 Sonnet for example.
If you want to build desktop apps with local LLMs, I would suggest using Cursor!
Thanks for your reply @ColeMedin ! I do appreciated this.
I have one more question. I love bolt.new and how it recognized the entire projects and keep adding more and more without getting what was built removed or deleted.
What is model that bolt.new using? Will the model you recommend would do something up to that?
Thanks
1 Like
Good question!
Bolt.new is using Claude 3.5 Sonnet, which you can also use for oTToDev and I think it really does perform the best. GPT-4o and DeepSeek Coder V2 236 through OpenRouter are also really strong options. For local Qwen 2.5 Coder 32b if you can run it is also really powerful!
Thanks @ColeMedin.
I have run OTTODev already. I tested between 7b, 14b, and 32b. Mine was not able to run 32b and 14b. my vga is only good for 7b at the moment.
but the problem is when i run the code (start application inside the bolt) it generate webcontainer.io which is not possible to view anything.
how to have it run?
Gotcha! So a 7b parameter model is pretty small, it’s likely the preview isn’t showing up because the LLM hallucinated bad code/commands. Do you see an error message in the “Bolt terminal” where the commands were run to install everything and run the site?
