Ollama is broken

bolt.diy [bolt.diy] Issues and Troubleshooting
21

I’m having the same issue on my Mac. I’m running Ollama directly on the host so that I can make use of the GPU. When running oTToDev in the container I needed to set

OLLAMA_API_BASE_URL=http://host.docker.internal:11434

If I open a terminal in the oTToDev container I can:

first time running a ps command I get:
→ curl {OLLAMA_API_BASE_URL}/api/ps

{“models”:}

but then running:

→ curl {OLLAMA_API_BASE_URL}/api/chat -d '{

  • “model”: “dolphin_2_9_2_lc:latest”*
    }’

{“model”:“dolphin_2_9_2_lc:latest”,“created_at”:“2024-11-19T21:49:37.519538Z”,“message”:{“role”:“assistant”,“content”:“”},“done_reason”:“load”,“done”:true}

Now ps works

→ curl {OLLAMA_API_BASE_URL}/api/ps

{“models”:[{“name”:“dolphin_2_9_2_lc:latest”,“model”:“dolphin_2_9_2_lc:latest”,“size”:102690629632,“digest”:“f586f65be437e2fca804b550a90715e227ec9106d6d67cffc3bd9a7553f7a782”,“details”:{“parent_model”:“”,“format”:“gguf”,“family”:“qwen2”,“families”:[“qwen2”],“parameter_size”:“72.7B”,“quantization_level”:“Q4_0”},“expires_at”:“2024-11-19T13:54:37.519822-08:00”,“size_vram”:102690629632}]}

But refreshing the web page still won’t find any models

Screenshot 2024-11-19 at 1.53.00 PM
Screenshot 2024-11-19 at 1.53.00 PM1370×928 72.7 KB

eventually the model unloads and I get back to:

→ curl {OLLAMA_API_BASE_URL}/api/ps

{“models”:}

1 Like
22

I’ve been having the exact same issues. Docker will absolutely not work. Even trying to allow direct access host network doesn’t work either. Have tried going into the container and directly modifying the env vars too. I double checked that they’re all correct. However, in the network requests it’s not even using the ones I enter.

I tried entering in random gibberish for the Ollama API URL and it still uses the same one every time.

23

For my return: I encountered the same problem when I avoided using Docker. Ollama didn’t work initially because I had configured an Anthropics key for Claude. I deleted all my keys and kept only OLLAMA_API_BASE_URL=http://localhost:11434. It works now, but it’s very slow. I’m using the CodeLlama 13B model, which may not be the most suitable.

24

How can I get this kind of UI?
I am getting old UI.

25

image
image809×1562 167 KB

26

This is what I ended up doing.

3 Likes
27

I’ve raised this issue as a potential source of the problem.

it appears lines 286-289 are causing this issue there was a recent commit to add these but once commented out it works again

28

I modified constants.py
//export const DEFAULT_MODEL = ‘claude-3-5-sonnet-latest’;
export const DEFAULT_MODEL = ‘qwen2.5-coder:7b’;

Also I commented out the 3 lines in getOllammaModels, and it STILL tries to use claude-sonnet:
bolt-ai-1 | requestBodyValues: {
bolt-ai-1 | format: undefined,
bolt-ai-1 | model: ‘claude-3-5-sonnet-latest’,

I did a recursive search through the whole tree and there is no way that it should be requesting to use this model. I am also sanitizing my container and images by deleting docker images and containers each time.

Any ideas on how its still managing to try and pick claude-sonnet?

29

Ok FINALLY got it to work. Here is my topology:
Server hostname=fractal:

  • Linux Ubuntu running Ollama and has various models loaded including Qwen. It’s running a 4070ti Super
  • Docker compose profile:production running Otto
    Client:
  • Macbook, using Chrome browser as a pure client

Changes made:

  • Create and update .env.local with :
    OLLAMA_API_BASE_URL=http://fractal.local.net:11434
  • Update bindings.sh to take file from app/.env.local instead of just .env.local since the docker build DOES NOT seem to include .env.local in the base directory
  • Changed DEFAULT_MODEL=‘qwen2.5-code:7b’; it seems like Bolt seems to somehow ignore this anyway (see my previous comment)
  • Modified the constants.py PROVIDER_LIST with, this effectively hard-codes the model with the Ollama selection:
    name: ‘Ollama’,
    staticModels: [
    {
    name: ‘qwen2.5-coder:7b’,
    label: ‘Qwen’,
    provider: ‘Alibaba’,
    maxTokenAllowed: 8000
    }
    ],

I can confirm from the log files that it was going out to the Ollama and Qwen model installed on Fractal server. Next major issue however is the fact that the pseudo-IDE that its supposed to update does not work and I see this error at the bottome message window:
"Failed to spawn bolt shell

Failed to execute ‘postMessage’ on ‘Worker’: SharedArrayBuffer transfer requires self.crossOriginIsolated."

I can see the results however only in the intermediate window above it.

Somehow my random incantations after a few nights of trying finally somehow partially worked.

2 Likes
30

I think it’s the model it’s trying to use that’s the problem.

31

If you are running bolt.new in side a docker just try to replace localhost by host.docker.internal

Refer here for details networking - How host.docker.internal works on Windows - Super User

Hope this help