Context length of only 4096 tokens

bolt.diy [bolt.diy] Issues and Troubleshooting
1

I’m getting the following error when running Bolt with LM Studio:

ERROR api.chat AI_TypeValidationError: Type validation failed: Value: “Trying to keep the first 6844 tokens when context the overflows. However, the model is loaded with context length of only 4096 tokens, which is not enough. Try to load the model with a larger context length, or provide a shorter input”.

I have set the context length in the .env file to DEFAULT_NUM_CTX=24576 but it doesn’t seem to be picking it up.

Can someone please explain how I can resolve this?

2

Not 100% sure at the moment but I think the default_num_ctx just works with ollama not LM Studio.
Cant you set the context size also in LM Studio directly?

3

I found the problem, LM Studio has default context size of 4096, you need to go into the model setting and adjust this to what ever you want. I set mine to 24 576 and everything is now working. (this is all in the GUI)

1 Like