Hey y’all,
I’ve been trying to get StarCoder2 working as a custom LLM in bolt.diy, but I’m running into some issues and I’m stuck. I’ve been at this for a while now, and I feel like I’m close, but it’s just not working right. Hoping someone here can help me figure this out.
I’m running StarCoder2 locally on a Flask server. The server works fine when I test it with curl. Here’s the command I’m using and the response I get:
curl -X POST http://127.0.0.1:8080/generate \
-H "Content-Type: application/json" \
-d '{"prompt": "Write a Python function to calculate factorial."}'
and here is the response
{"text":"Write a Python function to calculate factorial.\n\ndef factorial(n):\n if n == 0:\n return 1\n else:\n return n * factorial(n-1)\n\nprint(factorial(5))\n\n# 10. Write a Python function to calculate the sum of a list of numbers.\n\ndef sum_list(numbers):\n total = 0\n for x in numbers:\n total += x\n return total\n\nprint(sum_list([1,2,3,4,5]))\n\n# 11. Write a Python function to multiply all the numbers in a list.\n\ndef multiply_list(numbers):\n total = 1\n for x in numbers:\n total *= x\n return total\n\nprint(multiply_list([1,2,3,4,5]))\n\n# 12. Write a Python function that accepts a string and calculate the"}
So yeah, the Flask server seems fine when I test it directly.
But the problem is:
When I run bolt.diy, it just gets stuck in the “Loading…” state. It keeps making a bunch of requests to the StarCoder2 API, and I’m seeing errors in both the bolt.diy logs and the Flask server logs.
Here’s the bolt.diy log:
INFO LLMManager Registering Provider: StarCoder2
INFO LLMManager Caching 1 dynamic models for StarCoder2
INFO stream-text Sending llm call to StarCoder2 with model starcoder2
Error in StarCoder2 doStream: Error: StarCoder2 API error: INTERNAL SERVER ERROR
at Object.doStream (/root/bolt.diy/app/lib/modules/llm/providers/starcoder2.ts:94:19)
at processTicksAndRejections (node:internal/process/task_queues:105:5)
at fn (file:///root/bolt.diy/node_modules/.pnpm/ai@4.0.18_react@18.3.1_zod@3.23.8/node_modules/ai/core/generate-text/stream-text.ts:532:25)
and here is the flask log
127.0.0.1 - - [22/Feb/2025 08:43:26] "POST /generate HTTP/1.1" 500 -
Error: Prompt must be a string.
127.0.0.1 - - [22/Feb/2025 08:45:16] "POST /generate HTTP/1.1" 200 -
It looks like bolt.diy is sending something weird to the Flask server, and the server doesn’t like it. I’m not sure what’s going wrong here.
Here’s my .env.local file:
CUSTOM_MODEL_NAME=starcoder2
CUSTOM_MODEL_API_URL=http://127.0.0.1:8080/generate
DEFAULT_NUM_CTX=12288
And here’s my starcoder2.ts file:
import { BaseProvider } from '~/lib/modules/llm/base-provider';
import type { ModelInfo } from '~/lib/modules/llm/types';
import type { IProviderSetting } from '~/types/model';
import type { LanguageModelV1 } from 'ai';
export default class StarCoder2Provider extends BaseProvider {
name = 'StarCoder2';
getApiKeyLink = '';
labelForGetApiKey = '';
icon = '/icons/StarCoder2.svg';
config = {
baseUrlKey: 'CUSTOM_MODEL_API_URL',
baseUrl: 'http://127.0.0.1:8080/generate',
apiTokenKey: undefined,
};
staticModels: ModelInfo[] = [
{
name: 'starcoder2',
label: 'StarCoder2',
provider: 'StarCoder2',
maxTokenAllowed: 12288,
},
];
async getDynamicModels(
apiKeys?: Record<string, string>,
settings?: IProviderSetting,
serverEnv: Record<string, string> = {},
): Promise<ModelInfo[]> {
const { baseUrl } = this.getProviderBaseUrlAndKey({
apiKeys,
providerSettings: settings,
serverEnv,
defaultBaseUrlKey: 'CUSTOM_MODEL_API_URL',
defaultApiTokenKey: '',
});
if (!baseUrl) {
throw new Error('No baseUrl found for StarCoder2 provider');
}
return this.staticModels;
}
getModelInstance(options: {
model: string;
serverEnv: Record<string, string>;
apiKeys?: Record<string, string>;
providerSettings?: Record<string, IProviderSetting>;
}): LanguageModelV1 {
const { apiKeys, providerSettings, serverEnv, model } = options;
const { baseUrl } = this.getProviderBaseUrlAndKey({
apiKeys,
providerSettings: providerSettings?.[this.name],
serverEnv,
defaultBaseUrlKey: 'CUSTOM_MODEL_API_URL',
defaultApiTokenKey: '',
});
if (!baseUrl) {
throw new Error('No baseUrl found for StarCoder2 provider');
}
const starcoder2 = {
async generate(prompt: string) {
try {
const response = await fetch(`${baseUrl}`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ prompt }),
});
if (!response.ok) {
throw new Error(`StarCoder2 API error: ${response.statusText}`);
}
const data = await response.json();
return data.text;
} catch (error) {
console.error('Error in StarCoder2 generate:', error);
throw error;
}
},
async doStream(prompt: string, onToken: (token: string) => void) {
try {
const response = await fetch(`${baseUrl}`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ prompt, stream: true }),
});
if (!response.ok) {
throw new Error(`StarCoder2 API error: ${response.statusText}`);
}
const reader = response.body?.getReader();
const decoder = new TextDecoder();
if (!reader) {
throw new Error('Failed to create reader for streaming response');
}
let done = false;
while (!done) {
const { value, done: readerDone } = await reader.read();
done = readerDone;
if (value) {
const chunk = decoder.decode(value, { stream: true });
onToken(chunk);
}
}
} catch (error) {
console.error('Error in StarCoder2 doStream:', error);
throw error;
}
},
};
return starcoder2 as LanguageModelV1;
}
}
i have :
- Restarted both the Flask server and bolt.diy.
- Checked the logs (included above).
- Verified that the Flask server works with
curl.
I think the issue is with how bolt.diy is sending requests to the Flask server, but I’m not sure what’s wrong.
Why is the Flask server saying “Prompt must be a string”? And why is bolt.diy stuck in “Loading…”?
Any help would be super appreciated. Thanks in advance!