Internal corp.custom llm

martinb113 · December 11, 2024, 8:07pm

Hi there, is there potencial to connect also internal APIs for LLM interaction, where basically bearer auth is needed for interaction? I am sure there must be others who are dealing with similar chalange

thecodacus · December 11, 2024, 8:44pm

it its accessible from the browser, it should be, are you facing any issue ?

martinb113 · December 11, 2024, 10:14pm

I just updated my message, I need to be more verbouse i guess, I was curiouse if there is way how to connect LLM which is internal for companies whith all sorts of other things, no openAI like apis nor ollama server exc, just short list of links, auth and so on…

wonderwhy.er · December 12, 2024, 9:18am

I would say that its not super well supported yet.
We have so called “open ai like” provider where you can give urls, keys etc, and it will treat it as OpenAI API but on different server

I think we will get to having super configurable provider that can be used with whatever.
For now you will need to thinker a bit with openai like
I know people used it to add support for providers we do not have yet

martinb113 · December 14, 2024, 3:19pm

Open ai like would be fine, just to know how to create adapter for that

wonderwhy.er · December 14, 2024, 5:17pm

Well take a look here

github.com

stackblitz-labs/bolt.diy/blob/main/app/lib/.server/llm/model.ts#L68


      
          }
          
          export function getGoogleModel(apiKey: OptionalApiKey, model: string) {
            const google = createGoogleGenerativeAI({
              apiKey,
            });
          
            return google(model);
          }
          
          export function getGroqModel(apiKey: OptionalApiKey, model: string) {
            const openai = createOpenAI({
              baseURL: 'https://api.groq.com/openai/v1',
              apiKey,
            });
          
            return openai(model);
          }
          
          export function getHuggingFaceModel(apiKey: OptionalApiKey, model: string) {
            const openai = createOpenAI({

Groq is added with OpenAI client but url is provided to Groq server.
If you have other provider that has OpenAI like API then you can use same approuch.

You can also see that we use same approuch for HuggingFace, Deepseek, LM Studio, xAI.