@thecodacus I was unaware. Have to look at what you mean. So, it looks like you split up the providers into separate *.ts files within a Providers folder. And while it eliminates the PROVIDER_LIST as it was technically, it’s still kind of the same, just split up logically. I don’t see how that’s dynamic (unless you meant to say “static”). Are these TS files being generated somewhere? It kind of just ended up making more redundant code tbh.
P.S. And being that the models are not alphabetical, it’s clearly unlikely to be generated/dynamic. It’s just one more thing you have to maintain and spend time on updating, and it still won’t be as consistent as a code variant (typos, duplicates, unoptimized, etc.). Not here to rock the boat, but that’s my two cents.
now you just need to write one provider file to add a provider, the model manager will automatically find it from the folder and loads it.
plus you don’t have to go to any other place to update any code
like you showed in the above code to add github provider
you can load dynamic models by writing the function getDynamicModels function from the provider to load dynamic models
and when I am saying dynamic I am talking about runtime dynamic models and not written in the code base by a script as a list of models
I must have just scrolled right past the getDynamicModels code block. I do like that. I’ll need to checkout the PR.
Do you know how many models this returns from HuggingFace by chance? They have a lot and we only want to pull those with the Interface API (serverless) enabled.
And does this “Dynamic” list get compiled on Build?
I need to test. it does not compile on build but on start. because some provider needs an api key to get models list and if users set the apikey from UI then it cannot be done on build has to happen on runtime
Fair enough, my prototype was a hacky solution to get around needing the Keys. HuggingFace makes it easy but most of the rest do not. I’ll test tonight.
compiling it on build makes sense as this can be a huge list. also not sure it will be a goo UX to show a gigantic list in a dropdown. might need a dropdown with a search box to make it more user friendly
Well, it’s not only a size issue. Only the Interface API (serverless) support API calls. The serverless endpoint doesn’t support quantized models either (at least not right now), so those would get filtered out automatically. Stupid thing is there isn’t a straightforward filter for “serverless” API support that I have found. There’s also a lot of redundancy on HuggingFace, like forking on GitHub but as most people won’t setup an endpoint, filtering by “serverless” also resolves that. So, your list will go from ~1,213,033 to < 100.
That’s really an assumption though. Let me pull your PR and test this.
Hello @aliasfox , this looks awesome. Xai, Mistral, Antropic and Together seem all to be working well. Quick questions: can you confirm that Deepinfra goes into Openai_like_API, still working on this one. Also where did you enter the url end point for Github? Do we have to mod the constant file? I just finished setting up everything and things are looking good but I got to tweak github/Deepinfra some more. Finally I wonder if vite is supposed to be bundled in as a dependency as I was under the impression that it would load and execute automatically on new prompt build like in bolt.new so we don’t have to run it manually. Cheers and thanks again.
I think we can call it github Provider, as most people when we say azureAi, they think of azure cloud that has a totally different api and corporate uses that alot.
Yeah, I think the PR should be closed, merge your changes and then do a new one. Their docs and naming are misleading and there was an unused line for Azure already. Should have a meeting when grey areas like this come up. I originally called it GITHUB_API_KEY but they also do a paid service beyond GitHub (using the same endpoint) and their docs call it Azure Models (GitHub). The endpoint is Azure, etc.
I tried adding “azure cloud” but to list down deployed models you need oauth credentials. and I didn’t want to integrate oauth just to have one provider.
with azure cloud ai you can deploy any listed model with your own custom name and version. and when you want to call the api you have to use that exact name. so without dynamically listing the name it cannot be hardcoded into the tool like static models, cuz these names are chosen by the users who deployed the models.
I agree with naming the provider GitHub (at least for now). Microsoft just seems to make it misleading and unintuitive is all, lol. I do like having access to the 4o model for free though, it’s pretty good and they have a pretty generous free tier. Not to mention, if we get on GitHub trending, we could all probably get CoPilot for free. And I don’t like paying for things unnecessarily, lol.
I’m not sure I had it enabled in my testing, so I likely didn’t see it in the model list. And I only added the one’s I could confirm were working. That and it’s been discussed just to name the provider “GitHub”.
And @thecodacus has a PR that may conflict with this, which is restructuring how Providers are added, which should make it easier to add them.
So while I would like this now, I’ll update the PR once the new changes are made. I also did the same for testing Cerebras as well. It’s not bad adding new providers (I’ve been testing a bit), but there are a lot of files to update and room for making mistakes.
Hello all, so as things stand on this Christmas eve, do we simply need to pull the main branch to enable adding github api, multiple GPTLike APIs, Deepinfra etc. or do we still need to pull 836? What about 832 for now? Just asking as I am rebuilding my docker once more. Regarding my previous comment regarding Vite ( the browser server by Starblitz), I’ve read in another thread that deleting and rebuilding nodes should take care of the “preview” not loading reliably (even with Claude). Happy holidays to all!
PS Once things stabilize I’m more than willing participating in documenting the process to add API providers. As I am still learning all that stuff, if I can understanding it enough to explain it, everybody will:)
I probably wouldn’t pull PR’s if I were you, hopefully these things just get sorted out in main/stable. And that is curious about removing the node_modules and I’m not sure anyone recommended to clear their cache, but thinking about it now… it’s likely the issue, especially because of how pnpm works. @thecodacus I bet you that was the issue all along!
But you know, thinking about it… if you were able to say update the OpenAI-Like endpoint and it dynamically loaded “Models” (because the convention is /v1/models), then it should almost work for anything… and maybe say a dropdown of endpoints (so a list with endpoint and say alias). Not a fully thought out concept, but interesting.
