Hugging Face will only work with Models/Providers that have Interface API (serverless) enabled and not all of them are on that list. If you want to use models you download and run locally, use Ollama or LMStudio, they have a Phi-3.5 Model in both 3.8B and 14B parameters. And they take care of the technical bit (downloading, setting up the model card, defaulting to 4bit quantization, etc.) for you.
There are models not on the Ollama Index worth mention, like QwQ-LCoT-7B-Instruct, but that’s a little more involved to get setup. And as a note, I plan on adding the QwQ models to the LLama index shortly, to make them easier to suggest testing for people.
@aliasfox As my backend is functional I think this issue will be fixed if my HuggingFace downloaded model is shown in the list. ( Frontend )
I want to add this for fixing the issue:
{
“name”: “microsoft/phi-2”,
“provider”: “HuggingFace”,
“description”: “Microsoft’s Phi-2 Model”
}
But I’m not sure where I should add it in the project directory so that it can show in the list. Please guide me
Just use Ollama or LMStudio and read their docs. No reason to re-invent the wheel or manually install models they have available. They take care of all of this for you.
Yes, you are. You are over-complicating things, and making assumptions, for honestly no reason. You can download models that are not supported and get them working, but I would not recommend this. It’s more complicated and there are very few reasons for doing this.
And Phi-3 should perform better with the same system requirements. Just because it’s newer doesn’t mean anything, it’s all about Parameters and Quantization.
@aliasfox It doesn’t matter if we are using it in a standard way or adding it manually. It all comes down to which method will be more suitable for the system & based on my understanding of my system I prefer the harder way considering the system processor & specs, this is my biggest concern.
Please help me with this
Which file should I add the billow content so that my frontend dropdown shows the model
I want to be helpful, but I can’t in this case, because I don’t see value to the community in trying to replicate what you are doing. Sorry.
It is perfectly fine to do it however you want but then you have to do the research and figure it out. Watch some videos on setting things up and try some things out. We’ve all been there; that’s how you get knowledgeable.
I made my suggestion, so that’s where I will leave it. I just feel that you are making things harder for yourself then they need to be and the arbitrary constraint of system specs being your bottleneck, I believe, is simply not true. And even if it was, that doesn’t explain why not to do things the standard and documented way.
I’m all for it though; pave your own path by all means. It just might be a lonely one. But then again you do "prefer the harder way".