oTToDev with QwQ 32b - Surprising Results

bolt.diy
1

Hey everyone!

Recently, a new open source model developed by the Qwen team was released that has advanced reasoning (chain of thought) like o1 - QwQ:

I was pretty excited to try it out with oTTODev, but it seems like the advanced reasoning (or maybe something else?) actually prevents it from producing the artifacts needed to interact with the web container, so I just end up with a normal LLM chat:

qwqMessingUp
qwqMessingUp1896Ă—939 42.7 KB

And then when I ask it specifically to interact with the webcontainer it refuses:

qwqRefusingToHelp
qwqRefusingToHelp1894Ă—940 29.9 KB

Obviously would be exciting if this model worked well for oTToDev, but I honestly thought it was more interesting it failed so poorly so definitely worth sharing!

At some point, we need to develop a system for a prompt library since it’s becoming more and more obvious that different LLMs need different prompts to work well with oTToDev (or any AI coding assistant). And it seems QwQ is the best example of this because it is a pretty powerful model!

2 Likes
2

Yes, I agree. I tried QwQ 32b a few hours after it launched with oTToDev, but I experienced the same issue as shown in the screenshot above.

1 Like
3

Yeah, absolutely… A common library for system prompts to automatically benchmark models against, etc. I would be interested in assisting with writing that automation. Process improvement and automation is my jam, lol.

As for QwQ, I tested it within hours of release on oTTo…eh bolt.diy and found it impressive, despite not interacting fully with the system. Maybe this might get resolved with an “instruct” release of the model.

1 Like
4

Indeed an instruct version would probably solve a lot of the issues we are seeing with it!