Wow, that explains a lot. I was using OpenClaw with open router, but stopped because how inconsistent the results were over the same prompt. I thought OC is at fault, never would have guessed that there are differences between providers.
could not agree more one pet peeve of mine is that using strict json output also does not work on all the endpoints of the models so I have to go through them one by one try it out and then only whitelist them.
I've had opencode go + opencode work reliably, though I'm skeptical of how robust their data security claims are in practice because they suddenly blocked accessing Deepseek unless you were okay with the data going to China where true data privacy for something like that is illegal, which makes me wonder where it went before, which weakens my trust. It's also a lot less useful now that Deepseek is so much more expensive.
It's good to quantify the extent to which a lot of the stuff on there is just vibe-hosted. But I think it was always pretty apparent that this was the case. You've got numerous providers all running the gamut of:
I noticed some strange behavior when I was setting it up to avoid providers that collect data, hence blocking default providers. Definitely it is worth filtering well tested providers, which is a feature OpenRouter provides.
great post, I just put $100 in credits on open router to try different models, I think maybe using the "real" provider is the way forward once I've spent my credits and decided which one I want (for now).
Wow, that explains a lot. I was using OpenClaw with open router, but stopped because how inconsistent the results were over the same prompt. I thought OC is at fault, never would have guessed that there are differences between providers.
could not agree more one pet peeve of mine is that using strict json output also does not work on all the endpoints of the models so I have to go through them one by one try it out and then only whitelist them.
That looks absolutely horrifying. What are the alternatives ??
I've had opencode go + opencode work reliably, though I'm skeptical of how robust their data security claims are in practice because they suddenly blocked accessing Deepseek unless you were okay with the data going to China where true data privacy for something like that is illegal, which makes me wonder where it went before, which weakens my trust. It's also a lot less useful now that Deepseek is so much more expensive.
Excellent post, thanks for the article. This is very relevant for anyone using openrouter and similar services seriously.
I feel there is still a lot of progress to be made before we can really trust LLM providers.
It's good to quantify the extent to which a lot of the stuff on there is just vibe-hosted. But I think it was always pretty apparent that this was the case. You've got numerous providers all running the gamut of:
* Hardware availability
* Competency
* Scruples
I noticed some strange behavior when I was setting it up to avoid providers that collect data, hence blocking default providers. Definitely it is worth filtering well tested providers, which is a feature OpenRouter provides.
great post, I just put $100 in credits on open router to try different models, I think maybe using the "real" provider is the way forward once I've spent my credits and decided which one I want (for now).
Wish this had more insights on 'cost' - I found OpenRouter credits to burn up fast.
Good job
Yes