Remember when OpenAI was the default for basically everything? Yeah, that’s over. For the Winter 2026 YC batch, for the first time ever, Anthropic is the most-used API and there is a major rebalancing of preferences across models.
Remember what made an AI gateway back in the day? Weren’t we were all just trying to survive OpenAI’s rate limits?
This made sense when:
OpenAI was the only serious option
Models were roughly equivalent
The goal was uptime, not optimization
Nobody knew what they were building
None of these are true anymore.
What Changed
By the end of 2025, we have:
Claude that dominates coding (77% SWE-bench)
Gemini that processes 1M tokens (10x anyone else)
GPT that leads mathematical reasoning (100% AIME)
DeepSeek that matches GPT-4 at 1/30th the cost
Specialized models for translation, legal, medical, code review
The question isn’t “which model is up?”
The question is “which model is right?”
Your AI gateway perhaps can’t answer that. It doesn’t even know to ask.
The unengineered gains
If you’re pushing 1M requests a month, the “ Legacy Gateway” approach is a tax on your ignorance. Today, you can optimize for the Efficiency, Costs, and quality of Inference across different models
Here’s an example
By using a small, 5ms “classifier” model at the edge to detect intent, you can:
Boost Accuracy: 20% lift by matching task complexity to model “reasoning” levels.
Slash Burn: 40-60% cost reduction by offloading “commodity” prompts to cheap, fast models.
The new layer emerging above gateways
Gateways solved the 2023 problem brilliantly. Unified APIs, failover, observability, cost tracking. So much so that infrastructure is now table stakes. Portkey, LiteLLM, Helicone, OpenRouter all do this well. But a new layer is emerging above them.
let’s call it Intelligent routing.
Martian is pioneering this
Analyzes each prompt semantically in real-time.
Uses “model mapping” to understand what makes each LLM succeed or fail
Routes to the optimal model per request, not per rule
Their insight: if Model A answers 20% of queries best, Model B another 20%, and Model C another 20% then routing to the right model can outperform any single model alone.
vLLM Semantic Router (open source) shows the architecture
BERT embeddings convert prompts to intent vectors
Task/complexity/tool-aware classification
Mixture-of-Models selection in milliseconds
What To Do Right Now
If you’re building:
Don’t bake model selection into your application logic
Use a gateway for now, but abstract it behind an interface
Build your own intent classification (it’s just a small model + embeddings)
Track which requests go to which models and measure quality
Prepare to swap your gateway for a router when the time comes
If you’re operating:
Measure model-specific accuracy on your actual workload
Calculate cost per successful completion, not cost per request
Identify request categories where you’re using the wrong model
Experiment with routing 10% of traffic to better-matched models
If you’re investing:
Intelligent routing is an infrastructure layer that doesn’t exist yet
Gateways are commodity, the moat is in understanding models
The winner will have the best model capability graph, not the most integrations
This market is early but the signal is clear
The Bottom Line
Your AI gateway routes traffic. It doesn’t understand it.
For the last 3 years, the problem was uptime, and gateways solved uptime. But the problem has changed. We have multiple models that excel at different things. The new challenge is matching requests to capabilities.
There is a clear gap. Intelligent routing fills it.
And finally in some ‘unengineered’ words - The likes of martian, and vLLM should do exceeding well as It is easier to add infrastructure to intelligence than intelligence to infrastructure.