Managing API Costs and Abnormal Traffic in AI SaaS

Three shapes of cost runaway

Shape	Symptom	Real attribution
Bill-level	Month-end bill doubles while DAU is flat	Bot or free-tier farming traffic never shows up as active users
Per-tenant	A handful of accounts eat most of the budget	Free tier uncapped, or quotas counted per request instead of per cost
Per-endpoint	One expensive endpoint dominates spend	Video, high-resolution image and long-context calls cost tens of times more per call

Each shape has a different control point, but one shared root cause: you bill by request count while costs accrue in tokens and compute. Whenever the measurement unit and the cost unit diverge, cost runs away.

Why per-request rate limiting breaks on AI endpoints

Dimension	Conventional web API	AI endpoint
Cost per call	Roughly constant	Up to tens of times apart, driven by context length, resolution, steps
Natural frequency	Bounded by human interaction	Scripts can loop indefinitely; no natural ceiling
Retry amplification	Minor	One retry may re-consume an entire context window

A rule of “60 requests per minute” means 60 cheap queries on a conventional API — and 60 maximum-context inferences on an AI API. Switching the counting unit from requests to cost units is the first and most often missed step in AI cost control.

Five control points

1. Count in cost units. Assign each request class a point value (text: input plus output tokens; image: resolution × steps; video: seconds) and let the limiter count points, not requests. Cloudflare Rate Limiting rules can count on combinations of path, headers and body/JWT claims, and return custom responses — which makes “count by the plan tier inside the claim” possible.

2. Tiered quotas. Run three layers simultaneously: tenant (total budget per customer), endpoint (separate limits for expensive operations) and model (separate quota for premium models). Remove any layer and cost leaks through the gap.

3. Detect anomalies, then slow them down. Rate limiting caps speed; it cannot tell good from bad. Identify first, then degrade: traffic with concentrated origins, no browser environment, mechanical cadence, or that only calls generation endpoints and never reads results should be downgraded to a cheaper model or challenged — not blanket-429ed.

4. Cache and de-duplicate. Identical requests (same parameters for image generation, same prompt for completion) should return cached results — the cheapest optimisation there is. Combined with edge cache rules, popular requests never reach your origin.

5. Bill-level alerting and circuit breaking. Wire per-directory, per-tenant and per-endpoint consumption into alerting, and add a hard breaker: at threshold, automatically downgrade to a cheaper model or move to a queue. Without a breaker, alerting only tells you the money is already spent.

Deriving quota thresholds

Parameter	Derivation	Example
Monthly tenant quota	Plan gross margin ÷ unit cost × acceptable cost share	100 monthly fee, 60% margin, 0.02 unit cost, 30% acceptable → 900 calls
Per-endpoint limit	Scale by how many times costlier than the baseline	Video ≈ 40× text → limit at 1/40
Per-minute rate	Peak human cadence × 1.5	Human peak 20/min → limit 30/min
Circuit-breaker threshold	Alert at 80% of budget, degrade at 100%	5,000 budget → alert at 4,000

The point is that thresholds come from unit economics, not from defaults or guesswork. The same parameter set has a completely different optimum under a different margin structure.

Caveats and where this does not apply

• Long-running async endpoints (video generation, batch inference) should not be limited by per-minute rate. Limit by concurrent queue length instead, or legitimate queued jobs get rejected.
• Partner integrations have cadence dictated by the partner’s system; rate limits misfire. Use total budget plus dedicated alerting instead.
• Free trials must have a hard ceiling below customer acquisition cost, or more growth means more loss.

FAQ

Does returning 429 hurt real users? Yes — which is why 429 is the last resort. The correct order is cache hit → detected-degrade/queue → challenge → reject. Reaching for 429 first means the previous four layers are missing.

How far does the free tier get you? Basic edge rate limiting is available on free plans, but rule counts, counting windows and custom responses usually require paid tiers. The scaling test is whether you can count by business claims (plan, tenant) — not how many rules you get.

How do you separate anomalies from legitimate peaks? Watch three invariants: whether source distribution matches history, whether cadence fits human interaction intervals, and whether the endpoint mix is sensible. During peaks all three hold; during an attack at least one breaks.

We already overspent. What now? Trip the breaker first (downgrade model or queue) to stop the bleeding, then break the bill down by endpoint to locate the source, then add quotas. Reverse the order and you keep burning money while fixing it.