Rate Limiting for Lovable Apps: How to Stop Surprise OpenAI Bills

emory counters reset on cold starts and fragment across concurrent instances. A distributed key-value store is mandatory for consistent window tracking. Upstash Redis provides a serverless-native REST API with a free tier that comfortably handles standard application traffic.

Step 2: Build the Cost Guard Module

Instead of scattering rate-limit logic across route handlers, encapsulate it in a dedicated service. This improves testability and ensures consistent enforcement across all AI endpoints.

// lib/ai-cost-guard.ts
import { Ratelimit } from "@upstash/ratelimit";
import { Redis } from "@upstash/redis";

export type GuardResult = {
  allowed: boolean;
  headers: Record<string, string>;
  error?: string;
};

export class AICostGuard {
  private ipThrottle: Ratelimit;
  private userQuota: Ratelimit;
  private globalCap: Ratelimit;
  private redis: Redis;

  constructor() {
    this.redis = Redis.fromEnv();

    this.ipThrottle = new Ratelimit({
      redis: this.redis,
      limiter: Ratelimit.slidingWindow(8, "60 s"),
      prefix: "rl:ip",
    });

    this.userQuota = new Ratelimit({
      redis: this.redis,
      limiter: Ratelimit.fixedWindow(150, "1 d"),
      prefix: "rl:usr",
    });

    this.globalCap = new Ratelimit({
      redis: this.redis,
      limiter: Ratelimit.fixedWindow(5000, "1 d"),
      prefix: "rl:global",
    });
  }

  async evaluate(
    clientIp: string,
    userId?: string
  ): Promise<GuardResult> {
    const ipCheck = await this.ipThrottle.limit(clientIp);
    if (!ipCheck.success) {
      return {
        allowed: false,
        headers: this.formatHeaders(ipCheck),
        error: "IP throttle exceeded",
      };
    }

    if (userId) {
      const userCheck = await this.userQuota.limit(`user:${userId}`);
      if (!userCheck.success) {
        return {
          allowed: false,
          headers: this.formatHeaders(userCheck),
          error: "Daily user quota exhausted",
        };
      }
    }

    const globalCheck = await this.globalCap.limit("budget");
    if (!globalCheck.success) {
      return {
        allowed: false,
        headers: { "Retry-After": "3600" },
        error: "Global daily budget reached",
      };
    }

    return {
      allowed: true,
      headers: this.formatHeaders(ipCheck),
    };
  }

  private formatHeaders(check: {
    limit: number;
    remaining: number;
    reset: number;
  }): Record<string, string> {
    return {
      "X-RateLimit-Limit": String(check.limit),
      "X-RateLimit-Remaining": String(check.remaining),
      "X-RateLimit-Reset": String(check.reset),
    };
  }
}

Step 3: Integrate with API Route

The guard must execute before any provider SDK initialization. Input validation and token capping happen in the same pre-flight phase.

// app/api/chat/route.ts
import { NextRequest, NextResponse } from "next/server";
import OpenAI from "openai";
import { AICostGuard } from "@/lib/ai-cost-guard";

const guard = new AICostGuard();
const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });

export async function POST(req: NextRequest) {
  const forwarded = req.headers.get("x-forwarded-for") ?? "";
  const clientIp = forwarded.split(",")[0]?.trim() ?? "0.0.0.0";
  const userId = req.headers.get("x-user-id") ?? undefined;

  const assessment = await guard.evaluate(clientIp, userId);
  if (!assessment.allowed) {
    return new NextResponse(
      JSON.stringify({ error: assessment.error }),
      {
        status: 429,
        headers: {
          ...assessment.headers,
          "Content-Type": "application/json",
        },
      }
    );
  }

  const body = await req.json();
  const prompt = body.message as string;

  if (!prompt || prompt.length > 3000) {
    return NextResponse.json(
      { error: "Payload exceeds maximum character threshold" },
      { status: 400 }
    );
  }

  const completion = await openai.chat.completions.create({
    model: "gpt-4",
    messages: [{ role: "user", content: prompt }],
    max_tokens: 1024,
    temperature: 0.7,
  });

  return NextResponse.json({
    response: completion.choices[0].message.content,
  });
}

Architecture Rationale

• Sliding Window for IP: Prevents burst abuse while allowing natural usage patterns. A fixed window would create hard cutoffs that frustrate legitimate users during peak hours.
• Fixed Window for User/Global: Budget enforcement requires predictable daily ceilings. Fixed windows align with accounting cycles and simplify cost forecasting.
• Pre-Flight Evaluation: The guard runs before openai.chat.completions.create. This eliminates race conditions where concurrent requests bypass accounting during network latency.
• Explicit Token Capping: max_tokens: 1024 bounds response cost. Combined with the prompt.length check, the maximum cost per request becomes mathematically deterministic.

Pitfall Guide

1. Relying on req.connection.remoteAddress in Serverless

Explanation: Cloud platforms route traffic through internal load balancers. The connection address resolves to the platform's proxy IP, not the end user. Throttling by this value blocks all traffic or none. Fix: Parse x-forwarded-for and extract the first comma-separated value. Validate the header exists before splitting to prevent undefined errors.

2. Using In-Memory Counters on Stateless Runtimes

Explanation: JavaScript objects reset on cold starts. Concurrent function instances maintain separate counters, allowing attackers to bypass limits by triggering multiple instances. Fix: Always use an external distributed store. Upstash Redis, Vercel KV, or Cloudflare D1 provide consistent state across invocations.

3. Returning 200 OK with Error Payloads

Explanation: HTTP 200 signals success. Caching layers, monitoring tools, and client SDKs ignore error messages buried in successful responses. This breaks standard retry logic and obscures metrics. Fix: Return strict 429 status codes for throttling violations. Include Retry-After or X-RateLimit-Reset headers to guide client backoff strategies.

4. Accounting After the Provider Call

Explanation: Updating counters post-response creates a race condition. During network latency, multiple requests pass the guard simultaneously, all trigger paid API calls, and only then decrement the budget. Fix: Evaluate limits synchronously before SDK initialization. Reserve the token budget atomically, then execute the call.

5. Ignoring Streaming Disconnects

Explanation: When a client closes a streaming connection, the server-side OpenAI request continues until completion. You pay for tokens the user never received. Fix: Propagate AbortController signals. Listen for req.signal.aborted and cancel the upstream SDK call immediately.

6. Single-Window Throttling

Explanation: One limit cannot handle both burst protection and daily budgeting. A tight window blocks legitimate power users; a loose window allows financial bleed. Fix: Implement tiered limits. IP-level handles anonymous noise, user-level handles authenticated abuse, global-level enforces hard budget ceilings.

7. Missing Response Token Boundaries

Explanation: Omitting max_tokens allows the model to generate until context limits. A single request can consume 4,000+ output tokens, multiplying costs unpredictably. Fix: Always specify max_tokens. Align the value with your UI constraints and cost model. Use streaming with client-side cancellation for long-form generation.

Production Bundle

Action Checklist

• Audit all AI endpoints for missing throttle logic, auth checks, and token caps
• Provision a distributed KV store (Upstash Redis, Vercel KV, or equivalent)
• Implement sliding window limits for anonymous IP traffic
• Add fixed window quotas for authenticated user accounts
• Configure a global daily spend cap with 503 fallback behavior
• Enforce prompt.length validation and max_tokens on every SDK call
• Return strict 429/503 status codes with Retry-After headers
• Wire provider dashboard alerts for daily usage thresholds
• Test streaming endpoints with AbortController propagation
• Run load simulation against staging to verify limit enforcement

Decision Matrix

Scenario	Recommended Approach	Why	Cost Impact
Prototype / Internal Tool	Single IP sliding window (10 req/min)	Fastest implementation, prevents accidental loops	Low
Public SaaS (Free Tier)	IP throttle + per-user daily quota (50 req/day)	Balances UX with budget predictability	Medium
Enterprise / High-Traffic	Layered architecture + global cap + streaming aborts	Deterministic cost boundaries, abuse-resistant	High (infrastructure)
Budget-Constrained Startup	Global cap first, then IP throttle	Prevents catastrophic bills, sacrifices some UX	Low

Configuration Template

# .env.local
UPSTASH_REDIS_REST_URL=https://your-region.upstash.io
UPSTASH_REDIS_REST_TOKEN=your-rest-token
OPENAI_API_KEY=sk-proj-xxxxxxxxxxxx

# Optional: Provider alerting webhook
ALERT_WEBHOOK_URL=https://hooks.slack.com/services/xxxxx
DAILY_BUDGET_CAP=50.00

// lib/ratelimit-config.ts
export const THROTTLE_PROFILES = {
  anonymous: { window: "60 s", max: 8 },
  authenticated: { window: "1 d", max: 150 },
  global: { window: "1 d", max: 5000 },
} as const;

export const TOKEN_BOUNDS = {
  maxPromptChars: 3000,
  maxResponseTokens: 1024,
  temperature: 0.7,
} as const;

Quick Start Guide

1. Initialize Redis: Create a free Upstash database in the same region as your deployment. Copy the REST URL and token into your environment variables.
2. Install Dependencies: Run npm install @upstash/ratelimit @upstash/redis openai.
3. Deploy Guard Module: Copy the AICostGuard class into your project. Adjust window sizes and token bounds to match your budget model.
4. Wrap Endpoints: Replace direct SDK calls with the guard evaluation pattern. Ensure 429/503 responses include proper headers.
5. Verify Enforcement: Execute a rapid request loop against your staging endpoint. Confirm that requests transition from 200 to 429 after the configured threshold, and that provider dashboards show bounded usage.