Decoupling Market Data Collection from Frontend Delivery: A Production-Grade Pattern

Current Situation Analysis

Derivatives traders, quantitative researchers, and crypto infrastructure developers consistently face a fragmented data landscape. Public dashboards either lack historical depth, fail to normalize cross-exchange metrics, or rely on fragile live API proxies that throttle under load. The core issue isn't data availability; it's an architectural mismatch between how market data is consumed and how it's delivered.

This problem is frequently misunderstood because developers treat funding rates like standard REST resources. They build real-time fetchers that query exchange endpoints on every page load or render request. This approach ignores two critical realities: funding rates are discrete, scheduled events (typically settling every eight hours), and frontend rendering has zero business need for sub-second latency on this specific metric. Coupling ingestion directly to presentation creates unnecessary database load, exhausts exchange API quotas, and introduces single points of failure.

The mathematical handling of these rates is equally problematic. Most public tools apply linear scaling to interval rates, which introduces compounding errors that distort arbitrage signals. A raw eight-hour rate of 0.01% compounds to approximately 11.6% annually. Linear projection yields only 10.95%. While the difference appears marginal, it scales non-linearly across higher rates and directly impacts backtesting accuracy, capital efficiency calculations, and cross-exchange spread analysis. When comparing a 0.03% interval against a 0.05% interval, linear math suggests a 0.02% spread, while compound calculation reveals a materially larger divergence. Ignoring this mathematical reality leads to flawed strategy parameters.

WOW Moment: Key Findings

Decoupling data ingestion from frontend delivery fundamentally changes the operational profile of a market data application. By shifting from live API proxying to scheduled static snapshots, you eliminate runtime database queries, bypass exchange rate limits, and achieve global CDN distribution. The performance and cost implications are immediate and measurable.

Delivery Approach	Average Latency	Backend Compute Load	API Quota Consumption	Hosting Cost Model
Live API Proxy	200–800ms	High (per-request)	Exhausts daily limits	Scales with traffic
Scheduled Snapshots	<50ms (CDN)	Zero (build-time)	Fixed (ingestion only)	Flat/free tier

This finding matters because it transforms a rate-limited, latency-sensitive system into a globally distributed, zero-maintenance asset pipeline. The frontend no longer cares about database connections, connection pooling, or API keys. It simply consumes pre-validated JSON files served from edge locations. The ingestion layer operates independently, ensuring data integrity without impacting user experience. This separation of concerns is the foundation of resilient financial data architecture.

Core Solution

Building this pipeline requires separating three distinct concerns: time-series storage, scheduled ingestion, and static delivery. The following implementation uses TypeScript for type-safe financial calculations, TimescaleDB for efficient time-series queries, and a static site generator for edge delivery.

Step 1: Time-Series Schema Design

Standard relational tables degrade quickly under rolling window queries and period-over-period comparisons. TimescaleDB hypertables partition data automatically by time, enabling sub-millisecond aggregations without manual partition management.

// db/schema.ts
import { sql } from 'kysely';

export const createFundingHypertable = sql`
  CREATE TABLE IF NOT EXISTS funding_intervals (
    asset_symbol TEXT NOT NULL,
    exchange_id TEXT NOT NULL,
    raw_interval_rate NUMERIC(10, 8) NOT NULL,
    annualized_rate NUMERIC(10, 6) NOT NULL,
    settlement_timestamp TIMESTAMPTZ NOT NULL,
    created_at TIMESTAMPTZ DEFAULT NOW()
  );

  SELECT create_hypertable('funding_intervals', 'settlement_timestamp');
  CREATE INDEX idx_funding_asset_time ON funding_intervals (asset_symbol, settlement_timestamp DESC);
`;

Step 2: The Collector Service

The collector runs on a fixed interval. It fetches raw rates, applies compound annualization, and upserts results. Using TypeScript ensures strict type checking across exchange payloads and prevents silent type coercion errors that corrupt financial calculations.

// services/market-collector.ts
import { createClient } from '@supabase/supabase-js';

const EXCHANGES = ['bybit', 'binance'] as const;
const ASSETS = ['BTCUSDT', 'ETHUSDT', 'SOLUSDT'] as const;

interface RawRatePayload {
  symbol: string;
  fundingRate: string;
  fundingTime: number;
}

export async function ingestFundingData() {
  const db = createClient(process.env.DB_URL!, process.env.DB_KEY!);

  for (const exchange of EXCHANGES) {
    for (const asset of ASSETS) {
      const response = await fetch(`https://api.${exchange}.com/v5/market/funding-rate?symbol=${asset}`);
      const payload: RawRatePayload = await response.json();

      const rawRate = parseFloat(payload.fundingRate);
      // Compound annualization: 3 settlements/day * 365 days
      const annualized = (Math.pow(1 + rawRate, 3 * 365) - 1) * 100;

      await db.from('funding_intervals').upsert({
        asset_symbol: asset,
        exchange_id: exchange,
        raw_interval_rate: rawRate,
        annualized_rate: annualized,
        settlement_timestamp: new Date(payload.fundingTime).toISOString()
      }, {
        onConflict: 'asset_symbol, exchange_id, settlement_timestamp'
      });
    }
  }
}

Step 3: Snapshot Generation & Static Build

Instead of querying the database at runtime, a scheduled job extracts the latest state, calculates cross-exchange spreads, and writes JSON files. The static site reads these during its build phase, guaranteeing zero runtime database hits.

// scripts/generate-snapshots.ts
import { writeFileSync } from 'fs';
import { join } from 'path';

interface SnapshotEntry {
  asset: string;
  exchanges: Record<string, { raw: number; annualized: number }>;
  maxSpread: number;
  generatedAt: string;
}

export async function buildMarketSnapshots() {
  const db = createClient(process.env.DB_URL!, process.env.DB_KEY!);
  const { data: latestRates } = await db
    .from('funding_intervals')
    .select('asset_symbol, exchange_id, raw_interval_rate, annualized_rate')
    .order('settlement_timestamp', { ascending: false })
    .limit(100);

  const snapshots: SnapshotEntry[] = [];
  const assetMap = new Map<string, SnapshotEntry>();

  latestRates?.forEach(row => {
    if (!assetMap.has(row.asset_symbol)) {
      assetMap.set(row.asset_symbol, {
        asset: row.asset_symbol,
        exchanges: {},
        maxSpread: 0,
        generatedAt: new Date().toISOString()
      });
    }
    const entry = assetMap.get(row.asset_symbol)!;
    entry.exchanges[row.exchange_id] = {
      raw: Number(row.raw_interval_rate),
      annualized: Number(row.annualized_rate)
    };
  });

  // Calculate cross-exchange spreads
  assetMap.forEach(entry => {
    const rates = Object.values(entry.exchanges).map(e => e.annualized);
    entry.maxSpread = Math.max(...rates) - Math.min(...rates);
    snapshots.push(entry);
  });

  const outputDir = join(process.cwd(), 'public', 'data');
  writeFileSync(join(outputDir, 'funding-snapshots.json'), JSON.stringify(snapshots, null, 2));
}

Step 4: Static Site Integration

The frontend framework loads the JSON at build time. No runtime database calls occur. This eliminates connection pooling overhead, query planning latency, and database scaling concerns.

// src/pages/market-data.astro.ts
import fundingData from '../../public/data/funding-snapshots.json';

export async function getStaticPaths() {
  return fundingData.map(asset => ({
    params: { symbol: asset.asset },
    props: { data: asset }
  }));
}

// Template renders directly from props.data
// No async data fetching occurs during user navigation

Architecture Rationale

• TimescaleDB over vanilla PostgreSQL: Rolling averages and period-over-period comparisons require time-partitioned indexes. Hypertables handle this natively, reducing query complexity and improving execution plans.
• Scheduled ingestion over real-time polling: Funding rates are discrete events. Polling every eight hours aligns with settlement cycles, eliminating wasted API calls and reducing infrastructure costs.
• Static JSON over runtime queries: Frontend rendering and data collection have opposing reliability requirements. Decoupling them ensures that traffic spikes never impact database connections or API quotas. Build-time data loading guarantees consistent performance regardless of user load.

Pitfall Guide

1. Linear Annualization Fallacy

• Explanation: Multiplying the interval rate by settlements-per-day and days-per-year ignores compounding. This underestimates yields and distorts spread analysis, leading to incorrect arbitrage thresholds.
• Fix: Always apply geometric compounding: (1 + rate)^(settlements * 365) - 1. Validate calculations against known benchmarks before deploying to production.

2. Polling Real-Time Events via REST

• Explanation: Using scheduled REST calls for liquidation events or order book depth creates stale data and excessive latency. These metrics require persistent connections and cannot be accurately captured through polling.
• Fix: Reserve REST polling for discrete, scheduled metrics (funding rates, open interest). Use WebSockets or gRPC streams for tick-level or event-driven data. Maintain separate ingestion pipelines for batch vs. streaming data.

3. Ignoring Timezone Normalization

• Explanation: Exchange APIs return timestamps in varying formats (Unix milliseconds, ISO strings, local time). Storing them without normalization breaks chronological sorting, rolling window calculations, and cross-exchange alignment.
• Fix: Convert all incoming timestamps to UTC at ingestion. Store as TIMESTAMPTZ and enforce strict parsing rules in the collector. Never rely on client-side timezone conversion for analytical accuracy.

4. Coupling Ingestion to Rendering

• Explanation: Building frontend components that directly query the database or proxy exchange APIs creates tight coupling. Database migrations, schema changes, or API deprecations immediately break the UI.
• Fix: Treat the database as an internal artifact. Expose only versioned, pre-processed JSON or GraphQL schemas to the frontend. Maintain strict contract testing between ingestion and presentation layers.

5. Missing Idempotency in Collectors

• Explanation: Network retries, cron overlaps, or failed transactions can cause duplicate inserts. Without conflict resolution, aggregations double-count intervals, corrupting annualized calculations and spread metrics.
• Fix: Use UPSERT or INSERT ... ON CONFLICT with a composite unique key (asset, exchange, timestamp). Log duplicate attempts for monitoring and implement exponential backoff for API retries.

6. Overlooking Hypertable Compression

• Explanation: Time-series data grows linearly. Without compression, query performance degrades and storage costs spike after 90 days. Uncompressed hypertables also increase backup sizes and replication lag.
• Fix: Enable TimescaleDB compression on intervals older than 30 days. Set a retention policy to drop raw data beyond your analytical window, keeping only aggregated rollups. Monitor compression ratios and adjust chunk intervals accordingly.

7. Neglecting Data Validation at the Boundary

• Explanation: Exchange APIs occasionally return malformed payloads, zero rates during maintenance, or negative values during extreme market conditions. Blindly storing these values corrupts downstream calculations.
• Fix: Implement strict schema validation at the ingestion boundary. Reject or flag anomalous rates, log validation failures, and implement circuit breakers to pause ingestion during exchange outages.

Production Bundle

Action Checklist

• Define settlement cadence: Align collector frequency with exchange update cycles (e.g., 8h for funding rates).
• Implement compound annualization: Replace linear math with geometric compounding in all calculation modules.
• Provision hypertables: Initialize TimescaleDB extensions and partition tables by timestamp before ingesting data.
• Enforce UTC normalization: Parse all exchange timestamps to UTC at the ingestion boundary.
• Add idempotency keys: Configure database upserts using composite unique constraints to prevent duplicate intervals.
• Decouple frontend delivery: Generate static JSON snapshots and trigger builds instead of querying databases at runtime.
• Monitor ingestion health: Set up alerts for failed API calls, missing intervals, or schema drift.
• Validate data boundaries: Implement schema checks and anomaly detection before writing to the hypertable.

Decision Matrix

Scenario	Recommended Approach	Why	Cost Impact
Public analytics dashboard	Scheduled static snapshots	Eliminates runtime DB load, leverages CDN caching, handles traffic spikes gracefully	Near-zero hosting cost
Internal quant research	Direct TimescaleDB queries	Enables complex rolling windows, custom aggregations, and ad-hoc analysis	Higher compute, but justified by analytical depth
Real-time trading signals	WebSocket streams + in-memory cache	REST polling introduces unacceptable latency for execution systems	Requires dedicated infrastructure, higher operational cost
Historical backtesting	Compressed hypertables + rollups	Balances storage efficiency with query speed for multi-year datasets	Reduces storage costs by 60-80%

Configuration Template

Ready-to-deploy collector cron configuration and database setup script.

# docker-compose.yml
version: '3.8'
services:
  timescaledb:
    image: timescale/timescaledb:latest-pg15
    environment:
      POSTGRES_DB: market_data
      POSTGRES_USER: collector
      POSTGRES_PASSWORD: ${DB_PASSWORD}
    ports:
      - "5432:5432"
    volumes:
      - ts_data:/var/lib/postgresql/data

  collector:
    build: ./collector
    environment:
      DB_URL: postgresql://collector:${DB_PASSWORD}@timescaledb:5432/market_data
      CRON_SCHEDULE: "0 */8 * * *" # Every 8 hours
    depends_on:
      - timescaledb

volumes:
  ts_data:

// collector/cron-runner.ts
import { schedule } from 'node-cron';
import { ingestFundingData } from './services/market-collector';
import { buildMarketSnapshots } from './scripts/generate-snapshots';

// Ingest raw data
schedule('0 */8 * * *', async () => {
  console.log(`[${new Date().toISOString()}] Starting funding ingestion...`);
  await ingestFundingData();
});

// Generate static snapshots 30 mins after ingestion
schedule('30 */8 * * *', async () => {
  console.log(`[${new Date().toISOString()}] Building market snapshots...`);
  await buildMarketSnapshots();
  // Trigger CI/CD webhook or static build command here
});

Quick Start Guide

1. Initialize the database: Run docker compose up -d timescaledb and execute the hypertable creation script against the local instance.
2. Configure environment variables: Set DB_URL, DB_KEY, and exchange API base URLs in your collector service.
3. Run the ingestion cycle: Execute node cron-runner.ts manually to verify data flow, then enable the cron schedule.
4. Generate snapshots: Trigger the snapshot builder to populate the public/data/ directory with validated JSON files.
5. Deploy the frontend: Push to your static hosting provider. The build process will automatically consume the JSON files and render the dashboard.