, each containing a left (column identifier), operation (comparison operator), and right (threshold value). The platform's query engine evaluates these conditions server-side using indexed lookups, which is why pushing filters down is critical.

Step 2: Execute and Paginate

The range property defines the window of results to return. Since the endpoint does not support offset-based pagination natively, you must increment the window manually. The response includes a totalCount field, which serves as the termination condition for your loop. Failing to respect this boundary results in redundant requests and potential throttling.

Step 3: Normalize Positional Data

The API returns matched records as an array of objects, where each object contains a d array. The values in d correspond positionally to the columns array you requested. Mapping these indices to named properties prevents brittle code when column order changes or when the platform updates its internal schema.

Here is a production-ready TypeScript implementation that encapsulates these steps:

interface ScanCondition {
  left: string;
  operation: 'egreater' | 'eless' | 'greater' | 'less' | 'in_range';
  right: number | number[] | string[];
}

interface ScanRequest {
  filter: ScanCondition[];
  columns: string[];
  sort: { sortBy: string; sortOrder: 'asc' | 'desc' };
  range: [number, number];
}

interface ScanRecord {
  d: (number | string | null)[];
}

interface ScanResponse {
  totalCount: number;
  data: ScanRecord[];
}

class MarketScannerClient {
  private readonly baseUrl: string;
  private readonly batchSize: number;
  private readonly maxRetries: number;

  constructor(market: string, batchSize = 100, maxRetries = 3) {
    this.baseUrl = `https://scanner.tradingview.com/${market}/scan`;
    this.batchSize = batchSize;
    this.maxRetries = maxRetries;
  }

  async executeScan(
    conditions: ScanCondition[],
    fields: string[],
    sortField: string,
    sortOrder: 'asc' | 'desc' = 'desc'
  ): Promise<Record<string, unknown>[]> {
    const allResults: Record<string, unknown>[] = [];
    let windowStart = 0;
    let totalRecords = Infinity;

    while (windowStart < totalRecords) {
      const payload: ScanRequest = {
        filter: conditions,
        columns: fields,
        sort: { sortBy: sortField, sortOrder },
        range: [windowStart, windowStart + this.batchSize]
      };

      let response: Response | null = null;
      let attempt = 0;

      while (attempt < this.maxRetries) {
        try {
          response = await fetch(this.baseUrl, {
            method: 'POST',
            headers: { 'Content-Type': 'application/json' },
            body: JSON.stringify(payload),
            signal: AbortSignal.timeout(10000)
          });

          if (response.ok) break;
          if (response.status === 429) {
            const retryAfter = parseInt(response.headers.get('Retry-After') || '2', 10);
            await new Promise((res) => setTimeout(res, retryAfter * 1000));
          }
          attempt++;
        } catch (err) {
          attempt++;
          await new Promise((res) => setTimeout(res, 1000 * attempt));
        }
      }

      if (!response || !response.ok) {
        throw new Error(`Scanner request failed after ${this.maxRetries} attempts`);
      }

      const json: ScanResponse = await response.json();
      totalRecords = json.totalCount;

      const normalized = json.data.map((record) => {
        const obj: Record<string, unknown> = {};
        fields.forEach((col, idx) => {
          obj[col] = record.d[idx];
        });
        return obj;
      });

      allResults.push(...normalized);
      windowStart += this.batchSize;
    }

    return allResults;
  }
}

Architecture Decisions & Rationale

• Server-Side Filtering Over Client-Side: Pushing conditions to the filter array ensures the platform's indexing engine handles the heavy lifting. This eliminates network transfer of irrelevant tickers and reduces memory pressure on your runtime. Client-side filtering forces you to download, parse, and discard 90%+ of the dataset.
• Explicit Batch Sizing: The batchSize parameter prevents timeout errors on large datasets. Platforms typically cap single-response payloads to avoid gateway bottlenecks. Incremental fetching also allows graceful degradation if a request fails mid-stream.
• Dynamic Column Mapping: Hardcoding array indices creates maintenance debt. The normalization step iterates over your requested fields and maps them to record.d positions, ensuring the output remains stable even if the platform updates its internal column ordering.
• Market Abstraction: The constructor accepts a market identifier, making the client reusable across equities, cryptocurrencies, and forex without duplicating request logic. This decouples configuration from execution.
• Timeout & Retry Handling: Network calls to public endpoints can experience transient failures. Implementing AbortSignal.timeout and exponential backoff with jitter prevents thread starvation and respects implicit rate limits.

Pitfall Guide

1. Pulling Full Tables for Local Filtering

   • Explanation: Fetching all available tickers and filtering in-memory consumes excessive bandwidth and triggers rate limits. It also increases garbage collection pressure in Node.js or browser environments.
   • Fix: Always express constraints in the filter array. The platform's scanner is optimized for indexed lookups; client-side loops are not.

2. Misusing in_range for Set Membership

   • Explanation: Developers often assume in_range only accepts numeric bounds. It also accepts arrays of strings or numbers for exact set matching, which is critical for filtering by sectors, exchanges, or countries.
   • Fix: Pass an array to right when filtering by discrete categories. Example: { left: "sector", operation: "in_range", right: ["Technology", "Healthcare"] }.

3. Hardcoding Response Array Indices

   • Explanation: The d array in the response is positional. If you assume index 0 is always close, your code breaks when column order changes or when you add/remove requested fields.
   • Fix: Always map columns to d dynamically during normalization. Never rely on implicit ordering.

4. Ignoring Regional Endpoint Variations

   • Explanation: Column identifiers and available metrics differ across markets. market_cap_basic works for equities but may not apply to forex pairs or crypto tokens.
   • Fix: Validate column availability per market. Maintain a configuration map that restricts queries to supported fields for each region.

5. Assuming Real-Time Synchronization

   • Explanation: Public scanner endpoints refresh on platform-defined intervals, not exchange-level tick data. Delays of 15–60 minutes are common for fundamental metrics like P/E or dividend yield.
   • Fix: Treat this endpoint as a screening and discovery tool, not a high-frequency trading feed. Add timestamp validation or cache invalidation strategies if temporal precision matters.

6. Infinite Pagination Loops

   • Explanation: Failing to check totalCount against the current window causes unnecessary requests or infinite loops if the platform returns stale counts or if the dataset shrinks between calls.
   • Fix: Terminate the loop when windowStart >= totalRecords. Add a maximum iteration cap (e.g., 500 batches) as a safety valve against runaway requests.

7. Rate Limit Blindness

   • Explanation: Even unauthenticated endpoints enforce throttling. Rapid sequential requests trigger 429 Too Many Requests or temporary IP blocks. Public endpoints share infrastructure with millions of retail users.
   • Fix: Implement exponential backoff with jitter. Cache results for static screening criteria and respect platform usage norms. Monitor Retry-After headers when available.

Production Bundle

Action Checklist

• Validate column identifiers against your target market before constructing filters
• Implement dynamic column-to-value mapping to prevent index drift
• Set explicit batch sizes between 50–200 to balance latency and payload size
• Add totalCount boundary checks to prevent over-fetching
• Wrap fetch calls in retry logic with exponential backoff for transient failures
• Cache screening results when criteria remain static across runs
• Monitor response times and payload sizes to detect endpoint degradation early

Decision Matrix

Scenario	Recommended Approach	Why	Cost Impact
Rapid prototyping / internal dashboards	Public Scanner API	Zero cost, low latency, covers 90% of screening needs	$0
Compliance-heavy / audited reporting	Paid Data API	SLA guarantees, historical depth, regulatory compliance	$99–$299/mo
High-frequency trading / tick-level data	Institutional Feed	Sub-millisecond latency, direct exchange routing	$5k–$50k+/mo
Legacy platform integration	Web Scraping + Headless Browser	Fallback when APIs are unavailable, but fragile	$0–$50/mo (infra)

Configuration Template

// scanner.config.ts
export const SCANNER_CONFIG = {
  markets: ['america', 'crypto', 'forex', 'india', 'uk'] as const,
  defaultBatchSize: 100,
  maxRetries: 3,
  retryDelayMs: 1000,
  supportedColumns: {
    america: [
      'close', 'change', 'market_cap_basic', 'price_earnings_ttm',
      'dividend_yield_recent', 'RSI', 'Perf.YTD', 'sector', 'industry',
      'exchange', 'country', 'name'
    ],
    crypto: [
      'close', 'change', 'market_cap_basic', 'volume_24h',
      'RSI', 'Perf.YTD', 'name'
    ]
  },
  filterOperations: ['egreater', 'eless', 'greater', 'less', 'in_range'] as const
};

export type Market = typeof SCANNER_CONFIG.markets[number];
export type FilterOp = typeof SCANNER_CONFIG.filterOperations[number];

Quick Start Guide

1. Initialize the client: Import the MarketScannerClient class and instantiate it with your target market (e.g., new MarketScannerClient('america')).
2. Define your filters: Create an array of ScanCondition objects using supported column IDs and operations. Example: { left: "market_cap_basic", operation: "egreater", right: 10000000000 }.
3. Execute the scan: Call executeScan(conditions, columns, sortField) and await the normalized results.
4. Validate output: Log the first few records to confirm column mapping and data types. Adjust batch size or retry logic if latency spikes.
5. Integrate downstream: Pipe the normalized array into your portfolio tracker, alerting system, or database. Add caching if screening criteria remain static.