cessing.

Architecture Decisions

1. XML Parser Selection: fast-xml-parser is preferred over DOM-based parsers like Cheerio for this use case. It offers faster parsing speeds and lower memory overhead, which is critical when processing multiple feeds in a polling loop.
2. Locale Handling: Google News requires three parameters to control regional and language results: hl (language), gl (country), and ceid (country:language). The ceid parameter is the most critical for accurate localization and must be constructed dynamically.
3. Temporal Filtering: The when: operator is appended to the query string. Supported values include 1h, 1d, 7d, etc. This reduces the result set to relevant items and avoids processing stale data.
4. Title Normalization: Google News appends the publisher name to the title field (e.g., Headline - Publisher). The ingestion layer must strip this suffix to provide clean headlines, while preserving the publisher data from the <source> tag.

Implementation

import { XMLParser } from 'fast-xml-parser';

export interface NewsQuery {
  topic: string;
  language: string;
  region: string;
  recencyWindow?: string; // e.g., '1d', '7d', '1h'
}

export interface Article {
  headline: string;
  url: string;
  publisher: string;
  publishedAt: string;
}

export class GoogleNewsIngestor {
  private parser: XMLParser;
  private readonly BASE_URL = 'https://news.google.com/rss/search';

  constructor() {
    this.parser = new XMLParser({
      ignoreAttributes: false,
      trimValues: true,
    });
  }

  /**
   * Fetches and normalizes articles from Google News RSS.
   */
  async fetchArticles(query: NewsQuery): Promise<Article[]> {
    const feedUrl = this.constructFeedUrl(query);
    
    const response = await fetch(feedUrl);
    if (!response.ok) {
      throw new Error(`Feed request failed: HTTP ${response.status}`);
    }

    const xmlPayload = await response.text();
    const parsedData = this.parser.parse(xmlPayload);

    // RSS structure: rss -> channel -> item (array or single object)
    const channel = parsedData?.rss?.channel;
    if (!channel) {
      return [];
    }

    const rawItems = channel.item;
    if (!rawItems) {
      return [];
    }

    // Normalize single item vs array
    const items = Array.isArray(rawItems) ? rawItems : [rawItems];

    return items.map(item => this.normalizeItem(item));
  }

  private constructFeedUrl(query: NewsQuery): string {
    const params = new URLSearchParams();
    
    // Build query with optional recency operator
    const qValue = query.recencyWindow 
      ? `${query.topic} when:${query.recencyWindow}` 
      : query.topic;
    
    params.set('q', qValue);
    params.set('hl', query.language);
    params.set('gl', query.region);
    params.set('ceid', `${query.region}:${query.language}`);

    return `${this.BASE_URL}?${params.toString()}`;
  }

  private normalizeItem(item: any): Article {
    const rawTitle = item.title || '';
    const publisher = item.source || '';
    
    // Strip suffix "- Publisher" if present
    let headline = rawTitle;
    if (publisher && rawTitle.endsWith(` - ${publisher}`)) {
      headline = rawTitle.slice(0, -(publisher.length + 3));
    }

    return {
      headline: headline.trim(),
      url: (item.link || '').trim(),
      publisher: publisher.trim(),
      publishedAt: (item.pubDate || '').trim(),
    };
  }
}

Usage Example

const ingestor = new GoogleNewsIngestor();

const articles = await ingestor.fetchArticles({
  topic: 'Tesla',
  language: 'en',
  region: 'US',
  recencyWindow: '1d',
});

console.log(`Retrieved ${articles.length} articles.`);
articles.forEach(a => {
  console.log(`[${a.publisher}] ${a.headline}`);
});

Rationale: The constructFeedUrl method encapsulates the ceid logic, ensuring the country and language codes are synchronized. The normalizeItem method safely handles the title suffix removal, preventing errors if the suffix is missing or malformed. The array normalization handles XML parsers that return a single object when only one item exists in the feed.

Pitfall Guide

1. Locale Drift via Missing ceid

Explanation: Developers often set hl and gl but omit ceid. This can result in mixed-language results or feeds dominated by a region different from the target audience. Fix: Always construct ceid as ${region}:${language} and include it in every request.

2. XML Single-Item Edge Case

Explanation: When a feed returns only one article, some XML parsers return an object instead of an array. Iterating over an object causes runtime errors. Fix: Implement array normalization: const items = Array.isArray(raw) ? raw : [raw];.

3. Aggressive Polling Without Caching

Explanation: Polling the feed every few seconds without caching can trigger rate limiting or IP blocks from Google. Fix: Implement a caching layer with a TTL matching your polling interval. If polling every 5 minutes, cache results for 5 minutes and skip fetches if cache is valid.

4. HTML Entity Corruption

Explanation: Titles may contain HTML entities (e.g., &, '). While fast-xml-parser handles most entities, custom string manipulation can corrupt them if not careful. Fix: Rely on the parser's built-in entity handling. Avoid manual regex replacements on title strings unless decoding is explicitly required.

5. Recency Operator Syntax Errors

Explanation: Using invalid operators like when:24h (if unsupported) or spaces in the operator breaks the query. Fix: Use documented operators: 1h, 1d, 7d. Ensure the operator is appended directly to the topic string with a space separator: topic when:1d.

6. Assuming Infinite Pagination

Explanation: Unlike JSON APIs, RSS feeds do not support offset-based pagination. The feed caps at ~100 items. Fix: For high-volume topics, use time-window slicing. Poll with when:1d and store results. If you need older data, adjust the window or accept the 100-item limit per query.

7. Title Suffix Variance

Explanation: The - Publisher suffix is consistent but not guaranteed. Blindly slicing the string can truncate valid headlines. Fix: Always verify rawTitle.endsWith( - ${publisher}) before slicing. If the check fails, retain the original title.

Production Bundle

Action Checklist

• Define NewsQuery interface with topic, language, region, and recencyWindow.
• Implement GoogleNewsIngestor class with fast-xml-parser.
• Construct URL with ceid parameter dynamically from region and language.
• Append when: operator to query string for temporal filtering.
• Handle XML response where item may be a single object or array.
• Strip - Publisher suffix from titles only when suffix matches publisher.
• Add caching layer to prevent redundant fetches during polling loops.
• Implement error handling for HTTP failures and malformed XML.

Decision Matrix

Scenario	Recommended Approach	Why	Cost Impact
Real-time Brand Monitor	RSS + 5-min Polling	Low latency, covers recent news	Zero
Historical Trend Analysis	RSS + Time Windows	Slice data by when: ranges	Zero
High-Volume Ticker Feed	RSS + Redis Cache	Avoid rate limits, reduce latency	Infra cost
Enterprise Compliance	Commercial API	SLA guarantees, support	High
Rapid Prototyping	RSS + Direct Fetch	No auth setup, immediate results	Zero

Configuration Template

Use this configuration structure to parameterize your ingestion service across multiple topics.

{
  "monitoring": {
    "topics": [
      {
        "id": "tech_ticker",
        "query": "NVIDIA",
        "language": "en",
        "region": "US",
        "recencyWindow": "1d",
        "pollIntervalMinutes": 5
      },
      {
        "id": "competitor_eu",
        "query": "CompetitorX",
        "language": "de",
        "region": "DE",
        "recencyWindow": "7d",
        "pollIntervalMinutes": 15
      }
    ],
    "cache": {
      "ttlSeconds": 300,
      "provider": "redis"
    }
  }
}

Quick Start Guide

1. Install Dependencies:

   npm install fast-xml-parser
   

2. Copy Implementation: Add the GoogleNewsIngestor class and interfaces to your project.
3. Run Ingestion:

   const ingestor = new GoogleNewsIngestor();
   const results = await ingestor.fetchArticles({
     topic: 'Artificial Intelligence',
     language: 'en',
     region: 'US',
     recencyWindow: '1d'
   });
   console.log(results);
   

4. Verify Output: Check that headline is clean (no suffix), publisher is populated, and publishedAt contains a valid date string.
5. Deploy Polling: Wrap the fetch call in a scheduled job or cron task with appropriate caching to productionize the monitor.