Back to KB
Difficulty
Intermediate
Read Time
5 min

We are still grieving Google Reader.

By Codcompass Team··5 min read

Current Situation Analysis

The modern content consumption landscape suffers from algorithmic optimization that prioritizes engagement over readability. While Telegram successfully restored chronological, unranked feeds, it fragments subscriptions into isolated inboxes. Aggregating these feeds introduces severe architectural friction:

  • Bot API Limitations: Telegram bots operate as separate identities and require administrator privileges in target channels. This creates a hard dependency on channel owners, making third-party aggregation of public channels structurally impossible.
  • Polling Inefficiency: Traditional polling introduces N/2 average latency and scales poorly. Fetching N channels for M users generates N×M API calls per cycle, quickly exhausting Telegram's per-account rate limits.
  • Media Group Fragmentation: Telegram delivers albums as separate messages sharing a grouped_id. Naive forwarding breaks these into disjointed media posts, stripping captions and context.
  • Concurrency & Security Risks: Multi-user referral flows and session token management introduce race conditions and severe threat vectors if handled with standard ORM patterns or plaintext storage.

Traditional aggregation methods fail because they either lack user-level channel access (Bot API), cannot scale without hitting rate limits (Polling), or break native Telegram message semantics (Independent Forwarding).

WOW Moment: Key Findings

ApproachAvg LatencyAPI Call EfficiencyAlbum IntegrityRate Limit Compliance
Bot API + Polling (30s)~15sFails (Requires Admin)Fails (Splits Groups)Fails (Rapid Exhaustion)
Telethon + Polling (10s)~5sMedium (N×M Calls)Fails (Splits Groups)Fails (Per-Account Wall)
Televizor Architecture~15s (Intentional)High (1 Conn/User)100% (2s Debounce)100% (Redis Tumbling)

Key Findings:

  • Event-driven events.NewMessage reduces API overhead from N×M to 1 persistent connection per user, eliminating polling latency and rate limit collisions.
  • A 2-second debounce buffer keyed on (source_id, grouped_id) perfectly reconstructs Telegram albums without perceptible delay (parts arrive in 500ms–1.5s).
  • Redis tumbling windows (now // 3600) provide zero-maintenance, auto-expiring rate limiting that scales horizontally without cleanup jobs.
  • SQL-level atomic increments with deterministic SELECT FOR UPDATE locking completely eliminate referral double-counting and ABBA deadlocks.

Core Solution

1. Telethon vs Bot API: The Foundation

Telethon operates as an MTProto user client, authenticating with the same protocol as official Telegram apps. This grants access to every subscribed channel without requiring admin privileges. The trade-off is operating in a ToS grey area, mitigated by:

  • 15-second configurable forward delay
  • Per-source/per-feed rate limiting via Redis
  • Event-driven handlers (zero polling)
  • Graceful SessionRevokedError and AuthKeyError handling

2. Event-Driven Message Routing

Telethon's events.NewMessage establishes a persistent WebSocket-like connection. The routing table (source_to_feeds) maps source channel IDs to subscribed feeds and updates instantly via PostgreSQL LISTEN/NOTIFY.

on @client.on(events.NewMessage()) async def handler(event): normalized_id = utils.get_peer_id(event.message.peer_id, add_mark=False) if normalized_id not in source_to_feeds: return

valid_feeds = []
for feed in source_to_feeds[normalized_id]:
    if feed.filters and not check_filters(event.message, feed.filters):
        continue
    if not check_rate_limit(user_id, feed.id, feed.filters):
        continue
    valid_feeds.append(feed)

for dest_id, feeds in group_by_destination(valid_feeds):
    asyncio.create_task(
        forward_message(client, source_id, dest_id, event.message.id)
    )

### 3. Album Reconstruction via Debounce Buffer
Album parts arrive within 500ms–1.5s. A 2-second asyncio timer resets on each new part. After quiet, the batch flushes via `forward_messages(messages=[ids])`, preserving native grouping.

```python
if grouped_id:
    key = (source_channel_id, grouped_id)
    if key not in pending_albums:
        pending_albums[key] = {
            'message_ids': set(),
            'feeds': valid_feeds,
            'timer': None,
            'source_peer': await event.get_input_chat()
        }
    pending_albums[key]['message_ids'].add(event.message.id)
    if pending_albums[key]['timer']:
        pending_albums[key]['timer'].cancel()
    pending_albums[key]['timer'] = asyncio.create_task(
        wait_and_flush(key, flush_album)
    )

4. Concurrency & Rate Limiting

  • Referral Concurrency: Uses SELECT FOR UPDATE with deterministic lock ordering (user row → referrer row) to prevent ABBA deadlocks. Increments are executed at the SQL level (User.referral_count = User.referral_count + 1) to bypass ORM cache staleness.
  • Redis Tumbling Windows: Anti-flood logic uses hourly (now // 3600) and daily (now // 86400) counters. Keys auto-expire via TTL, eliminating cleanup overhead while maintaining strict per-user/action limits.

5. Threat Model & Session Security

Session strings function as auth tokens. The architecture stores phone numbers, Telegram IDs, session strings, feed configs, and tier data. It explicitly excludes message content, history, contacts, and media.

  • Encryption: Sessions are encrypted at rest using Fernet (AES-128-CBC + HMAC-SHA256), gated on SESSION_ENCRYPTION_KEY.
  • Revocation: Users can instantly revoke sessions via Telegram settings. 2FA passwords are never requested, preventing account takeover even if DB leaks.
  • Self-Hosting: Session data never leaves the user's machine.

6. Self-Host Deployment

git clone https://github.com/s0larpunk/televizor
cd televizor
cp .env.example .env  # TELEGRAM_API_ID / TELEGRAM_API_HASH from my.telegram.org
docker compose up -d

Four containers (frontend, backend, postgres, redis) idle at ~180MB RAM, active at ~350MB. Requires TLS reverse proxy and postgres_data backups.

Pitfall Guide

  1. Bot API for Public Channel Aggregation: Bots require admin rights in target channels, making third-party aggregation of public feeds structurally impossible. Always use MTProto user clients (Telethon/Pyrogram) for cross-channel routing.
  2. Polling-Based Message Fetching: Polling introduces N/2 latency and scales at N×M API calls. Telegram's per-account rate limits will throttle or ban aggressive polling loops. Switch to events.NewMessage for persistent, event-driven updates.
  3. Independent Album Message Forwarding: Forwarding album parts individually breaks Telegram's native media grouping. Implement a 2-second debounce buffer keyed on (source_id, grouped_id) and flush via forward_messages(messages=[ids]).
  4. Naive Concurrent Referral Updates: ORM-level read-modify-write operations cache stale values during concurrent transactions, causing lost updates. Use SELECT FOR UPDATE with deterministic lock ordering and SQL-level atomic increments.
  5. Unencrypted Session Storage: Telethon session strings are functionally equivalent to auth tokens. Always encrypt at rest using Fernet (AES-128-CBC + HMAC-SHA256) and enforce environment-gated key rotation.
  6. AI Agent Context Drift: Multi-session AI code generation causes module conflicts and inconsistent patterns. External review, manual refactoring for cross-module consistency, and explicit test coverage are mandatory when using agent-assisted development.

Deliverables

  • 📘 Televizor Architecture Blueprint: Complete system diagram detailing MTProto client initialization, PostgreSQL LISTEN/NOTIFY routing, Redis tumbling window implementation, Fernet session encryption flow, and Docker container orchestration.
  • ✅ Self-Host & Security Checklist: Step-by-step verification for API key generation, .env configuration, Fernet key creation, TLS reverse proxy setup, PostgreSQL backup scheduling, and session revocation protocols.
  • ⚙️ Configuration Templates: Production-ready .env.example structure, docker-compose.yml service definitions, Redis rate-limiting key schemas, and PostgreSQL routing table DDL with LISTEN/NOTIFY triggers.