We are still grieving Google Reader.
Current Situation Analysis
The modern content consumption landscape suffers from algorithmic optimization that prioritizes engagement over readability. While Telegram successfully restored chronological, unranked feeds, it fragments subscriptions into isolated inboxes. Aggregating these feeds introduces severe architectural friction:
- Bot API Limitations: Telegram bots operate as separate identities and require administrator privileges in target channels. This creates a hard dependency on channel owners, making third-party aggregation of public channels structurally impossible.
- Polling Inefficiency: Traditional polling introduces
N/2average latency and scales poorly. FetchingNchannels forMusers generatesN×MAPI calls per cycle, quickly exhausting Telegram's per-account rate limits. - Media Group Fragmentation: Telegram delivers albums as separate messages sharing a
grouped_id. Naive forwarding breaks these into disjointed media posts, stripping captions and context. - Concurrency & Security Risks: Multi-user referral flows and session token management introduce race conditions and severe threat vectors if handled with standard ORM patterns or plaintext storage.
Traditional aggregation methods fail because they either lack user-level channel access (Bot API), cannot scale without hitting rate limits (Polling), or break native Telegram message semantics (Independent Forwarding).
WOW Moment: Key Findings
| Approach | Avg Latency | API Call Efficiency | Album Integrity | Rate Limit Compliance |
|---|---|---|---|---|
| Bot API + Polling (30s) | ~15s | Fails (Requires Admin) | Fails (Splits Groups) | Fails (Rapid Exhaustion) |
| Telethon + Polling (10s) | ~5s | Medium (N×M Calls) | Fails (Splits Groups) | Fails (Per-Account Wall) |
| Televizor Architecture | ~15s (Intentional) | High (1 Conn/User) | 100% (2s Debounce) | 100% (Redis Tumbling) |
Key Findings:
- Event-driven
events.NewMessagereduces API overhead fromN×Mto1persistent connection per user, eliminating polling latency and rate limit collisions. - A 2-second debounce buffer keyed on
(source_id, grouped_id)perfectly reconstructs Telegram albums without perceptible delay (parts arrive in 500ms–1.5s). - Redis tumbling windows (
now // 3600) provide zero-maintenance, auto-expiring rate limiting that scales horizontally without cleanup jobs. - SQL-level atomic increments with deterministic
SELECT FOR UPDATElocking completely eliminate referral double-counting and ABBA deadlocks.
Core Solution
1. Telethon vs Bot API: The Foundation
Telethon operates as an MTProto user client, authenticating with the same protocol as official Telegram apps. This grants access to every subscribed channel without requiring admin privileges. The trade-off is operating in a ToS grey area, mitigated by:
- 15-second configurable forward delay
- Per-source/per-feed rate limiting via Redis
- Event-driven handlers (zero polling)
- Graceful
SessionRevokedErrorandAuthKeyErrorhandling
2. Event-Driven Message Routing
Telethon's events.NewMessage establishes a persistent WebSocket-like connection. The routing table (source_to_feeds) maps source channel IDs to subscribed feeds and updates instantly via PostgreSQL LISTEN/NOTIFY.
on @client.on(events.NewMessage()) async def handler(event): normalized_id = utils.get_peer_id(event.message.peer_id, add_mark=False) if normalized_id not in source_to_feeds: return
valid_feeds = []
for feed in source_to_feeds[normalized_id]:
if feed.filters and not check_filters(event.message, feed.filters):
continue
if not check_rate_limit(user_id, feed.id, feed.filters):
continue
valid_feeds.append(feed)
for dest_id, feeds in group_by_destination(valid_feeds):
asyncio.create_task(
forward_message(client, source_id, dest_id, event.message.id)
)
### 3. Album Reconstruction via Debounce Buffer
Album parts arrive within 500ms–1.5s. A 2-second asyncio timer resets on each new part. After quiet, the batch flushes via `forward_messages(messages=[ids])`, preserving native grouping.
```python
if grouped_id:
key = (source_channel_id, grouped_id)
if key not in pending_albums:
pending_albums[key] = {
'message_ids': set(),
'feeds': valid_feeds,
'timer': None,
'source_peer': await event.get_input_chat()
}
pending_albums[key]['message_ids'].add(event.message.id)
if pending_albums[key]['timer']:
pending_albums[key]['timer'].cancel()
pending_albums[key]['timer'] = asyncio.create_task(
wait_and_flush(key, flush_album)
)
4. Concurrency & Rate Limiting
- Referral Concurrency: Uses
SELECT FOR UPDATEwith deterministic lock ordering (user row → referrer row) to prevent ABBA deadlocks. Increments are executed at the SQL level (User.referral_count = User.referral_count + 1) to bypass ORM cache staleness. - Redis Tumbling Windows: Anti-flood logic uses hourly (
now // 3600) and daily (now // 86400) counters. Keys auto-expire via TTL, eliminating cleanup overhead while maintaining strict per-user/action limits.
5. Threat Model & Session Security
Session strings function as auth tokens. The architecture stores phone numbers, Telegram IDs, session strings, feed configs, and tier data. It explicitly excludes message content, history, contacts, and media.
- Encryption: Sessions are encrypted at rest using Fernet (AES-128-CBC + HMAC-SHA256), gated on
SESSION_ENCRYPTION_KEY. - Revocation: Users can instantly revoke sessions via Telegram settings. 2FA passwords are never requested, preventing account takeover even if DB leaks.
- Self-Hosting: Session data never leaves the user's machine.
6. Self-Host Deployment
git clone https://github.com/s0larpunk/televizor
cd televizor
cp .env.example .env # TELEGRAM_API_ID / TELEGRAM_API_HASH from my.telegram.org
docker compose up -d
Four containers (frontend, backend, postgres, redis) idle at ~180MB RAM, active at ~350MB. Requires TLS reverse proxy and postgres_data backups.
Pitfall Guide
- Bot API for Public Channel Aggregation: Bots require admin rights in target channels, making third-party aggregation of public feeds structurally impossible. Always use MTProto user clients (Telethon/Pyrogram) for cross-channel routing.
- Polling-Based Message Fetching: Polling introduces
N/2latency and scales atN×MAPI calls. Telegram's per-account rate limits will throttle or ban aggressive polling loops. Switch toevents.NewMessagefor persistent, event-driven updates. - Independent Album Message Forwarding: Forwarding album parts individually breaks Telegram's native media grouping. Implement a 2-second debounce buffer keyed on
(source_id, grouped_id)and flush viaforward_messages(messages=[ids]). - Naive Concurrent Referral Updates: ORM-level read-modify-write operations cache stale values during concurrent transactions, causing lost updates. Use
SELECT FOR UPDATEwith deterministic lock ordering and SQL-level atomic increments. - Unencrypted Session Storage: Telethon session strings are functionally equivalent to auth tokens. Always encrypt at rest using Fernet (AES-128-CBC + HMAC-SHA256) and enforce environment-gated key rotation.
- AI Agent Context Drift: Multi-session AI code generation causes module conflicts and inconsistent patterns. External review, manual refactoring for cross-module consistency, and explicit test coverage are mandatory when using agent-assisted development.
Deliverables
- 📘 Televizor Architecture Blueprint: Complete system diagram detailing MTProto client initialization, PostgreSQL
LISTEN/NOTIFYrouting, Redis tumbling window implementation, Fernet session encryption flow, and Docker container orchestration. - ✅ Self-Host & Security Checklist: Step-by-step verification for API key generation,
.envconfiguration, Fernet key creation, TLS reverse proxy setup, PostgreSQL backup scheduling, and session revocation protocols. - ⚙️ Configuration Templates: Production-ready
.env.examplestructure,docker-compose.ymlservice definitions, Redis rate-limiting key schemas, and PostgreSQL routing table DDL withLISTEN/NOTIFYtriggers.
