community-ai-pipeline.config.yaml

{ id: string; source: 'discord' | 'github' | 'slack'; author: string; timestamp: Date; content: string; metadata: Record<string, unknown>; }

class EventNormalizer { normalize(raw: unknown): CommunityEvent { // Platform-specific parsers omitted for brevity // Guarantees consistent schema before AI processing return { id: crypto.randomUUID(), source: (raw as any).platform, author: (raw as any).user?.id || 'anonymous', timestamp: new Date((raw as any).created_at), content: String((raw as any).message || (raw as any).body), metadata: { thread_id: (raw as any).thread_id } }; } }

### Step 2: AI Triage & Intent Classification

Route events using a lightweight classifier before invoking expensive LLMs. Cache results to control inference costs.

```typescript
import { createOpenAI } from '@ai-sdk/openai';
import { generateObject } from 'ai';
import { z } from 'zod';

const triageSchema = z.object({
  intent: z.enum(['bug_report', 'feature_request', 'documentation', 'spam', 'general_question']),
  priority: z.enum(['low', 'medium', 'high', 'critical']),
  requires_human_review: z.boolean(),
  suggested_channel: z.string()
});

class TriageEngine {
  private cache = new Map<string, z.infer<typeof triageSchema>>();

  async classify(event: CommunityEvent) {
    const cacheKey = `${event.source}:${event.author}:${event.content.slice(0, 50)}`;
    if (this.cache.has(cacheKey)) return this.cache.get(cacheKey)!;

    const openai = createOpenAI({ apiKey: process.env.OPENAI_API_KEY! });
    const { object } = await generateObject({
      model: openai('gpt-4o-mini'),
      schema: triageSchema,
      prompt: `Classify this community interaction:\n${event.content}`
    });

    this.cache.set(cacheKey, object);
    return object;
  }
}

Step 3: Semantic Routing & Knowledge Retrieval

Use vector embeddings to match incoming questions against existing documentation and resolved issues. Prevent duplicate routing and auto-suggest references.

import { createClient } from '@supabase/supabase-js';
import { openai } from '@ai-sdk/openai';
import { embed } from 'ai';

const supabase = createClient(process.env.SUPABASE_URL!, process.env.SUPABASE_KEY!);

class SemanticRouter {
  async findMatches(content: string, limit = 3) {
    const { embedding } = await embed({
      model: openai.embedding('text-embedding-3-small'),
      value: content
    });

    const { data, error } = await supabase.rpc('match_documents', {
      query_embedding: embedding,
      match_threshold: 0.75,
      match_count: limit
    });

    if (error) throw error;
    return data;
  }
}

Step 4: Closed-Loop Feedback Integration

Track routed items through GitHub Issues or Linear. Attach AI-generated summaries and confidence scores. Enable contributors to see status changes.

class FeedbackLoop {
  async createTrackedIssue(event: CommunityEvent, triage: any, matches: any[]) {
    const body = [
      `**Source:** ${event.source}`,
      `**Intent:** ${triage.intent} | **Priority:** ${triage.priority}`,
      `**Confidence:** ${triage.requires_human_review ? 'Needs Review' : 'Auto-Triaged'}`,
      `---`,
      event.content,
      matches.length ? `**Related:** ${matches.map(m => m.title).join(', ')}` : ''
    ].join('\n');

    // GitHub API call or Linear SDK integration
    // Returns issue URL for community visibility
    return { status: 'tracked', url: `https://github.com/org/repo/issues/${Date.now()}` };
  }
}

Architecture Decisions & Rationale

• Event-driven over polling: Webhooks from Discord/Slack/GitHub trigger processing only when activity occurs. Eliminates idle compute and reduces latency.
• Tiered AI routing: Lightweight classifier (gpt-4o-mini) handles triage. Expensive models (o1, claude-sonnet) reserved for complex debugging or code review. Cuts inference costs by 60–70%.
• Vector DB for semantic matching: Supabase/pgvector or Pinecone enables fast similarity search without full-text index bloat. Matches intent, not keywords.
• Explicit feedback tracking: Every routed event creates a traceable artifact. Contributors see status, engineers see context, product managers see demand signals.
• Human-in-the-loop gate: requires_human_review flag ensures critical or ambiguous inputs bypass automation. Prevents AI drift and maintains trust.

Pitfall Guide

1. Full LLM automation without human review LLMs hallucinate context and misclassify nuanced technical requests. Always route high-priority or ambiguous items to human moderators. Implement a confidence threshold that triggers manual review.

2. Ignoring data residency & compliance Community data often contains PII, code snippets, or proprietary discussions. Route sensitive content through on-prem or regional endpoints. Implement opt-out flags for training data. GDPR/CCPA compliance isn't optional; it's a product requirement.

3. Building monolithic community bots Single-purpose bots become maintenance nightmares. Decompose into composable services: ingestion, triage, routing, analytics. Each service scales independently and fails safely.

4. Feedback loop opacity Contributors abandon communities when they don't see their input tracked. Auto-create issues, attach AI summaries, and post status updates. Transparency drives retention.

5. Unbounded AI inference costs Without caching or tiered routing, inference bills scale with community growth. Implement semantic deduplication, response caching, and model routing based on complexity. Monitor cost-per-interaction.

6. Poor contributor onboarding New members drown in channels without clear paths. Provide structured templates for bug reports, feature requests, and discussions. Auto-attach contribution guidelines and code of conduct links.

7. Neglecting community governance AI tools don't replace policy. Define escalation paths, moderation tiers, and contributor roles. Toxicity and spam require policy enforcement, not just filtering.

Best Practices from Production:

• Route by intent, not channel. Let AI decide where conversations belong.
• Cache embeddings and triage results aggressively. Community content repeats.
• Expose feedback status publicly. Contributors stay when they see impact.
• Audit AI decisions weekly. Track misclassification rates and adjust prompts.
• Treat community data as a product asset. Instrument it like telemetry.

Production Bundle

Action Checklist

• Instrument community event ingestion: Deploy webhook handlers for Discord, Slack, and GitHub to normalize payloads into a unified schema.
• Deploy tiered AI triage: Implement lightweight classification with confidence thresholds and human-in-the-loop fallbacks.
• Configure semantic routing: Index documentation and resolved issues in a vector DB; enable similarity matching for duplicate prevention.
• Close the feedback loop: Auto-create tracked issues with AI summaries; post status updates to originating channels.
• Enforce compliance boundaries: Route sensitive content through regional endpoints; implement training opt-out flags and PII redaction.
• Monitor cost & quality: Track inference cost per interaction, misclassification rate, and contributor retention weekly.

Decision Matrix

Scenario	Recommended Approach	Why	Cost Impact
High-volume Q&A channels	Semantic routing + cached embeddings	Prevents duplicate answers; reduces moderator load	Low (vector search + cache hits)
Complex bug reports	Tiered LLM triage → human review	Preserves accuracy; avoids AI misdiagnosis	Medium (specialized model + manual time)
Feature request aggregation	Intent classification + roadmap sync	Surfaces demand signals; prevents backlog noise	Low (classification + sync automation)
Enterprise/community hybrid	Regional endpoints + PII redaction	Meets compliance; isolates sensitive data	Medium (regional compute + redaction pipeline)
Rapid MVP launch	Pre-built bot framework + manual overrides	Fast deployment; defers complex routing	Low (SaaS bot + manual triage)

Configuration Template

# community-ai-pipeline.config.yaml
ingestion:
  sources:
    - discord
    - github
    - slack
  normalization:
    schema_version: "2.1"
    pii_redaction: true
    cache_ttl: 3600 # seconds

triage:
  model: "gpt-4o-mini"
  fallback_model: "claude-3-haiku"
  confidence_threshold: 0.85
  human_review_on: ["critical", "ambiguous", "low_confidence"]

routing:
  vector_db:
    provider: "supabase"
    collection: "community_knowledge"
    match_threshold: 0.75
  deduplication:
    enabled: true
    max_age_days: 30

feedback_loop:
  tracker: "github"
  auto_create_issues: true
  status_updates: true
  contributor_visibility: "public"

compliance:
  data_residency: "us-east-1"
  training_opt_out: true
  audit_log_retention_days: 90

Quick Start Guide

1. Deploy the ingestion handler: Clone the starter repository, configure environment variables for Discord/GitHub webhooks, and run npm run start:ingest. Verify event normalization in the logs.
2. Initialize the triage engine: Set OPENAI_API_KEY and CLAUDE_API_KEY, run npm run start:triage, and test with sample community messages. Confirm classification output matches expected intent/priority.
3. Connect vector routing: Provision a Supabase project, run the provided migration to create match_documents, and execute npm run start:semantic. Validate similarity matches against existing documentation.
4. Enable feedback tracking: Configure GitHub personal access token with repo scope, run npm run start:feedback, and verify auto-created issues contain AI summaries and source attribution.
5. Monitor & iterate: Open the provided Grafana dashboard, track inference_cost_per_event, triage_accuracy, and contributor_retention_90d. Adjust confidence thresholds and cache TTL based on observed traffic.