Building an AI-powered product

c retrieveContext(queryVector: number[], query: string): Promise<ContextChunk[]> { const results = await this.vectorStore.similaritySearch({ vector: queryVector, topK: this.config.topK, filter: { active: true }, // Example metadata filter });

// Hybrid re-ranking or keyword filtering could be applied here
return results
  .filter(r => r.score >= this.config.minScore)
  .map(r => ({
    id: r.id,
    text: r.text,
    metadata: r.metadata,
    score: r.score,
  }));

}

private buildPrompt(query: string, chunks: ContextChunk[]): string { // Context window management: truncate if necessary const contextText = chunks .slice(0, this.config.maxContextTokens / 100) // Rough estimate .map(c => <context>${c.text}</context>) .join('\n');

return `
  You are a helpful assistant. Answer the user's question based *only* on the provided context.
  If the context does not contain the answer, state that you cannot answer based on the available information.
  
  Context:
  ${contextText}
  
  Question: ${query}
  Answer:
`;

}

private async evaluate(response: string, chunks: ContextChunk[]): Promise<EvaluationResult> { // LLM-as-a-Judge or deterministic metric evaluation const evalPrompt = ` Evaluate the following response based on the provided context. Criteria: 1. Groundedness: Is the response supported by the context? 2. Relevance: Does the response answer the query?

  Context: ${chunks.map(c => c.text).join(' ')}
  Response: ${response}
  
  Return JSON: { "passed": boolean, "score": number, "reasons": string[] }
`;

const evalOutput = await this.llm.generate(evalPrompt, {
  model: 'evaluation-model-v1', // Smaller, faster model
  response_format: { type: 'json_object' },
});

const schema = z.object({
  passed: z.boolean(),
  score: z.number(),
  reasons: z.array(z.string()),
});

return schema.parse(JSON.parse(evalOutput));

}

private handleFallback(query: string, evalResult?: EvaluationResult): AIResponse { // Implement fallback logic: e.g., return "I don't know" or route to human agent return { content: "I'm unable to provide a definitive answer based on the current information.", sources: [], confidence: 0, }; } }

### Rationale

*   **Low Temperature:** Setting `temperature: 0.1` minimizes creativity, ensuring the model adheres strictly to the context.
*   **Evaluation Step:** The `evaluate` method prevents hallucinations from reaching the user. It uses a dedicated evaluation model to keep latency low while maintaining rigorous checks.
*   **Context Management:** The `buildPrompt` method includes logic to slice chunks based on token estimates, preventing context window overflow errors.
*   **Fallback Strategy:** The pipeline degrades gracefully. If retrieval fails or evaluation rejects the response, a safe fallback is returned.

## Pitfall Guide

Production experience reveals specific failure modes that can derail AI products. Avoid these pitfalls to ensure stability and scalability.

1.  **Ignoring Evaluation Debt**
    *   *Mistake:* Shipping without automated evaluation. Relying on manual testing is insufficient for non-deterministic systems.
    *   *Best Practice:* Implement a continuous evaluation harness. Run evals on every model update and periodically on production traffic. Use metrics like groundedness, faithfulness, and answer relevance.

2.  **Hardcoding Prompts**
    *   *Mistake:* Embedding prompts directly in code. This makes updates difficult and prevents A/B testing.
    *   *Best Practice:* Externalize prompts to a versioned store. Use prompt management tools to update prompts without code deployments. Version control allows rollback if a prompt change degrades performance.

3.  **Context Window Overflow**
    *   *Mistake:* Retrieving too many chunks or using unbounded text, causing API errors or truncation.
    *   *Best Practice:* Implement dynamic context window management. Chunk documents semantically, retrieve a fixed top-K, and truncate based on token counts. Prioritize chunks with higher relevance scores.

4.  **Cost Blindness**
    *   *Mistake:* No monitoring of token usage or cost per query. Costs spiral due to prompt injection or inefficient pipelines.
    *   *Best Practice:* Implement cost tracking and budget caps. Use model routing to send simple queries to cheaper models. Monitor average tokens per query and alert on anomalies.

5.  **Data Leakage and PII**
    *   *Mistake:* Ingesting sensitive data into vector stores without redaction.
    *   *Best Practice:* Implement a PII redaction pipeline before embedding. Use access controls on the vector store to ensure retrieval respects user permissions. Audit data flows regularly.

6.  **Silent Failures**
    *   *Mistake:* The model returns a plausible but incorrect answer without indicating uncertainty.
    *   *Best Practice:* Require confidence scores. If confidence is below a threshold, trigger a fallback. Use the evaluation layer to detect low-confidence responses and suppress them.

7.  **Over-Reliance on Single Model**
    *   *Mistake:* Building the entire product around one LLM provider. Outages or API changes break the product.
    *   *Best Practice:* Abstract the LLM interface. Support multiple providers and models. Implement automatic fallback to secondary providers in case of primary provider failure.

## Production Bundle

### Action Checklist

- [ ] **Define Success Metrics:** Establish baseline metrics for latency, cost, hallucination rate, and user satisfaction before development.
- [ ] **Implement RAG Pipeline:** Deploy a retrieval-augmented generation architecture with vector search and context management.
- [ ] **Create Evaluation Harness:** Build an automated evaluation suite using LLM-as-a-judge or deterministic metrics to validate responses.
- [ ] **Set Up Observability:** Integrate tracing and logging for all AI interactions. Monitor token usage, latency, and error rates.
- [ ] **Configure Fallbacks:** Implement graceful degradation strategies for retrieval failures, evaluation rejections, and API outages.
- [ ] **Establish Cost Controls:** Set budget alerts and implement model routing to optimize cost per query.
- [ ] **Secure Data Pipeline:** Apply PII redaction and access controls to all data ingestion and retrieval processes.
- [ ] **Version Prompts and Models:** Use version control for prompts and model configurations to enable safe rollouts and rollbacks.

### Decision Matrix

| Scenario | Recommended Approach | Why | Cost Impact |
|----------|---------------------|-----|-------------|
| **Domain-Specific Q&A** | RAG with Hybrid Search | Grounded answers reduce hallucination; easy to update knowledge base. | Medium per-query; low upfront. |
| **Structured Data Extraction** | Fine-Tuned Model | High accuracy on specific formats; faster inference than RAG. | High upfront; low per-query. |
| **Creative Content Generation** | Prompting with Guardrails | Flexibility required; RAG may constrain creativity unnecessarily. | Low upfront; medium per-query. |
| **Real-Time Chatbot** | RAG + Streaming + Eval | Low latency streaming with evaluation ensures quality without blocking. | Medium per-query; requires infra. |
| **Legacy System Integration** | Model Router + API Wrapper | Abstracts AI complexity; allows gradual migration to full RAG. | Low upfront; scales with usage. |

### Configuration Template

Copy this TypeScript configuration to bootstrap your AI product settings. Adjust values based on your specific requirements and model capabilities.

```typescript
// ai.config.ts
export const AIConfig = {
  models: {
    embedding: {
      provider: 'openai',
      model: 'text-embedding-3-large',
      dimensions: 1536,
    },
    generation: {
      primary: {
        provider: 'anthropic',
        model: 'claude-3-5-sonnet-20240620',
        maxTokens: 1024,
        temperature: 0.1,
      },
      fallback: {
        provider: 'openai',
        model: 'gpt-4o-mini',
        maxTokens: 512,
        temperature: 0.1,
      },
      evaluation: {
        provider: 'openai',
        model: 'gpt-4o-mini',
        maxTokens: 256,
      },
    },
  },
  retrieval: {
    vectorStore: 'pinecone',
    topK: 5,
    minScore: 0.75,
    maxContextTokens: 4000,
  },
  evaluation: {
    enabled: true,
    thresholds: {
      groundedness: 0.8,
      relevance: 0.85,
    },
  },
  observability: {
    tracing: true,
    logging: 'verbose',
    costTracking: true,
  },
};

Quick Start Guide

Follow these steps to initialize your AI product pipeline in under 5 minutes.

1. Install Dependencies: Run npm install @vectorstore/sdk @ai-providers/sdk zod to install the required libraries for vector search, LLM interaction, and validation.

2. Configure Environment: Create a .env file with your API keys and vector store URL:

   LLM_API_KEY=sk-...
   EMBEDDING_API_KEY=sk-...
   VECTOR_DB_URL=https://your-vector-store...
   

3. Initialize Pipeline: Import the AIProductPipeline class and instantiate it with your configuration:

   import { AIProductPipeline } from './pipeline';
   import { AIConfig } from './ai.config';
   
   const pipeline = new AIProductPipeline(AIConfig.retrieval);
   

4. Run Test Query: Execute a test query to verify the pipeline:

   const response = await pipeline.generateResponse("How do I reset my password?");
   console.log(response.content);
   

5. Verify Evaluation: Check the evaluation logs to ensure the evaluation layer is functioning. Confirm that the response passed the groundedness and relevance checks before deployment.