One Open Source Project per Day #74: ai-engineering-from-scratch - Build AI Full-stack Skills from Ground Up

, implementing backpropagation manually to understand gradient flow. 3. Generative Systems: Explore the principles behind image, video, and audio generation, focusing on diffusion models and autoregressive architectures. 4. LLM Engineering: Move to large language models, covering training loops, fine-tuning techniques (LoRA/QLoRA), quantization strategies (GGUF/AWQ), and deployment optimization. 5. Agent Engineering: Architect autonomous systems using ReAct loops, external memory, and multi-agent coordination patterns.

Code Example: Raw Linear Transformation

To demonstrate the "from scratch" philosophy, consider implementing a linear layer without framework dependencies. This example uses TypeScript to illustrate how mathematical operations map directly to code, reinforcing the underlying mechanics.

// Raw Linear Layer Implementation
// Demonstrates matrix multiplication and bias addition without frameworks

interface Tensor {
  data: Float32Array;
  shape: [number, number];
}

function createTensor(shape: [number, number], values: number[]): Tensor {
  return {
    data: new Float32Array(values),
    shape,
  };
}

function matMul(a: Tensor, b: Tensor): Tensor {
  const [rowsA, colsA] = a.shape;
  const [rowsB, colsB] = b.shape;

  if (colsA !== rowsB) {
    throw new Error(`Incompatible shapes: ${a.shape} and ${b.shape}`);
  }

  const resultData = new Float32Array(rowsA * colsB);

  for (let i = 0; i < rowsA; i++) {
    for (let j = 0; j < colsB; j++) {
      let sum = 0;
      for (let k = 0; k < colsA; k++) {
        sum += a.data[i * colsA + k] * b.data[k * colsB + j];
      }
      resultData[i * colsB + j] = sum;
    }
  }

  return createTensor([rowsA, colsB], Array.from(resultData));
}

function addBias(input: Tensor, bias: Tensor): Tensor {
  if (input.shape[1] !== bias.shape[1]) {
    throw new Error("Bias dimension mismatch");
  }

  const resultData = new Float32Array(input.data.length);
  for (let i = 0; i < input.data.length; i++) {
    resultData[i] = input.data[i] + bias.data[i % bias.shape[1]];
  }

  return createTensor(input.shape, Array.from(resultData));
}

// Usage: Linear transformation y = xW + b
const input = createTensor([2, 3], [1.0, 2.0, 3.0, 4.0, 5.0, 6.0]);
const weights = createTensor([3, 4], [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0, 1.1, 1.2]);
const bias = createTensor([1, 4], [0.01, 0.02, 0.03, 0.04]);

const linearOutput = addBias(matMul(input, weights), bias);
console.log("Linear Output Shape:", linearOutput.shape);

Architecture Decisions:

• Raw Implementation: Using Float32Array and manual loops forces the engineer to manage memory layout and computational complexity explicitly. This knowledge is critical when optimizing inference engines or writing custom CUDA kernels later.
• Type Safety: TypeScript interfaces enforce tensor shape contracts, reducing runtime errors during complex pipeline construction.
• Modularity: Separating matMul and addBias allows for independent testing and optimization, mirroring the structure of production neural network libraries.

Pitfall Guide

1. Math Paralysis

   • Explanation: Engineers spend excessive time studying linear algebra theory without implementing code, leading to burnout before reaching practical applications.
   • Fix: Time-box mathematical study. Immediately implement every concept in code, even if the implementation is inefficient. The goal is intuition, not proof mastery.

2. Framework Dependency Crutch

   • Explanation: Jumping straight to PyTorch or TensorFlow obscures the underlying mechanics. Engineers may struggle to debug issues that frameworks abstract away.
   • Fix: Adopt a "raw-first" policy. Implement algorithms using NumPy or raw Python/TypeScript before using high-level APIs. Only introduce frameworks after the manual implementation is complete.

3. Artifact Neglect

   • Explanation: Treating lessons as theoretical exercises and failing to save outputs as reusable tools. This wastes the opportunity to build a personal engineering toolkit.
   • Fix: Maintain a structured repository for artifacts. Save every prompt, skill file, agent script, and MCP server configuration. Tag artifacts by phase and utility for easy retrieval.

4. Agent Complexity Overload

   • Explanation: Attempting to build multi-agent swarms without mastering single-agent patterns like ReAct loops and memory management. This leads to unstable and unpredictable systems.
   • Fix: Progress incrementally. Master single-agent tool use and memory retrieval before introducing inter-agent communication. Validate each component in isolation.

5. Quantization Blindness

   • Explanation: Deploying full-precision (FP32) models in production, resulting in unnecessary latency and hardware costs.
   • Fix: Integrate quantization early in the learning path. Experiment with GGUF, AWQ, and INT8 formats. Measure accuracy degradation and latency improvements to understand trade-offs.

6. Language Lock-in

   • Explanation: Relying solely on Python for all AI tasks, missing performance benefits of other languages for specific components.
   • Fix: Explore multi-language implementations. Use Rust or C++ for performance-critical inference engines, TypeScript for agent orchestration in web environments, and Julia for numerical experimentation.

7. Evaluation Neglect

   • Explanation: Building models and agents without rigorous evaluation metrics, leading to deployment of unverified systems.
   • Fix: Implement evaluation loops from the start. Define metrics for accuracy, latency, cost, and safety. Automate testing for agent behaviors and model outputs.

Production Bundle

Action Checklist

• Audit Knowledge Gaps: Run a diagnostic quiz to identify weak areas and map them to the appropriate phase in the curriculum.
• Initialize Artifact Repository: Create a version-controlled directory structure to store skills, agents, and MCP servers generated during learning.
• Implement Math Primitives: Write raw implementations of vector operations, matrix multiplication, and gradient descent without frameworks.
• Build Custom MCP Server: Develop an MCP server that exposes a unique tool or data source, integrating it into your agent workflow.
• Optimize Model Deployment: Quantize a model using GGUF or AWQ and benchmark inference performance on target hardware.
• Architect Agent Swarm: Design a multi-agent system with distinct roles, communication protocols, and conflict resolution mechanisms.
• Create Skill Files: Document reusable prompts and workflows as .md skill files for integration into AI assistants like Cursor or Claude.

Decision Matrix

Scenario	Recommended Approach	Why	Cost Impact
Rapid Prototyping	API Wrapper + Pre-trained Models	Speed of implementation; low initial overhead.	Low upfront, high per-call costs.
Production Optimization	First-Principles Build + Quantization	Control over latency, cost, and security; deep debugging capability.	High upfront, low operational costs.
Complex Task Automation	Multi-Agent Swarm	Handles diverse tasks via specialization and coordination.	Moderate infrastructure, high development complexity.
Simple Workflow Automation	Single Agent with Tools	Sufficient for linear tasks; easier to debug and maintain.	Low infrastructure, low development complexity.

Configuration Template

Skill File Template for AI Assistants

# Skill: Custom Data Analysis Agent

## Description
An agent specialized in analyzing CSV datasets using statistical methods and generating visualizations.

## Instructions
1. Load the dataset using the `load_csv` tool.
2. Perform exploratory data analysis (EDA) using `calculate_stats`.
3. Generate insights based on correlations and distributions.
4. Output a summary report with key findings.

## Tools
- `load_csv(path: string): DataFrame`
- `calculate_stats(df: DataFrame): Stats`
- `generate_chart(data: any, type: string): Image`

## Constraints
- Do not expose raw data in the output.
- Ensure all statistical tests meet a 95% confidence level.
- Handle missing values by imputation or exclusion based on context.

MCP Server Configuration

{
  "mcpServers": {
    "custom-analysis": {
      "command": "node",
      "args": ["./dist/mcp-server.js"],
      "env": {
        "DATA_DIR": "./data",
        "LOG_LEVEL": "info"
      },
      "tools": [
        {
          "name": "analyze_dataset",
          "description": "Analyze a dataset and return insights.",
          "parameters": {
            "type": "object",
            "properties": {
              "path": { "type": "string" },
              "metrics": { "type": "array", "items": { "type": "string" } }
            },
            "required": ["path"]
          }
        }
      ]
    }
  }
}

Quick Start Guide

1. Clone and Setup: Retrieve the curriculum repository and install dependencies. Ensure Python, TypeScript, and Rust toolchains are configured for multi-language support.
2. Run Diagnostic: Execute the /find-your-level command in your AI assistant to generate a personalized starting phase based on your current expertise.
3. Execute Phase 1: Begin with the mathematical foundations. Implement the provided primitives in raw code and verify results against expected outputs.
4. Generate First Artifact: Save your implementation as a reusable script or skill file. Integrate it into your development environment to validate immediate utility.
5. Iterate and Expand: Progress through subsequent phases, continuously building artifacts. Focus on transitioning from theoretical understanding to production-ready components.