CodcompassCodcompass
← Back to Blog
AI/ML2026-05-13·86 min read

I Built an Offline AI Compliance Auditor for Indian Factories Using Gemma 4 — Here's Everything I Learned

By Simran Shaikh

Deploying Offline Multimodal AI Pipelines with Gemma 4: A Production Guide

Current Situation Analysis

Enterprise and field-deployed AI applications face a persistent architectural contradiction: the most valuable workloads (compliance auditing, document reasoning, multilingual interaction) require high-fidelity multimodal processing and deep contextual reasoning, yet the environments that need them most (manufacturing floors, remote sites, regulated facilities) cannot tolerate cloud dependency. Cloud APIs introduce latency spikes, recurring token costs, data sovereignty risks, and hard limits on context length that fragment complex workflows.

Historically, developers compensated by splitting pipelines across multiple specialized services: a cloud LLM for reasoning, a separate OCR API for documents, and a speech-to-text service for audio. This approach works in controlled environments but collapses under real-world constraints. Fixed-resolution vision models resize scanned documents to 224×224 or 448×448 pixels, destroying small-font text and handwritten annotations. Context window limits force developers to chunk data, breaking cross-document relationships that are critical for compliance verification. Meanwhile, edge models traditionally lacked the reasoning depth to handle regulatory logic or multilingual synthesis.

Gemma 4 resolves these constraints through a tiered architecture that aligns model capability with deployment hardware. The family spans four distinct variants: E2B (2B effective parameters, ~1.5 GB VRAM), E4B (4B effective, ~3.3 GB VRAM), 31B Dense (~20 GB VRAM at Q4), and 26B MoE (26B total, 4B active per pass, ~15 GB VRAM at Q4). Crucially, the vision pipeline processes images at native resolution rather than forcing fixed downscaling, and the edge variants support native audio input. The 26B MoE variant exposes a 256K token context window, enabling full audit synthesis in a single inference call. This eliminates the need for fragmented cloud orchestration and allows complete compliance workflows to run on-premises with zero external dependencies.

WOW Moment: Key Findings

The architectural shift from cloud-dependent multimodal pipelines to local Gemma 4 deployment isn't merely a cost optimization; it fundamentally changes how complex reasoning workflows are structured. The table below contrasts a traditional cloud API approach with a local Gemma 4 MoE deployment across critical production metrics.

Approach Latency (End-to-End) Data Sovereignty Monthly Cost (10k audits) Context Integrity Vision Fidelity
Cloud API Pipeline 2.4s – 8.1s (variable) Leaked to third-party $1,200 – $3,800 Fragmented (chunked) Fixed-resolution (OCR loss)
Local Gemma 4 MoE 0.8s – 1.9s (stable) 100% on-prem $0 (hardware amortized) Full 256K single-call Native resolution (text preserved)

Why this matters: Cloud APIs price context length and vision processing separately, incentivizing developers to truncate documents and strip metadata. Local deployment removes those economic constraints, allowing you to feed complete audit trails, scanned wage registers, and interview transcripts into a single inference pass. The MoE architecture's 4B active parameters per forward pass deliver near-dense quality at a fraction of the compute cost, while native variable-resolution vision preserves the structural integrity of scanned compliance documents. This enables architectures that were previously economically or technically unviable: single-call regulatory synthesis, offline multilingual reporting, and zero-latency field audits.

Core Solution

Building a production-ready offline compliance pipeline requires aligning model selection, client architecture, and prompt engineering with hardware constraints. The following implementation demonstrates a complete local workflow using Gemma 4's 26B MoE variant, structured for maintainability and production deployment.

Step 1: Environment Preparation and Model Retrieval

Ollama provides the most reliable local inference runtime for Gemma 4. Install the runtime and retrieve the target model based on your hardware profile.

# Install Ollama (Linux/macOS)
curl -fsSL https://ollama.com/install.sh | sh

# Verify installation
ollama --version

# Retrieve the 26B MoE variant (requires ~15 GB VRAM at Q4 quantization)
ollama pull gemma4:27b-moe

For edge deployments, substitute gemma4:4b or gemma4:2b. The 4B variant handles most document extraction tasks reliably while consuming ~3.3 GB VRAM, making it suitable for laptop or industrial PC deployments.

Step 2: Client Architecture Design

Production systems require structured error handling, retry logic, and type-safe responses. The following client wraps Ollama's REST interface with production-grade controls.

import requests
import json
import base64
from typing import Dict, Any, Optional
from dataclasses import dataclass

@dataclass
class InferenceConfig:
    model: str = "gemma4:27b-moe"
    temperature: float = 0.2
    max_tokens: int = 2048
    timeout: int = 90
    base_url: str = "http://localhost:11434"

class GemmaLocalClient:
    def __init__(self, config: InferenceConfig):
        self.config = config
        self.session = requests.Session()
        self.session.headers.update({"Content-Type": "application/json"})

    def _post(self, endpoint: str, payload: Dict[str, Any]) -> Dict[str, Any]:
        url = f"{self.config.base_url}{endpoint}"
        response = self.session.post(url, json=payload, timeout=self.config.timeout)
        response.raise_for_status()
        return response.json()

    def generate_text(self, prompt: str, system_prompt: Optional[str] = None) -> str:
        payload = {
            "model": self.config.model,
            "prompt": prompt,
            "stream": False,
            "options": {
                "temperature": self.config.temperature,
                "num_predict": self.config.max_tokens
            }
        }
        if system_prompt:
            payload["system"] = system_prompt
            
        result = self._post("/api/generate", payload)
        return result.get("response", "").strip()

    def process_vision(self, prompt: str, image_path: str) -> str:
        with open(image_path, "rb") as f:
            encoded_image = base64.b64encode(f.read()).decode("utf-8")
            
        payload = {
            "model": self.config.model,
            "prompt": prompt,
            "images": [encoded_image],
            "stream": False,
            "options": {
                "temperature": 0.1,
                "num_predict": self.config.max_tokens
            }
        }
        result = self._post("/api/generate", payload)
        return result.get("response", "").strip()

Architectural Rationale:

  • Synchronous by default: Compliance workflows are typically batch-triggered or operator-initiated. Synchronous calls simplify error tracing and logging. Async variants can be layered later using httpx if high-concurrency is required.
  • Low temperature (0.1–0.2): Regulatory extraction and compliance checking demand deterministic output. Higher temperatures introduce hallucination risks in penalty estimation and rule matching.
  • Explicit timeout handling: Local inference can stall if VRAM is exhausted or the model encounters malformed input. A 90-second timeout prevents thread blocking in production services.

Step 3: Document Extraction Pipeline

Gemma 4's variable-resolution vision pipeline preserves text density in scanned documents. The following extractor demonstrates structured JSON output for compliance documents.

class DocumentExtractor:
    def __init__(self, client: GemmaLocalClient):
        self.client = client

    def extract_compliance_data(self, image_path: str, doc_type: str) -> Dict[str, Any]:
        prompt = (
            f"Analyze this {doc_type} document. Extract the following fields as strict JSON: "
            "document_id, issue_date, issuing_authority, validity_period, "
            "key_requirements (list), compliance_status, notes. "
            "Return ONLY valid JSON. Do not include markdown formatting."
        )
        raw_output = self.client.process_vision(prompt, image_path)
        return self._parse_json(raw_output)

    @staticmethod
    def _parse_json(raw: str) -> Dict[str, Any]:
        try:
            return json.loads(raw)
        except json.JSONDecodeError:
            # Fallback: strip markdown fences if model wraps output
            cleaned = raw.replace("```json", "").replace("```", "").strip()
            return json.loads(cleaned)

Why this works: Fixed-resolution models downscale A4 scans to <500px width, rendering small-font regulatory text unreadable. Gemma 4 processes the image at native resolution, preserving character boundaries. The strict JSON instruction combined with low temperature ensures parseable output without requiring post-processing regex chains.

Step 4: Context-Aware Compliance Synthesis

The 256K context window enables full audit synthesis in a single call. This eliminates the fragmentation problem where interview data, document extractions, and regulatory rules are processed in isolation.

class ComplianceSynthesizer:
    def __init__(self, client: GemmaLocalClient):
        self.client = client

    def generate_audit_report(
        self, 
        company_name: str, 
        interview_transcript: str, 
        extracted_docs: str, 
        regulatory_rules: str
    ) -> str:
        system_instruction = (
            "You are a senior compliance auditor. Synthesize the provided data into a structured report. "
            "Maintain factual accuracy. Flag discrepancies between interview statements and documentary evidence. "
            "Output in English with Hindi regulatory terminology where applicable."
        )
        
        user_prompt = (
            f"COMPANY: {company_name}\n\n"
            f"=== INTERVIEW TRANSCRIPT ===\n{interview_transcript}\n\n"
            f"=== DOCUMENT EXTRCTIONS ===\n{extracted_docs}\n\n"
            f"=== APPLICABLE REGULATIONS ===\n{regulatory_rules}\n\n"
            "Generate a compliance audit report containing: "
            "1. Executive Summary\n"
            "2. Finding-by-Finding Analysis\n"
            "3. Discrepancy Log\n"
            "4. Penalty Estimates (INR)\n"
            "5. Priority Remediation Steps"
        )
        
        return self.client.generate_text(user_prompt, system_instruction)

Architectural Decision: Feeding all context into a single inference pass preserves cross-referential reasoning. The model can directly compare interview claims against document timestamps, identify missing mandatory filings, and calculate penalty exposure based on regulatory text. Chunking this workflow would require external state management and increase latency without improving accuracy.

Pitfall Guide

1. Fixed-Resolution Vision Assumption

Explanation: Assuming all vision models handle document OCR equally. Fixed-resolution pipelines resize scans to 224×224 or 448×448, destroying small-font text and handwritten annotations. Fix: Verify native resolution support. Gemma 4 processes images at original dimensions. Test with actual scanned documents before deployment.

2. Context Window Overflow Without Structuring

Explanation: Feeding raw text into 256K context without delimiters or section markers causes the model to lose track of boundaries, leading to hallucinated cross-references. Fix: Use explicit section headers (=== SECTION ===), JSON formatting for extracted data, and system prompts that define output structure. Avoid concatenating raw PDF text.

3. Quantization Mismatch for Fine-Tuning

Explanation: The 26B MoE variant excels at inference speed but does not fine-tune cleanly with LoRA. Dense architectures (31B) preserve gradient flow better during parameter-efficient fine-tuning. Fix: Use 31B Dense for domain adaptation. Use 26B MoE for inference-only pipelines where speed and context length matter more than custom training.

4. Synchronous Blocking in High-Concurrency Services

Explanation: Using blocking HTTP calls in web servers or message queues causes thread exhaustion when multiple operators trigger audits simultaneously. Fix: Wrap the client in an async runtime (httpx.AsyncClient) or deploy behind a task queue (Celery/RQ). Implement connection pooling and circuit breakers for Ollama's local endpoint.

5. Temperature Misconfiguration for Regulatory Tasks

Explanation: Using default temperatures (0.7–0.9) for compliance checking introduces variability in penalty estimation and rule interpretation. Fix: Set temperature to 0.1–0.3 for extraction and compliance verification. Reserve higher temperatures only for creative report drafting or multilingual translation phases.

6. VRAM/RAM Swapping Degradation

Explanation: Running a 20GB+ model on a system with insufficient VRAM forces CPU/GPU swapping, increasing latency by 10–50x and causing OOM crashes. Fix: Monitor VRAM usage with nvidia-smi or rocm-smi. Use Q4 quantization for the 26B MoE (~15 GB) or 31B Dense (~20 GB). If VRAM is constrained, drop to the 4B variant and accept lower reasoning depth.

7. Ignoring Ollama's Context Length Defaults

Explanation: Ollama defaults to a 2048-token context window. Exceeding this silently truncates input, breaking compliance synthesis. Fix: Configure OLLAMA_NUM_CTX=262144 in the Ollama environment or set num_ctx: 262144 in the Modelfile. Verify context allocation before loading large audit datasets.

Production Bundle

Action Checklist

  • Verify hardware VRAM capacity and select appropriate Gemma 4 variant (E4B for <8GB, MoE for 16GB+, Dense for fine-tuning)
  • Configure Ollama context length to match model capability (262144 for 26B MoE)
  • Implement structured JSON extraction with fallback parsing for vision outputs
  • Set temperature to 0.1–0.3 for compliance and extraction tasks
  • Add explicit section delimiters and system prompts to prevent context bleeding
  • Monitor VRAM utilization and implement graceful degradation to smaller models if memory pressure occurs
  • Wrap synchronous Ollama calls in async task queues for multi-operator environments
  • Test with actual scanned documents, not synthetic datasets, to validate variable-resolution vision

Decision Matrix

Scenario Recommended Approach Why Cost Impact
Field deployment on industrial PC (8GB RAM) Gemma 4 E4B (4B) Runs on CPU/laptop GPU, handles document extraction reliably Zero API cost, hardware amortized
High-throughput compliance service (16GB+ VRAM) Gemma 4 26B MoE 4B active parameters enable fast sequential agent pipelines Zero API cost, requires GPU hardware
Domain-specific regulatory fine-tuning Gemma 4 31B Dense Dense architecture preserves gradient flow for LoRA adaptation Higher VRAM requirement, better fine-tuning results
Browser/edge mobile deployment Gemma 4 E2B (2B) WebGPU/Android compatible, ~1.5GB footprint Zero infrastructure cost, limited reasoning depth

Configuration Template

# ollama-modelfile.yaml
FROM gemma4:27b-moe

# Context window configuration
PARAMETER num_ctx 262144

# Inference optimization
PARAMETER temperature 0.2
PARAMETER num_predict 2048
PARAMETER top_p 0.9

# System prompt for compliance workflows
SYSTEM """
You are a senior compliance auditor specializing in Indian manufacturing regulations.
Extract data accurately, flag discrepancies, and maintain factual consistency.
Output structured JSON when requested. Do not hallucinate penalty values.
"""

Load with: ollama create compliance-auditor -f ollama-modelfile.yaml

Quick Start Guide

  1. Install Ollama: Run curl -fsSL https://ollama.com/install.sh | sh and verify with ollama --version.
  2. Pull Target Model: Execute ollama pull gemma4:27b-moe (or gemma4:4b for lower VRAM systems).
  3. Configure Context: Set OLLAMA_NUM_CTX=262144 in your environment or use the provided Modelfile.
  4. Initialize Client: Instantiate GemmaLocalClient with temperature=0.2 and max_tokens=2048.
  5. Run First Extraction: Pass a scanned compliance document to process_vision() with a strict JSON prompt. Verify output structure before scaling to full audit pipelines.

Deploying offline multimodal AI is no longer a compromise between capability and connectivity. Gemma 4's tiered architecture, native variable-resolution vision, and 256K context window enable complete compliance workflows to run on-premises with deterministic accuracy. The architectural shift from fragmented cloud orchestration to local synthesis reduces latency, eliminates data leakage, and restores full control over regulatory reasoning pipelines.