Running Local AI Models in .NET with Ollama (Step-by-Step Guide)

n:** Download Ollama from the official distribution channel.

• Model Management: Pull models using the CLI. For general-purpose tasks, llama3 offers a balance of performance and capability. For code-specific tasks, codellama or mistral variants may be preferred.

  ollama pull llama3
  

• Verification: Confirm the service is running and responsive.

  curl http://localhost:11434/api/tags
  

2. .NET Integration Architecture

The integration follows a layered approach: Configuration → Resilient Client → Service Abstraction → Controller. This separation allows swapping inference providers or adding caching without modifying business logic.

Configuration Model Define settings to externalize model selection and endpoint configuration.

public sealed class InferenceSettings
{
    public const string SectionName = "Inference";
    
    public string BaseUrl { get; set; } = "http://localhost:11434";
    public string DefaultModel { get; set; } = "llama3";
    public int TimeoutSeconds { get; set; } = 120;
    public bool EnableStreaming { get; set; } = false;
}

Resilient HTTP Client Use IHttpClientFactory with Polly policies to handle transient failures and timeouts. Local models may experience latency spikes during context loading or resource contention.

using Polly;
using Polly.Extensions.Http;

// In Program.cs or Startup
builder.Services.AddHttpClient<IInferenceClient, OllamaInferenceClient>()
    .SetHandlerLifetime(TimeSpan.FromMinutes(5))
    .AddPolicyHandler(HttpPolicyExtensions
        .HandleTransientHttpError()
        .WaitAndRetryAsync(3, retryAttempt => 
            TimeSpan.FromSeconds(Math.Pow(2, retryAttempt))))
    .ConfigureHttpClient(client =>
    {
        var settings = builder.Configuration
            .GetSection(InferenceSettings.SectionName)
            .Get<InferenceSettings>();
        
        client.BaseAddress = new Uri(settings!.BaseUrl);
        client.Timeout = TimeSpan.FromSeconds(settings.TimeoutSeconds);
    });

Service Interface and Implementation Abstract the inference logic behind an interface. This supports testing and future provider swaps.

public interface IInferenceClient
{
    Task<InferenceResult> GenerateAsync(
        string prompt, 
        string? modelOverride = null, 
        CancellationToken ct = default);
}

public sealed class OllamaInferenceClient : IInferenceClient
{
    private readonly HttpClient _http;
    private readonly InferenceSettings _settings;
    private readonly ILogger<OllamaInferenceClient> _logger;

    public OllamaInferenceClient(
        HttpClient http, 
        IOptions<InferenceSettings> settings,
        ILogger<OllamaInferenceClient> logger)
    {
        _http = http;
        _settings = settings.Value;
        _logger = logger;
    }

    public async Task<InferenceResult> GenerateAsync(
        string prompt, 
        string? modelOverride = null, 
        CancellationToken ct = default)
    {
        var model = modelOverride ?? _settings.DefaultModel;
        
        var payload = new 
        {
            model,
            prompt,
            stream = false
        };

        _logger.LogInformation("Sending inference request to model {Model}", model);

        var response = await _http.PostAsJsonAsync(
            "api/generate", 
            payload, 
            ct);

        response.EnsureSuccessStatusCode();

        var result = await response.Content
            .ReadFromJsonAsync<OllamaGenerateResponse>(ct);

        if (result?.Response is null)
        {
            throw new InvalidOperationException("Empty response from inference engine.");
        }

        return new InferenceResult(result.Response, model);
    }
}

public sealed record InferenceResult(string Content, string ModelUsed);

public sealed class OllamaGenerateResponse
{
    [JsonPropertyName("response")]
    public string? Response { get; set; }
}

Controller Implementation Expose the capability via a dedicated endpoint with validation and cancellation support.

[ApiController]
[Route("api/v1/inference")]
public class InferenceController : ControllerBase
{
    private readonly IInferenceClient _inferenceClient;

    public InferenceController(IInferenceClient inferenceClient)
    {
        _inferenceClient = inferenceClient;
    }

    [HttpPost("complete")]
    [ProducesResponseType(typeof(CompletionResponse), StatusCodes.Status200OK)]
    [ProducesResponseType(StatusCodes.Status400BadRequest)]
    public async Task<IActionResult> Complete(
        [FromBody] CompletionRequest request, 
        CancellationToken ct)
    {
        if (string.IsNullOrWhiteSpace(request.Prompt))
        {
            return BadRequest("Prompt cannot be empty.");
        }

        try
        {
            var result = await _inferenceClient.GenerateAsync(
                request.Prompt, 
                request.Model, 
                ct);

            return Ok(new CompletionResponse(result.Content, result.ModelUsed));
        }
        catch (OperationCanceledException)
        {
            return StatusCode(StatusCodes.Status408RequestTimeout, 
                "Inference request timed out.");
        }
        catch (Exception ex)
        {
            // Log exception details in production
            return StatusCode(StatusCodes.Status503ServiceUnavailable, 
                "Inference service unavailable.");
        }
    }
}

public sealed record CompletionRequest(string Prompt, string? Model);
public sealed record CompletionResponse(string Content, string Model);

3. Architecture Rationale

• Typed Clients: Using AddHttpClient<T> isolates configuration and policies per service, preventing cross-contamination of settings.
• Polly Integration: Retries and exponential backoff are essential for local models that may temporarily block due to GPU memory management or context swapping.
• Abstraction Layer: The IInferenceClient interface allows implementing fallback strategies, such as routing to a cloud provider if the local model fails or is overloaded.
• Cancellation Tokens: Propagating CancellationToken ensures that long-running inference requests can be aborted if the client disconnects, freeing resources on the Ollama server.

Pitfall Guide

Production deployments of local AI introduce unique failure modes. Address these proactively.

Pitfall	Explanation	Fix
Context Window Overflow	Local models have fixed context limits. Sending prompts exceeding this causes silent truncation or errors.	Implement token counting and prompt truncation logic before sending requests. Validate input length against model specs.
Resource Exhaustion	Ollama loads models into RAM/VRAM. Concurrent requests can exhaust memory, causing OOM kills.	Monitor Ollama metrics. Implement request queuing or concurrency limits in .NET. Use smaller models for high-concurrency scenarios.
Hardcoded Model Names	Embedding model names in code prevents runtime switching and testing.	Externalize model selection via configuration. Allow API consumers to specify models within a whitelist.
Blocking Async Calls	Using .Result or .Wait() on inference tasks can deadlock threads, especially under load.	Enforce async/await throughout the call stack. Use ConfigureAwait(false) in library code.
Ignoring Streaming	Non-streaming responses block until generation completes, leading to poor UX for long outputs.	Implement streaming support using api/generate with stream: true for interactive applications.
Lack of Health Checks	Ollama may crash or become unresponsive without detection.	Add ASP.NET Core Health Checks that ping http://localhost:11434/api/tags. Integrate with load balancers.
Prompt Injection	Local models are still susceptible to adversarial inputs that manipulate behavior.	Sanitize user inputs. Implement system prompts to constrain behavior. Consider output validation layers.

Production Bundle

Action Checklist

• Install Ollama: Deploy Ollama runtime on target hardware. Verify GPU utilization if applicable.
• Pull Models: Pre-download required models (ollama pull llama3) to avoid cold-start delays.
• Configure Settings: Add Inference section to appsettings.json with base URL, model, and timeout.
• Register Services: Implement IHttpClientFactory with Polly policies and register IInferenceClient.
• Add Health Checks: Configure health endpoints to monitor Ollama availability.
• Implement Caching: Add response caching for deterministic prompts to reduce inference load.
• Enable Telemetry: Instrument requests with OpenTelemetry for latency and error tracking.
• Security Review: Validate inputs, restrict model access, and ensure Ollama is not exposed to untrusted networks.

Decision Matrix

Scenario	Recommended Approach	Why	Cost Impact
Internal Documentation Search	Local Inference	Data sensitivity; high volume; predictable queries.	Low (Hardware amortization)
Customer-Facing Chatbot	Cloud Inference	Requires state-of-the-art reasoning; global scale.	High (Token usage)
Code Generation Pipeline	Local Inference	Proprietary code; cost control; integration with CI/CD.	Low (Hardware amortization)
Ad-Hoc Analysis Tool	Hybrid	Route sensitive data locally; use cloud for complex reasoning.	Medium (Optimized)
High-Concurrency API	Cloud or Hybrid	Local hardware may bottleneck; cloud scales elastically.	Variable

Configuration Template

appsettings.json

{
  "Inference": {
    "BaseUrl": "http://localhost:11434",
    "DefaultModel": "llama3",
    "TimeoutSeconds": 120,
    "AllowedModels": [
      "llama3",
      "mistral",
      "codellama"
    ]
  }
}

Program.cs Registration

var inferenceSettings = builder.Configuration
    .GetSection(InferenceSettings.SectionName)
    .Get<InferenceSettings>();

builder.Services.Configure<InferenceSettings>(
    builder.Configuration.GetSection(InferenceSettings.SectionName));

builder.Services.AddHttpClient<IInferenceClient, OllamaInferenceClient>()
    .AddPolicyHandler(GetRetryPolicy())
    .ConfigureHttpClient(client =>
    {
        client.BaseAddress = new Uri(inferenceSettings!.BaseUrl);
        client.Timeout = TimeSpan.FromSeconds(inferenceSettings.TimeoutSeconds);
    });

builder.Services.AddHealthChecks()
    .AddUrlGroup(
        new Uri($"{inferenceSettings.BaseUrl}/api/tags"),
        name: "ollama-health",
        timeout: TimeSpan.FromSeconds(5));

static IAsyncPolicy<HttpResponseMessage> GetRetryPolicy() =>
    HttpPolicyExtensions
        .HandleTransientHttpError()
        .OrResult(msg => msg.StatusCode == System.Net.HttpStatusCode.ServiceUnavailable)
        .WaitAndRetryAsync(3, retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt)));

Quick Start Guide

1. Initialize Ollama: Run ollama pull llama3 and ensure the service is active on port 11434.
2. Create Project: Execute dotnet new webapi -n LocalInferenceApi.
3. Add Dependencies: Install Microsoft.Extensions.Http.Resilience for Polly integration.
4. Implement Code: Copy the configuration, client, and controller patterns from this guide.
5. Test: Run the application and send a POST request to /api/v1/inference/complete with a JSON payload containing a prompt. Verify the response returns generated content.