C# LINQ performance

[MemoryDiagnoser]
[SimpleJob(RuntimeMoniker.Net90)]
public class LinqPerformanceBenchmarks
{
    private int[] _data = Enumerable.Range(0, 1_000_000).ToArray();

    [Benchmark]
    public List<int> StandardLinq() =>
        _data.Where(x => x % 2 == 0).Select(x => x * x).ToList();

    [Benchmark]
    public List<int> PreallocatedForeach()
    {
        var result = new List<int>(_data.Length / 2);
        foreach (var x in _data)
            if (x % 2 == 0) result.Add(x * x);
        return result;
    }

    [Benchmark]
    public Span<int> SpanLoop()
    {
        var span = _data.AsSpan();
        var result = new int[span.Length / 2];
        int idx = 0;
        for (int i = 0; i < span.Length; i++)
        {
            if (span[i] % 2 == 0)
                result[idx++] = span[i] * span[i];
        }
        return result.AsSpan(0, idx);
    }
}

// ❌ Hidden cost: Enumerates _source twice
var evenCount = _source.Where(x => x % 2 == 0).Count();
var evenSums = _source.Where(x => x % 2 == 0).Sum();

// ✅ Materialize once
var evenNumbers = _source.Where(x => x % 2 == 0).ToArray();
var evenCount = evenNumbers.Length;
var evenSums = evenNumbers.Sum();

public ref struct EvenPredicate
{
    public readonly bool I

nvoke(int value) => value % 2 == 0; }

// Custom extension using Span + ref struct public static Span<T> Filter<T, TPredicate>(this Span<T> source, ref TPredicate predicate) where TPredicate : struct { var result = new T[source.Length]; int idx = 0; for (int i = 0; i < source.Length; i++) { if (predicate.Invoke(source[i])) result[idx++] = source[i]; } return result.AsSpan(0, idx); }

### Step 4: Integrate `ArrayPool<T>` for Hot Paths
When LINQ readability is required but allocation budgets are tight, reuse buffers.

```csharp
using System.Buffers;

public static List<T> ToListWithPool<T>(this IEnumerable<T> source, int estimatedCapacity)
{
    var pool = ArrayPool<T>.Shared;
    var buffer = pool.Rent(estimatedCapacity);
    var list = new List<T>(estimatedCapacity);
    int index = 0;
    foreach (var item in source)
    {
        if (index == buffer.Length)
        {
            list.AddRange(buffer);
            pool.Return(buffer);
            buffer = pool.Rent(buffer.Length * 2);
            index = 0;
        }
        buffer[index++] = item;
    }
    list.AddRange(buffer.AsSpan(0, index));
    pool.Return(buffer);
    return list;
}

Step 5: Async LINQ Optimization

IAsyncEnumerable<T> replaces Task<IEnumerable<T>> to avoid buffering entire result sets in memory.

// ❌ Buffers all results before returning
public async Task<List<int>> GetAsyncData() =>
    await _dbContext.Orders.Where(o => o.Status == "Pending").ToListAsync();

// ✅ Streams results, reduces peak memory
public async IAsyncEnumerable<int> StreamAsyncData()
{
    await foreach (var order in _dbContext.Orders.Where(o => o.Status == "Pending").AsAsyncEnumerable())
        yield return order.Id;
}

Architecture Rationale:

• Keep LINQ for cold paths, configuration parsing, and low-frequency business logic where readability outweighs allocation cost.
• Switch to Span<T>/Memory<T> + manual loops for hot paths processing >100K items/sec or requiring sub-10ms latency.
• Use ArrayPool<T> when LINQ composition is non-negotiable but allocation budgets are strict.
• Reserve AsParallel() for CPU-bound operations on datasets >50K elements with independent computations.

Pitfall Guide

1. Chaining .ToList() or .ToArray() Unnecessarily

Materializing intermediate results breaks deferred execution and forces full enumeration. Each .ToList() allocates a new array and copies elements. Fix: Keep sequences deferred until the final materialization point. Use .AsEnumerable() or .AsQueryable() to preserve pipeline composition.

2. Multiple Enumeration of Deferred Sequences

IEnumerable<T> does not cache results. Calling .Count(), .Any(), or iterating twice executes the underlying query/provider multiple times. Fix: Materialize once with .ToArray() or .ToList() when multiple operations are required. Document enumeration expectations in method contracts.

3. AsParallel() Misuse on Small or Contention-Heavy Workloads

ParallelEnumerable partitions work across thread pool threads. For datasets under 10K elements, partitioning and synchronization overhead exceeds parallel gain. Shared state or database calls inside AsParallel() cause lock contention. Fix: Profile with and without parallelism. Use Parallel.ForEach with ParallelOptions.MaxDegreeOfParallelism for controlled concurrency. Prefer IAsyncEnumerable for I/O-bound streams.

4. Ignoring Enumerator Allocation Differences

foreach over List<T> allocates a reference-type enumerator. foreach over arrays or Span<T> uses a value-type enumerator with zero allocations. Chaining LINQ methods always allocates enumerator wrappers. Fix: Prefer Span<T> or arrays in hot loops. Use CollectionsMarshal.AsSpan() for List<T> when direct memory access is safe and performance-critical.

5. SelectMany with Nested LINQ Creating Closure Allocations

SelectMany(x => x.Items.Where(...)) captures outer variables and allocates delegate chains per iteration. Fix: Flatten manually or use struct-based projections. Pre-allocate target collections when the total count is predictable.

6. String Comparisons in LINQ Without ReadOnlySpan<char>

Where(s => s.Contains("value")) allocates substrings and uses culture-sensitive comparisons by default. Fix: Use ReadOnlySpan<char> with IndexOf or SequenceEqual. Specify StringComparison.Ordinal for case-sensitive, culture-invariant matches.

7. Assuming FirstOrDefault Always Bypasses Allocation

FirstOrDefault exits early, but the query pipeline still allocates enumerators and delegates up to the matching element. In tight loops, this overhead accumulates. Fix: For hot-path lookups, switch to Dictionary<TKey, TValue> or Span<T> binary search. Reserve FirstOrDefault for infrequent or non-latency-sensitive queries.

Production Bundle

Action Checklist

• Instrument LINQ-heavy paths with BenchmarkDotNet and measure allocations, not just execution time
• Audit deferred sequences for multiple enumeration; materialize once when reused
• Replace Func<T, bool> delegates with ref struct predicates or compile-time expressions in hot loops
• Integrate ArrayPool<T> for LINQ chains that cannot be refactored to Span<T>
• Validate AsParallel() usage with load tests; disable when p99 latency increases
• Enforce StringComparison.Ordinal in all LINQ string predicates
• Document allocation budgets and LINQ usage thresholds in team architecture guidelines
• Run production memory profiling under sustained load to detect Gen 2 promotion patterns

Decision Matrix

Scenario	Recommended Approach	Why	Cost Impact
< 10K items, infrequent execution	Standard LINQ chains	Readability outweighs allocation cost	Negligible
10K–100K items, moderate throughput	Pre-allocated List<T> + foreach	Eliminates enumerator/delegate overhead	~30% reduction in Gen 0 collections
> 100K items, sub-10ms latency requirement	Span<T> + manual loop	Zero allocation, JIT inlining, cache-friendly	~60% faster, zero heap pressure
LINQ composition required but budget tight	ArrayPool<T> hybrid materialization	Reuses buffers, maintains query readability	Caps allocation growth, reduces GC frequency
I/O-bound streaming (DB, API, files)	IAsyncEnumerable<T> + yield return	Avoids full result buffering, backpressure-aware	Reduces peak memory by 40–70%
CPU-heavy independent computations > 50K items	ParallelEnumerable with controlled degree	Leverages multi-core without thread pool starvation	Linear scaling until memory bandwidth limit

Configuration Template

// BenchmarkDotNet Configuration for LINQ Performance Testing
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Jobs;

[MemoryDiagnoser]
[GCServer(true)]
[HardwareCounters(HardwareCounter.CacheMisses)]
[SimpleJob(
    runtimeMoniker: RuntimeMoniker.Net90,
    launchCount: 3,
    warmupCount: 5,
    iterationCount: 20)]
[RankColumn]
[Orderer(BenchmarkDotNet.Order.SummaryOrderPolicy.FastestToSlowest)]
public class LinqOptimizationBenchmarkConfig
{
    // Apply this attribute to benchmark classes
    // Ensures consistent GC mode, hardware counter tracking, and statistical reliability
}

// Reusable Hot-Path LINQ Replacement Template
public static class HotPathDataProcessor
{
    public static Span<TOutput> Process<TInput, TOutput>(
        ReadOnlySpan<TInput> input,
        Func<TInput, bool> predicate,
        Func<TInput, TOutput> selector,
        TOutput[] outputBuffer)
    {
        int idx = 0;
        for (int i = 0; i < input.Length; i++)
        {
            if (predicate(input[i]))
                outputBuffer[idx++] = selector(input[i]);
        }
        return outputBuffer.AsSpan(0, idx);
    }
}

Quick Start Guide

1. Install BenchmarkDotNet: Run dotnet add package BenchmarkDotNet in your project. Create a benchmark class mirroring your LINQ pipeline.
2. Add Memory Diagnostics: Apply [MemoryDiagnoser] and [GCServer(true)] to capture allocation counts and generation behavior accurately.
3. Execute Baseline: Run dotnet run -c Release to generate baseline metrics. Record execution time, bytes allocated, and GC collection counts.
4. Apply Targeted Refactor: Replace the LINQ chain with the recommended approach from the Decision Matrix. Keep the method signature identical to ensure behavioral parity.
5. Validate & Deploy: Re-run benchmarks. If allocations drop ≥30% and latency improves ≥20%, integrate the pattern. Add allocation thresholds to CI pipelines using dotnet-counters or PerfView to prevent regression.