Backend service discovery

ritical or unresponsive before routing.

Implementation Steps

1. Define Service Metadata Schema: Standardize how services register themselves (host, port, metadata, tags).
2. Implement Registry Client: Create a client that handles registration, deregistration, and querying.
3. Build Discovery Manager: Develop a manager that caches results, subscribes to updates, and provides resolved endpoints.
4. Integrate Load Balancer: Apply a strategy (Round-Robin, Random, or Least-Connections) within the client.

TypeScript Implementation

The following code demonstrates a robust ServiceDiscoveryClient with caching, health filtering, and a configurable load balancer.

import { EventEmitter } from 'events';

// Types
interface ServiceInstance {
  id: string;
  host: string;
  port: number;
  tags: string[];
  status: 'passing' | 'warning' | 'critical';
  metadata: Record<string, string>;
}

interface DiscoveryConfig {
  registryUrl: string;
  cacheTTL: number; // milliseconds
  healthCheckInterval: number; // milliseconds
}

// Load Balancing Strategy
type LoadBalancer = (instances: ServiceInstance[]) => ServiceInstance;

const roundRobin = (): LoadBalancer => {
  let index = 0;
  return (instances) => {
    if (instances.length === 0) throw new Error('No healthy instances');
    const instance = instances[index % instances.length];
    index++;
    return instance;
  };
};

export class ServiceDiscoveryClient extends EventEmitter {
  private cache: Map<string, { instances: ServiceInstance[]; timestamp: number }> = new Map();
  private config: DiscoveryConfig;
  private loadBalancer: LoadBalancer;

  constructor(config: DiscoveryConfig, loadBalancer: LoadBalancer = roundRobin()) {
    super();
    this.config = config;
    this.loadBalancer = loadBalancer;
  }

  /**
   * Register a service instance with the registry.
   */
  async register(instance: ServiceInstance): Promise<void> {
    const payload = {
      id: instance.id,
      address: instance.host,
      port: instance.port,
      tags: instance.tags,
      meta: instance.metadata,
      check: {
        http: `http://${instance.host}:${instance.port}/health`,
        interval: `${this.config.healthCheckInterval}ms`,
        timeout: '5s',
        deregister_critical_service_after: '90s',
      },
    };

    try {
      await fetch(`${this.config.registryUrl}/v1/agent/service/register`, {
        method: 'PUT',
        headers: { 'Content-Type': 'application/json' },
        body: JSON.stringify(payload),
      });
      this.emit('registered', instance);
    } catch (error) {
      this.emit('error', 'Registration failed', error);
      throw error;
    }
  }

  /**
   * Deregister a service instance.
   */
  async deregister(instanceId: string): Promise<void> {
    await fetch(`${this.config.registryUrl}/v1/agent/service/deregister/${instanceId}`, {
      method: 'PUT',
    });
    this.emit('deregistered', instanceId);
  }

  /**
   * Resolve healthy instances for a service name.
   * Implements local caching with TTL and health filtering.
   */
  async resolve(serviceName: string): Promise<ServiceInstance[]> {
    const cached = this.cache.get(serviceName);
    const now = Date.now();

    // Return cache if valid
    if (cached && now - cached.timestamp < this.config.cacheTTL) {
      return cached.instances;
    }

    // Fetch from registry
    try {
      const response = await fetch(
        `${this.config.registryUrl}/v1/health/service/${serviceName}?passing=true`
      );
      
      if (!response.ok) {
        throw new Error(`Registry query failed: ${response.statusText}`);
      }

      const data = await response.json();
      const instances: ServiceInstance[] = data.map((entry: any) => ({
        id: entry.Service.ID,
        host: entry.Service.Address,
        port: entry.Service.Port,
        tags: entry.Service.Tags,
        status: 'passing', // Filtered by ?passing=true
        metadata: entry.Service.Meta || {},
      }));

      this.cache.set(serviceName, { instances, timestamp: now });
      this.emit('resolved', serviceName, instances);
      return instances;
    } catch (error) {
      this.emit('error', 'Resolution failed', error);
      // Fallback to stale cache if available and within extended TTL
      if (cached && now - cached.timestamp < this.config.cacheTTL * 3) {
        return cached.instances;
      }
      throw error;
    }
  }

  /**
   * Get a single instance using the load balancer.
   */
  async getInstance(serviceName: string): Promise<ServiceInstance> {
    const instances = await this.resolve(serviceName);
    return this.loadBalancer(instances);
  }

  /**
   * Clear cache for a service.
   */
  invalidate(serviceName: string): void {
    this.cache.delete(serviceName);
  }
}

Usage Example

const discovery = new ServiceDiscoveryClient({
  registryUrl: 'http://consul:8500',
  cacheTTL: 30000,
  healthCheckInterval: 10000,
});

// Register current service
await discovery.register({
  id: 'user-service-1',
  host: '10.0.1.5',
  port: 3000,
  tags: ['v1', 'production'],
  status: 'passing',
  metadata: { region: 'us-east-1' },
});

// Discover another service
const orderServiceInstance = await discovery.getInstance('order-service');
console.log(`Routing to: ${orderServiceInstance.host}:${orderServiceInstance.port}`);

Pitfall Guide

1. Stale Cache Propagation:

   • Mistake: Caching discovery results indefinitely or with excessive TTLs without a watch mechanism.
   • Impact: Clients continue routing to terminated instances, causing connection timeouts and user-facing errors.
   • Fix: Implement TTL-based expiration combined with registry watches or short-lived leases to invalidate caches immediately upon topology changes.

2. Health Check Storms:

   • Mistake: All health checks in a large cluster fire simultaneously.
   • Impact: Registry overload, CPU spikes on target services, and potential denial of service.
   • Fix: Jitter health check intervals. Use randomized offsets so checks are distributed over time.

3. Ignoring CAP Theorem Trade-offs:

   • Mistake: Choosing a strongly consistent (CP) registry for a system requiring high availability (AP) during network partitions.
   • Impact: Service discovery becomes unavailable during partitions, halting all new service communication.
   • Fix: Align registry choice with system requirements. Use AP systems (like Eureka) for high availability or CP systems (like etcd/Consul) where consistency is paramount. Configure fallbacks for partitions.

4. Synchronous Blocking Resolution:

   • Mistake: Performing discovery lookups synchronously on the request path without caching.
   • Impact: High latency per request, registry bottleneck, and cascading failures if the registry slows down.
   • Fix: Always use local caching. Pre-warm caches during service startup. Ensure resolution is asynchronous and non-blocking.

5. Security Exposure:

   • Mistake: Exposing the service registry to public networks or unauthenticated internal traffic.
   • Impact: Attackers can enumerate internal services, identify vulnerable endpoints, or deregister services to cause outages.
   • Fix: Restrict registry access to mTLS or API tokens. Never expose registry ports externally. Use network policies to limit access to authorized services only.

6. Version Mismatch Routing:

   • Mistake: Routing traffic between incompatible service versions without metadata filtering.
   • Impact: Protocol errors, data corruption, or feature failures.
   • Fix: Use metadata tags (e.g., version, api-compat) in service registration. Clients must filter instances based on required compatibility metadata during resolution.

7. Lack of Observability:

   • Mistake: No metrics on discovery latency, cache hit rates, or resolution failures.
   • Impact: Inability to diagnose routing issues or performance degradation.
   • Fix: Instrument the discovery client. Export metrics for cache hits/misses, registry query latency, and instance count changes. Set alerts on resolution failures.

Production Bundle

Action Checklist

• Define Metadata Schema: Establish standard tags and metadata fields (version, region, tier) for all services.
• Select Discovery Pattern: Choose client-side, server-side, or DNS based on latency, complexity, and stack constraints.
• Implement Health Checks: Configure HTTP/TCP health endpoints with jittered intervals and automatic deregistration.
• Deploy Registry Cluster: Run the registry in a high-availability configuration with quorum requirements.
• Secure Access: Enforce authentication and authorization on registry API endpoints.
• Add Client Caching: Ensure all discovery clients implement local caching with TTL and invalidation.
• Instrumentation: Add metrics for discovery latency, cache performance, and instance health status.
• Failure Testing: Conduct chaos engineering tests to verify behavior during registry unavailability and node failures.

Decision Matrix

Scenario	Recommended Approach	Why	Cost Impact
Kubernetes Native	Server-Side (K8s Services/Ingress)	Platform manages IP/DNS; minimal client code required.	Low infrastructure cost; potential LB hop latency.
Polyglot Microservices	Client-Side with Sidecar (e.g., Envoy)	Decouples logic from app code; supports diverse languages.	Medium complexity; sidecar resource overhead.
Ultra-Low Latency	Client-Side Direct	Eliminates proxy hop; local cache enables sub-millisecond resolution.	High client complexity; requires library maintenance.
Legacy Monolith Migration	DNS-Based + Service Mesh	Eases transition; DNS is familiar; mesh adds gradual discovery capabilities.	Low immediate cost; mesh adds long-term operational overhead.
Multi-Region Active-Active	Global Service Mesh	Handles cross-region routing, latency awareness, and failover automatically.	High infrastructure cost; complex configuration.

Configuration Template

Consul Service Definition (service.hcl):

service {
  name = "payment-service"
  port = 8080
  tags = ["v2", "production", "pci-dss"]
  
  meta = {
    region     = "us-west-2"
    api-version = "2.1"
  }

  check {
    http     = "http://localhost:8080/health"
    interval = "10s"
    timeout  = "2s"
    deregister_critical_service_after = "60s"
  }
}

Docker Compose for Local Registry:

version: '3.8'
services:
  consul:
    image: consul:1.15
    command: agent -dev -ui -client=0.0.0.0
    ports:
      - "8500:8500"
      - "8600:8600/udp"
    environment:
      CONSUL_BIND_INTERFACE: eth0

Quick Start Guide

1. Spin up Registry: Run docker compose up -d using the template above to start a local Consul agent. Access the UI at http://localhost:8500.
2. Initialize Client: Instantiate ServiceDiscoveryClient in your TypeScript application pointing to http://localhost:8500.
3. Register Service: Call discovery.register() with your service details. Verify the service appears in the Consul UI.
4. Resolve & Route: Call discovery.getInstance('target-service'). The client will query the registry, cache the result, and return a healthy instance.
5. Verify Resilience: Stop the target service container. Wait for the health check interval. Query getInstance again; the client should return an error or empty list, confirming health filtering works.