5. Infrastructure — System Design Concepts

🏗️ 5 · INFRASTRUCTURE

Load Balancer

L4 (TCP — fast, IP+port) vs L7 (HTTP — smart, URL/headers). ALB, NLB, NGINX, HAProxy

▸ What is a Load Balancer?

▸ Layer 4 vs Layer 7

L4 — Transport Layer

L7 — Application Layer

▸ Load Balancing Algorithms

Round Robin

Weighted RR

Least Conn

IP Hash

▸ L4 vs L7 Comparison

Aspect	L4 (Transport Layer)	L7 (Application Layer)
Works On	IP + TCP/UDP — raw bytes, no content inspection	HTTP, gRPC, WebSocket — reads headers, URL, cookies
Routing By	IP address + Port only	URL path, host, method, headers, cookies
Speed	Faster — no parsing overhead	Slightly slower — inspects every request
SSL	Passthrough — encrypted traffic forwarded as-is	Termination — decrypts, inspects, re-encrypts
Sticky Sessions	IP-based only	Cookie / header based — more reliable
Content Awareness	None — blind to payload	Full — can rewrite headers, redirect, A/B route
Use Case	TCP proxying, DB connections, raw throughput, gaming	API gateway, microservices routing, canary deploys
Examples	AWS NLB, HAProxy (TCP mode), NGINX (stream)	AWS ALB, NGINX (http), Envoy, Traefik, Istio
Health Check	TCP connect only (is port open?)	HTTP /health endpoint — checks actual app response
WebSocket	Native — TCP passthrough, no extra config	Needs explicit Upgrade header proxying config

Guarantees: No single point of failure (if LB is HA). Health checks auto-remove unhealthy backends. SSL termination at L7 offloads crypto from backends.

Sticky Sessions: Route same client to same backend. Problem: hot spots, breaks on death. Better: externalize session to Redis.

API Gateway

Single entry point — auth, rate limiting, routing, SSL. Kong, Apigee, AWS API Gateway

Guarantee: API Gateway provides a single enforcement point for cross-cutting concerns — auth, rate limiting, logging, circuit breaking happen once at the edge, not duplicated in every service.

Forward & Reverse Proxy

Forward = hide client (corporate proxy). Reverse = hide servers (NGINX, Cloudflare)

Reverse Proxy Guarantees: Backend isolation — clients never see internal IPs. SSL termination — decrypt at proxy, HTTP internally. Caching — serve cached responses without hitting origin. DDoS absorption at edge.

NGINX

Reverse proxy, LB, web server, API gateway. ~34% of all websites. Event-driven, non-blocking.

Architecture

Master manages config. Workers handle all connections via event loop — no thread-per-request.

Web Server

Serves static content under high traffic. Event-driven — massive concurrency with minimal resources.

Reverse Proxy + LB

Hides backends, distributes traffic. Round Robin / Least Conn / IP Hash / Weighted.

SSL Termination

Content Cache

proxy_cache serves repeated requests without hitting backend. TTL-based eviction.

upstream backend {
    least_conn;
    server app1:8080 weight=3;
    server app2:8080;
}
server {
    listen 443 ssl;
    location /api/ { proxy_pass http://backend; }
    location / { root /var/www/html; }  # Static files directly
}

Real-world: Netflix (video delivery), Dropbox (replaced Apache, cut servers 75%), Kubernetes (default Ingress Controller). Solves C10K — event-driven workers handle 100K+ concurrent connections vs Apache's thread-per-connection.

Docker & Kubernetes

Docker packages apps into containers. Kubernetes (K8s) orchestrates them at scale.

Docker Concept	Detail
Image	Immutable template with app + dependencies. Built from Dockerfile. Stored in registry (Docker Hub, ECR).
Container	Running instance of image. Lightweight isolation (shared kernel, not full VM). Starts in seconds.
Volume	Persistent storage that survives container restarts.

K8s Concept	Detail
Pod	Smallest unit. 1+ containers sharing network/storage. Ephemeral.
Service	Stable network endpoint for pods (ClusterIP, NodePort, LoadBalancer).
Deployment	Declarative desired state. Rolling updates, rollbacks.
HPA	Horizontal Pod Autoscaler — scale on CPU/memory/custom metrics.
StatefulSet	Ordered, stable pod identities. For DBs, Kafka, ZooKeeper.
Ingress	HTTP routing rules (NGINX Ingress, Traefik). External traffic → services.

Deployment Strategies: Rolling (gradual, default) · Blue-Green (swap envs) · Canary (5% traffic first) · A/B (feature flags, Istio traffic split)

Guarantees: K8s guarantees desired state reconciliation — if a pod dies, controller restarts it. Self-healing via liveness/readiness probes. Service discovery via DNS.

Real-world: Google (Borg predecessor). Spotify 2000+ services on K8s. Managed: EKS (AWS), GKE (Google), AKS (Azure).

Service Mesh

Istio, Linkerd — sidecar proxy (Envoy) for service-to-service networking

Guarantees: mTLS everywhere (zero-trust). Automatic observability (metrics, traces per call). Traffic management (canary, retries, circuit breaking) via YAML, not code. vs API Gateway: Gateway = north-south (external→internal). Mesh = east-west (internal→internal).

Multi-Region & Multi-Tenant

Deploying across regions for low latency, disaster recovery, and compliance

Pattern	How	Trade-off
Active-Passive	Primary region serves traffic; standby for failover	Simple but standby idle; failover delay (minutes)
Active-Active	Both regions serve traffic; data replicated	Low latency globally but conflict resolution needed
Follow-the-Sun	Route to region where it's business hours	Good for support/ops workloads

Guarantees: Multi-region provides disaster recovery (entire region can fail) and data residency compliance (GDPR: EU data stays in EU). Trade-off: cross-region replication lag and conflict resolution complexity.

▸ Multi-Tenant Architecture Types

Shared App, Shared DB

Pro Cheapest, simple ops
Con Noisy neighbor, data leak risk
Ex: Salesforce, Slack

Shared App, Multi DB

Pro Strong data isolation, shared compute
Con More DB ops, connection pooling
Ex: Shopify, GitHub Enterprise

Multi App, Multi DB

Pro Full isolation, no noisy neighbor
Con Expensive, complex ops at scale
Ex: AWS accounts, dedicated SaaS

Choose: Shared/Shared for cost (Salesforce) → Shared/Multi-DB for data isolation (Shopify) → Multi/Multi for full isolation (enterprise/compliance). Most SaaS starts shared and migrates to hybrid as they scale.

Service Discovery

How services find each other in a dynamic fleet — IPs change constantly as containers scale, restart, and migrate

▸ Service Discovery — Client-Side vs Server-Side

Pattern	Who Resolves	Examples	Pros	Cons
Client-side	App library / SDK	Eureka + Ribbon, Consul SDK, gRPC name resolver	No extra hop, client LB	SDK per language, stale cache
Server-side	Load balancer / proxy	AWS ALB + Cloud Map, Envoy, Istio	Simple client, language-agnostic	Extra hop, LB is SPOF
DNS-based	Stdlib DNS resolver	K8s CoreDNS, Consul DNS, AWS Route 53	Zero SDK, universal	TTL caching, no health-aware LB
Service Mesh	Sidecar proxy (transparent)	Istio/Envoy, Linkerd, Consul Connect	Zero app changes, mTLS, observability	Complexity, resource overhead

▸ Registry Implementations

Tool	Consensus	Health Check	Key Feature
Consul	Raft	HTTP, TCP, gRPC, script	Multi-DC, service mesh (Connect), KV store
etcd	Raft	Lease-based TTL	K8s backbone, strong consistency, watch API
Eureka	AP (peer replication)	Heartbeat (30s default)	Netflix OSS, self-preservation mode
ZooKeeper	ZAB	Ephemeral nodes	Mature, Kafka/Hadoop ecosystem
AWS Cloud Map	Managed	Route 53 health checks	Native AWS, API + DNS discovery
K8s (built-in)	etcd	Liveness + readiness probes	Zero setup, CoreDNS, Endpoints API

Health checks remove dead instances within seconds — critical for fast failover. Use liveness (is it alive?) + readiness (can it serve traffic?) probes. Deregister unhealthy instances immediately, don't wait for TTL.

K8s patterns: ClusterIP — virtual IP, kube-proxy routes (default). Headless — returns all pod IPs (for stateful sets, client-side LB). ExternalName — CNAME to external service. Service Mesh — Envoy sidecar intercepts all traffic transparently.

Anti-patterns: Hardcoded IPs — breaks on any scale event. Long DNS TTL — routes to dead instances. No health checks — registry serves stale entries. Single registry without replication — SPOF.

CI/CD & Deployment Strategies

Ship safely without taking the site down — automate everything from commit to production

▸ Deployment Strategies Compared

Strategy	Downtime	Rollback Speed	Risk	Best For
Rolling	Zero	Minutes (re-roll)	Mixed versions during rollout	Stateless services, K8s default
Blue/Green	Zero	Instant (flip router)	2× cost, DB schema must be compatible	Critical services, instant rollback needed
Canary	Zero	Fast (route 0% to canary)	Slow rollout, needs good observability	High-traffic services, gradual confidence
Feature Flag	Zero	Instant (toggle off)	Flag debt, testing matrix grows	Per-user rollout, A/B testing, kill switch
Recreate	Yes	Redeploy old version	Downtime during swap	Dev/staging, stateful apps that can't run mixed

Pipeline: commit → build → unit tests → image → deploy staging → e2e → promote prod (canary 5% → 25% → 100%) → auto-rollback on SLO breach. Use GitOps (ArgoCD/Flux) for declarative, auditable deployments.

Anti-patterns: Manual deploys — error-prone, no audit trail. No rollback plan — "we'll fix forward" fails at 3am. Big-bang releases — all changes at once = impossible to debug. No staging environment — prod is your test environment.

Real-world: Netflix — Spinnaker canary with automated analysis (Kayenta). Google — 1% → 10% → 50% → 100% over days. Amazon — one-box deployment (single host first). GitHub — feature flags + Scientist for safe refactoring.

Serverless / FaaS

Pay-per-invocation compute that scales to zero — no servers to manage, auto-scales per request

▸ Serverless Execution Model — Cold Start & Warm Invocation

Good For	Bad For	Why
Event-driven glue	Long-running jobs (>15 min)	Timeout limits, cost per duration
Spiky / low-volume traffic	Sustained high RPS	Cost exceeds containers at ~1M req/day
Image/video processing	Stateful sessions / WebSockets	Stateless by design, no persistent connections
Cron jobs + queue workers	Low-latency APIs (p99 < 10ms)	Cold start adds 100ms-10s
Prototyping / MVPs	Complex orchestration	Use Step Functions for multi-step workflows

▸ Cold Start Mitigation Strategies

Reduce Cold Start

Provisioned Concurrency: pre-warm N instances ($$)
SnapStart: snapshot after init, restore on invoke (Java)
Slim runtimes: Go, Rust (10-50ms cold start)
Smaller packages: tree-shake, no unused deps
Keep-warm pings: scheduled invoke every 5 min
Init outside handler: DB connections in global scope

Serverless Platforms

AWS Lambda: most mature, 15 min max, SnapStart
GCP Cloud Functions: gen2 (Cloud Run based), 60 min
Azure Functions: durable functions for orchestration
Cloudflare Workers: V8 isolates, 0ms cold start, edge
Vercel/Netlify: frontend-focused, edge functions
Knative: K8s-native serverless (scale to zero)

Mitigate cold start: Provisioned concurrency for latency-sensitive paths. Slim runtimes (Go, Rust: 10-50ms cold start vs Java: 3-10s). SnapStart (Lambda Java). Init outside handler — DB connections, SDK clients in global scope (reused across warm invocations).

Anti-patterns: Lambda monolith — one giant function doing everything. Synchronous chains — Lambda → Lambda → Lambda (use Step Functions). VPC without NAT — adds 6-10s cold start. Ignoring concurrency limits — throttled at 1000 default.

Real-world: Netflix — Lambda for encoding pipeline triggers. Coca-Cola — vending machine backend (spiky, event-driven). iRobot — IoT event processing. Capital One — real-time fraud detection. BBC — on-demand video transcoding.

Infrastructure as Code

Version-control your cloud just like your app — reproducible, auditable, reviewable infrastructure

▸ IaC Tools — Declarative vs Imperative

Tool	Language	Approach	State	Strength
Terraform	HCL (declarative)	Plan → Apply	S3 + DynamoDB lock / TF Cloud	Multi-cloud, huge provider catalog, modules
OpenTofu	HCL (declarative)	Plan → Apply	Same as Terraform	Open-source fork, community-driven
CloudFormation	YAML/JSON	Stack-based	AWS-managed (free)	Native AWS, drift detection, StackSets
Pulumi	TS/Python/Go/C#	Real code	Pulumi Cloud / self-managed	Loops, tests, abstractions, type safety
AWS CDK	TS/Python/Java/Go	Synthesizes to CFN	CloudFormation	L2/L3 constructs, AWS-blessed patterns
Crossplane	YAML (K8s CRDs)	Reconciliation loop	K8s etcd	GitOps-native, K8s-first, compositions

▸ IaC Workflow & Best Practices

GitOps Workflow

PR: change infra code → terraform plan in CI
Review: team reviews plan diff (what will change)
Merge: terraform apply runs automatically
State: remote backend (S3 + DynamoDB lock)
Drift: detect with scheduled plan runs
Modules: reusable components (VPC, EKS, RDS)

Best Practices

Environments: separate state per env (dev/staging/prod)
Least privilege: CI role has only needed permissions
No manual changes: all changes through code
Blast radius: small stacks, not one mega-stack
Secrets: never in code — use Vault, SSM, SOPS
Testing: tflint, checkov, terratest

// Terraform example — S3 bucket with versioning
resource "aws_s3_bucket" "logs" {
  bucket = "acme-logs-prod"
  tags   = { env = "prod", team = "platform" }
}

resource "aws_s3_bucket_versioning" "logs" {
  bucket = aws_s3_bucket.logs.id
  versioning_configuration { status = "Enabled" }
}

Workflow: PR → plan in CI (diff visible in PR comment) → team review → apply on merge. State in S3 + DynamoDB lock or Terraform Cloud. Use workspaces or directory structure for environment separation.

Testing IaC: tflint — lint HCL for errors. checkov / tfsec — security scanning (open S3 buckets, missing encryption). terratest — integration tests (deploy, validate, destroy). OPA/Sentinel — policy-as-code (enforce tagging, region restrictions).

Anti-patterns: ClickOps — manual console changes that drift from code. Mega-stack — one state file for everything (slow, risky). Secrets in state — state file contains sensitive values (encrypt it). No locking — concurrent applies corrupt state.

Real-world: HashiCorp — Terraform manages millions of cloud resources globally. Shopify — CDK for AWS infrastructure. Uber — custom IaC for multi-cloud. GitLab — Terraform + GitOps for all infrastructure changes.