Zero-Downtime Singleton: How We Solved the Currency Conversion SPOF for a Global Digital Wallet
By Motti Dadison · June 10, 2025 · 10 min read
Our client serves hundreds of thousands of wallet users globally — but a currency conversion service with a single open WebSocket was their biggest single point of failure. Even 3 seconds of downtime meant financial damage. Here is how we solved it with Kubernetes leader election, hot standby, and voluntary lease handoff.
The Platform: A Global Digital Wallet at Scale
Our client operates a digital wallet platform serving hundreds of thousands of active users across multiple continents — processing payments, peer-to-peer transfers, and real-time currency conversions in over 40 fiat currencies and several cryptocurrencies. Peak traffic spikes unpredictably: a marketing campaign in Brazil, a sports event in Europe, a public holiday in Southeast Asia — and the platform must absorb all of it without a hiccup.
When they approached Softech, the infrastructure was a mix of VM-based services and some early Kubernetes workloads, but it hadn't been designed to scale gracefully under load — and one component in particular was keeping the engineering team up at night.
┌─────────────────────────────────────────────────────────┐ │ Global User Traffic │ │ (EU · US · APAC · LATAM · ME) │ └────────────────────────┬────────────────────────────────┘ │ ┌──────▼──────┐ │ CloudFlare │ CDN + DDoS protection │ WAF + LB │ └──────┬──────┘ │ ┌────────────▼────────────┐ │ AWS ALB / Ingress │ └─────┬──────────────┬───┘ │ │ ┌────────▼───┐ ┌──────▼────────┐ │ API GW │ │ Static SPA │ │ (Node.js) │ │ (CloudFront) │ └────────┬───┘ └───────────────┘ │ ┌────────────▼────────────────────┐ │ Kubernetes Cluster │ │ ┌──────────┐ ┌─────────────┐ │ │ │ Wallet │ │ Payment │ │ │ │ Service │ │ Service │ │ │ │ (HPA 3-20│ │ (HPA 2-10) │ │ │ └──────────┘ └─────────────┘ │ │ ┌──────────────────────────┐ │ │ │ Currency Conversion Svc │ │ ← The problem │ │ (exactly 1 pod, socket) │ │ │ └──────────────────────────┘ │ └─────────────────────────────────┘
Smart Scaling: The Easy Win
Most of the migration was straightforward: we containerized the stateless services, deployed them on Kubernetes with Horizontal Pod Autoscalers (HPA) tied to both CPU and custom Prometheus metrics (requests-per-second per service), and added cluster autoscaling on the node group so the underlying EC2 fleet could grow and shrink automatically.
We also implemented a KEDA (Kubernetes Event-Driven Autoscaler) setup for the payment processing queue — the number of consumer pods scales directly with the depth of the SQS queue, meaning a sudden flood of transactions doesn't cause latency spikes; it just spins up more workers within 30 seconds.
Load (req/s) │ ▲ │ ╭──────╮ │ ╭────╯ ╰────╮ │ ╭────╯ ╰────╮ │ ╭────╯ ╰── │ ─────╯ └──────────────────────────────────────────▶ time Pods ──2──╮──4──╮────8────╮──6──╮──4──╮──2── (HPA) │ │ │ │ │ scale up peak scale down idle Node ──2──╮───3───╮─────3─────╮──2──╮──2── count │ │ │ │ +1 node steady -1 node steady
The Hard Problem: The Currency Conversion Service
Currency conversion sounds simple — but the implementation had a critical constraint that made it one of the hardest engineering challenges on the project.
The service connects to a financial market data provider via a persistent WebSocket — a long-lived, stateful connection that streams live exchange rates. The data provider's contract enforced a hard rule: only one active connection per account at any time . A second connection from the same credentials immediately terminated the first. Violating this even momentarily meant:
Both connections dropping simultaneously
A reconnect storm causing temporary rate blindness
Transactions during that window being processed at stale or incorrect rates
Potential financial liability and reconciliation overhead
So the service had to run as a singleton — exactly one pod, always. But a singleton in Kubernetes is a single point of failure . A node eviction, an OOM kill, a rolling update — any of these would cause downtime. And on a financial platform serving users 24/7, even 3–5 seconds of rate unavailability translates directly into failed transactions, support tickets, and in worst cases, regulatory exposure.
The naive approach (what they had): ┌──────────────┐ WebSocket ┌───────────────────┐ │ Currency │ ══════════════════════▶ │ Market Data Feed │ │ Svc Pod │ 1 connection allowed │ (rate provider) │ └──────────────┘ └───────────────────┘ Pod crashes or evicted: ┌──────────────┐ ✗ no connection ┌───────────────────┐ │ (dead pod) │ │ Market Data Feed │ └──────────────┘ └───────────────────┘ ↓ Kubernetes creates a new pod (10–30 seconds later) ┌──────────────┐ ══════════════════════▶ ┌───────────────────┐ │ new pod │ reconnects │ Market Data Feed │ └──────────────┘ └───────────────────┘ Gap: 10–30 seconds of no exchange rates → transactions fail
The Solution: Kubernetes Leader Election with Hot Standby
The answer was to run two pods simultaneously but ensure only one of them — the elected leader — holds the WebSocket connection at any given time. The second pod stays warm (process running, dependencies initialized) but passive — it does not open a socket.
We implemented this using Kubernetes Lease objects ( coordination.k8s.io/v1 ), the same mechanism Kubernetes uses internally for control-plane leader election.
Normal operation: ┌─────────────────────┐ │ Kubernetes API │ │ Lease: "curr-conv" │ │ holder: pod-A │ │ renewTime: now │ │ leaseDuration: 15s │ └──────┬──────────────┘ │ holds lease │ watching lease ┌──────▼──────────┐ ┌──────▼──────────┐ │ Pod A (LEADER) │ │ Pod B (STANDBY)│ │ ● WS Connected │ │ ○ No socket │ │ ● Serving reqs │ │ ○ Waiting │ │ ● Renews lease │ │ ○ Ready probe │ │ every 5s │ │ returns 503 │ └──────────────────┘ └─────────────────┘ │ ══════▶│ WebSocket │ ┌──────▼───────────┐ │ Market Data Feed │ └──────────────────┘ Kubernetes Service routes ONLY to Pod A (readiness probe)
How Leader Election Works
Each pod runs a leader election loop using the Kubernetes client library. The loop works as follows:
On startup, each pod tries to create or acquire the Lease object for currency-conversion-leader in its namespace.
The pod that wins sets itself as holderIdentity and starts renewing the lease every 5 seconds .
The losing pod enters a watch loop — it monitors the lease and checks whether the current holder has missed its renewal window (15-second lease duration + 5-second buffer).
The readiness probe of each pod returns 200 only when it is the active leader. This means the Kubernetes Service endpoint only routes traffic to the leader — the standby is running but not in the load balancer rotation .
When the leader fails to renew (pod crash, node failure, OOM kill), the lease expires. The standby detects this within ~10 seconds , acquires the lease, opens the WebSocket, passes its readiness probe, and begins serving traffic.
Pod A crashes at T=0: T=0s Pod A crashes, lease stops renewing T=10s Lease expires (15s TTL - last renewal ~5s before crash) T=10s Pod B detects expired lease T=10s Pod B acquires lease, opens WebSocket T=11s Pod B readiness probe → 200 T=11s K8s Service adds Pod B to endpoints T=11s Traffic resumes ✓ Meanwhile, Kubernetes restarts Pod A: T=30s Pod A restarts, tries to acquire lease T=30s Lease held by Pod B → Pod A becomes standby T=30s Pod A readiness → 503 (not leader) Result: ~10-11 second failover window (vs 10-30s with no standby) Socket continuity: guaranteed (only one socket ever open)
Handling Rolling Updates: The Trickiest Case
Unplanned crashes are one thing — but planned rolling updates were actually the trickier scenario. The default Kubernetes rolling update behavior for a 2-replica deployment would:
Start a new pod (Pod C)
Wait for it to pass readiness
Terminate one of the old pods
But since the new pod is a standby (not leader), its readiness probe returns 503 indefinitely — Kubernetes would never consider it ready, and the rollout would stall.
We solved this with a two-phase update strategy :
Rolling update flow: Phase 1: Deploy new standby ┌───────┐ ┌───────┐ ┌───────┐ │ Pod A │ │ Pod B │ │ Pod C │ ← new version, standby │ LEADER│ │STANDBY│ │STANDBY│ │ ✓ ws │ │ ✗ ws │ │ ✗ ws │ └───────┘ └───────┘ └───────┘ Phase 2: Drain old standby (Pod B) ┌───────┐ ┌───────┐ │ Pod A │ │ Pod C │ │ LEADER│ │STANDBY│ │ ✓ ws │ │ ✗ ws │ └───────┘ └───────┘ Phase 3: Gracefully step down leader (Pod A) - Pod A releases lease voluntarily before terminating - Pod C acquires lease immediately (no expiry wait) - Pod C opens WebSocket → passes readiness → serves traffic ┌───────┐ ┌───────┐ │ Pod A │ │ Pod C │ │(term) │ │ LEADER│ │ ✗ ws │ │ ✓ ws │ └───────┘ └───────┘ Gap: < 2 seconds (voluntary handoff, no TTL wait) Phase 4: New standby for Pod A's slot ┌───────┐ ┌───────┐ │ Pod D │ │ Pod C │ ← new version fully deployed │STANDBY│ │ LEADER│ │ ✗ ws │ │ ✓ ws │ └───────┘ └───────┘
The key was adding a pre-stop lifecycle hook to the pod spec that triggers a graceful lease release:
lifecycle: preStop: exec: command: ["/app/release-leader-lock.sh"] # release-leader-lock.sh sends SIGUSR1 to the app process, # which triggers voluntary lease release before pod terminates. # terminationGracePeriodSeconds: 30
The application catches SIGUSR1 , releases the Kubernetes Lease, closes the WebSocket gracefully, and exits cleanly. The standby pod detects the released lease immediately (no TTL wait), acquires it, and opens its own socket — all within 1–2 seconds .
Additional Resilience Layers
Beyond the leader election, we added several supporting measures:
Rate caching with TTL: The leader writes each exchange rate to Redis with a 30-second TTL. If the leader goes down and the standby is still spinning up, the API falls back to the cached rate rather than failing the transaction outright — with a clear stale-rate flag in the response for the application layer to handle conservatively.
PodDisruptionBudget: A PDB ensures Kubernetes never evicts both pods simultaneously during node maintenance: minAvailable: 1 .
Pod affinity rules: The two pods are scheduled on different nodes (anti-affinity), so a single node failure never takes out both.
Alerting: A Prometheus alert fires if the leader hasn't renewed its lease for more than 8 seconds — giving the on-call engineer a heads-up before an actual failover happens.
Full resilience architecture: ┌─────────────────────────────────────────────────────┐ │ Kubernetes Cluster │ │ │ │ Node 1 Node 2 │ │ ┌─────────────────┐ ┌────────────────────┐ │ │ │ Pod A (LEADER) │ │ Pod B (STANDBY) │ │ │ │ ● WS open │ │ ○ WS closed │ │ │ │ ● writes rates │ │ ○ reads lease │ │ │ └────────┬────────┘ └────────────────────┘ │ │ │ │ │ ┌─────▼──────┐ ┌──────────────┐ │ │ │ Redis │ │ K8s Lease │ │ │ │ (rate cache│ │ coordination │ │ │ │ 30s TTL) │ │ API object │ │ │ └─────┬──────┘ └──────────────┘ │ │ │ │ │ ┌─────▼──────────────────────────┐ │ │ │ Prometheus + Alertmanager │ │ │ │ lease-age > 8s → PagerDuty │ │ │ └────────────────────────────────┘ │ │ │ │ PodDisruptionBudget: minAvailable: 1 │ │ Anti-affinity: pods on separate nodes │ └─────────────────────────────────────────────────────┘ │ ══════════▼═════════════ Market Data WebSocket Feed
Results
After deploying this architecture:
Zero financial incidents related to rate unavailability in the first year of production
Failover time reduced from 10–30 seconds (single pod restart) to under 12 seconds for unplanned failures, and under 2 seconds for planned deployments
The stale-rate Redis fallback covered the remaining gap — no transaction was ever rejected due to rate unavailability
Deployments went from high-stress events (manual socket management, off-hours windows) to routine CI/CD pipeline pushes during business hours
The platform now handles 3× the original peak load via HPA + KEDA without any manual intervention
Key Takeaways
The currency conversion challenge is a good example of a class of problems that appear often in financial systems: a resource that must be a singleton for correctness , but where that singleton is also a critical path. The pattern — leader election with a warm standby, voluntary handoff on graceful shutdown, and a cache-based fallback — is applicable to many similar situations: database change-data-capture connectors, rate limiters that maintain shared state, IoT gateways with hardware-pinned connections.
Kubernetes gives you the building blocks (Lease objects, readiness probes, PDBs, lifecycle hooks) — the engineering is in assembling them in the right order. If you're running a stateful singleton in production and your current plan is "we restart it quickly," it's worth revisiting.
Topics: Kubernetes, DevOps, FinOps, AWS, Microservices