How to Build Resilient WebSocket Workers on AWS with DynamoDB Leases

Why Long-Lived WebSocket Connections Are Hard to Manage

HTTP requests are stateless: a request arrives, gets processed, and returns a response. WebSocket connections are the opposite. They are persistent, bidirectional channels that must stay open, process incoming messages, and respond to keep-alive pings. This statefulness introduces serious operational challenges when you run a fleet of workers:

• Worker failures: When a worker crashes, it drops all its WebSocket connections. Without a mechanism to detect the failure and reassign those connections, data is lost.
• Rolling deployments: ECS terminates old tasks and starts new ones. Each terminated task drops its connections, creating a window where connections have no owner.
• Horizontal scaling: Adding workers is easy, but removing them requires draining connections and verifying other workers take over before the task exits.
• Double-claiming: If two workers both think they own the same connection, you get duplicate data processing or protocol errors.

Because workers are WebSocket clients initiating outbound connections, you need coordination at the application layer, not the network layer. The solution: DynamoDB conditional writes as a distributed lease.

This guide is based on the AWS Architecture Blog post Building resilient real-time streaming workers with Amazon DynamoDB leases.

The Lease Pattern: Core Implementation

A lease is a DynamoDB item that tracks who owns a connection and when that ownership expires. The table schema:

A GSI on desired_state (partition key) and lease_expires_at_ms (sort key) lets you query for orphaned connections efficiently.

Acquire: Conditional Write as a Distributed Lock

The critical piece is the ConditionExpression. It succeeds only if the lease doesn't exist yet (attribute_not_exists) or has already expired (lease_expires_at_ms < :now). DynamoDB evaluates this atomically, so only one worker wins.

import boto3
from botocore.exceptions import ClientError

dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('connections')

def try_acquire_lease(conn_id: str, worker_id: str, lease_seconds: int = 20):
    now_ms = int(time.time() * 1000)
    try:
        table.update_item(
            Key={'Pk': f'CONN#{conn_id}'},
            UpdateExpression='SET lease_owner = :w, lease_expires_at_ms = :exp',
            ConditionExpression='attribute_not_exists(lease_owner) OR lease_expires_at_ms < :now',
            ExpressionAttributeValues={
                ':w': worker_id,
                ':exp': now_ms + lease_seconds * 1000,
                ':now': now_ms,
            },
        )
        return True
    except ClientError as e:
        if e.response['Error']['Code'] == 'ConditionalCheckFailedException':
            return False  # 다른 워커가 이미 소유 중
        raise

Renew: Heartbeat Loop

The owning worker renews its lease every HEARTBEAT_EVERY seconds. If renewal returns False, the worker knows it lost ownership and exits cleanly.

async def heartbeat_loop(conn_id: str, worker_id: str):
    while not shutdown_event.is_set():
        ok = await renew_lease(conn_id, worker_id)
        if not ok:
            print(f'Lease lost for {conn_id}, exiting')
            return
        await asyncio.sleep(HEARTBEAT_EVERY)  # 5초

Release: Graceful Shutdown

During graceful shutdown, set lease_expires_at_ms = 0 so other workers pick up the connection on their next reconciliation cycle instead of waiting for natural expiration.

async def graceful_shutdown():
    shutdown_event.set()
    await asyncio.gather(*[
        release_lease(conn_id, worker_id) for conn_id in active_connections
    ])

Orphan Reconciliation and Scaling

The reconciliation loop is the safety net. Every 60 seconds, each worker queries the GSI for records where desired_state = STARTED and lease_expires_at_ms < now. These are orphaned connections — the previous owner crashed without releasing. Any healthy worker can then call try_acquire_lease on them. Because the acquire uses a conditional write, multiple workers can safely run reconciliation concurrently.

Recovery time budget: 20s lease expiry + up to 60s reconciliation interval = ~80s worst-case. Graceful shutdown cuts this to one reconciliation cycle.

Scaling with CloudWatch Metrics

Each worker publishes an ActiveConnections custom metric. An Application Auto Scaling target tracking policy scales the fleet based on the average. Scale-in is safe because SIGTERM triggers graceful shutdown, releasing leases for immediate reacquisition.

Cost Optimization

Heartbeat writes dominate cost: one update_item per connection per 5s = 1 WCU. At 500 connections, that's ~$325/month on-demand vs ~$47/month provisioned. For sustained high counts, use provisioned capacity with Auto Scaling. Doubling the heartbeat interval to 10s halves WCU consumption — just double the lease duration to maintain the 4:1 ratio.

⚠️ Limitations and Caveats

• Clock skew: Lease expiration uses worker-supplied now, not a server-side clock. AWS Fargate provides Amazon Time Sync, keeping skew within milliseconds, but if you deploy outside AWS, verify NTP and increase lease duration by max expected skew.
• Not a general lock client: The amazon-dynamodb-lock-client library exists for Java, but it doesn't integrate domain state into the lock record. This pattern combines lock + metadata in one item to reduce reads.
• SQS is not enough alone: SQS delivers a unit of work to one consumer, but WebSocket ownership is a continuous state that must be renewed. SQS cannot query for connections with no active owner.

Next Steps

• Add distributed tracing with AWS X-Ray for end-to-end visibility across workers.
• Implement reconnection logic with upstream replay or offset-based resumption using last_seq to handle data gaps.
• For deeper insights on minimizing runtime inference costs in similar distributed systems, see Stop Wasting GPU Cycles: Why Code Agents Beat Tool-Calling for In-Game AI.

Key Takeaways

1. DynamoDB conditional writes are a distributed lock — no external coordination service needed. The ConditionExpression on update_item guarantees exactly one worker owns each connection.
2. Heartbeat + reconciliation covers the full failure spectrum — lease expiration handles crashes, graceful shutdown handles deployments, and new workers acquire leases while departing workers release them.
3. This pattern applies broadly — real-time transcription, IoT ingestion, financial feeds, live event streaming — any system managing long-lived WebSocket connections at scale.

Related Reading

If you're optimizing cloud infrastructure performance, you might also enjoy PostgreSQL on Azure Just Got a Serious Upgrade Inside VS Code — a deep dive into database tooling improvements that pair well with the backend patterns discussed here.

The complete implementation — worker application, Lambda event router, and Terraform templates — is available in the AWS samples repository. Deploy the infrastructure and validate the lease lifecycle with a small set of test connections before scaling up.