Netflixs Solution to Wide Partitions in Cassandra for Time Series Workloads

The Challenge of Wide Partitions

When you're storing petabytes of time series data in Apache Cassandra, few things can ruin your day like a wide partition. As events accumulate over time, partitions can balloon into gigabytes, causing read latencies to spike into seconds, timeouts, and even cluster instability.

Netflix's TimeSeries Abstraction team faced this exact problem. Their system ingests millions of events per second, and while Cassandra handles the write throughput admirably, the read side starts to suffer when certain partitions grow too large. The typical response might be to throw more hardware at it, but that's expensive and doesn't address the root cause.

Instead, they built a dynamic partitioning system that automatically detects and splits wide partitions at the individual ID level. This approach reduced average read latency for wide partitions from seconds to low double-digit milliseconds, and tail latency from several seconds to around 200ms.

Why Wide Partitions Are a Problem

Cassandra is designed for high write throughput, but reads can become problematic when a single partition holds too much data. Here's what happens:

• High read latency: Reading a multi-gigabyte partition requires scanning a lot of data, pushing latency into the seconds.
• Garbage collection pauses: Large heap allocations during reads can trigger frequent GC pauses.
• Thread queueing: When many requests target the same wide partition, threads pile up waiting for the scan to finish.

These issues can cascade into timeouts and even unavailability. For time series data, partitions naturally grow over time, making this a persistent challenge.

Solution 1: Time Slice Re-Partitioning

The first approach was to adjust partitioning at the table level. Netflix uses discrete time slices, where each slice can have its own partitioning strategy. By monitoring partition sizes via Cassandra's nodetool tablehistograms, they could detect when partitions were too small or too large.

A background worker computes an adjustment factor and updates the time bucket interval for future slices. Here's a simplified example of how the detection and adjustment logic might work:

import subprocess
import json

def get_partition_histogram(table_name):
    """Fetch partition size histogram from nodetool"""
    output = subprocess.check_output(["nodetool", "tablehistograms", table_name])
    return parse_histogram(output)

def adjust_time_bucket(histogram, target_size_mb):
    """Compute new time bucket interval based on observed partition sizes"""
    p99_size_mb = histogram["p99"] / (1024 * 1024)
    if p99_size_mb < target_size_mb:
        # Increase bucket interval to make partitions larger
        new_interval = current_interval * (target_size_mb / p99_size_mb)
    else:
        # Decrease interval to make partitions smaller
        new_interval = current_interval / (p99_size_mb / target_size_mb)
    return new_interval

This worked well for datasets where most partitions were misconfigured. But it failed when only a small percentage of IDs generated excessive data. In those cases, re-partitioning the entire table would over-partition the majority of normal IDs.

Solution 2: Dynamic Partitioning per ID

For the outlier problem, Netflix built an asynchronous pipeline that splits wide partitions at the ID level. It has three stages:

1. Detection: Every read tracks bytes read per partition. If the bytes exceed a threshold, an event is sent to Kafka.
2. Planning & Splitting: A planner reads the entire partition to compute an optimal split plan, then delegates splitting to a strategy that distributes data across multiple buckets.
3. Serving Reads: The server uses Bloom filters to quickly check if a partition has been split, then routes reads to the smaller pieces.

Here's a conceptual example of the detection event and split metadata:

{
  "time_slice": "data_20260328",
  "time_series_id": "profileId:123",
  "time_bucket": 7,
  "event_bucket": 2,
  "immutable": true,
  "version": "0"
}

{
  "pre_split_data": {
    "time_slice": "data_20260328",
    "time_series_id": "6313825",
    "time_bucket": 0,
    "event_bucket": 2
  },
  "post_split_data": {
    "time_slice": "wide_data_20260328_0",
    "event_bucket_partition_strategy": {
      "target_event_buckets": 2,
      "start_event_bucket": 32
    }
  }
}

Checksums ensure split integrity, and the original partition is never deleted, providing a safe fallback.

Key Lessons and Takeaways

Netflix's journey offers valuable insights for anyone dealing with similar scalability challenges:

• Reduce surface area: Start with simpler solutions that still deliver impact. They first tried table-level re-partitioning before tackling per-ID splitting.
• Build confidence: Invest in validation mechanisms like checksums and phased rollouts to ensure correctness before full deployment.
• Monitor and adapt: Use introspection tools to continuously monitor partition health and adjust strategies dynamically.

Limitations and Considerations

This approach is not a silver bullet. Splitting mutable partitions is still complex and not yet supported. Also, detection relies on reads, so there's a short window where some reads may still hit the wide partition. For extreme cases, they implemented a 'Partial Return' feature that aborts requests exceeding latency SLOs.

Next Steps for Learning

If you're dealing with wide partitions in Cassandra, start by monitoring your partition sizes and read latencies. Consider whether you can adjust your partitioning strategy at the table level first. If you have outlier IDs, think about implementing a similar detection and split pipeline. For more on the underlying partitioning strategy, check out their previous Netflix Tech Blog.

Also, explore related topics like Python 3.15 Alpha 5 features or Python 3.14.3 release highlights for more tech insights.