How Meta Turns Idle AI Inference Capacity Into Better Ads (Without Breaking Baseline)

The Inference Capacity Paradox

Every large-scale inference fleet lives with a contradiction: the fleet is globally constrained while meaningful pockets of capacity sit locally idle. Peak provisioning, model-development timing, regional resource ratios, and accelerator heterogeneity all create mismatches that strand capacity.

At Meta's scale, these temporary pockets add up to substantial compute. The question isn't whether idle capacity exists—it's whether you can use it without compromising baseline serving. This post breaks down the architecture Meta built to answer that question, and the design constraints that make it work.

The core insight: opportunistic capacity is preemptible, volatile, and carries no guarantees. Only workloads with high elasticity, sustained quality at scale, and limited cross-service dependencies can consume it safely.

(Source: Meta Engineering)

Two Complementary Elasticity Levers

Meta identified two levers that meet the structural criteria for consuming idle capacity. Each consumes a different resource mix, so together they cover more of the opportunistic pool than either alone.

Rank More Ads (RMA)

For each user request, the ads system selects a handful of ads from millions of candidates in under a second. Ranking stages progressively shrink the candidate set, with later stages evaluating only the top candidates from earlier stages.

RMA increases the number of candidates evaluated by ranking stages when opportunistic capacity is available. More candidates → higher chance of finding relevant ads. Because the extra candidates flow through CPU-heavy upstream (retrieval, feature processing) and accelerator-heavy inference, RMA needs both CPU and accelerator surplus.

Serve Better Models (SBM)

SBM changes the share of requests routed to a more powerful model as accelerator capacity fluctuates. It keeps candidate volume constant, so upstream CPU demand stays stable—meaning SBM can consume accelerator-only surplus that RMA cannot.

# Conceptual illustration of the two levers' resource profiles
levers = {
    "RMA": {"cpu": "high", "accelerator": "high", "upstream_impact": "significant"},
    "SBM": {"cpu": "low", "accelerator": "high", "upstream_impact": "minimal"},
}

# Combined coverage: RMA handles CPU+GPU surplus, SBM handles GPU-only surplus
def usable_capacity(cpu_surplus, gpu_surplus):
    rma_usable = min(cpu_surplus, gpu_surplus)   # bounded by the scarcer resource
    sbm_usable = gpu_surplus - rma_usable        # leftover GPU-only capacity
    return rma_usable + sbm_usable

Why Both Are Needed

The incremental value of each lever shifts with traffic, current coverage, and resource mix. At any moment, the next unit of opportunistic capacity might be better spent ranking more candidates or routing more requests to a powerful model. Meta periodically rebalances the allocation based on measured performance.

Key architectural takeaway: Don't build one elasticity mechanism. Build a small set of levers with disjoint resource profiles and allocate between them dynamically.

Safely Using Preemptible Capacity

Opportunistic capacity can be reclaimed faster than control loops can react. Meta's design centers on one hard requirement:

Losing opportunistic capacity may reduce incremental work—but it must never overload baseline serving capacity.

Three mechanisms enforce this:

1. Physical vs. Logical Isolation

• SBM → physical isolation. Baseline and powerful models use separate routing targets. Dedicated replicas serve baseline; opportunistic replicas serve the powerful model. When capacity contracts, traffic falls back to the baseline model whose capacity is already provisioned.
• RMA → logical isolation. The system computes how much extra ranking work opportunistic replicas can support, then sets a global budget. Baseline traffic is protected by capping total incremental work—no physical request separation needed.

2. Workload-Specific Feedback Loops

• RMA uses a central PID controller watching tail latency. Latency below target → rank more ads. Latency rising → rank fewer. Clients enforce the target on new requests.
• SBM uses a central controller for target coverage plus a client-side model router. The router segments requests by predicted engagement value and runs explore-and-exploit to measure the incremental benefit of the powerful model per segment—because incremental benefit doesn't always track absolute request value.

3. Multi-Layer Value-Aware Load Shedding

Central loops operate on the order of minutes. Preemption can hit in milliseconds. So both server and client shed load locally:

Both shedding layers use value signals to choose what to drop, minimizing quality impact during transitions.

Limitations and Cautions

• Not a general-purpose pattern. Only workloads with high elasticity and limited cross-service dependencies qualify. A tightly coupled service with strict consistency guarantees cannot safely consume preemptible capacity this way.
• Value estimation is a bottleneck. Both SBM routing and load shedding depend on request-value signals. Mis-calibrated value models will shed the wrong work.
• Explore-and-exploit has a cost. Continuously probing segments consumes real capacity and can temporarily serve suboptimal models.
• Fragmentation limits ROI. Capacity stranded on incompatible accelerator types or in regions without matching model deployments remains unusable until portability improves.

Next Steps

If you're designing a similar system, start with these questions:

1. Which of your workloads are self-contained and elastic? Batch precomputation, ranking expansion, and model-tier routing are common candidates.
2. Can you isolate baseline from optional work? Physical isolation is simpler; logical isolation requires accurate capacity accounting.
3. What's your fast-path shedding story? If your central controller is the only defense, you have a multi-minute exposure window.
4. How will you measure incremental value? Without a value signal, you can't prioritize what to shed or route.

Meta's roadmap points toward precomputation elasticity (work done ahead of request time, less bound to live traffic location), broader model portability across accelerator types, and an optimization-based allocator that automatically distributes inference capacity across elasticity levers.

The Bottom Line

Meta's system is a masterclass in using volatile resources without betting the house on them. The pattern generalizes beyond ads: isolate baseline, add elastic levers with disjoint resource profiles, close the loop with fast local shedding, and let central controllers converge slowly.

The hardest part isn't the ML—it's the capacity accounting and value signaling that make grace degradation possible.

Further reading:

• Stop Writing Monolithic AI Agent Prompts: Treat Them Like Build Artifacts
• Data Commons MCP Now Hosted on Google Cloud: Query Public Data with AI, No Setup Required