From 6 Hours to 24 Hours How Adobe Firefly Scaled GPU Observability with Amazon Managed Service for Prometheus

Observability Is the Hidden Bottleneck of AI Training

Scaling AI training infrastructure is not just about adding more GPUs. While most of the spotlight falls on model architecture and compute allocation, the unsung hero of efficient AI development is observability. Without a clear view into GPU utilization, memory consumption, and network throughput, engineers are flying blind, unable to identify bottlenecks or optimize resource allocation.

Adobe Firefly, the company's generative AI suite now powering creative tools in Photoshop and Illustrator, faced this exact challenge. Their GPU training infrastructure, built on Amazon EKS, generates telemetry at a staggering scale. With training jobs running across thousands of nodes and tens of thousands of GPUs, the sheer volume of metrics quickly overwhelmed their initial observability setup.

This deep dive explores how Adobe evolved from a self-managed Prometheus deployment to a hybrid architecture incorporating Amazon Managed Service for Prometheus. The result: 28x faster queries and a 4x extension in observability windows, changing how infrastructure teams can support AI workloads.

The Challenge: High-Cardinality Metrics at Massive Scale

GPU training clusters are not typical IT workloads. The telemetry they produce is high-cardinality, meaning it spans multiple dimensions simultaneously. A single training job might need to track GPU health, compute utilization, memory pressure, and network I/O simultaneously.

To understand the scale, consider this: a training job running on 2,000 nodes with 16,000 GPUs, scraped every 30 seconds, can generate over 1 billion data points in a single query window. Traditional monitoring approaches, even when using a self-managed Prometheus, struggle to deliver query results at this scale within acceptable timeframes.

Adobe's original infrastructure relied on a self-hosted Prometheus setup, sending data to remote storage for long-term retention. As Firefly adoption grew, this architecture began to show critical limitations:

• Query timeouts: Complex queries over large time ranges often exceeded the 60-second limit.
• Operational overhead: Maintaining Prometheus at scale required dedicated engineering resources.
• Limited visibility: Practical monitoring windows were capped at around 6 hours, insufficient for long-running training jobs.

"The challenge wasn't just about storing more data; it was about being able to query it quickly enough to make real-time decisions during training runs."

The Evolution to Managed Services

Adobe's path to Amazon Managed Service for Prometheus was iterative rather than a complete overhaul. This strategy is worth noting for anyone considering similar migrations. They didn't rip out their existing infrastructure. Instead, they used Amazon Managed Service for Prometheus collectors (managed scrapers) alongside their existing self-managed setup.

This incremental approach offered several key advantages:

1. Zero disruption: Existing monitoring workflows remained intact.
2. Targeted migration: Only critical metrics were moved initially.
3. Direct comparison: The team could measure performance improvements side-by-side.

# Example: Configuring a managed scraper alongside existing Prometheus
apiVersion: v1
kind: ConfigMap
metadata:
  name: managed-scraper-config
  namespace: observability
data:
  scrape_configs: |
    - job_name: 'gpu-training-metrics'
      kubernetes_sd_configs:
        - role: pod
      relabel_configs:
        - source_labels: [__meta_kubernetes_pod_label_app]
          regex: 'training-.*'
          action: keep
      metric_relabel_configs:
        # Only forward critical metrics to the managed workspace
        - source_labels: [__name__]
          regex: '(gpu_utilization|gpu_memory_used|network_transmit_bytes_total)'
          action: keep

Measurable Results and Critical Considerations

The migration to Amazon Managed Service for Prometheus yielded dramatic improvements, particularly in query performance. The table below compares the performance of the managed service versus the self-managed infrastructure:

The Real-World Impact

These performance numbers translate to tangible operational benefits:

• Extended visibility: Infrastructure users can now view metrics across 24-hour windows, compared to the previous practical limit of 6 hours. This is critical for long-running jobs spanning 256+ nodes, where identifying when performance degraded requires seeing the full job lifecycle.
• Automation enablement: Fast, reliable queries make it possible to build automated systems that respond to infrastructure events in real-time.
• Reduced operational burden: With fully managed data and control components, engineering resources are freed up for infrastructure development rather than monitoring maintenance.

Limitations and Considerations

While Amazon Managed Service for Prometheus delivers significant benefits, it's important to consider potential limitations:

• Cost structure: This is a billable service. Costs are based on metrics ingested, stored, and queried. Organizations with massive metric volumes need to carefully estimate costs before migration.
• Migration strategy: A complete migration is not a one-step process. Adobe's incremental approach worked well, but it requires maintaining two systems during the transition.
• Workspace limits: Each workspace supports up to 50 million active time series. While this provides headroom, organizations approaching this limit will need to consider sharding strategies.

Next Steps for Your Observability Journey

If you're managing GPU training infrastructure or other high-cardinality workloads, consider these steps:

1. Audit your current metrics: Identify which metrics are truly critical for decision-making.
2. Test with a pilot workload: Follow Adobe's example by moving a targeted set of metrics first.
3. Measure before and after: Establish baseline performance metrics before migration to quantify improvements.

The Future of AI Infrastructure Observability

Adobe's experience demonstrates that observability infrastructure must evolve alongside AI training capabilities. The 28x query performance improvement and extended monitoring windows aren't just technical metrics—they represent a fundamental shift in what's possible for infrastructure teams supporting AI workloads.

As AI models grow more complex, the demand for sophisticated observability will only increase. The collaboration between Adobe and AWS to extend managed Prometheus to remaining metric tiers signals a move toward multi-tenant, highly available observability stacks that can support full-scale telemetry.

For teams building AI infrastructure, the lesson is clear: invest in observability architecture as seriously as you invest in compute infrastructure. The ability to see and understand what's happening across thousands of GPUs isn't a nice-to-have—it's essential for efficient model development.

If you're interested in related infrastructure challenges, check out how Pantone built an AI-ready data foundation for agentic creativity, or explore Spotify's approach to scaling developer experience in an AI-augmented world.