How Netflix Manages Millions of Data Assets with Data Projects

The Scale of the Problem

Netflix’s data platform is massive. Millions of tables in the data warehouse, tens of thousands of scheduled workloads running across orchestration systems. Each asset is owned by an engineer, a team, or an initiative — and behind each one are decisions about who can access what, and how those workloads execute.

Traditional access control models break down at this scale. When every table has its own Access Control List (ACL) and every workflow runs under a human identity, organizational changes become a nightmare. Reorgs, role changes, and departures trigger cascading permission failures.

Two persistent problems emerge:

• Permissions can’t keep up: Support teams get flooded with requests to update table permissions en masse. Teams often resort to granting overly broad access just to avoid the overhead.
• Workloads tied to human identities: When the engineer who authored a workflow leaves or changes teams, the workflow breaks. The only fix is to swap in another human identity, which starts a game of "permissions whack-a-mole."

The Solution: Data Projects

Netflix introduced Data Projects to tackle both problems. A Data Project is two things:

1. A container to manage a set of related assets (tables, workflows, etc.) under a single logical umbrella.
2. A synthetic, durable, assumable identity — one that asynchronous and scheduled workloads can execute under, independent of any human’s lifecycle.

Think of it as hoisting the granularity of management from the individual asset to a meaningful container: the project. Instead of managing permissions on 500 tables, you manage them on one project that contains those 500 tables.

How Data Projects Work

Grants and Roles

Each Data Project has a set of grants managed by the owning team. Different identity types can be added: users, groups, applications, and CI jobs. Each grant has a role that determines what the grantee can do within the project.

# Example: Managing project grants via API (simplified)
# Python-like pseudocode for illustration

class DataProject:
    def __init__(self, project_id, owner_team):
        self.project_id = project_id
        self.owner_team = owner_team
        self.grants = {}  # identity -> role mapping
        self.contained_assets = set()

    def add_grant(self, identity, role):
        # role can be 'contributor' (read/write) or 'viewer' (read-only)
        self.grants[identity] = role

    def remove_grant(self, identity):
        self.grants.pop(identity, None)

    def assume_identity(self):
        # Returns the project's durable identity token
        return generate_project_token(self.project_id)

# Usage: when a user joins a team, just update one project grant
project = DataProject("analytics_qoe", "streaming_team")
project.add_grant("alice@netflix.com", "contributor")

The Identity Umbrella

Every Data Project is provisioned with a Netflix application identity (and optionally an AWS IAM role). This is the "identity umbrella" that makes workloads durable:

• The project’s Netflix identity executes async workloads (e.g., Maestro workflows). It belongs to the project, not to any person.
• The project’s IAM role supports specialized use cases like Spark jobs on Amazon EMR. The IAM role can be exchanged for the project’s Netflix identity cryptographically.

Gravity: Automatic Asset Association

One elegant property is gravity. When a workload running under a project’s identity creates a new asset — say a Maestro workflow creates three tables — those assets are automatically added to the project. The project becomes the center of mass for everything produced under its identity. This eliminates future challenges of discovering relevant assets and gaining access to them.

Securing Workflows with Durable Identity

Maestro is Netflix’s primary workflow orchestrator for batch analytics. Because workflows run on schedules without the original user present, Maestro is a Trusted Workload Manager (TWM) that can mint fresh identity tokens on behalf of the workloads it manages.

With Data Projects, the workflow runs under the project’s identity, not a human’s. Tables created during execution are automatically associated with the project through gravity. Secrets are scoped to project policies. Access is managed once at the project level.

This model is stable, auditable, and built to survive organizational changes.

Limitations and Considerations

While Data Projects solve many problems, they’re not a silver bullet. Here are some considerations:

• Migration overhead: Moving existing workflows to a Data Project requires setting up the project with appropriate permissions before changing execution identity. Netflix is building infrastructure to track access patterns and recommend precise permission updates.
• Granularity trade-off: Projects group assets logically, but sometimes you need per-asset exceptions. Teams must decide whether to create a new project or use a broader one.
• Organizational adoption: Teams need to shift from thinking about individual assets to thinking about project boundaries. This requires cultural change and tooling support.

Next Steps and Learning Path

If you’re inspired by Netflix’s approach, here’s how to start applying similar principles:

1. Audit your current permission model: Identify where human-tied identities cause workflow fragility.
2. Experiment with project-based access: Use cloud-native tools like AWS IAM roles, Azure Managed Identities, or GCP service accounts as durable identities.
3. Explore orchestration best practices: Look into workflow orchestrators like Apache Airflow or Prefect that support identity scoping.

For more on managing data assets at scale, check out this related article on Building a Multi-Tenant, Sovereign Carbon Footprint Exchange on Catena-X with AWS. And for a deeper dive into on-device AI patterns, see On-Device Function Calling Goes Cross-Platform Inside Google AI Edge Gallerys Latest Update.

Conclusion

Data Projects is an answer to a simple observation: at Netflix’s scale, the unit of identity and access management can’t be the individual asset or the individual human. It has to be something larger, something durable, something that matches the way teams actually think about the work they own.

A project is that unit. By hoisting access management to the project level and providing a durable identity, Netflix has eliminated a major source of operational pain. The concept is already being generalized beyond data — to software assets, studio assets, and more.

The key takeaway: think in containers, not individuals. Whether you’re managing a few tables or millions, a project-based approach can save you from the endless cycle of permission whack-a-mole.

This article is based on insights from Netflix's engineering blog.