How CSIRO Built a Serverless Genomics Platform That Queries Billions of Variants in Seconds

The Challenge: Genomic Data at Biobank Scale

Genomic research is exploding. Studies now require cohorts of hundreds of thousands—sometimes millions—of individuals to uncover the genetic basis of complex diseases. Traditional monolithic databases buckle under the weight of billions of genomic locations, and centralized data lakes raise privacy and ownership concerns.

CSIRO, Australia's national science agency, tackled this head-on by building sBeacon, a production-ready, serverless implementation of the GA4GH Beacon protocol. It queries genomic variant data across federated datasets without copying raw sequences, achieving sub-second response times at a fraction of a cent per query.

This deep dive breaks down the architecture, security model, and performance characteristics that make sBeacon a compelling pattern for any large-scale, privacy-sensitive data sharing platform. For a broader look at how platform choices are evolving, see our analysis of Microsoft Foundry's agent platform model.

Architecture Deep Dive: Two Core Workflows

sBeacon splits into two independent processes: data onboarding (indexing metadata) and data querying (retrieving variant information). The raw VCF files never leave their original S3 bucket.

Data Onboarding

Metadata (disease status, age, location) is ingested via API Gateway → Lambda → S3 (ORC format). Athena CTAS queries build indexed tables, and DynamoDB stores ontology term mappings for fast lookups.

# Simplified pseudocode for metadata indexing (actual implementation uses Terraform + Lambda)
import boto3

s3 = boto3.client('s3')
athena = boto3.client('athena')

def onboard_metadata(dataset_id, metadata_uri):
    # 1. Write metadata to S3 in ORC format
    s3.copy_object(Bucket='sbeacon-metadata', Key=f'{dataset_id}/raw.orc', CopySource=metadata_uri)
    # 2. Run CTAS query to build indexed table
    athena.start_query_execution(
        QueryString=f"CREATE TABLE {dataset_id}_indexed AS SELECT * FROM raw_{dataset_id}",
        ResultConfiguration={'OutputLocation': f's3://sbeacon-metadata/{dataset_id}/indexed/'}
    )
    # 3. Update DynamoDB ontology mapping (handled by separate Lambda)
    return {'status': 'indexed'}

Data Querying

Queries hit API Gateway, which invokes a Microservice Lambda. That Lambda resolves ontology terms via DynamoDB, queries Athena for metadata, and—if needed—fans out to a Variant Querying Module. The module uses bcftools with htslib HTTP byte-range requests to read only the relevant tabix-indexed region from S3. Results are aggregated and returned in Beacon protocol format.

# Core variant query logic (simplified)
import subprocess

def perform_query(vcf_s3_uri, region):
    # bcftools reads only the byte range for the region via htslib
    cmd = ['bcftools', 'view', '-r', region, vcf_s3_uri]
    result = subprocess.run(cmd, capture_output=True, text=True, timeout=10)
    # Parse VCF lines into Beacon response
    variants = [line for line in result.stdout.split('\n') if not line.startswith('#')]
    return {'exists': len(variants) > 0, 'variants': variants}

Performance snapshot (1000 Genomes, chr1, 2504 samples):

• Ingestion: 18 seconds, $0.00052
• Query (10k bp region): 1.52 seconds, $0.00013
• Idle storage: $0.03/month for 1.1 GB VCF + metadata

Security, Privacy, and Limitations

Zero-Trust Model

Every request carries a Cognito-issued JWT. API Gateway validates signature, expiry, and audience before any Lambda runs. Authorization is enforced inside Lambda via Cognito groups that map to disclosure tiers:

Lambda execution environments are ephemeral—no shared memory, /tmp cleared between cold starts. S3 buckets are private with no public policies. VPC-isolated Lambdas have no inbound network access.

Privacy by Design

Each institution deploys its own Terraform stack into its own AWS account. There is no central data lake. VCF files stay in the deploying institution's S3 bucket. performQuery passes the S3 URI directly to bcftools, which uses HTTP byte-range requests to read only ~1 KB per query. Raw sequence bytes never pass through Lambda memory as returnable data.

Critical Limitations

• Whole-genome range queries time out at API Gateway. The architecture is optimized for targeted region queries, not full-genome scans.
• Concurrency exhaustion: A single fan-out query spawns many parallel Lambdas. Without monitoring ConcurrentExecutions, burst queries can starve other functions.
• Silent throttle failures: Synchronous Lambda invokes don't auto-retry on 429. You must set CloudWatch alarms on Throttles for performQuery or risk losing results.
• Cold starts: After idle periods, burst fan-out may hit cold starts. Provisioned concurrency helps but increases idle cost.
• No built-in retry: The application must handle 429 responses explicitly; otherwise, results are silently lost.

Key Takeaways and Next Steps

sBeacon demonstrates that serverless architectures can handle petabyte-scale genomic data with sub-second query times and near-zero idle costs. The pattern—separating metadata indexing (Athena/DynamoDB) from raw data access (S3 byte-range reads via bcftools)—is applicable beyond genomics to any domain requiring federated, privacy-preserving data sharing.

Actionable advice:

1. Start with a single chromosome to validate cost and performance before scaling to whole genomes.
2. Monitor concurrency and throttles from day one—set CloudWatch alarms on ConcurrentExecutions and Throttles.
3. Use provisioned concurrency only for latency-sensitive query paths; keep ingestion on on-demand.
4. Federate gradually: Begin with internal datasets, then connect to external Beacons as trust frameworks mature.

For a complementary perspective on platform strategy and model choice, explore our React Conf 2025 takeaways on the compiler and native future.

Further reading: The sBeacon source code and Terraform modules are available on GitHub. The original architecture write-up is available as 근거자료.