Why You Need Automated PII Detection

If you work in finance, healthcare, or any regulated industry, you already know the pain of identifying sensitive data scattered across your S3 buckets. Names, addresses, policy numbers, member IDs—manually classifying this data doesn't scale, and compliance frameworks like GDPR and HIPAA demand you know exactly where PII lives.

Amazon Macie gives you managed sensitive-data discovery out of the box, but what about the domain-specific identifiers unique to your business? A standard engine won't recognize your internal policy number format. That's where custom identifiers come in.

In this guide, you'll build a complete event-driven pipeline that:

  • Triggers the moment a file lands in S3
  • Scans with both built-in and custom Macie identifiers
  • Generates timestamped CSV and JSON compliance reports
  • Sends real-time SNS alerts for high-severity findings

Let's get started.

Developer configuring custom PII detection rules in Amazon Macie console with AWS Step Functions diagram Software Concept Art

Building the Pipeline: Step-by-Step

Architecture Overview

The pipeline uses five core AWS services:

ServiceRole
S3Three-bucket storage layer (raw, staged, scanned)
EventBridgeDetects uploads and triggers workflow
Step FunctionsOrchestrates the entire scan lifecycle
MacieDetects PII with built-in + custom identifiers
SNSSends real-time alerts

The three-bucket pattern isolates data by processing state—unscanned data never mixes with validated data.

Step 1: Deploy the CloudFormation Stack

Download the template and deploy it using the AWS CLI:

# Deploy the stack with a custom bucket prefix
aws cloudformation create-stack \
  --stack-name pii-pipeline \
  --template-body file://template.yaml \
  --parameters ParameterKey=BucketNamePrefix,ParameterValue=mycorp \
                ParameterKey=CustomIdentifierPatterns,ParameterValue="\\bPOL-\\d{8}\\b" \
                ParameterKey=NotificationEmail,ParameterValue=security@mycorp.com

Expected result: Stack reaches CREATE_COMPLETE in 3–5 minutes.

Step 2: Configure Custom Data Identifiers

The stack automatically provisions custom identifiers from the CustomIdentifierPatterns parameter. Verify they appear in the Macie console under Settings > Custom data identifiers.

Step 3: Test the Pipeline

Upload a test file to the raw bucket:

# Upload a sample file to trigger the pipeline
aws s3 cp sample-data.csv s3://mycorp-raw-bucket/

Within seconds, a Step Functions execution starts. The state machine runs through five states:

  1. TriggerScan - Copies file to staging bucket, creates Macie job
  2. WaitForMacie - Pauses 60 seconds for job processing
  3. CheckStatus - Polls DescribeClassificationJob API
  4. GetFindings - Retrieves results, generates reports, publishes SNS
  5. MoveFiles - Relocates original to scanned bucket, cleans staging

Step 4: Verify Reports

Each scan produces two timestamped files in the scanned bucket:

# List generated reports
aws s3 ls s3://mycorp-scanned-bucket/reports/
# Output: 2025-01-15-143022-findings.csv
# Output: 2025-01-15-143022-findings.json

For high-severity findings, you'll also receive an SNS notification email with details.

Cloud architecture diagram showing event-driven pipeline with S3 buckets and AWS Step Functions for PII scanning Development Concept Image

Critical Considerations and Hardening for Production

Quotas and Throttling

Be aware of Macie's limits before deploying at scale:

  • Custom identifiers: Up to 10,000 per account, but max 30 per classification job
  • CreateClassificationJob throttle: 0.1 requests/second (one job every 10 seconds)

For high-volume workloads, batch multiple objects into a single Macie job to avoid hitting API limits.

Security Hardening

Before moving to production, implement these measures:

  • Encryption: Enable SSE-S3 or customer-managed KMS keys via the KmsKeyArn parameter
  • Multi-tenant isolation: Deploy separate stacks per tenant with distinct BucketNamePrefix
  • Audit trail: Enable S3 CloudTrail data events to track every object access
  • Cross-account: Configure EventBridge rules for data sources spanning multiple accounts

Known Limitations

  • Scan duration varies: Jobs can take 15–20+ minutes depending on data volume, compression, and identifier count
  • No native retry for Macie jobs: You must rely on Step Functions retry logic
  • Staging bucket auto-expiry: Objects expire after 7 days by default—adjust if you need longer retention

Next Steps for Learning

Now that your pipeline works, consider extending it:

  • Route JSON findings to Athena for trend analysis and build QuickSight dashboards
  • Add a Step Functions remediation branch to quarantine high-risk files
  • Publish findings to AWS Security Hub for unified posture management

For deeper context on building reliable automation, check out our guide on engineering predictable AI coding agents through feedback loops. And if you're evaluating enterprise AI tools, see how Claude Opus 4.6 on Azure is setting new standards for autonomous coding.

Analyst reviewing compliance reports with CSV and JSON files generated by automated PII detection pipeline Algorithm Concept Visual

Conclusion

You now have a fully automated PII detection pipeline that responds to S3 uploads in seconds, runs comprehensive scans with both managed and custom identifiers, and produces audit-ready compliance reports without manual intervention.

The key takeaway: combining Macie's managed classification with your own custom identifiers—orchestrated by Step Functions and triggered via EventBridge—gives you complete visibility into sensitive data across your organization.

Your next move: Clone the sample GitHub repository and deploy this in a sandbox environment. Start with a small dataset, verify the reports, then scale up with additional custom identifiers for your organization's unique data formats.

This content was drafted using AI tools based on reliable sources, and has been reviewed by our editorial team before publication. It is not intended to replace professional advice.