Offline-First Generative AI A Reference Architecture for Edge Deployments

Why Offline-First AI Matters

In industrial settings, connectivity is often unreliable. According to Siemens, unplanned downtime costs Fortune 500 companies $1.4 trillion annually. Generative AI can help, but deploying it where cloud access is spotty requires a different approach: move inference to the edge, and use the cloud for training, orchestration, and continuous improvement.

This post explores a reference architecture that does exactly that, using AWS services to build a self-contained edge AI stack.

Architectural Decisions for Edge AI

Choosing the right model customization strategy is critical. The options range from simple fine-tuning to a hybrid approach combining fine-tuning, continued pre-training, and RAG. For our use case, we used fine-tuning + RAG to balance capability and complexity.

Fine-Tuning + RAG in Practice

Here’s how we implemented the hybrid approach:

• Fine-tuning: We used Amazon SageMaker AI Pipelines to fine-tune gpt-oss-20b on domain-specific Q&A pairs generated by Amazon Nova Pro on Amazon Bedrock.
• RAG: We used ChromaDB (SQLite + HNSW) with a sentence-transformer embedding model (384 dimensions) running on CPU. Documents are chunked at 512 tokens with 50-token overlap, keeping retrieval latency under 50 ms.

# Example: Setting up ChromaDB for edge RAG
import chromadb
from sentence_transformers import SentenceTransformer

# Load embedding model
embedder = SentenceTransformer('all-MiniLM-L6-v2')

# Create Chroma client (persistent storage)
client = chromadb.PersistentClient(path='/edge/rag_db')
collection = client.get_or_create_collection('docs')

# Add documents with embeddings
collection.add(
    documents=["Cookie machine manual: check temperature..."],
    ids=["doc1"],
    embeddings=embedder.encode(["Cookie machine manual: check temperature..."])
)

# Query
results = collection.query(query_embeddings=embedder.encode(["How to fix cookie machine?"])[:1], n_results=1)
print(results['documents'])

Deployment Options

On a g4dn.12xlarge with 4× NVIDIA T4 GPUs, we had two choices:

Tensor parallelism supports the full 128K context window, but replication maximizes throughput.

Security and Operational Considerations

Moving to the edge shifts the security perimeter. You own the full stack, so implement:

• Authentication: Integrate with your IdP (SAML/OIDC) or use mutual TLS.
• Encryption: Full-disk encryption (LUKS/BitLocker) and TLS 1.2+ for all communication.
• Guardrails: Validate inputs and block prompt injection attempts.
• Network segmentation: Isolate components into separate zones.

Also, plan for continuous improvement: when connectivity is available, sync feedback to the cloud to refine your model. The architecture is designed to queue feedback locally and synchronize opportunistically.

Next Steps

Start with a clear use case, work backward from hardware constraints, and choose a customization strategy. The hybrid FT + RAG approach is a solid starting point. For more on this, check out our detailed guide as the source material.

Conclusion

Offline-first generative AI is not just a workaround—it’s a strategic advantage for industries where downtime is costly and connectivity is unreliable. By using AWS services like SageMaker, Bedrock, and IoT Greengrass, you can build a robust edge AI stack that operates independently of the cloud.

Key takeaways:

• Use cloud for training data generation and model fine-tuning.
• Choose deployment strategies based on your latency and concurrency needs.
• Implement security controls tailored to edge environments.
• Continuously improve your model with a feedback loop.

Together with this article, you might find these useful:

• Behind the Scenes The Engineering Challenges of Building Meta Ray-Ban Display
• Bending the Curve How Metas Adaptive Ranking Model Serves LLM-Scale AI at Sub-Second Latency