Building a Scalable AI Platform for LLMs on AWS
By Or Hazout · October 29, 2025 · 5 min read
Running LLMs in production is complex and expensive. We built a scalable, cost-efficient AI platform on AWS EKS using Terraform, with smart, event-driven scaling for GPU nodes via Karpenter and KEDA. The entire system is automated with a Helm Umbrella Chart and GitHub Actions CI/CD for seamless, zero-downtime deployments.
Artificial Intelligence, especially Large Language Models (LLMs), is changing the world. But ask any engineer, and they'll tell you: running these models in production is hard.
It’s not like deploying a simple website. LLMs demand massive resources (hello, expensive GPUs!), handle unpredictable traffic, and process huge amounts of data. Just "lifting and shifting" won't work. You need an infrastructure that is smart, scalable, and automated from the ground up.
At [Your Company Name], we recently tackled this exact challenge for a cutting-edge LLM product. We built a robust, production-grade AI infrastructure on AWS. Here’s a breakdown of how we did it, component by component.
1. The Foundation: Infrastructure as Code with Terraform
Before you can build a house, you need a solid foundation and blueprints. For cloud infrastructure, that blueprint is Terraform .
We used Terraform to define everything as code. This means no manual clicking in the AWS console.
VPC (Virtual Private Cloud): We started by creating a secure, isolated private network for the entire application.
Core EKS Cluster: We laid the groundwork for our Kubernetes cluster (more on that next).
S3 Buckets & IAM Roles: We securely set up storage for artifacts and the exact permissions everything needed to talk to each other safely.
Why this matters: Using Terraform makes our infrastructure repeatable, auditable, and easy to modify. Need to deploy the exact same platform in a new region? We can do it in minutes, not weeks.
2. The Brain: EKS with Smart Scaling
The "brain" of our operation is Amazon EKS (Elastic Kubernetes Service) . Kubernetes is the industry standard for managing containerized applications, and EKS gives us all its power without the headache of managing the control plane.
But for AI, a standard EKS setup isn't enough. We added three specialized tools to handle the unique demands of LLMs:
Kafka: LLMs often need to process data in streams (like user prompts or data for fine-tuning). We use Kafka as the high-throughput "central nervous system" to buffer and manage all this data reliably.
Karpenter: This is our GPU cost-saver. Powerful GPU nodes are incredibly expensive, and you don't want them running 24/7 if you don't need them. Karpenter is a modern node autoscaler that watches for pending work. The moment our LLM needs a GPU, Karpenter provisions a new GPU node on-the-fly. When the work is done, it terminates the node. This "just-in-time" scaling is a game-changer for managing costs.
KEDA (Kubernetes Event-Driven Autoscaling): So, we have Kafka for data and Karpenter for nodes. How do we scale the application pods themselves? That's KEDA. It watches our Kafka queues. If the queue of user prompts gets long, KEDA automatically scales up our LLM application pods. When the queue is empty, it scales them back down to zero.
This trio (Kafka + KEDA + Karpenter) creates a highly efficient, event-driven system that scales from zero to massive, and back down again, all based on real-time demand.
3. The Front Door: Nginx Ingress Controller
Now that we have a powerful application, we need a secure and reliable way for users to access it.
We deployed the Nginx Ingress Controller using its popular Bitnami Helm chart. In the AWS environment, this chart automatically sets up and configures an AWS Load Balancer . This acts as the "front door," directing all incoming user traffic to the right services inside our Kubernetes cluster, handling SSL termination, and ensuring high availability.
4. The "Big Red Button": Automated Deployment with Helm & GitHub Actions
A complex platform with many parts (Kafka, KEDA, our app, ingress) can be a nightmare to deploy and update. We streamlined this entirely.
Helm Umbrella Chart: We packaged every single component of our application into a single "Umbrella Chart." This one chart holds all the other charts (Nginx, KEDA, our app) and their configurations as dependencies.
GitHub Actions CI/CD: We built a fully automated CI/CD (Continuous Integration/Continuous Deployment) pipeline using GitHub Actions.
Here’s the complete, automated workflow:
A developer pushes new code to GitHub.
GitHub Actions automatically triggers.
The pipeline runs tests, builds the new application image, and pushes it to our container registry.
Terraform runs to apply any infrastructure-level changes.
Finally, Helm takes our Umbrella Chart and safely upgrades the application in the EKS cluster with zero downtime.
This "commit-to-production" pipeline means our developers can focus on building better AI features, not on manual, error-prone deployment steps.
5. The Eyes: Full-Stack Observability
You can't manage what you can't see. For a system this dynamic, visibility is non-negotiable. We implemented a "two-pronged" observability strategy:
Datadog for Application Monitoring: Datadog gives us deep insights into the application itself. We can trace individual requests, monitor LLM performance (like time-to-first-token), and check the health of our Kafka queues.
Coralogix for EKS & Infrastructure: Coralogix plugs directly into EKS and our AWS environment. It collects all the logs and metrics from the infrastructure level. Are our nodes scaling correctly? Are there any errors in the Kubernetes system? Coralogix tells us instantly.
With this setup, we have a complete, top-to-bottom view of the entire platform, from the user's web request all the way down to the AWS infrastructure.
Building AI Infrastructure the Right Way
Building a platform for LLMs is complex, but it doesn't have to be chaotic.
By combining the power of Terraform for solid foundations, EKS with smart scalers like Karpenter and KEDA, and a fully automated Helm & GitHub Actions pipeline, we built a system that is not only powerful but also cost-effective and easy to manage.
Don't let infrastructure be the bottleneck for your AI ambitions. If you're looking to build a scalable, production-ready platform, let's talk.
Topics: LLM, AWS, K8S