Shared AI platform
I designed a shared AI platform on EKS where every app and workflow calls one LiteLLM gateway and only the gateway talks to Bedrock, with per-team keys, cost caps and audit logs. I picked the AWS region, set up Bedrock and n8n and proposed Argo CD for deploys, which the design review adopted.
17 internal apps and workflows and 6 teams outside my own run on one governed platform, with spend tracked per team.
- 17internal apps and workflows on one gateway
- 6teams outside my own using it
Context
AI workflows called Bedrock with their own API keys, one per workflow or team. 2 separate n8n cloud instances ran side by side.
What I did
- Wrote the Bedrock setup plan for production and QA.
- Picked the AWS region by Bedrock model coverage and latency to the teams using it.
- Built an n8n workflow that attributed AI usage to each workflow before the gateway existed.
- Demoed a working Argo CD deploy of n8n in about 10 minutes at the design review to make the case for Argo CD.
- Wrote the LiteLLM execution plan.
How it works
- Every consumer calls LiteLLM. Only LiteLLM talks to Bedrock, over a private VPC endpoint.
- Cost caps, rate limits and guardrails are set per key.
- Argo CD syncs the workloads to EKS with Helm. Terraform owns the AWS resources and TeamCity builds the images.
- Audit logs go to S3. Container logs go to ClickHouse with Fluent Bit. LiteLLM metrics feed the engineering metrics dashboard.
- The cluster runs LiteLLM, n8n, Onyx with its vector store and the MCP gateway.