All projects
Coda Payments2025 - 2026

Jenkins retirement

Jenkins got slower the longer it ran. Stuck processes, a main process at 18GB, scheduled restarts to keep it alive and big repositories timing out on checkout. I moved the jobs onto TeamCity and the internal deployment system team by team until no jobs were left, then shut the servers down.

One delivery path, one less platform to babysit and $54k a year saved on servers and agents. In on-demand test environments an app deploy now takes about 5 seconds, where Jenkins took minutes.

JenkinsTeamCityAWSPython

Context

Jenkins staging slowed down over several weeks at a time. The causes were stuck AWS CLI processes, a runaway permissions script and a main process that grew to 18GB. Staging needed scheduled restarts and big repositories timed out on checkout.

What I did

  • Wrote a Python script that dumps every job on the staging and production instances to CSV.
  • Sorted 97+ jobs by deployment type. Only 8 needed a new path: 3 AMI jobs, 3 EC2 jobs for analytics and 2 Terraform jobs.
  • Wrote the plan for the remaining jobs. Serverless stacks and test environments moved under the serverless migration and faster test environments.
  • Moved product operations jobs to self-service portals.
  • Switched off the last jobs by the end of Q2 2026 and shut down the servers and agents.

Numbers

  • 62 production jobs and 99 staging jobs, 161 in all. The staging count includes test environment jobs.

Diagrams

Delivery path now
Before and after
Timeline