I’d love to hear some stories about how you or your organization is using Kubernetes for development! My team is experimenting with using it because our “platform” is getting into the territory of too large to run or manage on a single developer machine. We’ve previously used Docker Compose to enable starting things up locally, but that started getting complicated.
The approach we’re trying now is to have a Helm chart to deploy the entire platform to a k8s namespace unique to each developer and then using Telepresence to connect a developer’s laptop to the cluster and allow them to run specific services they’re working on locally.
This seems to be working well, but now I’m finding myself concerned with resource utilization in the cluster as devs don’t remember to uninstall or scale down their workloads when they’re not active any more, leading to inflation of the cluster size.
Would love to hear some stories from others!
We use monitoring to check the state of our applications to see if they’re idle. For example, one of our healthchecks looks at if data was placed into the database within a period of time. We know the database is idle if there is no data there within the last month for example. We use Prometheus/AlertManager to let us know in slack about idle resources.
Also we make use of HPA to scale resources, it takes a lot of time to find optimal settings for HPA in my experience. Each product team has a namespace in the cluster. These clusters have like 200 nodes.
We have hundreds of installations of our application across ~8 data centers, we use both Ansible and Golang operators for managing our many resources.