I implemented a system that included the OP functionality (plus a whole lot more.) It was for on-premise deployment at customers. It can also be used to spin up stand-alone instances of our system in the cloud, for development, testing, etc. While you could, in theory, do many deployments on a single k8s cluster, there are some benefits to the automatic isolation you get from deploying on a standalone VM.
I'm doing many deployments in my single k8s cluster. I just put them each in a different namespace.
The only piece that's maybe a little dicey is the single load balancer/gateway. If there's a hiccup in that, then everything goes down.
But I've only blown up my cluster once in like 8 years or something, that's not too bad. It was a learning experience :-)
What other kinds of isolation do you want? I can see maybe a separate staging environment if you want to test gnarly things like that ahead of rolling them out to prod. And I guess maybe they can eat eachother's resources if you don't have request limits nor auto-scaling enabled.
But I'm cheap and managing more clusters sounds like a pain. Then I'd have to deal with more kubectl credentials and what not too.
What you’re saying makes it sound like you’re a one-person operation, or somewhere close to that scale. That obviously doesn’t have same requirements as much larger organizations.
I ran a job last night which provisioned a cluster with 4TB of RAM and nearly 1000 vCPUs. It ran for 20 minutes, ingested about 800 GB of data from nearly an million files, and was then deleted. To do that on a single cluster that’s also used for serving production requests would be unnecessarily complex and risky. Our production system has users in every timezone using the system 24x7. At the very least you’d have to provision separate node pools anyway, but why would you bother to do that?
