As in, no human should run a package manager in prod? (But salt/ansible/etc. running it is fine) Same idea as "if you're SSHing to prod, something is wrong" (where provisioning tools make all changes, logs are all aggregated and delivered in their own tool, and even debugging is built into the app or logging system).
More as, you should not modify images running in production. By human or machine.
Rather build new images and roll over the fleet. If you need to debug, remove from production (quarantine) and work on it there.
Don't run master / agent setups for ansible / salt anymore. You can still use them for creating images, which are later turned into running VMs. Think about it like containers. Do you update the contents of your running containers, or log into your containers to make changes?
But golden images on Linux are, well, messy. It's very annoying to make a clean VM template without some post-provisioning like cloud-init. And for most shops if you're running cloud-init you could do that post-provisioning with Ansible or Salt. And since your images are built with Ansible/Salt in the first place you might as well just build each VM fresh and use the vendor's ISO. One less thing to maintain and update.
Plus when you're in a pinch, which never happens of course, you can make changes without having to roll your VMs.
I feel like Atomic distributions are basically the happy medium between the two worlds.
It's a trade-off. The point is the matrix security lapse turned worse because they ran this master / agent setup. You can still use ansible (or similar), just do it localhost during the build process.
Yea it's easier to not do these things, because good security posture takes work to set up. Once you're on the immutable train, you'll find it's not actually harder day to day. You learn to deal with issues in the pinch another way.
On the point of building VMs fresh each time vs building golden images, you'll find you boot time reduced, your roll over more reliable and autoscaling more responsive. Why build the same thing dozens or hundreds of times? What happens if a remote package is updated in the middle of your upgrade? Does this sound messier to you?
