That's a weird one. When you say it brings it down, what do you mean is actually occurring? I have a USB ethernet adapter in use on my M1 MBP running Ventura as I write this, and it's been completely fine. But if there's a gotcha out there lurking, I'd be interested to know what triggers it.
We actually had a very similar issue in the office but maybe a bit different. From time to time an entire part of the office would start having slow internet or almost no internet. Took us a few months to fix (it wasn't really on the priority list and was super random when it happened).
Turned out that it happened when one specific USB-C dongle with ethernet and PD was left powered on but not connected to the computer (I think there were other preconditions as well like it was powered-off and you connect power to it but not a computer to it).
The device will then decide to go into DHCP server mode and start sending advertisement packets and the switch would start broadcasting them, the internal firewall would see this and start blocking the switch and in turn turn off the AP that was connected to it so 26 ports would be offline essentially.
Most of us were on wifi and everyone who was on an ethernet (3 ppl) would go "FFS internet is unstable again. Stupid switch is probably buggy", disconnect their cable and go on wifi. Plus they never really reported it as they had work to do and Wifi kinda worked so not a big issue.
It was just a really cool moment when we found out what caused it and we fired up Wireshark and looked at the traffic and we were right. I haven't had a moment of so much joy in a really long time.
That is fascinating, thanks for the link! Hopefully I'll remember that if either of my USB/Ethernet adapters seems to be related to a network problem. Sometimes I think USB-C hubs are simultaneously the best thing ever, and the bane of my existence. I'm finally happy with the CalDigit one I use now, but it took a couple attempts to find something that didn't have some glaring problem. One of my MacBooks just uses a plan USB->Ethernet adapter successfully, but that's because I don't need it to do anything more complicated.