I had a bit of a problem over the last 8-9 months with 2 Supermicro X9SRH-7TF systems and the cheap and cheerful 10G Netgear XS708E switch. Somewhat sporadically (weeks would go by), the onboard X540 based 10G NICs would disconnect and the motherboard would require a full power cut (PSU switch off/on) before they would re-link again. If you follow me on twitter you would have heard numerous rants.
To begin with, I thought it was just one of the boards, so it was sent it back to Supermicro (no fault found) but then the other one started doing it too. Fast forward a few months and the servers were moved from 220v power to 110v (Australia -> USA move)… suddenly the problem disappeared. I thought it must have been some dodgy voltage step-down issue, but I didn’t think much more of it. I was simply happy the issue had been resolved and the HA wasn’t working overtime starting up failed VMs.
A couple of weeks ago, the servers were each upgraded with an Icy Dock MB994SP 4S to enable VSAN. Some older HP 10K 146GB SAS drives (with Samsung EVO SSDs) were used with the onboard LSI 2308. As soon as the servers were powered up, the NICs once again started disconnecting. At once I knew it must be a power issue, and I upgraded the PSUs in both servers. While calculating the total power draw, the originals (430W) should have been enough, but I now have plenty more in reserve (750W).
The moral of the story? Power Supplies are a relatively cheap component within your home lab system, so get one a little better than your current needs.