The last few systems have been patched after significant delays from CloudLinux. But throughout the day from early morning, multiple possibility mitigations have been put in place. While not 100% effective, it limits exposure. Happy to report no customers faced issues.
We had roughly 5 minutes of downtime on nlsh06 due to a webserver crash that needed manual intervention. We are investigating the root cause.
We had a short alarm trigger on nlcp01 due to a connection flood which enabled our protection that served a Splashscreen / Proof-of-work to all visitors (including monitoring) until the connection count dropped. Sites continued to work during the 3.5 minutes alert.
We will do a quick maintenance of the hosting-panel.net control panel which requires we take it offline. We expect 1-2 minutes.
We will perform emergency patching later today in the evening of servers:
nlsh04,nlsh05,nlsh06,nlsh07,nlcp01,nlcp02,nlcp03
nash02 will be patched now.
Downtime per server is expected to be a couple of minutes, but can be slightly longer.
perfgrid.com will be unavailable for a bit, we're fixing some network issues. It does not have any direct customer impact. Support can be reached at [email protected] as usual, and billing.perfgrid.com works for any orders and panel access.
We're aware of the unavailability of our website perfgrid.com - this is due to a hypervisor issue which is being worked on as we speak. No customer sites/services are affected.