sto3 compute nodes down

Incident Report for Safespring

Resolved

at 00:01 1 compute node in sto3 was reported down by our monitoring system
at 00:06 3 more compute nodes in sto3 were reported down
00:13 hosts were verified as unavailable
00:27 first host rebooted
00::43 last host rebooted
00:58 all instances back up
~08:30 some instances were reported as not reachable
~09:00 issues found in iptables on rebooted hosts
~09:15 iptables issue fixed on all hosts

Posted Jan 26, 2023 - 00:00 CET