Resolved -
This incident has been resolved.
Sep 6, 18:19 CEST
Monitoring -
A fix has been implemented and we are monitoring the results.
Sep 6, 13:19 CEST
Update -
unfortunately the affected drive vanished after a reboot, which means we are not unable to recover the localdrive (l2. flavor) instances on this host.
b2 flavored instances are booted back up and are being migrated to other compute hosts.
Sep 6, 13:19 CEST
Update -
copying files consitantly failed, so we are skipping that and trying to reboot the host
Sep 6, 12:29 CEST
Update -
we are currently trying to copy the instances' data over to a working compute host, this will at the current speed take around 70 minutes.
following that we will do a reboot of the affected host to see if that will make it possible to get the drive back to a r/w state so we can migrate off.
if that does not work we will attempt recreating instances on other hosts manually.
Sep 6, 12:14 CEST
Identified -
one of the compute hosts in the rise1 site has issues with its local drives. instances using these drives will not be able to write to them.
Sep 6, 11:44 CEST