rise1 compute host r/o filesystem

Incident Report for Safespring

Resolved

This incident has been resolved.
Posted Sep 06, 2025 - 18:19 CEST

Monitoring

A fix has been implemented and we are monitoring the results.
Posted Sep 06, 2025 - 13:19 CEST

Update

unfortunately the affected drive vanished after a reboot, which means we are not unable to recover the localdrive (l2. flavor) instances on this host.
b2 flavored instances are booted back up and are being migrated to other compute hosts.
Posted Sep 06, 2025 - 13:19 CEST

Update

copying files consitantly failed, so we are skipping that and trying to reboot the host
Posted Sep 06, 2025 - 12:29 CEST

Update

we are currently trying to copy the instances' data over to a working compute host, this will at the current speed take around 70 minutes.

following that we will do a reboot of the affected host to see if that will make it possible to get the drive back to a r/w state so we can migrate off.

if that does not work we will attempt recreating instances on other hosts manually.
Posted Sep 06, 2025 - 12:14 CEST

Identified

one of the compute hosts in the rise1 site has issues with its local drives. instances using these drives will not be able to write to them.
Posted Sep 06, 2025 - 11:44 CEST
This incident affected: Safespring Private Cloud.