How to restore a read-only file system
- Reboot the instance
- Log into the instance as root
- Run fsck
- Reboot the instance
Why does it happen?
The default behavior or many Linux operating systems is to set a file system to read-only if critical information can't be written to disk. If this happens the file system is corrupt and fsck has to be run to fix it. The default behavior can be changed but it is not advisable. If it is changed to "continue" errors can accumulate and corrupt the file system. If it is changed to "panic" the system will halt and fsck will need to be run on the file system before it can be restarted.
How can I detect when the file system has been changed to read-only?
The following command can be run to find read-only file system in your instance.
grep "\sro[\s,]" /proc/mounts
Details
Reboot the instance through the OpenStack Horizon dashboard.
Browse to the Nebula dashboard at nebula.ncsa.illinois.edu.
In the drop-down "Actions" menu to the right of your instance select "Soft Reboot Instance".
Confirm the reboot.
Logging in as root
To log in as root you must previously have set a password for root in the instance.
Browse to the Nebula dashboard at nebula.ncsa.illinois.edu.
In the drop-down "Actions" menu to the right of your instance select "Console".
You will be given a login prompt.
Running fsck
There is no guaranteed way to restore an instance with disk corruption. Good backups are the only way to avoid losing data.
The fsck command can recover your disk partition but there is no guarantee that it will work correctly. The fsck operation can occasionally cause data corruption on active disks. For this reason, the fsck procedure should only be performed on unmounted or read-only file system to minimize this risk. Problems can still occur in cases of severe damage though, so consider these last-resort methods for data recovery.
Use the "mount" command to verify the file system you wish to fix is read-only.
Run fsck on the file device you wish to check, for example "fsck "/dev/vdb1". You may be prompted.
After fsck has been run it is important to check the /lost+found directory. This is where fsck puts partially recovered files. Sometimes, fsck is able to recover file data, but it cannot find a reference to the file on the filesystem.When this happens, fsck places the files in the /lost+found directory so that you can manually try to figure out what the file is. If there are files in this directory check to see if you can identify them. Often these are files that were previously deleted but were still being used when the system crashed. It is worth checking them though to be sure.