MD. Nazmul Alam

Recover from read-only file system

MD. Nazmul Alam's photo
MD. Nazmul Alam
·

2 min read

We have faced system "read-only" issue after unusual power-off the VM.

Issue syndrome.

  • The VM has suddenly gone read-only, all services have crashed and giving errors when pressing tab to complete file name as: "bash: cannot create temp file for here-document: Read-only file system"

To recover this issue, we had followed the below checklists.

  • [x] Check the HDD list as mounted and find the mount point on /root directory.
  • [x] Find the UUID for /root mount point
  • [x] Remove the orphan files.
  • [x] Reboot

Check the HDD list as mounted and find the mount point on /root directory

#df-hT

Filesystem                                             1K-blocks    Used Available Use% Mounted on
rootfs                                                 249773956 2328024 234758164   1% /
udev                                                       10240       0     10240   0% /dev
tmpfs                                                     406336     200    406136   1% /run
/dev/disk/by-uuid/e45e30eb-efa4-4cd9-aaf9-c6cbe46aa41c 249773956 2328024 234758164   1% /
tmpfs                                                       5120       0      5120   0% /run/lock
tmpfs                                                    2489760       0   2489760   0% /run/shm
/dev/sdb1                                              249773956 2303784 234782404   1% /mnt/sdb1

Find the UUID for /root mount point

From "df-hT" command we can find the mount point for /root. Here the mount point is "/dev/disk/by-uuid/e45e30eb-efa4-4cd9-aaf9-c6cbe46aa41c"

For find the UUID, we will use blkid command

#blkid

/dev/sr0: UUID="2022-06-14-15-29-52-00" LABEL="config-2" TYPE="iso9660"
/dev/vda1: LABEL="cloudimg-rootfs" UUID="88baefc0-b766-40e9-865f-a671c43d0628" TYPE="ext4" PARTUUID="bb7d4753-c5cd-435f-b473-fb12c4fa2aa2"
/dev/vda15: LABEL_FATBOOT="UEFI" LABEL="UEFI" UUID="69C1-6F66" TYPE="vfat" PARTUUID="7f6ef26a-860a-4ef0-867f-916c55eb246f"
/dev/loop0: TYPE="squashfs"
/dev/loop1: TYPE="squashfs"
/dev/loop2: TYPE="squashfs"
/dev/loop3: TYPE="squashfs"
/dev/loop4: TYPE="squashfs"
/dev/loop5: TYPE="squashfs"
/dev/loop6: TYPE="squashfs"
/dev/vda14: PARTUUID="9448161e-b0b9-4265-8361-c2b986a774a7"
/dev/loop8: TYPE="squashfs"
/dev/loop9: TYPE="squashfs"
/dev/loop10: TYPE="squashfs"

Remove the orphan files.

fsck -f /dev/disk/by-uuid/88baefc0-b766-40e9-865f-a671c43d0628
Linuxread-onlyrecovery