Identify damaged files
This article gives details on how to find out which file owns a given disk sector. The main purpose for doing so is finding out which file was damaged in the event a storage device develops any bad sectors (that way you will know if you lost anything important).
For most of these commands you will have to be either root or a user that has direct read access to the drive you are checking (being a member of the disk group should be enough). As usual, a current backup is always a good idea, especially if imminent drive failure is suspected. S.M.A.R.T. can help determining that.
Filesystems
btrfs
Unlike other filesystem types, btrfs has native support for reporting on damaged files. When scrubbing a partition, btrfs reads all data and metadata blocks and verifies checksums. It automatically repairs corrupted blocks if there is a correct copy available in a RAID configuration. btrfs also reports any unreadable sector along with the affected file via system log.
Find damaged files
First of all, let us scrub the damaged partition:
# btrfs scrub start -Bd /dev/sdxy
The same partition can be scrubbed when mounted by providing its mount point:
# btrfs scrub start -Bd /mnt/point
Scrub reports can be retrieved as many times as need with the status subcommand:
# btrfs scrub status /mnt/point
scrub status for e11013b3-b244-4d1a-a9c7-3956db1a699c
scrub started at Thu Apr 23 19:07:45 2015 and finished after 372 seconds
total bytes scrubbed: 301.13GiB with 1 errors
error details: read=1
corrected errors: 0, uncorrectable errors: 1, unverified errors: 0
When done, search the kernel log for I/O errors:
# journalctl --output=cat --grep='BTRFS .* i/o error' | sort | uniq | less
Jan 16 23:14:19 my_server kernel: BTRFS: i/o error at logical 398602014720 on dev /dev/sdxy, sector 4621320, root 5, inode 23839, offset 4378624, length 4096, links 1 (path: my/damaged/file)
The output above reveals that sector 4621320 could not be read and it contains data for file my/damaged/file. The sector number can be directly used with hdparm when reallocating (see #Force the disk to reallocate bad block).
Ext2, ext3, and ext4
Debug the filesystem
The tune2fs command will give you access to all the low level structures within any ext2/ext3/ext4 filesystem.
The first thing we want to do is get the block size from the filesystem in question. Just run:
# tune2fs -l /dev/sdxy | grep Block
Block count: 29119820 Block size: 4096
In this case 4096 is the block size being used (it appears to be the default).
If you did not run badblocks using the block size that your filesystem is using then you will need to convert your block number(s) to match it (remember to use the block number(s) relative to the partition they are on).
i.e. block number 100 with a block size of 1024 bytes becomes block number 25 at 4096 bytes. The formula is:
(original block number) / ((filesystem block size) / (badblocks block size))
Now the entire point of running this program (for the purpose of this article) is to get the inode number. Before continuing to debug it, unmount the partition. Then, run the command:
# debugfs
Then in the debugfs console, use the open command on the ext partition containing the bad sector:
debugfs: open /dev/sdxy
Finally, use the testb command to get information about the block in question (in this example block 1000):
debugfs: testb blocknumber
debugfs says that block is not in use then that means the block is not allocated and is being used as free space. In that case this is a good thing as it means nothing important was damaged.If the block is in use then run this command to get the inode number
icheck blocknumber
This will return two numbers. The block number and the inode number.
Find damaged files
Use the inode number (second number from the icheck command) with the ncheck command:
ncheck inodenumber
The debugfs console will give you the full pathname to the file using the bad block. Now you will know what was actually damaged.
If the inode number is very small and ncheck fails to return a path then it is probably the journal itself that is damaged. To delete the journal simply run this command on the partition:
# tune2fs -O ^has_journal /dev/sdxy
Run the testb command again from the debugfs console on the bad block and it should be no longer marked as used if it was indeed used by the journal. To build a new journal run:
# tune2fs -j /dev/sdxy
JFS
Debug the filesystem
The jfs_debugfs command will give you access to all the low level structures within any JFS filesystem. Other filesystems such as the ext3 and ext4 filesystems have similar tools. It is probably a good idea to umount any filesystem before you run this on them. To use it just run:
# jfs_debugfs /dev/sdxy
This puts you into a command console. The first thing you should note is your aggregate block size. This is (presumably) the block size the filesystem is using. JFS seems to default to 4096 bytes.
If you did not run badblocks using the block size that your filesystem is using then you will need to convert your block number(s) to match it (remember to use the block number(s) relative to the partition they are on).
i.e. block number 100 with a block size of 1024 bytes becomes block number 25 at 4096 bytes. The formula is:
(original block number) / ((filesystem block size) / (badblocks block size))
Now the entire point of running this program (for the purpose of this article) is to get the inode number. To do this run the command:
d blocknumber 0 i
The syntax is the d command for display, the block number, the offset (just set it to 0), and the display format i for inode.
The decimal number that di_number is set to is the one we want. From here you type x to exit out of the display mode. Repeat the display command for each bad block that you have and note all of their inode numbers. For more info on the inode such as permissions and filetype type:
i inodenumber
When you have all the inode numbers type q to quit.
Find damaged files
Finally to find the damaged file you can simply use the gnu find utility. Mount your filesystem and run:
# find / -inum inodenumber
Substitute / for the mountpoint of the filesystem that the inode belongs to. If you search root and have more than one filesystem mounted (who does not?) you can find multiple files with the same inode number on different filesystems, plus find will take significantly longer. Remember, an inode is only unique to the filesystem that it is in.
ReiserFS and Reiser4
Debug the filesystem
The debugreiserfs command will give you access to all the low level structures within any ReiserFS/Reiser4 filesystem.
The first thing we want to do is get the block size from the filesystem in question. Just run:
# debugreiserfs /dev/sdxy | grep '^Blocksize'
Blocksize: 4096
In this case 4096 is the block size being used (it appears to be the default).
If you did not run badblocks using the block size that your filesystem is using then you will need to convert your block number(s) to match it (remember to use the block number(s) relative to the partition they are on).
i.e. block number 100 with a block size of 1024 bytes becomes block number 25 at 4096 bytes. The formula is:
(original block number) / ((filesystem block size) / (badblocks block size))
Now the entire point of running this program (for the purpose of this article) is to get the inode number. Before continuing to debug it, unmount the partition. Then, run the command:
Find damaged files
- debugreiserfs -1 blocknumber /dev/sdxy
This will return the information for what reiser filesystems refer to as a "leaf node". If you see a line stating Looks like unformatted then likely this means that the bad block belongs to free space and nothing important was damaged. If that line is absent and there is more output but no filenames then it likely means damage was done to a filesystem structure (at which point it might be a good idea to run at least a read-only fsck to see what is going on).
If you see a Name column with filenames returned under it then it is possible any of these could be damaged (note: there may or may not be a way to more precisely narrow down the exact file damaged). The inode number(s) for listed file(s) seems to be the second number within square brackets listed under the Object key column (this is solely for informational purposes since we already have the filenames).
Realistically if you do have a damaged block then you will likely receive an Input/Output error. In this case you can try mounting the bad partition and then running this command on the mount point (per the smartmontools guide):
# tar -cO /mydir | cat >/dev/null
XFS
Debug the filesystem
The xfs_info command will give you basic information about an XFS formatted partition and the xfs_db command will give you access to all the low level structures within any XFS filesystem.
The first thing we want to do is get the block size from the filesystem in question. Just run:
# xfs_info /dev/sdxy | grep bsize
data = bsize=4096 blocks=127739, imaxpct=25 naming =version 2 bsize=4096 ascii-ci=0 ftype=1 log =internal bsize=4096 blocks=855, version=2
In this case 4096 is the block size being used (it appears to be the default).
If you did not run badblocks using the block size that your filesystem is using then you will need to convert your block number(s) to match it (remember to use the block number(s) relative to the partition they are on).
i.e. block number 100 with a block size of 1024 bytes becomes block number 25 at 4096 bytes. The formula is:
(original block number) / ((filesystem block size) / (badblocks block size))
Now the entire point of running this program (for the purpose of this article) is to get the inode number. Before continuing to debug it, unmount the partition. Then, run the command:
# xfs_db -c 'blockget -b blocknumber' /dev/sdxy
You should get output similar to this:
setting block 0/9 to data setting inode to 131 for block 0/9 inode 131 block 9 at offset 0
In this example we now know the inode is 131 and can proceed to the next section
If instead you do not get an inode number in your output but do see the word free then likely this means that the bad block belongs to free space and nothing important was damaged.
Find damaged files
Finally to find the damaged file you can simply use the gnu find utility. Mount your filesystem and run:
# find / -inum inodenumber
Substitute / for the mountpoint of the filesystem that the inode belongs to. If you search root and have more than one filesystem mounted (who does not?) you can find multiple files with the same inode number on different filesystems, plus find will take significantly longer. Remember, an inode is only unique to the filesystem that it is in.
Note: According to the xfs_db man page the blockuse -n command can be used in conjuction with blockget -n -c blocknumber in order to print file names. Testing this did not produce the desired results though it may be worth trying anyway since it is faster than the find method.
# xfs_db -c 'blockget -n -b blocknumber' -c 'blockuse -n' /dev/sdxy
Force the disk to reallocate bad block
First you will want to see how many badblocks the harddrive is aware of through the smartctl command:
# smartctl -t long /dev/sdx
Wait until test completes, then:
# smartctl -l selftest /dev/sdx
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 5 Reallocated_Sector_Ct 0x0033 100 100 005 Pre-fail Always - 0 196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0 197 Current_Pending_Sector 0x0022 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0008 100 100 000 Old_age Offline - 0
To make the harddrive transparently map out the badblock with a spare good sector you will have to simply write zeros to the bad block using the dd command as root. Remember that with this command you have to work with the same block size as your filesystem and the block as to be relative to the partition the filesystem is on and not the harddrive as a whole:
# dd if=/dev/zero of=/dev/sdxy bs=4096 count=1 seek=2269012 $ sync
Alternatively, hdparm provides a couple of nice and simple options to read and write a given sector (4621327, in the following example):
# hdparm --read-sector 4621327 /dev/sdxy # hdparm --repair-sector 4621327 --yes-i-know-what-i-am-doing /dev/sdxy
You can see if the harddrive did indeed map out an additional bad sector by checking with the smartctl command and seeing if the reallocated sector or event count went up:
# smartctl -A /dev/sdx
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 5 Reallocated_Sector_Ct 0x0033 100 100 005 Pre-fail Always - 1 196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 1 197 Current_Pending_Sector 0x0022 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0008 100 100 000 Old_age Offline - 1
To get Offline_Uncorrectable to go back to 0 you need to run a SMART long test and a selftest:
# smartctl -t long /dev/sdx
Wait until test completes, then:
# smartctl -l selftest /dev/sdx
See also
- Tracing a disk's sector number through md and LVM to a filesystem by Andy Smith of BitFolk
- EXT2/3 badblocks howto