HDFS health and FSCK_Hadoop 2.x Administration Cookbook-QQ阅读男生中文轻小说网

上QQ阅读APP看书，第一时间看更新

HDFS health and FSCK

The health of the filesystem is very important for data retrieval and optimal performance. In a distributed system, it becomes more critical to maintain the good health of the HDFS filesystem so as to ensure block replication and near-parallel streaming of data blocks.

In this recipe, we will see how to check the health of the filesystem and do repairs, if any are needed.

Getting ready

Make sure you have a running cluster that has already been up for a few days with data. We can run the commands on a new cluster as well, but for the sake of this lab, it will give you more insights if it is run on a cluster with a large dataset.

How to do it...

ssh to the master1.cyrus.com Namenode and change the user to hadoop.
To check the HDFS root filesystem, execute the hdfs fsck / command, as shown in the following screenshot:
We can also check the status of just one file instead of the entire filesystem, as shown in the following screenshot:
The output of the fsck command will show the blocks for a file, the replication status, whether blocks are corrupted, and many more details, as shown in the following screenshot:
We can also look at how the blocks of a file are laid across the cluster using the commands as shown in the following screenshot:
In the cluster named cyrus, you can see that there are some corrupt blocks. We can simulate this by manually deleting a block of a file on the lower filesystem. Each of the HDFS blocks, is a file at the lower filesystem such as EXT4.
The corrupt blocks can be fixed by deleting them, and for an under replicated block we can use the hdfs dfs -setrep 2 /input/new.txt command, so that a particular file is set to the desired number of replications. If we need to set many files to a specified number of replications, just loop through the list and do a setrep on them.

How it works...

The hdfs fsck /command is similar to the Linux fsck command. In Hadoop, it does not repair the filesystem automatically and needs a manual intervention. To see what options there are for this command, please use the hdfs fsck –help help command.