This is an HDFS service-level health check that checks that the number of under-replicated blocks does not rise above some percentage of the cluster’s total blocks. A failure of this health check may indicate a loss of DataNodes. Use the HDFS fsck command to identify which files contain under replicated blocks.
There are reasons for managing the replication level of data on a running Hadoop system. For example, if you don’t have even distribution of blocks across your DataNodes, you can increase replication temporarily and then bring it back down. Another example, if you are running a YARN job and want to give local access to a file or dataset set replication equal to the number of DataNodes/ResourceManagers.
To set replication of an individual file to 4:
sudo -u hdfs hadoop dfs -setrep -w 4 /path/to/file
You can also do this recursively. To change replication of entire HDFS to 1:
sudo -u hdfs hadoop dfs -setrep -R -w 1 /
To script a fix for under-replicated blocks in HDFS, try the following:
su - hdfs
hdfs fsck / | grep 'Under replicated' | awk -F':' '{print $1}' >> /tmp/under_replicated_files
for hdfsfile in `cat /tmp/under_replicated_files`; do echo "Fixing $hdfsfile :" ; hadoop fs -setrep 3 $hdfsfile; done</div>
<div>
