10 Most Used Hdfs Commands Part-II

Datetime:2016-08-23 01:46:46          Topic: HDFS           Share

1. Objective

In this tutorial we are going to learn remaining important and frequently used HDFS commands with the help of which we will be able to perform HDFS file operations like copying file, changing files permissions, viewing the file contents, changing files ownership, creating directories, etc. To learn more about world’s most reliable storage layer follow this HDFS introductory guide

2. HDFS Commands

Hadoop file system shell commands are used to perform various Hadoop HDFS operations and in order to manage the files present on HDFS clusters. All the Hadoop file system shell commands are invoked by the bin/hdfs script.

2.1. moveFromLocal

Command Usage

moveFromLocal <localSrc> <dest>

Command Example

hdfs dfs -moveFromLocal /home/dataflair/Desktop/sample /user/dataflair/dir1

Description

Copies the file or directory from the local file system identified by local source to destination within HDFS, and then deletes the local copy on success.

Learn more about Internals of HDFS Data Write Pipeline and File write execution flow

2.2. moveToLocal

Command Usage

moveToLocal <src> <localDest>

Command Example

hdfs dfs -moveToLocal /user/dataflair/dir2/sample /user/dataflair/Desktop

Description

Works like -get, but deletes the HDFS copy on success.

2.3. tail

Command Usage

hdfs dfs -tail [-f] <filename>

Command Example

"hdfs dfs -tail /user/dataflair/dir2/purchases.txt
hdfs dfs -tail -f /user/dataflair/dir2/purchases.txt"

Description

Shows the last 1KB of file on console or stdout.

2.4. rm

Command Usage

rm <path>

Command Example

hdfs dfs -rm /user/dataflair/dir2/sample

Description

Removes the file or empty directory present on the path provided by the user.

Command Example

hdfs dfs -rm -r /user/dataflair/dir2

Description

Recursive version of delete.

2.5. expunge

Command Usage

hdfs dfs -expunge

Command Example

hdfs dfs -expunge

Description

Used to empty the trash.

2.6. chown

Command Usage

hdfs dfs -chown [-R] [OWNER][:[GROUP]] URI [URI ]

Command Example

hdfs dfs -chown -R dataflair /opt/hadoop/logs

Description

Changes the owner of files. With -R, changes are made recursively by way of the structure of directory. User should be the superuser.

2.7. chgrp

Command Usage

hdfs dfs -chgrp [-R] <NewGroupName> <file or directory name>

Command Example

hdfs dfs -chgrp [-R] New Group sample

Description

hadoop chgrp shell command is used to change the files group association. Also you can try -R option to make changes recursively by the way of structure of directory.

2.8. setrep

Command Usage

setrep [-R] [-w] rep <path>

Command Example

hdfs dfs -setrep -w 3 /user/dataflair/dir1

Description

It is used to change the replication factor of a file. If the entered path is a directory, then this command changes the replication factor of all the files present on the directory tree rooted at path provided by user recursively.

Options:

The -w flag requests that the command wait for the replication process to get completed. This may likely take a very long time to get completed.

The -R flag is accepted for backwards compatibility. It does not makes any changes.

2.9. du

Command Usage

du <path>

Command Example

hdfs dfs -du /user/dataflair/dir1/sample

Description

Shows disk usage, in bytes, for all the files present on the path provided by the user; reporting of filenames are done with the full HDFS protocol prefix.

Command Example

hdfs dfs -du -s /user/dataflair/dir1/sample

Description

Like -du, but it prints a summary of amount of disk usage of all files/directories in the path.

2.10. df

Command Usage

hdfs dfs -df [-h] URI [URI ...]

Command Example

hdfs dfs -df -h

Description

Displays free space.





About List