Play with HDFS using Commands Part-III

Datetime:2016-08-23 01:46:43          Topic: HDFS           Share

1. Objective

In this tutorial we are going to learn remaining important and frequently used HDFS commands with the help of which we will be able to perform HDFS file operations like copying file, changing files permissions, viewing the file contents, changing files ownership, creating directories, etc. To learn more about world’s most reliable storage layer follow this HDFS introductory guide

Looking to learn HDFS, follow these detailed tutorials: HDFS  High Availability HDFS Fault Tolerance

2. HDFS Commands

Hadoop file system shell commands are used to perform various Hadoop HDFS operations and in order to manage the files present on HDFS clusters. All the Hadoop file system shell commands are invoked by the bin/hdfs script.

2.1. touchz

Command Usage

touchz <path>

Command Example

hdfs dfs -touchz /user/dataflair/dir2

Description

It creates a file at the path containing the current time as a timestamp. Fails if a file already exists at path, unless the file is already size 0.

2.2. test

Command Usage

hdfs dfs -test -[ezd] URI

Command Example

"hdfs dfs -test -e sample
hdfs dfs -test -z sample
hdfs dfs -test -d sample"

Description

The hadoop test is used for file test operations.

It gives 1 output if path exists; it has zero length; or it is a directory or otherwise 0.

Options:

-d: if the path given by the user is a directory, then it gives 0 output.

-e: if the path given by the user exists, then it gives 0 output.

-f: if the path given by the user is a file, then it gives 0 output.

-s: if the path given by the user is not empty, then it gives 0 output.

-z: if the file is zero length, then it gives 0 output.

2.3. text

Command Usage

hdfs dfs -text <source>

Command Example

hdfs dfs -text /user/dataflair/dir1/sample

Description

Takes a source file and outputs the file in text format. The allowed formats are zip and TextRecordInputStream.

2.4. stat

Command Usage

hdfs dfs -stat path

Command Example

hdfs dfs -stat /user/dataflair/dir1

Description

Prints information about path.

%b: If the format is a string which accepts file size in blocks.

%n: Filename

%o: Block size

%r: replication

%y, %Y: modification date.

2.5. tail

Command Usage

hdfs dfs -tail [-f] <filename2>

Command Example

"hdfs dfs -tail /user/dataflair/dir1/sample
hdfs dfs -tail -f /user/dataflair/dir1/sample"

Description

Shows the last 1KB of file on console or stdout.

2.6. chown

Command Usage

hdfs dfs -chown [-R] [OWNER][:[GROUP]] URI [URI ]

Command Example

hdfs dfs -chown -R dataflair /opt/hadoop/logs

Description

Changes the owner of files. With -R, makes the change according to the order of the directory structure recursively. The user must be the superuser.

2.7. chmod

Command Usage

chmod [-R] mode,mode,... <path>...

Command Example

hdfs dfs -chmod 777 /user/dataflair/dir1/sample

Description

Changes the permissions of files. With -R, makes the change recursively by way of the directory structure. The user must be the owner of the file or the superuser.

2.8. appendToFile

Command Usage

hadoop fs -appendToFile <localsource> ... <dst>

Command Example

hadoop fs -appendToFile /home/dataflair/Desktop/sample /user/dataflair/dir1

Description

Append single sources, or multiple sources from local file system to the file system at the destination. It also reads input from standard input and add it to to destination file system.

2.9. checksum

Command Usage

hadoop fs -checksum URI

Command Example

hadoop fs -checksum /user/dataflair/dir1/sample

Description

Returns the checksum information of a file.

2.10. count

Command Usage

hdfs dfs -count [-q] <paths>

Command Example

hdfs dfs -count /user/dataflair

Description

Counts the number of directories, number of files present and bytes under the paths that match the specified file pattern.





About List