HDFS Essentials commands: 101

Create, move files bidirectional — between HDFS and Local, and more….

ls command

This command is used to get the list of files for the given directory path
hdfs dfs -ls path

hdfs dfs -ls /user/akashpatel/

mkdir command

This command is used to create a directory in HDFS

hdfs dfs -mkdir path+name of a new directory

hdfs dfs -mkdir /user/akashpatel/hdfscommandexercises

touchz command

This command will help us to create an empty file in HDFS

hdfs dfs -touchz path+name of a new file

hdfs dfs -touchz /user/akashpatel/hdfscommandexercises/myNewFile.txt

put | copyFromLocal command

Both commands used to copy files from local to HDFS directories

hdfs dfs -copyFromLocal source destination OR hdfs dfs -put source destination

hdfs dfs -copyFromLocal /home/akashpatel/akash/MyTextFile.txt /user/akashpatel/hdfscommandexercises/

cat command

It is used to read the content of the file

hdfs dfs -cat /user/akashpatel/hdfscommandexercises/MyTextFile.txt

copyToLocal | get command

Both commands used to get files from HDFS to the local system

hdfs dfs -get source destination

hdfs dfs -get /user/akashpatel/hdfscommandexercises/myNewFile.txt /home/akashpatel/akash/getfiles/

moveFromLocal command

It used to move local files from HDFS

hdfs dfs -moveFromLocal /home/akashpatel/akash/MyAnotherTextFile.txt /user/akashpatel/hdfscommandexercises/

cp command

This command is used to copy a file from one location to other in HDFS

hdfs dfs -cp /user/akashpatel/hdfscommandexercises/MyTextFile.txt /user/akashpatel/hdfscommandexercises/MyTextFile_copied.txt

mv command

This command is used to move a file from one location to other (similar to cut-paste) location in HDFS.

hdfs dfs -mv /user/akashpatel/hdfscommandexercises/MyTextFile_copied.txt  /user/akashpatel/hdfscommandexercises/moveexample/

rmr command

This command is used to recursively deleting the files from the directory

hdfs dfs -rm -r /user/akashpatel/hdfscommandexercises/moveexample

ds command

It used to get the size of the files.

hdfs dfs -du /user/akashpatel/hdfscommandexercises/

-du -s command

It used to get the size of the entire directory.

hdfs dfs -du -s /user/akashpatel/hdfscommandexercises

stat command

It used to get modified date and time

hdfs dfs -stat /user/akashpatel/hdfscommandexercises

setrep command

This command is used to increase or decrease replication of the file or directory (By default it sets though core-site.xml).

hdfs dfs -setrep -R -w 6 /user/akashpatel/hdfscommandexercises/MyTextFile.txt

Jupyter Notebook file: Source code is available here.

Please Clap!! 👏 See you all in my next blog. Follow me to get more updates about data engineering.