Find and replace is a common requirement, for example it is easier to find files using a GNU/Linux server with the help of the fdupes utility. Fdupes is a Linux utility that finds duplicate files in a given directory and subdirectory, identifies duplicate files by comparing the file's MD5 signature and then doing a byte-to-byte comparison. With fdupes, you can pass many items to list, delete, and replace files that have duplicate file hardlinks.

The fdupes are compared in the following order:

Size Comparison > Partial MD5 Signature Comparison > Full MD5 Signature Comparison > byte-to-byte comparison.

Run the following command on a Debian-based system in Linux, such as Ubuntu and Linux Mint. Install fdupes (1.51) :

$ sudo apt-get install fdupes

On CentOS/RHEL and Fedora based systems, you need to open the epel repository to install the fdupes package:

# yum install fdupes

# dnf install fdupes    [On Fedora 22 onwards]

Starting with Fedora 22, the default package manager yum was replaced by dnf. Create several duplicate files under a directory, such as tecmint, as follows:

$ mkdir /home/"$USER"/Desktop/tecmint && cd /home/"$USER"/Desktop/tecmint && for i in {1.. 15}; do echo "Jtti is an ideal IDC service provider." > tecmint${i}.txt ;  done

After running the above command, let's use the ls command to verify that duplicate files have been created.

$ ls -l

total 60

-rw-r--r-- 1 tecmint tecmint 65 Aug  8 11:22 tecmint10.txt

-rw-r--r-- 1 tecmint tecmint 65 Aug  8 11:22 tecmint11.txt

-rw-r--r-- 1 tecmint tecmint 65 Aug  8 11:22 tecmint12.txt

-rw-r--r-- 1 tecmint tecmint 65 Aug  8 11:22 tecmint13.txt

-rw-r--r-- 1 tecmint tecmint 65 Aug  8 11:22 tecmint14.txt

-rw-r--r-- 1 tecmint tecmint 65 Aug  8 11:22 tecmint15.txt

-rw-r--r-- 1 tecmint tecmint 65 Aug  8 11:22 tecmint1.txt

-rw-r--r-- 1 tecmint tecmint 65 Aug  8 11:22 tecmint2.txt

-rw-r--r-- 1 tecmint tecmint 65 Aug  8 11:22 tecmint3.txt

-rw-r--r-- 1 tecmint tecmint 65 Aug  8 11:22 tecmint4.txt

-rw-r--r-- 1 tecmint tecmint 65 Aug  8 11:22 tecmint5.txt

-rw-r--r-- 1 tecmint tecmint 65 Aug  8 11:22 tecmint6.txt

-rw-r--r-- 1 tecmint tecmint 65 Aug  8 11:22 tecmint7.txt

-rw-r--r-- 1 tecmint tecmint 65 Aug  8 11:22 tecmint8.txt

-rw-r--r-- 1 tecmint tecmint 65 Aug  8 11:22 tecmint9.txt

The above script creates 15 files, which are tecmint1.txt, tecmint2.txt... tecmint15.txt, each file contains the same data, namely:

"Jtti is an ideal IDC service provider."

Now search for duplicate files in the tecmint folder.

$ fdupes /home/$USER/Desktop/tecmint  

/home/tecmint/Desktop/tecmint/tecmint13.txt

/home/tecmint/Desktop/tecmint/tecmint8.txt

/home/tecmint/Desktop/tecmint/tecmint11.txt

/home/tecmint/Desktop/tecmint/tecmint3.txt

/home/tecmint/Desktop/tecmint/tecmint4.txt

/home/tecmint/Desktop/tecmint/tecmint6.txt

/home/tecmint/Desktop/tecmint/tecmint7.txt

/home/tecmint/Desktop/tecmint/tecmint9.txt

/home/tecmint/Desktop/tecmint/tecmint10.txt

/home/tecmint/Desktop/tecmint/tecmint2.txt

/home/howtoing/Desktop/howtoing/howtoing5.txt

/home/howtoing/Desktop/howtoing/howtoing14.txt

/home/howtoing/Desktop/howtoing/howtoing1.txt

/home/howtoing/Desktop/howtoing/howtoing15.txt

/home/howtoing/Desktop/howtoing/howtoing12.txt

Recursively search for duplicates under each directory (including its subdirectories) with the -r option. All files and folders are searched recursively, and depending on the number of files and folders, it takes some time to scan for duplicates. During this time, you will see the total progress in the terminal, as shown below.

$ fdupes -r /home

Progress [37780/54747] 69%

Use the -S option to view the size of duplicates found in the folder.

$ fdupes -S /home/$USER/Desktop/tecmint

65 bytes each:                          

/home/tecmint/Desktop/tecmint/tecmint13.txt

/home/tecmint/Desktop/tecmint/tecmint8.txt

/home/tecmint/Desktop/tecmint/tecmint11.txt

/home/tecmint/Desktop/tecmint/tecmint3.txt

/home/tecmint/Desktop/tecmint/tecmint4.txt

/home/tecmint/Desktop/tecmint/tecmint6.txt

/home/tecmint/Desktop/tecmint/tecmint7.txt

/home/tecmint/Desktop/tecmint/tecmint9.txt

/home/tecmint/Desktop/tecmint/tecmint10.txt

/home/tecmint/Desktop/tecmint/tecmint2.txt

/home/tecmint/Desktop/tecmint/tecmint5.txt

/home/tecmint/Desktop/tecmint/tecmint14.txt

/home/tecmint/Desktop/tecmint/tecmint1.txt

/home/tecmint/Desktop/tecmint/tecmint15.txt

/home/tecmint/Desktop/tecmint/tecmint12.txt

You can use both the -S and -r options to see the size of duplicate files in each directory and subdirectory you encounter, as follows:

$ fdupes -Sr /home/avi/Desktop/

65 bytes each:                          

/home/tecmint/Desktop/tecmint/tecmint13.txt

/home/tecmint/Desktop/tecmint/tecmint8.txt

/home/tecmint/Desktop/tecmint/tecmint11.txt

/home/tecmint/Desktop/tecmint/tecmint3.txt

/home/tecmint/Desktop/tecmint/tecmint4.txt

/home/tecmint/Desktop/tecmint/tecmint6.txt

/home/tecmint/Desktop/tecmint/tecmint7.txt

/home/tecmint/Desktop/tecmint/tecmint9.txt

/home/tecmint/Desktop/tecmint/tecmint10.txt

/home/tecmint/Desktop/tecmint/tecmint2.txt

/home/tecmint/Desktop/tecmint/tecmint5.txt

/home/tecmint/Desktop/tecmint/tecmint14.txt

/home/tecmint/Desktop/tecmint/tecmint1.txt

/home/tecmint/Desktop/tecmint/tecmint15.txt

/home/tecmint/Desktop/tecmint/tecmint12.txt

107 bytes each:

/home/tecmint/Desktop/resume_files/r-csc.html

/home/tecmint/Desktop/resume_files/fc.html

In addition to searching recursively in one folder or all folders, you can also choose between two folders or three folders as desired. Not to mention, you can also use the options -S and/or -r if desired.

$ fdupes /home/avi/Desktop/ /home/avi/Templates/

To delete duplicate files and keep copies, use option -d. Use this option to be extremely small, otherwise you may lose necessary files/data and the process will not be recoverable.

$ fdupes -d /home/$USER/Desktop/tecmint

[1] /home/tecmint/Desktop/tecmint/tecmint13.txt

[2] /home/tecmint/Desktop/tecmint/tecmint8.txt

[3] /home/tecmint/Desktop/tecmint/tecmint11.txt

[4] /home/tecmint/Desktop/tecmint/tecmint3.txt

[5] /home/tecmint/Desktop/tecmint/tecmint4.txt

[6] /home/tecmint/Desktop/tecmint/tecmint6.txt

[7] /home/tecmint/Desktop/tecmint/tecmint7.txt

[8] /home/tecmint/Desktop/tecmint/tecmint9.txt

[9] /home/tecmint/Desktop/tecmint/tecmint10.txt

[10] /home/tecmint/Desktop/tecmint/tecmint2.txt

[11] /home/tecmint/Desktop/tecmint/tecmint5.txt

[12] /home/tecmi