Systems administrators need a bag of tricks to ensure that everything runs smoothly without any hitches. Linux has a fine set of utilities and commands to assist sysadmins in their task. Mastering these tools will take the efficiency levels of Linux admins to a whole new level.
GNU/Linux is one of the most popular operating systems for servers. Today, most of the operating systems use advanced and modern graphical user interfaces (GUIs) but the CLI (command line interface) is still popular. Using the CLI/scripts, you can automate complicated tasks and execute them in a repetitive manner. In this article, we will discuss the most common CLI utilities. If you are familiar with GNU/Linux and want to become more productive, then this article is for you.
1) find
Searching for a file in a file system is a very common task and we have to do it quite often every day. GNU/Linux provides the find command which searches for files in a directory hierarchy. Given below is the syntax of the find command:
<span style="color: #000000;">find [STARTING-POINT] [EXPRESSION]</span>
In the above example:
• STARTING-POINT represents the directory’s location, from where the search will start. Note that if STARTING-POINT is omitted, then the search will begin from the current directory.
• EXPRESSION is evaluated to search the file. EXPRESSION can be a name, type, size, permission, owner, and so on.
Let us search for a file with a given name. Here, we’ll be using the file name as an EXPRESSION.
<span style="color: #000000;">$ find src-dir -name hello.txt</span>
In the above example, src-dir is the STARTING-POINT and hello.txt is the EXPRESSION. Here the -name option indicates that we are searching for the file by name. If you want to perform a case-insensitive search operation, then use the -iname option instead.
For the find command, EXPRESSION can be a pattern as well. Many times we want to search certain types of files like .txt, .jpg, .mp3 and so on. The example below shows how to use a pattern in EXPRESSION:
<span style="color: #000000;">$ find src-dir -name “*.txt”</span>
The above example will search and list all .txt files recursively from src-dir.
We can use the file type as an EXPRESSION with the find command. For instance, use the command below to search only directories:
<span style="color: #000000;">$ find src-dir -type d</span>
In the above example:
• The type option indicates we are performing a search based on the file type
• Argument d indicates the directory file type
In addition to the directory, the find command supports the following file types:
<span style="color: #000000;">• b: block device</span> <span style="color: #000000;">• c: character device</span> <span style="color: #000000;">• f: regular file</span> <span style="color: #000000;">• l: symbolic links</span> <span style="color: #000000;">• p: named pipe</span> <span style="color: #000000;">• s: socket</span>
The find command allows you to perform a search based on the file size. We can provide an EXPRESSION which will compare files that are greater than, less than or equal to the provided size. We can also perform a search based on permissions, for which we have to use the -perm option. For instance, the command below searches for files with the 664 permission:
<span style="color: #000000;">$ find src-dir -type f -perm 664</span>
The find command allows you to perform some additional operations while searching. For instance, it provides the -delete option, which will remove a file that matches with EXPRESSION. The following example shows the usage of the -delete option:
<span style="color: #000000;">$ find src-dir -type f -size +2k -delete</span>
In the above example, all files greater than 2KB in size will be deleted.
We can also execute bash commands while performing a search. We can achieve this using the -exec option.
2) diff
We often need to compare the contents of files. Doing this manually is a tiresome and error-prone task. But fortunately, GNU/Linux provides a command for this, which will compare files, line by line, and report if any differences are found. When the diff command is combined with the patch command, it makes a powerful combination. With this command, we can apply changes from one file to the other. This section describes both these commands.
First, create two files with the following content:
<span style="color: #000000;"># file-1: version1.txt</span> <span style="color: #000000;">str1</span> <span style="color: #000000;">str2</span> <span style="color: #000000;">str3</span> <span style="color: #000000;"># file-2: version2.txt</span> <span style="color: #000000;">str1</span> <span style="color: #000000;">str3</span>
Now compare these files using the -u option, which stands for unified diff:
<span style="color: #000000;">$ diff -u version1.txt version2.txt</span> <span style="color: #000000;">--- version1.txt 2017-12-30 14:06:38.120849370 +0530</span> <span style="color: #000000;">+++ version2.txt 2017-12-30 14:06:46.976750148 +0530</span> <span style="color: #000000;">@@ -1,3 +1,2 @@</span> <span style="color: #000000;">str1</span> <span style="color: #000000;">-str2</span> <span style="color: #000000;">str3</span>
The above output shows that the line ‘str2’ is not present in the version2.txt file. We can store this diff output in a file and apply it as a patch. To create a patch file, just redirect the output to some file as shown below:
<span style="color: #000000;">$ diff -u version1.txt version2.txt > diff.patch</span>
If we apply this patch to the version1.txt file, then it will remove the ‘str2’ line from this file. The example below shows this:
<span style="color: #000000;">$ patch -p1 version1.txt < diff.patch</span> <span style="color: #000000;">patching file version1.txt</span> <span style="color: #000000;">$ diff -u version1.txt version2.txt</span>
After applying the patch, both files will be identical; hence, the diff command does not show any differences here.
To revert the patch, execute the commands given below and follow the on-screen instructions shown below:
<span style="color: #000000;">$ patch -p1 version1.txt < diff.patch</span> <span style="color: #000000;">patching file version1.txt</span> <span style="color: #000000;">Reversed (or previously applied) patch detected! Assume -R? [n] y</span>
The combination of diff and patch is really powerful. Many version control systems like Git, Subversion and CVS use this feature.
3) rename
Renaming multiple files is one of the common tasks of a sysadmin. GNU/Linux provides the rename command which will serve our purpose. It is particularly useful when we want to rename multiple files with a specific pattern. For instance, the command below renames all .TXT files to .txt:
<span style="color: #000000;">$ rename ‘s|.TXT|.txt|’ *</span>
Storing similar types of files in a directory is also a very common task. We can do it very easily with a combination of find and the mv command. The command below moves all MP3 files to a target-dir directory:
<span style="color: #000000;">$ find src-dir -type f -name “*.mp3” -exec mv {} target-dir \;</span>
4) tar
Sometimes it is convenient to operate on a single file rather than multiple files and here, the tar command comes into the picture. tar is a short form for ‘tape archive’. As the name suggests, it is an archiving utility that stores multiple files into a single one. Given below is the syntax of the tar command:
<span style="color: #000000;">tar [OPTIONS] [TAR NAME] [FILES TO BE INCLUDED IN TAR]</span>
To create a tar bundle, execute the command given below in a terminal:
<span style="color: #000000;">$ tar cvf archive.tar 1.txt 2.txt 3.txt</span>
In the above example:
• c option stands for create archive
• v option stands for verbose mode
• f option stands for file names mentioned for archive
The tar command allows us to manipulate tar bundles without recreating them again. For instance, to add a new file into an archive, use the -r option as shown below:
<span style="color: #000000;">$ tar rvf archive.tar 4.txt</span> <span style="color: #000000;">4.txt</span> <span style="color: #000000;">$ tar tf archive.tar #list the content of tar file</span> <span style="color: #000000;">1.txt</span> <span style="color: #000000;">2.txt</span> <span style="color: #000000;">3.txt</span> <span style="color: #000000;">4.txt</span>
By default, tar only archives multiple files; it doesn’t do any compression. There are various compression utilities available like bzip2, gzip, zip and so on. To compress a tar bundle using bzip2, execute the command show below:
<span style="color: #000000;">$ bzip2 archive.tar</span>
After compression, it will append the .bz2 extension to the tar bundle. If you compare sizes, before compression, the tar bundle was 20KB and after compression, it gets reduced to 4KB, as shown below:
<span style="color: #000000;"># Size before compression</span> <span style="color: #000000;">$ ls -sh archive.tar</span> <span style="color: #000000;">20K archive.tar</span> <span style="color: #000000;"># Size after compression</span> <span style="color: #000000;">$ ls -sh archive.tar.bz2</span> <span style="color: #000000;">4.0K archive.tar.bz2</span>
5) fdisk
We partition disks for better management and utilisation of available storage. GNU/Linux provides the gnome-disk utility which is a GUI based application. However, we can do similar things with fdisk, which is a CLI based utility and can be used to manipulate the disk partition table.
Manipulating disk partitions recklessly will cause data loss; hence, we are going to use the fdisk command with a pseudo disk. We’ll use a file as a disk, using the losetup command. Perform the steps given below to create a pseudo disk.
First, create a file of size 200MB using the dd command:
<span style="color: #000000;">$ dd if=/dev/zero of=disk.img bs=1M count=200</span>
Next, set up this file as a loop-back device so that, hereafter, we can use /dev/loop0 as a device:
<span style="color: #000000;">$ sudo losetup /dev/loop0 disk.img</span>
We can perform various actions with fdisk like printing the partition table, creating new partitions, deleting existing partitions, writing the partition table to disk, and so on. Let us perform all these actions, one by one.
To start the fdisk utility, use the command given below:
<span style="color: #000000;">$ sudo fdisk /dev/loop0</span>
After entering the above command, you will be shown a welcome message and the system will wait for a command to be entered. The section below describes various actions that can be performed using fdisk.
- Print the partition table – To print the partition table, type p and press Enter. This will display information about the disk and its partitions. As we haven’t created any partition yet, it will show only information about the disk.
- Create a new partition – To create a new partition, type n and press Enter. Then follow the on-screen instructions.
- Validate a created partition – To view the created partitions, type p and press Enter.
- Delete a partition – To delete a partition, type d and press Enter. Follow the on-screen instructions to complete the procedure.
- Write a partition table – To make the changes permanent, we need to write this partition table to the disk. Type w and press Enter to complete this action.
- Quit –Type q and press Enter any time to quit the fdisk utility.
6) Networking related commands
Networking is an essential part of a computer system. However, it is complex and can be unstable sometimes. This section discusses a few open source utilities that will help to debug networking related issues.
- ping
We can use the ping command to check connectivity between hosts. It uses the Internet Control Message Protocol (ICMP) to check connectivity. Given below is the syntax of the command:
<span style="color: #000000;">$ ping [ADDRESS OF HOST]</span>
If you are connected to the Internet and the host is reachable, it’ll start displaying ping statistics. Press Ctrl+c to abort it. We can also specify the packet count for ping. It’ll stop automatically after sending count packets. For instance, the command below will stop after sending four packets:
<span style="color: #000000;">$ ping -c 4 google.com</span>
- host
This is the DNS lookup utility, which can be used to convert host names to IP addresses and vice versa. For instance, the command below prints all the IP addresses attached to the google.com domain:
$ host google.com google.com has address 216.58.203.206 google.com has IPv6 address 2404:6800:4009:806::200e google.com mail is handled by 30 alt2.aspmx.l.google.com. google.com mail is handled by 10 aspmx.l.google.com. google.com mail is handled by 50 alt4.aspmx.l.google.com. google.com mail is handled by 20 alt1.aspmx.l.google.com. \google.com mail is handled by 40 alt3.aspmx.l.google.com.
Alternatively, you can also use the nslookup utility for DNS lookup.
- route
The route command is used to display routing table information. This table is maintained by the operating system. Execute the command given below to display the routing table on your host:
$ route -n
- traceroute
When we send a packet from source to destination, it may travel through multiple gateways. If we want to find those intermediate gateways, then we can use the traceroute command as follows:
$ traceroute -n google.com
The above command will show all the intermediate gateways between your host and google.com.
7) wget
Often, we download contents from the Internet/network. Most of the time, we use the browser to do this. However, GNU/Linux provides the wget utility, which can be used as a network downloader. This section describes a few examples. Given below is the syntax of the wget command:
wget [OPTIONS] [URL]
While downloading, it displays the progress bar, which shows the following:
— Percentage of the download completed
— Total amount of bytes downloaded so far
— Current download speed
— Remaining time to download
Like other utilities, wget is also a powerful utility. It provides various facilities to make our life easy. If your Internet connection is not stable, then downloading may be interrupted. In that case, we can provide a retry count. For instance, in the example that follows, we have provided the retry count as 3:
$ wget -t 3 <URL>
It’ll retry three times before throwing out an error. To provide infinite retries, set the retry count to 0 as shown in the example below:
$ wget -t 0 <URL>
Being a very flexible utility, wget can also be used to restrict the downloading speed. It provides the –limit-rate option for this. For instance, use the command given below to set the downloading rate to 512KB:
$ wget --limit-rate=512K <URL>
One of the nice things about wget is that if downloading is interrupted, then it can be resumed from that point. Use the command given below to resume an interrupted download:
$ wget -c <URL>
8) Working with a remote host
We often interact with remote hosts to download or upload content. This section discusses command line utilities that will perform these tasks.
- scp
One of the common tasks is to transfer files between the remote and the local host. GNU/Linux provides a remote copy program, namely ‘scp’, which stands for ‘secure copy’. It uses ssh for data transfer, and uses the same authentication and provides the same security as ssh. Given below is the syntax of the scp command:
scp [OPTIONS] user@src-host:/src-dir user@dst-host:/target-dir
To copy the contents from a remote host to the local one, execute the command given below in a terminal:
$ scp -r user@remote-host.com:/remote-dir-path local-dir-path
In the above example:
— r option stands for recursive. It will be useful while copying directories
— user is the user name of the remote host
— remote-host.com is the IP address/DNS of the remote host
- ssh
Sometimes, we need to execute a command on the remote host. Obviously, we can log in to that server and execute the command there, but what if we want to capture the output of that command and use it on the local machine? In such a scenario, we can instruct SSH to execute the command on the remote host using the syntax given below:
$ ssh user@remote-host.com [COMMAND]
For instance, the command given below executes the ls command on the remote host:
$ ssh user@remote-host.com ls
- rsync
rsync is a remote as well as local file-copying tool. The rsync utility is used to synchronise the files and directories from one location to another in an effective way.
To synchronise directories on the local host, execute the rsync command as follows:
$ rsync -zvr src-dir target-dir
In the above example:
— z option stands for ‘enable compression’
— v option stands for ‘verbose mode’
— r option stands for ‘recursive mode’
To synchronise the remote directory, we have to provide an IP address and user name for that host. For instance, the following command synchronises the local directory with the remote host:
$ rsync -zvr src-dir user@remote-host.com:target-dir
9) cron
We perform many kinds of tasks on a day-to-day basis; for instance, taking backup of important data, checking for updates, and many more. Wouldn’t it be great if we automate these tasks? We can achieve this using cron. We can write cron jobs, which will be scheduled periodically. This section provides practical examples of cron.
To list all available cron jobs, execute the command given below:
$ crontab -l
If any cron job is configured, then it’ll be listed here; otherwise, the output will be empty.
Cron jobs are stored in plain text files. To edit those files, we have to use the crontab command. But before that, let us understand the cron job format.
A cron job consists of the following six entries:
M H DOM MON DOW COMMAND
In the above example:
— M stands for ‘minutes’
— H stands for ‘hour’
— DOM stands for ‘day of the month’
— MON stands for ‘month’
— DOW stands for ‘day of the week’
— COMMAND field indicates the command/script to be executed periodically
For instance, to run a job at 5.00 am every week, we can add the following entry:
0 5 * * 1 script.sh
To add the above entry into cron, perform the following steps:
— Enter the crontab -e command in the terminal and follow the on-screen instructions:
$ crontab -e
— Add the cron job entry and save the file.
0 5 * * 1 script.sh
That’s it; and cron will schedule this job at the right time.
10) System monitoring
GNU/Linux provides many utilities to monitor the system. We can monitor memory usage, disk usage, CPU usage and so on. This section discusses some of the popular utilities that can be used to monitor memory and disk usage.
- free
GNU/Linux provides the free command to check memory usage. It displays the total amount of free and used physical and swap memory in the system, as well as the buffers and caches used by the kernel. Shown below is a sample output of the free command:
$ free total used free shared buff/cache available Mem: 8117768 1267836 3718996 153112 3130936 6393176 Swap: 2097148 0 2097148
In the above output:
— ‘total’ stands for the total installed memory on current system
— ‘used’ stands for the used memory. It is calculated as follows: [total – (free + buffers + cache) memory]
— ‘free’ stands for unused memory
— ‘shared’ stands for the shared memory used by tmpfs
— ‘buffers’ stands for the memory used by kernel buffers
— ‘cache’ stands for the memory used by the page cache and slabs
— ‘buff/cache’ stands for the sum of the buffers and cache memory
— ‘available’ stands for an estimation of how much memory is available for starting new applications, without swapping.
- du
As the name suggests, the du command is used to calculate disk usage. It summarises disk usage of the set of files/directories.
To calculate the size of the directory, execute the command given below:
$ du -sh DIR-PATH
In the above example:
— option s is used to display only a total for each argument
— option h is used to show the output in human readable format (K for KB, M for MB and so on)
- df
As the name suggests, the df command is used to get information about free disk space. It reports the file system’s disk space usage. If no file name is given, the space available on all currently mounted file systems is shown:
$ df -h
In the above example:
— option h is used to show output in a human readable format (K for KB, M for MB and so on)
In this article, we have discussed some of the popular GNU/Linux utilities briefly. Mastering these utilities will take your knowledge to the next level. To know more about each utility, do refer to the official documentation.
