Scripting a Simple Download Scheduler

Logging in

Command-line browsing in Lynx is all about moving the cursor in between hyperlinks and text fields (user name and password) in the login page by using the Tab key, and entering text into the text fields. In order to follow a link, we select the link by using Tab and press Enter. So it is understood that we will keep a text file created by Lynx, which it will use while running the start.sh script. To create the file, run the following command:

lynx -cmd_log=dir_in <URL of your login page>

The browser will open the login page. Now move the cursor to the text field where we are supposed to enter the user name and password, and enter the necessary data. After that, try to log in by moving to the login link and pressing Enter.

Remember that the process of logging in may vary with ISPs and the type of connection you have. For example, those with always-on (aided by a dedicated router/modem) connections can skip building the start.sh script, as they will get automatically logged in after they boot their computer. Here I am sticking to the core idea of a Web-based login method—i.e., reading keystrokes from a file, and even that, only if users have to submit their profile to authenticate their identity. Note that the name of the file given here is dir_in keeping in mind it contains directions for the browser to log in.

Now that you have logged in, you can close your browser. Press q and it will ask for confirmation. Enter y. That’s it. Our dir_in file is ready.

You can see that dir_in has got all the keystrokes you’ve made. Also, you can see your user name and password. So, make sure you have changed the read, write and execution permissions of the file:

chmod 700 dir_in

Hereafter, if you need to log in, you can simply run the following command:

lynx -cmd_script=dir_in <URL of your login page>

…which will fill your authentication details, submit the data, and will finally quit from the browser to return to the terminal.

Finally, save the above command in a file named login.sh.

What if you’re not able to connect the first time you invoked login.sh. In that case, you will have to run login.sh repeatedly till the connection is established. For that, embed the above command in the script within a while loop as follows:

#login.sh
#!bin/bash
status=1
while [ $status -ne 0 ]
do
    lynx -cmd_script=dir_in <URL of login page>
    status=$?         #  $? returns 0 if previous command was executed successfully.
done

That makes our final login.sh script file ready for use.

Downloading

Now that the system is connected, the next step is downloading the required files. We will use wget as our download utility, which accepts the URL of the file to be downloaded as its argument. We can invoke wget as follows:

wget <URL>

You can specify the folder to which wget saves the downloaded material. Simply add an option, --directory-prefix=<target folder>. For example,

wget --directory-prefix=/media/new_volume <URL>

Here, /media/new_volume is the mount point for my backup drive, which contains all my downloaded files.

However, before we create the download scheduler script, we should create a collection of URLs of the files we wish to download, one after the other.

Let’s now write another script called download.sh, which contains a set of wget commands pointing to the required URLs. Here’s an example:

#download.sh
#!bin/bash
wget –directory-prefix=<target folder> <URL of file 1>
wget –directory-prefix=<target folder> <URL of file 2>
##[append as many wget commands as you wish]
wget –directory-prefix=<target folder> <URL of file n>

The above script is, naturally, the most important part of our start.sh script. It includes login.sh and download.sh and, of course, is the script you need to run in any one of the three terminal windows as mentioned earlier. The function of start.sh is simple—run login.sh and then download.sh. Here’s what it should look like:

#start.sh
#!bin/bash
bash login.sh
bash download.sh