PGTS PGTS Pty. Ltd.   ACN: 007 008 568               Mobile Version Coming Soon

point Site Navigation







Valid HTML 4.01!






   Give Windows The Boot!
   And Say Goodbye To Viruses!

   Ubuntu

   If you own a netbook/laptop~
   Download Ubuntu Netbook!





Great Expectations!

By Gerry Patterson

I had a requirement for a simple script that checks connectivity. If the connection was not up, I wanted that script to telnet to my Dynalink RTA020 modem/router and reset it.

I tried out "expect" and was pleasantly surprised to find a neat package that was easy to install and use.

My Connection Is Down Again! ...

Since I started running this web site, there has been a niggling problem with the connection. Sometimes it just goes belly up. It usually comes good of its' own accord, however this may take some time. In the best case it will be a matter of only ten minutes. The worst case of eight hours is more serious. If I am aware of the problem, I can reset the modem/router (switch on/off). On rare occasions, I need to repeatedly reset it, but usually the simple act of recycling the router brings the connection back up.

I have tried quizzing my ISP about this, but they have been evasive. Initially this would only happen once a month or so. At the time I just accepted that occasionally I would lose the connection. Gradually the problem has become more frequent and lately it has become a regular (daily) occurrence!

Although I suspect that it has something to do with the ISP or the router (which was supplied by the ISP), it has the potential to be hard to pin down, since there are three parties involved (myself, the carrier and the provider). It has occurred to me that it might be possible to come up with a simple software fix.

One possibility would be automation of the process of recycling the router, which I currently perform manually (by pressing the on/off switch).

For potential software solutions that are integrated with a shell script, it is easy to detect when the connection has been lost. A simple command like the following will suffice: 

ping -c 4 123.123.123.123
This will send 4 packets to the specified address, and return a status. A suitable address needs to be chosen on the provider's network. Good candidates would be:
  1. The (immediate) other side of the connection (i.e. the carrier's modem -- in my case this is Telstra). If this is not reachable then this is almost a smoking gun.
  2. The ISP's primary DNS or primary MX host. If this fails, it indicates a network problem, most probably related to the ISP, although not necessarily the fault of the carrier (in my case the ISP's call centre staff, when asked about the reason for the outage, usually blame the carrier, or talk vaguely of the need to reset the modem "regularly").

There may be other candidates, though generally speaking either of these two alternatives would be preferable to pinging other addresses that are not on the ISP's network. It would be best to avoid resetting the line unnecessarily.

Scripting Options.

The next thing to do was choose a suitable scripting tool. In fact it would need to be more like a keyboard macro tool i.e. something which is capable of imitating user (keyboard) input. It is possible to perform a soft reset. However, it entails connecting via telnet to the router and issuing a "restart" command.

In recent years I have settled on perl as the scripting tool of choice. On the face of it perl seems to be a likely candidate for this task. There is a package called Net::Telnet which offers the required functionality.

However, as it turned out, there was a problem with Net::Telnet. The Dynalink modem does not ask for a username during the login sequence, and the Net::Telnet application was not able to connect. In fact, this type of behaviour (i.e. password only) is not uncommon for routers. It occurred to me that the package, Net::Telnet::Cisco, might have been a better choice, since it made allowance for this. However I could not get this package to work either, possibly because the device was not a Cisco router.

Having spent some time trying to get a result with Net::Telnet and Net::Telnet::Cisco, I then started to think about the possibility of hacking one of these packages. In principal, I do not prefer this as an option. Modifying standard program components for an application can lead to complications, including:

While searching for examples of other users' experiences with this problem, I found many references to the Expect scripting language. I have heard of it often in the past, and it seems the language has been around for quite a long time (more than a decade). However, I am usually reluctant to commit to learning yet another language. Generally I am not quick to adopt new technology, it was at least ten years before I considered perl. I was quite happy to plod along with shell/awk/sed.

However when I found a reference which described Expect as similar to TCL, I decided to give it a try, since I reasoned that the learning curve would not be all that steep.

In fact it appears that Expect has been embedded in the TCL language similar to the manner that the procedural language for the postgres engine had been inserted in TCL. In fact, the postgres extensions were the reason that I had eventually tried using TCL. It seemed that there were so many examples of programming with TCL when dealing with postgres databases, that I decided it would be easiest if I went with the flow.

As it turned out Expect was a breeze to download, install and program. The language is indeed very TCL-ish and easy to use (see bibliography). If you are already familiar with TCL, you will have little to learn. Note: TCL must be installed on your system, otherwise Expect will not work.

Also included with the download (version 5.39.0) were a number of excellent example programs. By referring to these, I was able to construct the simple reset program below, which amazingly, worked first time! -- Now I have to say that this is a very rare occurrence! The remarkably simple but effective code fragment that I used was as follows: 

#!/usr/local/bin/expect
# reset Dynalink RTA020 modem/router

# set these variables for your network
set router "123.123.123.123"
set passwd "password"
set rta_prompt "00:99:99:99:99:99>"

eval spawn telnet "$router"
expect "password:"	{send "$passwd\r"}
expect	"$rta_prompt"	{send "restart\r"}
sleep 2
send "\r"

There are three parameters which must be set first. They are:

(or you could just use them in the script as quoted literals).

In order to drive the script, I used a bash shell script like the following: 

#!/bin/bash
echo -e "From: webhost@mydomain.com\nTo: admin@mydomain.com
Subject: Dynalink Router Reset\n" > /tmp/myfile.tmp
ping -c 4 123.123.123.123 >> /tmp/myfile.tmp
if [ $? -ne 0 ] ; then
	/usr/sbin/sendmail admin@mydomain.com < /tmp/myfile.tmp
	/mypath/reset_line > /dev/null
fi

Where:

A possible modification of this script would include testing for 100% package loss. This would require code similar to the following: 

grep -q '0 packets received, 100% packet loss' /tmp/myfile.tmp
if [ $? -eq 0 ] ; then
	..
	..
This would ensure that the router would only be "restarted" if there was a 100% packet loss from the ping command.

Of course, Expect is quite smart enough to handle the tasks carried out by the shell script (i.e. testing the interface with ping and sending an e-mail when the router is reset). I decided to put them in a bash shell script because I already have a shell script that is run by cron every two minutes. It was easy to add this code to the existing cron job. The reason for re-directing stdout was to prevent messages being sent to the root user on the webhost. If it is ok to have the e-mail sent to this address, then just remove the redirection to the bit bucket (/dev/null) for the reset_line script.

Also, in the above example, the output from the ping command is sent as part of the e-mail notification. Once again, this is only if you want to send an e-mail.

Update.

My first impressions of this script were very good. It seemed to be reliable and the maximum downtime was less than three minutes. Then I noticed that it appeared to have a problem on odd occasions ...

Sometimes when the line (or the ISP's network) went bad it would take a couple of minutes to recover itself after being re-cycled. I should add that the normal behaviour was not like this. Most of the time the network would bounce straight back up after the modem was recycled. However, if this more worrisome behaviour manifested itself, there was a potential to become stuck in grim repetitive loop, constantly recycling the modem. This occurs because the latency for recovery is longer than the interval between runs. (i.e. it takes longer than two minutes to recover, but in two minutes it resets the modem -- rather a nasty loop).

If all the smarts had been inside the Expect script, the fix would have been simple. Since I was using the shell, the fix was still simple, but it looked ugly. (Ok, ok ... just about anything in a shell script looks ugly). The shell script now looks something like: 

#!/bin/bash
if [ -f /tmp/reset_count.tmp ] ; then
	reset_count=`awk '{print $1+1}' /tmp/reset_count.tmp`
else
	reset_count=0
fi
echo -e "From: webhost@mydomain.com\nTo: admin@mydomain.com
Subject: Dynalink Router Reset\n" > /tmp/myfile.tmp
ping -c 1 123.123.123.123 >> /tmp/myfile.tmp
if [ $? -ne 0 ] ; then
	echo $reset_count > /tmp/reset_count.tmp
	echo $reset_count | awk '{exit($1%4)}'
	if [ $? -eq 0 ] ; then
		/mypath/reset_line >> /tmp/myfile.tmp
		/usr/sbin/sendmail admin@mydomain.com < /tmp/myfile.tmp
	fi
else
	rm -f /tmp/reset_count.tmp
fi

Now, whenever the modem is recycled it waits 8 minutes before attempting to restart it. This is the reason for using the variable $reset_count (above). The first time the line goes bad, this variable will have a value of "0". It will be incremented only if the line is still bad on a successive run. The modem is restarted whenever the value ($reset_counter % 4) is zero (i.e. every fourth run). This is achieved with two awk one-liners. The first one increments the counter, the second tests the result of the modulus 4 operation.

The ping command now sends only one packet.

Conclusion.

Expect is a powerful tool that imitates user interaction with a console, and is well suited to a purpose such as automating some tasks that require telnet from a Unix console. I was able to download, install and write a simple script and have it working within half an hour. The script has continued to work reliably without many complications. So far, there has been a marked reduction in downtime.

Since writing this article, I have churned to another ISP. See the bibliography (below) for details of using the shell and expect to restart a Netgear DM111 ADSL2+ Modem.

Note: to install expect on Ubuntu use: 

	sudo apt-get install expect

You are welcome to use these ideas and/or scripts for your own purposes, whatever they may be, unless such purposes include spamming or other similar anti-social and/or illegal activities.

BIBLIOGRAPHY:

The following are some helpful links about the problem of automating or scripting telnet.
PGTS Jan 2008 Restarting a Netgear Modem. Since writing this, I have changed my ISP. The new ISP recommends the Netgear DM111 ADSL2+ Modem. Also the setup is a bit more complex.

Rob Manning Automation examples. Found with Google. Some examples of using Net::Telnet in perl -- This link is dead.

SourceForge Net::Telnet::Cisco. The official page ... gives details of the perl telnet package that has been written specifically to interact with a Cisco router.

Don Libes Expect Home Page. Expect seems to be the work of a team at NIST. And a great bit of work it is. Congratulations Don Libes et al! It's good to see a government agency putting taxpayers' money to a constructive use.

Don Libes Expect README. A quick bootstrap for the impatient, also available online at the NIST site. This README is a brief introduction to the language, and is included with the package when it is downloaded. It explains what Expect is and how to get started.