PGTS Humble Blog
Thread: Tips/Tricks For Programming etc
|Gerry Patterson, The World's Most Humble Blogger|
Problems With /etc/fstab And Server Upgrade
Chronogical Blog Entries:
Date: Tue, 27 Dec 2011 19:03:52 +1100
For a considerable amount of time one my workstations has been throwing up errors about not being able to determine the root device. These would often appear during an APT update.
As far as I can remember such errors started to occur after updating to Ubuntu 11.10 (Oneiric). After a recent update which involved replacing the Linux headers the following error appeared:
Cannot determine root device. Assuming /dev/hda1 This error is probably caused by an invalid /etc/fstab
Until recently I had chosen to ignore the errors, because they didn't seem to prevent the machine from working ... And I am usually cautious about fiddling with anything that mentions "the root device" ... Fiddling with such things can sometimes have disastrous consequences!
However, a few weeks ago, after running another APT upgrade I decided to look at what I might do in order to fix the errors. Since the message mentioned /etc/fstab, I thought that might be a good place to start.
I should add that after many version upgrades and several additional hard disks, and a complete system board replacement, my /etc/fstab was a bit of a dog's dinner. The problem is that adding and/or removing disks can change the order in which the system identifies them and this can change the device name. And when I make modifications to /etc/fstab on the fly I usually use the device name. Also changing components on your system board can disturb the order in which devices are identified.
It seems that at one stage during the software upgrade process, Ubuntu started to identify partitions with their UUID (Universal Unique Identifier). Which actually solves the problems of different device names ... Or it would have if my /etc/fstab had not been so badly hacked about by yours truly.
The theory behind using the UUID is that it uses a unique lable to identify devices rather than a device name and this makes it possible to alter the order of how devices attach to the system board without altering their mount points. And it is now the preferred way to identify mount points in Ubuntu.
Somewhere along the line I had lost my mount point for the root folder. It seems the kernel was still able to work out how to mount the root folder, even without explicit instructions in /etc/fstab. And it seems I had lost the mount point for my swap partition (Oops!). I was surprised that my poor little workstation was working at all.
You can view your UUID values by entering this command:
ls -l /dev/disk/by-uuid/
According to the man pages it is more reliable to use the blkid command:
sudo blkid -p /dev/sda1
Also, once I located the swap partition, I thought it might be a good idea to read up on the swapon command in the man pages.
Having identified and obtained the UUID of the various devices that should be in /etc/fstab, I then replace the mount points with the new ones such as follows:
# /etc/fstab: static file system information. # # /dev/sdb1 / ext3 errors=remount-ro 0 1 UUID=12e24a98-3108-49f7-a3ce-4965874752af / ext3 errors=remount-ro 0 1 proc /proc proc defaults 0 0 # /dev/sdb3 /data ext2 defaults 0 0 UUID=caa1bb55-ce45-4a82-9813-6df5d12f37bc /data ext2 defaults 0 0 UUID=b58c2c74-9280-4fea-8acd-e9035fe82311 none swap sw 0 0 # # nfs # lan:/exports/nfs /nfs nfs defaults 0 0 smtp:/pgts /pgts nfs defaults 0 0 pgts07:/Volumes/d1/pgts07 /pgts07 nfs defaults 0 0
The tense part comes after making such changes and rebooting ... Devout sysadmins might be inclined to say a quick "hail mary" as the boot sequence starts.
Fortunately in my case the problems were all solved ... No more weird errors in the log files and my swap partition is back!
Problems With Server Upgrade Ubuntu 10.4
A few weeks ago I got a very worrying message whilst running an upgrade on an Ubuntu server:
Setting up linux-headers-server (188.8.131.52.43) ... dpkg: dependency problems prevent configuration of linux-server: linux-server depends on linux-image-server (= 184.108.40.206.43); however: Package linux-image-server is not configured yet. dpkg: error processing linux-server (--configure): dependency problems - leaving unconfigured Errors were encountered while processing: linux-image-2.6.32-37-server linux-image-server linux-server
After this APT upgrades would not work.
Eventually, I discovered the cause in /etc/apt/term.log. This contained the following error message:
update-initramfs: Generating /boot/initrd.img-2.6.32-37-server gzip: stdout: No space left on device update-initramfs: failed for /boot/initrd.img-2.6.32-37-server Failed to create initrd image.
It seems that when I loaded the software, I had created a separate mount point for /boot (it was probably one of the defaults in the installation).
Getting it to work, was simply a matter of cleaning up /boot.
I tried ls -lt, and obtained a list of old files ...
And I removed the old files (actually moved them to a temporary area), which made enough space on /boot ... Files like the following:
-rw-r--r-- 1 root root 635002 2010-04-16 22:02 abi-2.6.32-21-server -rw-r--r-- 1 root root 645886 2010-09-17 04:55 abi-2.6.32-24-server -rw-r--r-- 1 root root 646217 2010-10-17 07:51 abi-2.6.32-25-server -rw-r--r-- 1 root root 646217 2010-11-24 22:13 abi-2.6.32-26-server -rw-r--r-- 1 root root 646217 2010-12-02 16:22 abi-2.6.32-27-server -rw-r--r-- 1 root root 110473 2010-04-16 22:02 config-2.6.32-21-server -rw-r--r-- 1 root root 110566 2010-09-17 04:55 config-2.6.32-24-server -rw-r--r-- 1 root root 110708 2010-10-17 07:51 config-2.6.32-25-server -rw-r--r-- 1 root root 110709 2010-11-24 22:13 config-2.6.32-26-server -rw-r--r-- 1 root root 110709 2010-12-02 16:22 config-2.6.32-27-server -rw-r--r-- 1 root root 9071177 2010-07-28 09:09 initrd.img-2.6.32-21-server -rw-r--r-- 1 root root 9079212 2010-09-18 16:21 initrd.img-2.6.32-24-server -rw-r--r-- 1 root root 9079997 2010-11-16 09:32 initrd.img-2.6.32-25-server -rw-r--r-- 1 root root 9080318 2010-12-01 08:54 initrd.img-2.6.32-26-server -rw-r--r-- 1 root root 2173738 2010-04-16 22:02 System.map-2.6.32-21-server -rw-r--r-- 1 root root 2175735 2010-09-17 04:55 System.map-2.6.32-24-server -rw-r--r-- 1 root root 2176796 2010-10-17 07:51 System.map-2.6.32-25-server -rw-r--r-- 1 root root 2177232 2010-11-24 22:13 System.map-2.6.32-26-server -rw-r--r-- 1 root root 2177586 2010-12-02 16:22 System.map-2.6.32-27-server -rw-r--r-- 1 root root 1335 2010-04-16 22:05 vmcoreinfo-2.6.32-21-server -rw-r--r-- 1 root root 1335 2010-09-17 04:58 vmcoreinfo-2.6.32-24-server -rw-r--r-- 1 root root 1335 2010-10-17 07:53 vmcoreinfo-2.6.32-25-server -rw-r--r-- 1 root root 1335 2010-11-24 22:15 vmcoreinfo-2.6.32-26-server -rw-r--r-- 1 root root 1335 2010-12-02 16:28 vmcoreinfo-2.6.32-27-server -rw-r--r-- 1 root root 4093216 2010-04-16 22:02 vmlinuz-2.6.32-21-server -rw-r--r-- 1 root root 4101952 2010-09-17 04:55 vmlinuz-2.6.32-24-server -rw-r--r-- 1 root root 4102624 2010-10-17 07:51 vmlinuz-2.6.32-25-server -rw-r--r-- 1 root root 4104672 2010-11-24 22:13 vmlinuz-2.6.32-26-server -rw-r--r-- 1 root root 4104448 2010-12-02 16:22 vmlinuz-2.6.32-27-server
Note: when removing old files, make sure you leave the memtest86 files in the boot partition.