Into.the.Void.

The Task
Last year, me and comzeradd set up a Gentoo server for HELLUG according to our plot to help Gentoo conquer the world. Unfortunately Gentoo is out of HELLUG’s administration policy, all servers must be Debian. We didn’t know that, so after a small flame :), we decided that we should take back the server to somebody’s home and re-install Debian over it, the problem was that the server was located at University of Athens campus which is a bit far from downtown Athens where comzeradd lives. I also live 500km away so we were pretty much stuck. Months passed and nobody actually had enough free time to go to UOA‘s campus and take the server to their house. …In the meantime manji joined us as an extra root for the server.

One Saturday night while chatting at IRC (what else could we be doing on saturday night ??) we had an inspiration, why not install Debian remotely, without taking the server home. Even if everything got eventually borked it couldn’t get any worse than going there, taking the server home and fixing it, just like we would do any way. So we gathered on a new IRC channel with some more friends that are really good with Debian and started the conversion progress.

The Server
The interesting part about the server was that it had 2x250Gb IDE disks. The Gentoo setup had these disks partitioned to 4 software raid devices + swap partitions.

(Gentoo) # fdisk -l
Disk /dev/hda: 250.0 GB, 250059350016 bytes
255 heads, 63 sectors/track, 30401 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x431bd7b7
Device Boot Start End Blocks Id System
/dev/hda1 * 1 6 48163+ fd Linux raid autodetect
/dev/hda2 7 130 996030 82 Linux swap / Solaris
/dev/hda3 131 27964 223576605 fd Linux raid autodetect
/dev/hda4 27965 30401 19575202+ 5 Extended
/dev/hda5 27965 29183 9791586 fd Linux raid autodetect
/dev/hda6 29184 30401 9783553+ fd Linux raid autodetect
Disk /dev/hdb: 250.0 GB, 250059350016 bytes
255 heads, 63 sectors/track, 30401 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00000000
Device Boot Start End Blocks Id System
/dev/hdb1 * 1 6 48163+ fd Linux raid autodetect
/dev/hdb2 7 130 996030 82 Linux swap / Solaris
/dev/hdb3 131 27964 223576605 fd Linux raid autodetect
/dev/hdb4 27965 30401 19575202+ 5 Extended
/dev/hdb5 27965 29183 9791586 fd Linux raid autodetect
/dev/hdb6 29184 30401 9783553+ fd Linux raid autodetect


md1 was RAID1 with hda1+hdb1 for /boot/
md3 was RAID1 with hda3+hdb3 for /
md5 was RAID1 with hda5+hdb5 for /var/db/
md6 was RAID0 with hda6+hdb6 for /usr/portage/

SUMMARY
What we had to do was:
A)break all RAID1 and RAID0 devices, set all hdbX partitions as faulty and remove them from the RAID.
B)repartition hdb, create new RAID1 arrays with LVM on top and format the new partitions
C)install debian on hdb
D)configure grub to boot debian

HOWTO
In order to be extra cautious for every command we gave we all logged in inside Gentoo and one of us set up a “screen” and the others joined that screen session using # screen -x

Now everything that one typed could be seen realtime by all the others.
PART A) RAID Manipulation
Check the status of the raid devices: cat /proc/mdstat
Copy /usr/portage/ to / as /usr/portage2 so that we can completely delete md6 (RAID0).
(Gentoo) # mkdir /usr/portage2/
(Gentoo) # cp -rp /usr/portage/* /usr/portage2/
(Gentoo) # umount /usr/portage
(Gentoo) # mv /usr/portage2 /usr/portage
(Gentoo) # mdadm --stop /dev/md6

Reminder: There’s no need to mdadm --remove /dev/md6 /dev/hdb6 since RAID0 can’t live with only one disk. The mdadm –remove command does nothing at all for RAID0.

We continued by breaking the rest of the RAID1 arrays.
(Gentoo) # mdadm --set-faulty /dev/md1 /dev/hdb1
(Gentoo) # mdadm --remove /dev/md1 /dev/hdb1
(Gentoo) # mdadm --set-faulty /dev/md3 /dev/hdb3
(Gentoo) # mdadm --remove /dev/md3 /dev/hdb3
(Gentoo) # mdadm --set-faulty /dev/md5 /dev/hdb5
(Gentoo) # mdadm --remove /dev/md5 /dev/hdb5

Checked on the current RAID status. Every RAID array should have been failed and with only one disk:
(Gentoo) # cat /proc/mdstat
Personalities : [raid0] [raid1]
md1 : active raid1 hda1[0]
48064 blocks [2/1] [U_]
md3 : active raid1 hda3[0]
223576512 blocks [2/1] [U_]
md5 : active raid1 hda5[0]
9791488 blocks [2/1] [U_]
done

We were now ready to repartition /dev/hdb.
PART B) Repartition hdb
(Gentoo) # fdisk hdb
Created 3 partitions: a) 128Mb for /boot, b) 1Gb for Swap and c) the rest for LVM
In order to re-read the partition table we issue:
(Gentoo) # hdparm -z /dev/hdb
Check if everything is OK
(Gentoo) # cat /proc/partitions | grep hdb

PART C) Install Debian on /dev/hdb
We first had to install the proper tools to do that. In order to create LVM partitions we needed the lvm userspace tools:
(Gentoo) # emerge -avt lvm2
Then we needed to install the tools to create the Debian system, the package is called debootstrap.
(Gentoo) # emerge -avt debootstrap
Created the new RAID1 arrays:
(Gentoo) # mdadm --create /dev/md11 --level=1 -n 2 /dev/hdb1 missing
(Gentoo) # mdadm --create /dev/md12 --level=1 -n 2 /dev/hdb2 missing
(Gentoo) # mdadm --create /dev/md13 --level=1 -n 2 /dev/hdb3 missing
Checked the new RAID arrays:
(Gentoo) # cat /proc/mdstat
Created some basic LVM partitions on top of md13. We didn’t use the whole space of hdb3 because we are going to create more partitions when and where we need to in the future:
(Gentoo) # pvcreate /dev/md13
(Gentoo) # vgcreate local /dev/md13
(Gentoo) # vgdisplay
(Gentoo) # lvcreate -n root -L 10G local
(Gentoo) # lvcreate -n tmp -L 2G local
(Gentoo) # lvcreate -n home -L 20G local

Formatted the LVM partitions and mounted them someplace.
(Gentoo) # mkfs.ext2 /dev/md11
(Gentoo) # mkfs.ext3 /dev/local/root
(Gentoo) # mkfs.ext3 /dev/local/home
(Gentoo) # mkfs.ext3 /dev/local/tmp
(Gentoo) # tune2fs -c 0 -i 0 /dev/local/root
(Gentoo) # tune2fs -c 0 -i 0 -m 0 /dev/local/home
(Gentoo) # tune2fs -c 0 -i 0 /dev/local/tmp
(Gentoo) # mkdir /mnt/newroot
(Gentoo) # mkdir /mnt/newroot/{boot,home,tmp}
(Gentoo) # mount /dev/local/root /mnt/newroot/
(Gentoo) # mount /dev/md11 /mnt/newroot/boot/
(Gentoo) # mount /dev/local/home /mnt/newroot/home/
(Gentoo) # mount /dev/local/tmp /mnt/newroot/tmp/

Then it was time to install Debian on /mnt/newroot using debootstrap:
(Gentoo) # debootstrap --arch=amd64 lenny /mnt/newroot/ http://ftp.ntua.gr/debian

After a while, when it was over we chrooted to the Debian install:
(Gentoo) # cd /mnt/newroot/
(Gentoo) # mount -o bind /dev dev/
(Gentoo) # mount -t proc proc proc
(Gentoo) # chroot . /bin/bash
(Debian) #

We created the network config,
(Debian) # vi /etc/network/interfaces
(contents)
auto eth0
iface eth0 inet static
address X.Y.Z.W
netmask 255.255.255.240
gateway A.B.C.D
auto lo
iface lo inet loopback
(/contents)

We fixed /etc/apt/sources.list:
(Debian) # vim /etc/apt/sources.list
(contents)
deb http://ftp.ntua.gr/debian lenny main contrib non-free
deb http://volatile.debian.org/debian-volatile lenny/volatile main contrib non-free
deb http://ftp.informatik.uni-frankfurt.de/debian-security/ lenny/updates main contrib
deb-src http://security.debian.org/ lenny/updates main contrib
(/contents)

We upgraded the current system and installed various usefull packages.
(Debian) # aptitude update
(Debian) # aptitude full-upgrade
(Debian) # aptitude install locales
(Debian) # vi /etc/locale.gen
(contents)
el_GR ISO-8859-7
el_GR.UTF-8 UTF-8
en_US.UTF-8 UTF-8
(/contents)
(Debian) # locale-gen
(Debian) # aptitude install openssh-server
(Debian) # aptitude install linux-image-2.6.26-1-amd64
(Debian) # aptitude install lvm2 mdadm
(Debian) # aptitude purge citadel-server exim4+
(Debian) # aptitude purge libcitadel1
(Debian) # aptitude install grub less
(Debian) # vi /etc/kernel-img.conf
(contents)
do_symlinks = Yes
do_initrd = yes
postinst_hook = update-grub
postrm_hook = update-grub
(/contents)
(Debian) # vi /etc/hosts
(Debian) # vi /etc/fstab
(contents)
proc /proc proc defaults 0 0
/dev/local/root / ext3 defaults,noatime 0 0
/dev/local/tmp /tmp ext3 defaults,noatime,noexec 0 0
/dev/local/home /home ext3 defaults,noatime 0 0
/dev/md11 /boot ext2 defaults 0 0
/dev/md12 none swap sw 0 0
(/contents)
(Debian) # update-initramfs -u -k all
(Debian) # passwd

And we logged out of Debian to go back to Gentoo to fix grub.
PART D) Configure Grub on Gentoo (hda) to boot Debian /em>
Since we didn’t have physical access to the server we had to boot Debian by using Grub on hda, where Gentoo’s Grub was.
We copied the kernel from debian:
(Gentoo) # cp /mnt/newroot/boot/vmlinuz-2.6.26-1-amd64 /boot/
(Gentoo) # cp /mnt/newroot/boot/initrd.img-2.6.26-1-amd64 /boot/
We edited grub config to add an entry for debian and set it as default! Otherwise the system would reboot back to Gentoo.
(Gentoo) # vi /boot/grub/menu.lst
(contents)
default 1
fallback 0
timeout 10
title=Gentoo
root(hd0,0)
kernel /gentoo-kernel ........
initrd /gentoo-initrd
title=debian (hdb)
root(hd1,0)
kernel /vmlinuz-2.6.26-1-amd64 root=/dev/mapper/local-root ro
initrd /initrd.img-2.6.26-1-amd64
(/contents)

Then we unmounted all partitions from /mnt/newroot/, we crossed our fingers and rebooted!
Voila! We could ssh to our new debian install 🙂 And there was much rejoicing…

What was left to be done, was to mount the old RAID arrays of Gentoo (md1,md3) take backups of configs and place them inside Debian. Then we could kill the old RAID arrays entirely, recreate partitions on hda and add those to RAID arrays of Debian (md11,md12,md13). Of course there should be special attention to re-install grub seperately on hda and hdb!!

Debian-izing the disk with the Gentoo
After a couple of days I decided to move on, kill Gentoo completely and make Debian use both disks.
First thing I did was to stop the old RAID1 arrays.
(Debian) # mdadm --stop /dev/md6
(Debian) # mdadm --stop /dev/md3
(Debian) # mdadm --stop /dev/md1
Then I repartitioned /dev/sda (the Debian kernel uses the modules that all disks appear as /dev/sdX) and created partitions the same size as /dev/sdb’s.:
(Debian) # fdisk /dev/sda
That was the point of no-return 🙂

There’s a risk involved here. The original sda1 was 64Mb and the newer sdb1 was 128Mb. I couldn’t add sda1 to md11 without extending the sda1 partition. If completely scratched /dev/sda1 to create a new partition of 128Mb in size and a power failure occurred while this process was going on, the server could become unbootable, because it wouldn’t find a proper sda1 to boot from. If someone wanted to minimize that risk, he would have to repartition sda, extend sda1 to the size of sdb1, extend the old /dev/md1 to fit the new sda1 size and extend the fs beneath it. Of course there is still a problem of what would happend if a power failure occured while extending the fs…so I chose to skip that “risk” and pretend it’s not there 🙂

Re-read the partition table:
(Debian) # hdparm -z /dev/sda
Add the new partitions to the Debian RAID1 arrays.
The first array I fixed was the /boot RAID1 array because it would only take some seconds to sync and minimizes the risk of a power failure while there’s no boot manager on the MBR and the rest of partitions are still syncing:
(Debian) # mdadm --add /dev/md11 /dev/sda1
When the sync is over I installed Grub on both sda1 and sdb1:
(Debian) # grub
grub> device (hd0) /dev/sda
grub> root (hd0,0)
Filesystem type is ext2fs, partition type 0xfd
grub> setup (hd0)
Checking if "/boot/grub/stage1" exists... no
Checking if "/grub/stage1" exists... yes
Checking if "/grub/stage2" exists... yes
Checking if "/grub/e2fs_stage1_5" exists... yes
[...snip...]
grub> quit
(Debian) # grub
grub> device (hd1) /dev/sdb
grub> root (hd1,0)
Filesystem type is ext2fs, partition type 0xfd
grub> setup (hd1)
Checking if "/boot/grub/stage1" exists... no
Checking if "/grub/stage1" exists... yes
Checking if "/grub/stage2" exists... yes
Checking if "/grub/e2fs_stage1_5" exists... yes
[...snip...]
grub> quit

Then we fix the rest RAID1 arrays:
(Debian) # mdadm --add /dev/md12 /dev/sda2
(Debian) # mdadm --add /dev/md13 /dev/sda3
The last sync took a while (approx 1h).

Make some final checks:
a) Check that grub is installed on every disk’s MBR
(Debian) # dd if=/dev/sda of=test.file bs=512 count=1
1+0 records in
1+0 records out
512 bytes (512 B) copied, 5.4721e-05 s, 9.4 MB/s
(Debian) # grep -i grub test.file
Binary file test.file matches
(Debian) # dd if=/dev/sdb of=test2.file bs=512 count=1
1+0 records in
1+0 records out
512 bytes (512 B) copied, 5.4721e-05 s, 9.4 MB/s
(Debian) # grep -i grub test2.file
Binary file test2.file matches

b) Make sure you have the correct entries in grub config:
(Debian) # cat /boot/grub/menu.lst
default 0
timeout 10
title=debian
root(hd0,0)
kernel /vmlinuz-2.6.26-1-amd64 root=/dev/mapper/local-root ro
initrd /initrd.img-2.6.26-1-amd64

c) Check the RAID1 arrays
(Debian) # cat /proc/mdstat
Personalities : [raid0] [raid1]
md13 : active raid1 sdb3[0] sda3[1]
243071360 blocks [2/2] [UU]
md12 : active (auto-read-only) raid1 sdb2[0] sda2[1]
987904 blocks [2/2] [UU]
md11 : active raid1 sdb1[0] sda1[1]
136448 blocks [2/2] [UU]
unused devices:

That’s all. Only a reboot will show whether everything went right.
Good luck!

P.S. The struggle of Gentoo taking over the world is not over. We may have lost a battle but we haven’t lost the war!

References:
a) HOWTO – Install Debian Onto a Remote Linux System
Pretty old but was the base of our efforts
b) RAID1 on Debian Sarge
c) growing ext3 partition on RAID1 without rebooting
d) Remote Conversion to Linux Software RAID-1 for Crazy Sysadmins HOWTO
e) Gentoo LVM2 installation

Filed by kargig at 20:21 under Gentoo,Linux
Tags: debian, debootstrap, Gentoo, grub, HOWTO, mdadm, no psysical access, raid, raid0, raid1, remote install
11 Comments | 21,043 views