Tuesday, May 12, 2009

Building Ubuntu .deb packages on Amazon AWS

I just got my virtual build farm working. It uses Amazon AWS and currently consists of 6 Ubuntu AMI: Hardy, Intrepid and Jaunty, all 32 and 64 bit. Starting with Fwbuilder v3.0.5 build 926, I'll be building Ubuntu .deb packages using these machines.

A bit of a background: until now, I've been running build using virtual machines in the VMWare server 2.0.0 on Ubuntu Hardy 64bit. I still have it, but it reached capacity because besides of the build virtual machines I also use it for all sorts of testing and experiments. I have FreeBSD, OpenBSD, CentOS, a couple of Fedora machines, Vyatta virtual appliance and few others running on it. They do not run all at once, but sometimes I have 6 or 7 virtual machines running which is a bit too much. The server has 4 Gb of RAM and 2.3GHz Intel Core-2 Duo CPU which is pretty good, but not enough if I start several 64bit virtual machines. Now that my scripts work and I can run builds on AWS, I am going to use this VMWare server mostly for the virtual "lab".

General Idea

The whole process is controlled by a few scripts that I run on my development machine. I am on Mac but these are just shell scripts (may be a few bash-specific features) and will work on Linux just the same.

First, I identified several AMI that suite my needs. For now these are three Ubuntu versions, each in 32 and 64 bit architecture. The AMI are just basic minimal installs with no desktop environment. I launch them using AWS command line tools and provide a script that runs at boot time. They call this parameterized launch, see detailed explanation here. This script installs missing packages and does other things on the machine to make it ready for use. My build scripts then wait for the machine to come up and start sshd, then log in, check out source code form svn and do the build. In the end they upload generated packages to the nightly builds site and shut down virtual machine.

I have one 4Gb AWS volume that I attach to the build machine when it starts up. I use this volume to store apt cache so that package installation does not take network bandwidth and works faster. I also use ccache to speed up build and keep ccache repository on this volume as well, so that it persists between builds.


Virtual build farm setup

To configure and control virtual machines, I am using both online AWS Management Console and command line tools. Online console is convenient way to check the status and start or stop machines manually, but actual build is automated and uses command line tools.

First of all, I had to download and configure AWS command line tools. They offer the for download here:

AWS API Tools

Follow documentation to set up certificate and private key and make these tools work. You'll need to configure several environment variables: EC2_HOME, EC2_CERT, EC2_PRIVATE_KEY, JAVA_HOME My scripts expect these variables to be configured.

Directory structure of my build environment looks like this:

src/fwb3
src/fwb3/aws
src/fwb3/tools
src/fwb3/source


Libfwbuilder and fwbuilder modules are checked out into src/fwb3/source. Bunch of Python scripts in src/fwb3/tools orchestrate build process and do all platform-dependent things so that I can use the same scripts on Linux, Mac and Windows. Directory src/fwb3/aws is for the AWS scripts and configuration files. I have the following scripts in the aws directory:

aws/ami_list
aws/build-all-ami.sh
aws/setup-ami-ubuntu.sh
aws/start_ami.sh
aws/stop_ami.sh


the "start" and "stop" scripts have obvious purpose. Script setup-ami-ubuntu.sh is the one used as a parameter when starting AMI, this script is actually copied over to the machine and then runs on it when it boots. Script build-all-ami.sh is a wrapper that starts all machines one by one, performs build and shuts them down. File ami_list is a configuration file that lists AMI I am using.

Here is how ami_list file looks like:


ami01 ami-005db969 m1.large setup-ami-ubuntu.sh Hardy 64bit
ami02 ami-ef48af86 m1.small setup-ami-ubuntu.sh Hardy 32bit


The first column is an alias I use internally to refer to a particular AMI, next goes AMI ID, AMI type, the name of the setup script (I have only one at this time, but when I add Fedora machines I expect to have another) and finally there is a comment.

Script start_ami.sh takes one parameter, it is AMI alias (from the first column in ami_list) and uses ec2-run-instances to start it. Here is how this script looks like:



#!/bin/sh
#
# This script starts AMI, waits for it to come up and saves its DNS name
# in the machine name mapping file in aws directory
#
# EC2 environment variables must be set up for this script to work.
# We could automatically find EC2_HOME, but there is no way to know
# where certificate and key are located.
#
# Usage:
#
# start_ami.sh machine_name
#
# Where machine_name is our internal name such as ami01
# Mapping of the machine name to AMI ID is done in the file aws/ami_list


AMI_LIST="aws/ami_list"
AWS_LOG="aws.log"
MACHINE_NAME=$1
MACHINE_DNS_FILE="aws/$MACHINE_NAME"
VOLUME1_ID="VOLUME-ID"
VOLUME_ZONE="us-east-1a"

if test -z "$EC2_HOME"
then
echo "Set up EC2_HOME EC2_PRIVATE_KEY EC2_CERT environment vairables"
exit 1
fi

if test -z "$MACHINE_NAME"
then
echo "Usage: start_ami.sh machine_name"
echo "Machine name is defined in file aws/ami_list"
exit 1
fi

if test -d "aws"
then
if test -f $AMI_LIST
then
set $(grep $MACHINE_NAME $AMI_LIST)
AMI=$2
TYPE=$3
SETUP_SCRIPT=$4
COMMENT="$5 $6 $7 $8"
else
echo "AMI list file $AMI_LIST not found"
exit 1
fi
else
echo "Run this script from the top of build environment directories"
exit 1
fi

cat /dev/null > $MACHINE_DNS_FILE

echo "Starting AMI $MACHINE_NAME $AMI $COMMENT"

cat aws/$SETUP_SCRIPT | sed "s/@MACHINE@/$MACHINE_NAME/" > /tmp/ami-setup.sh

# Note: launch instance in the same availability zone where our volume is.
$EC2_HOME/bin/ec2-run-instances -k ec2key -z $VOLUME_ZONE -t $TYPE -f /tmp/ami-setup.sh $AMI > $AWS_LOG 2>&1

# output looks like this:
#
# RESERVATION r-0757c76e 424466753135 default
# INSTANCE i-56ff8f3f ami-005db969 pending ec2key 0 m1.large 2009-05-11T01:46:34+0000 us-east-1c aki-b51cf9dc ari-b31cf9da

grep INSTANCE $AWS_LOG > /dev/null || {
cat $AWS_LOG
echo
echo "Instance failed to start, aborting"
exit 1
}

set $(grep INSTANCE $AWS_LOG)
INSTANCE=$2

echo "Instance $INSTANCE started"

# when instance is running, the output of ec2-describe-instances is like this:
#
# RESERVATION r-0757c76e 424466753135 default
# INSTANCE i-56ff8f3f ami-005db969 ec2-67-202-12-64.compute-1.amazonaws.com domU-12-31-35-00-2C-D2.z-2.compute-1.internal running ec2key 0 m1.large 2009-05-11T01:46:34+0000 us-east-1c aki-b51cf9dc ari-b31cf9da

CNTR=""
while :;
do
S=$($EC2_HOME/bin/ec2-describe-instances $INSTANCE | grep $INSTANCE)
echo $S | grep running >/dev/null && {
set $S
MACHINE_DNS=$4
break
}
sleep 10
CNTR="${CNTR}O"
if test "$CNTR" == "OOOOOOOOOOOO"
then
echo "Instance failed to start after 2 min timeout"
exit 1
fi
done

echo "Instance $INSTANCE is running"

echo "Attaching volume $VOLUME1_ID to instance $INSTANCE as /dev/sdf"
$EC2_HOME/bin/ec2-attach-volume $VOLUME1_ID -i $INSTANCE -d /dev/sdf

# Remove existing host key for this machine
ssh-keygen -R $MACHINE_DNS

# Wait for ssh to come up and read host key

TIMEOUT="O"
while :;
do
echo "Waiting for ssh access..."
if test "$TIMEOUT" = "OOOOOOOOOOOO"
then
echo "Timeout waiting for ssh access"
exit 1
fi
sleep 20
ssh -AX -o StrictHostKeyChecking=no root@$MACHINE_DNS 'uname -a' && break
TIMEOUT="${TIMEOUT}O"
done

echo $MACHINE_DNS > $MACHINE_DNS_FILE
exit 0




This script starts with checking the argument and sets up global variables. Note that the volume ID I am using is hard-coded at the beginning. It is also important to know the zone name this volume is in because AWS does not let you attach volume to the instance running in a different zone. I therefore have to start my instances in the same zone where my volume is.

Script runs ec2-run-instances to start the instance, then uses ec2-describe-instances to determine when it starts running. There is a timeout in this wait so it won't get stuck there forever. After instance has come up and started running, the script attaches volume and tries to log in via ssh. Sshd comes up a little later, so the script has to wait in another loop for that. Once sshd is up, the script stores dns name of the new instance in the file aws/$MACHINE_NAME (e.g. aws/ami01) so that it can be used by other scripts to access this machine.

Note how it removes ssh host key before trying ssh for the first time. This is because AWS assigns ip addresses and therefore dns names to instances dynamically and you may get the same address and name for a machine you already had in the past, but ssh host key will have changed. To avoid conflicts, I remove ssh host key.


The most interesting part perhaps is the setup script that is used for the parameterized launch and runs on the machine itself. This script makes it possible to use generic AMI and turn it into a build server (or anything else) automatically. Here is this script for Ubuntu machines:




#!/bin/sh

# This script sets up AWS instance running Ubuntu
#

MACHINE_ALIAS=@MACHINE@
SVN_SERVER=""
SVN_USER="vadim"
NIGHTLY_BUILDS_SERVER=""
EXT_VOLUME_DEV="/dev/sdf"
EXT_VOLUME_PART="${EXT_VOLUME_DEV}1"

exec 3>&1
exec 1> /root/ami_setup.log
exec 2>&1

KNOWN_HOSTS="SSH HOST KEY HERE"

cd /root

cat <<-EOF > /root/wait.sh
TIMEOUT="O"
while :;
do
test -f /root/machine_ready && break
if test "\$TIMEOUT" = "OOOOOOOOOOOO"
then
echo "Machine setup takes over 2 min, aborting"
exit 1
fi
TIMEOUT="\${TIMEOUT}O"
sleep 10
done
echo "Successful machine setup, can continue"
EOF
chmod +x /root/wait.sh

echo $KNOWN_HOSTS > /root/.ssh/known_hosts

echo "Running ssh to pick up ssh host keys"
# Note that since this is a boot-time setup script and ssh agent
# is not running when it is executed, these two ssh commands will in
# fact fail to log in. But they pick up host keys anyway.
ssh -o StrictHostKeyChecking=no ${SVN_USER}@${SVN_SERVER} 'uname -n'
ssh -o StrictHostKeyChecking=no ${NIGHTLY_BUILDS_SERVER} 'uname -n'

# wait for the volume to attach
while :;
do
e2fsck -y $EXT_VOLUME_PART > /dev/null && break
sleep 5
done

mkdir -p /data
mount $EXT_VOLUME_PART /data

# Set up ccache dir
mkdir -p /data/$MACHINE_ALIAS/.ccache
ln -s /data/$MACHINE_ALIAS/.ccache /var/tmp/.ccache
ln -s /data/$MACHINE_ALIAS/.ccache /root/.ccache

# Set up directory for apt cache
mkdir -p /data/$MACHINE_ALIAS/cache/apt
mkdir -p /data/$MACHINE_ALIAS/cache/apt/archives
mkdir -p /data/$MACHINE_ALIAS/cache/apt/archives/partial
mkdir -p /data/$MACHINE_ALIAS/cache/debconf/
mkdir -p /data/$MACHINE_ALIAS/cache/ldconfig/
mkdir -p /data/$MACHINE_ALIAS/cache/man
chown man /data/$MACHINE_ALIAS/cache/man
mv /var/cache /var/cache.bak
ln -s /data/$MACHINE_ALIAS/cache /var/cache

echo "Installing packages"

aptitude update

aptitude -y install g++ \
libqt4-dev libqt4-gui libqt4-core qt4-dev-tools \
libsnmp-dev \
subversion python-svn \
make libtool autoconf libxml2-dev libxslt1-dev python-paramiko \
fakeroot checkinstall ccache

mkdir -p src

# Set host name to machine alias so that our build scripts can
# recognize it by this name. In particular this will be the name
# scripts will use to find per-host override files under
# fwb3/machines/

hostname $MACHINE_ALIAS

touch /root/machine_ready




Note that machine name in this script is defined using a macro @MACHINE_NAME@. start_ami.sh script replaces it with machine alias before it is passed to ec2-run-instances.

First thing this script does, is it creates another script /root/wait.sh which we use later to determine if the AMI completed setup process.

then it connects to the svn an download servers using ssh to pick up their host keys. For the actual build process to be fully unattended, the machine should already have the host keys, otherwise it would stop during svn checkout and package upload asking operator if the key should be accepted. From the security standpoint, it would be better to copy these keys into the body of the script and transfer them that way. I am going to have to try this later.

After that, the script tries to mount external volume. It is possible that this script will run before the volume is attached, so first it runs e2fsck to check if the volume is available.

Once volume is mounted as /data, the script tries to create some directories on it. Note that directories are created in the tree that starts with machine alias, so that the same volume can be used with different virtual build machines. Script creates directory cache with subdirectories used by apt, man and few others. It helps minimize network bandwidth usage if apt cache is persistent since we download and install lots of packages every time we start virtual build machine. I also keep ccache directory on this volume to speed up repetitive builds.

Once this is done, the script runs "aptitude update" to download packge spec files and then installs bunch of dependency packages I need to actually do build of fwbuilder.

In the end, it sets hostname to the machine alias and touches file /root/machine_ready which serves as a flag indicating that setup process is complete.

This script is a good place to do other operations in preparation of the machine, for example you can set up svn tunnel configuration if access to your svn repository requires it or may be create some directories on the file system.

Finally, here is the fragment of the script build-all-ami.sh that performs actual build:




build_on_ami() {
machine_name=$1
build_script=$2
MACHINE=$(< aws/$machine_name)
ssh -AXt root@$MACHINE /root/wait.sh && \
ssh -AXt root@$MACHINE "cd src; svn co $SVN_URL fwb3" && \
scp $CLIENT_FILE root@${MACHINE}:src/fwb3 && \
ssh -AXt root@$MACHINE "cd src/fwb3 && ./tools/${build_script} $UPLOAD"
}

{
for a in $MACHINE_LIST
do
aws/start_ami.sh $a && build_on_ami $a build-deb.py
aws/stop_ami.sh $a
done




Note how function build_on_ami read machine's dns name from the file aws/$MACHINE_NAME created by start_ami.sh It then logs in to the machine and runs script /root/wait.sh that waits for the setup process to complete. After that, it checks out build environment and performs build.

Generally, I am quite happy with this system. It is not too complicated and is easy to extend. To add fedora systems, I should only need to write setup setup script and probably do minor changes to the build-all-ami.sh script.

Amazon AWS machines are not very fast. The /proc/cpuinfo shows quite a bit of nominal horsepower, but compile process seems to be slower than I would expect it do be at that speed. "Large" type machines (64 bit) are more expensive ($0.40 per hour, compared to $0.10 per hour for the "small" 32 bit machines) so I suggest all debugging of the scripts should be done with 32bit machines. It looks like my build on 6 machines cost me about $1.50 which is pretty good.

6 comments:

chenmeinv0 said...

adidas superstar trainers
pandora charms
ugg outlet online
adidas yeezy 350
louboutin sale
louis vuitton handbags uk
uggs for women
rolex watches for sale
coach outlet
true religion jeans
tory burch sale
true religion
gucci handbags
coach factory outlet
mcm handbags
seahawks jerseys
uggs outlet
ugg boots
sac louis vuitton
ugg boots
coach outlet
gucci handbags
nike roshe run pas cher
louis vuitton outlet
heat jerseys
nfl jerseys wholesale
borse louis vuitton
gucci bags
michael kors handbags
ugg slippers
ray bans
coach outlet online
toms shoes
fitflop sandals
louis vuitton purses
ugg outlet
wizards jerseys
coach outlet store online
kate spade outlet
adidas originals store
2016.10.31xukaimin

qqq said...

adidas shoes
pandora charms
air max 95
red bottoms
nike air max 90
michael kors outlet
louboutin pas cher
coach factory outlet
clarks shoes
rolex watches
170715yueqin

raybanoutlet001 said...

ugg boots
dolphins jerseys
ugg boots
adidas nmd
louboutin shoes
oakland raiders jerseys
michael kors handbags outlet
true religion outlet
oakley sunglasses
ugg outlet

قمم التميز said...

شركة تنظيف
تهتم شركة قمم التميز باعمال وخدمات التنظيف فى اى مكان فى تنظيف الشقق – البيوت – المنازل – المصانع واى مكان يحتاج الى اعمال التنظيف من اجل ان تساعد فى الوصول الى افضل ما تجدة من نتائج مميزة شركة تنظيف كنب بخميس مشيط
تنظيف يحتاج للاشياء معقدة من الادوات ، مثل الغرف المنزل لها ادوات التنظيف الخاصة، الصالون المدهب يحتاج لمنظف الخاص ، وسطح فى المنزل يتطلب مواد متخلفة ، فكثير من المنظفات المطلوبة حتى تكون عملية التنظيف سهلة ومن المنظفات : مبيض التواليت ، منظف الحمام، المطهرات ، المنظف للنوافذ شركة تنظيف موكيت بخميس مشيط
والارضيات والغسيل ، سائل الغسيل واعمال المنزل الصغيرة يمكن انجازاها بفوطة جافة ومنظف جيد ، المنظف المصنوع بقل الامكنيات ، معلقتين من الخل الابيض مع لتر من الماء الساخن وضعهم فى بخاخة . بيكربونات الصوديم لتنظيف البلاط يمكن مزج ثلاثه اجزاء من الماء الساخن مع جزء من الصودا لتنطيف الفرن والثلاجة ، وبيكربونات الصوديم بقليل من سائل الجلى يتكون معجون سميك صالح لتنظيف الحمام ، يمكن استخدام بيكربونات الصوديم وضعه فى صحن صغير ولطرد الروائح الكريهة من الثلاجة ، كربونات الصوديم فعالة فى ازالة البقع الدهنية لانها قلوية ويجب لبس قفازات لاستخدامها شركة تنظيف مجالس بخميس مشيط
الخل الابيض وعصير الليمون يمكن تنظيف السطوح الزجاجية والخشبية المصقولة ، ويمكن استخدام الخل الابيض والليمون للتخلص من الروائح الكريهة والتعطير .
عند التنظيف يجب لبس القفازات لحماية الايدى من المواد المنظفه ، ويمكن لبس الفقازات عند غسيل الصحون وتكون مخصص لعمل فى المطبخ .
والمنزل النظيفة من غير فوضوى أجمل بكثير وافضل ،وإذا توفر جميع الادوات النظافة داخل بيت تكون عمليه النظافة سهلة شركة تنظيف مساجد بخميس مشيط
للتنظيف المنزل من اعلى الى اسفل ، اى تنظيف الغبار من اعلى الى اسفل ويمكن استخدام الاجهزة الكهربائية فى عملية التنظيف .
تنظيف النوافذ باستخدام قطعة قطنية فى مسح الزجاج وللتجفيف بورقة من الجرائد ، وتنظيف الاسطح الزجاجية للمنضدة باستعمال عصير ليمون ودعكها ثم تجفيفها بفوطة ورقية. ويمكن استعمال معجون الاسنان فى ازالة الخدوش الصغيرة من الزجاج . شركة تنظيف سجاد بخميس مشيط
لتنظيف الاثاث استعمال منظف على قليل من نشادر ، عدم وضع الاثاث فى أشعه الشمس فالشمس تجفف الأثاث ، للتخلص من الحلقات التى تكون على المنضدة بقليل من المنظف مع النشادر ثم مسحها بفوطة جافة ثم تلميعها بالملمع .
لتنظيف الحمام باستخدام الادوات النظافة الخاصة للحمام فى الحمام ، تنظيف السيراميك والمرايا باستخدام مطهر الجراثيم وتنظيف كل شئ فى الحمام مثل مقبض الباب ومفتاح الكهرباء ، والتخلص من القاذورات فى الاركان الضيقة فى الحمام ، تنظيف الخلاطات المياه ومسحها بقطعة قماش مغموسة فى خل او زيت الطعام ، لتنظيف الحمام باستخدام المناديل المعقمة ومسح منطقة الدش والمرحاض .
يمكن وضع جدول زمنى لتنظيف المنزل : ترتيب السرير وتغير ملايات السرير كل اسبوع ، غسيل الملابس اذا كانت العائلة مكونة من الاطفال يقومون بتوسيخ ملابسهم بشكل متكرر فيجب تنظيف البقع سريعا .
وغسل الاوانى الطعام باليد او فى غسالة الاطباق وغسلها يوميا لا نها تشكل العفن وقد تنيجة خطر ويسبب امراض .
ومسح اسطح المطبخ بمنظف مضاد للجراثيم للوقاية من تكاثرها ، وهى افضل طريقة لتجنب الجراثيم التى تنتقل عن طريق الطعام .

jeje said...

retro jordans
yeezy boost 350
michael kors
nike dunk low
light up shoes
stephen curry shoes
nike huarache
nike mercurial vapor
golden goose
nmd r1

Candy Sim said...

The article you have shared here very good. This is really interesting information for me. Thanks for sharing!
hotmail.com login |hotmail log out |gmail login