Dealing with a rogue DHCP server

While working with virtual machines running on XenServer, I had one issue with the new machines not picking up their addresses from the DHCP server I was running but instead getting some strange addresses from somewhere else. As it was on a network I had no control over, I had to find a solution on the software side. Provided that I could access the network, configuring the switch would have been a much easier journey!

By the way I would expect the DHCP client to find that the closest DHCP server was my own given that it was on the same switch. But it seems that the DHCP client was receiving a NAK packet from the rogue DHCP before finishing the handshake with the my DHCP server.

On the Web you will find plenty of forums and mailing lists where people explain how to block DHCP packets with iptables. It seems a good idea except that it does not work! In the end you will find people explaining that DHCP packets are raw packets and can’t be blocked by iptables. I also tried with ebtables but did not managed to block the DHCP packets. Tcpdump is very useful when troubleshooting these kind of problems to understand what is really going through the network interfaces.

In the end I switched the network configuration of XenServer to openvswitch instead of Linux bridging capability and managed to get it working by applying some specific rules. I basically dropped every traffic from and to the rogue DHCP server IP address. Finally! The rogue DHCP server was no more and the virtual machines were picking up the correct addresses!

How to Install a ProActive Node on a Virtual Machine only accessible via the VDI image file.

If we have access to the VDI (Virtual Disk Image) file, we will be able to install ProActive and configure the VM to automatically start a ProActive Node at next boot. In order to do that, we will use the libguestfs free library. Libguestfs is a set of tools for accessing and modifying virtual machine (VM) disk images.

The following diagram shows the steps required to prepare a VM image (Linux or Windows) using libguestfs :

Fig 1. Automatic Installation Diagram

We suppose that the libguestfs is installed (or can be installed) on a controller host OS Linux Fedora 16 that will be used to prepare images.

We will follow a similar procedure between Linux and Windows OS, we will detail now this procedure for both Linux and Windows VMs. We implemented and tested successfully this procedure for a few Windows and Linux virtual machines.

The naked image is a Linux OS

We will describe all the commands used and the scripts written in order to install ProActive on a naked Linux image. The image that we used here is a Linux Ubuntu 12.10 image.

We will use libguestfs via the command guestfish. This command connects to the file system of a Virtual Machine image and allow the user to execute bash-like commands which can for example, add directories and files inside the file system, edit existing files, listing directories, modifying files permissions, etc...

Guestfish is an interactive interpreter but commands can as well be executed all-at-once by providing to the guestfish command a script. Here is the command-line that we launched in order to connect to our ubuntu image, and execute our installation procedure :

guestfish --rw -a "/home/fviale/fviale/VirtualBox VMs/ubuntu-12.10-desktop-i386.vdi" -i -f guestfish_script_linux

The guestfish_script_linux definition with the explanation of each step is shown below :

# Insure that all created files are accessible

umask 000

# Equivalent of the Linux command mkdir -p

mkdir-p /opt/tools

# Upload Java packages in the image

tgz-in jdk-7u10-linux-i586.gz /opt/tools/

# Upload ProActive Scheduling packages

tgz-in ProActiveScheduling-3.3.2_bin.tgz /opt/tools/

# Create the logs directory

mkdir-p /opt/tools/ProActiveScheduling-3.3.2_bin/.logs

# Insure the logs directory is writable

chmod 0777 /opt/tools/ProActiveScheduling-3.3.2_bin/.logs

# Copy the service definition into the

copy-in proactive-node /etc/init.d/

# Backup previous services definitions

cp /etc/rc.local /etc/rc.local.old

# Copy the rc.local file from guest to host

download /etc/rc.local /tmp/rc.local

# Remove the exit 0 from last line

! head -n -1 /tmp/rc.local > /tmp/rc.local.new

# Add service startup at boot of the image

! echo /etc/init.d/proactive-node start >> /tmp/rc.local.new

# Add the mandatory exit 0

! echo exit 0 >> /tmp/rc.local.new

# Copy the new rc.local from host to guest

upload /tmp/rc.local.new /etc/rc.local

chmod 0775 /etc/init.d/proactive-node

# Restrict service file to root only

chown 0 0 /etc/init.d/proactive-node

exit

After executing our guestfish script, our naked linux Virtual Machine now contains ProActive. It has a new service called proactive-node which will create a ProActive Node when the VM will boot.

The proactive-node service definition is shown below :

#!/bin/sh

# chkconfig: 98 02

# description: This is a daemon for automatically

# starting a proactive node

#

# processname: proactive-node

##

prefix=/usr

exec_prefix=/usr

sbindir=/usr/sbin

SCRIPT_BASE=/opt/tools/ProActiveScheduling-3.3.2_bin/bin/unix

SCRIPT_LOG_LOCATION=/opt/tools/ProActiveScheduling-3.3.2_bin/.logs/

START_UP_LOG=${SCRIPT_LOG_LOCATION}/startup

SHUT_DOWN_LOG=${SCRIPT_LOG_LOCATION}/shutdown

# Sanity checks.

# so we can rearrange this easily

RETVAL=0

start() {

echo "Starting my service "

echo " log location is:" ${START_UP_LOG}

export JAVA_HOME=/opt/tools/jdk1.7.0_10

nohup ${SCRIPT_BASE}/rm-start-node -r rmi://node0.entreprise.com:1099 >> ${START_UP_LOG} &

echo "Done starting my service! "

}

stop() {

echo "Stopping my service "

echo " log location is:" ${SHUT_DOWN_LOG}

killall -s 9 java>>${SHUT_DOWN_LOG}

echo "Done stopping my service! "

}

# See how we were called.

case "$1" in

start)

start

;;

stop)

stop

;;

status)

status $pidfile $processname

RETVAL=$?

;;

restart)

stop

start

;;

condrestart)

if [ -f /var/lock/subsys/$servicename ]; then

stop

start

fi

;;

*)

echo $"Usage: $0 {start|stop|status|restart|condrestart}"

;;

esac

exit $RETVAL

At next boot, the ProActive node will be created and will connect to our resource manager : rmi://node0.entreprise.com:1099
The node will then be usable by the Resource Manager as a Linux computing resource.

The naked image is a Windows OS

Similarly to our Linux VM, we will upload ProActive binaries on the Windows VM by using guestfish. We used for the example a Windows 7 SP1 32 bits image. We run our installation procedure by the following command :

guestfish --rw -a "/home/fviale/fviale/VirtualBox VMs/Windows 7 (32)/Windows 7 (32).vdi" -i -f guestfish_script_windows

The guestfish_script_windows definition with the explanation of each step is shown below :

# creates the directory c:\tools

mkdir /tools

# upload to the VM and unpack Java Virtual Machine 1.6

tgz-in jre1.6.tgz /tools/

# upload to the VM and unpack the ProActive library

tgz-in ProActiveScheduling-3.3.2_bin.tgz /tools/

# upload the RHSrvAny program to convert any windows command to a service

upload rhsrvany.exe /tools/rhsrvany.exe

# upload a modified DOS batch file which will be used to start the node

upload start-node-with-jre.bat /tools/ProActiveScheduling-3.3.2_bin/bin/windows

# exit from the guestfish interpreter

exit

In order to add the ProActive node as a service, the RHSrvAny  tool is used to wrap as a service the command that starts the ProActive node. After we upload the tool into the VM, we will have to edit the VM’s Windows Registry in order to install our service. This is done by using another libguestfs command called virt-win-reg :

virt-win-reg --merge "/home/fviale/fviale/VirtualBox VMs/Windows 7 (32)/Windows 7 (32).vdi" service.reg

The service.reg file must contain:

[HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\services\RHSrvAny]

"Type"=dword:00000010

"Start"=dword:00000002

"ErrorControl"=dword:00000001

"ImagePath"="c:\\tools\\rhsrvany.exe"

"DisplayName"="RHSrvAny"

"ObjectName"="LocalSystem"

[HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\services\RHSrvAny\Parameters]

"CommandLine"="c:\\tools\\ProActiveScheduling-3.3.2_bin\\bin\\windows\\start-node-with-jre.bat"

"PWD"="c:\\tools\\ProActiveScheduling-3.3.2_bin\\bin\\windows"

Since no JAVA_HOME variable is defined on the naked image, the start-node-with-jre.bat script is uploaded and used on the VM to start a node, the script sets the JAVA_HOME variable and the Resource Manager URL used :

set JAVA_HOME=c:\Program Files\Java\jdk1.6.0_32

rm-start-node.bat -r rmi://node0.entreprise.com:1099

After these steps, at next boot, the ProActive node will be created and will connect to our resource manager : rmi://node0.entreprise.com:1099
The node will then be usable by the Resource Manager as a Windows computing resource.

Appendix

Automatic Installation Diagram Legend

1. Controller HOST: a machine on which the installation procedure will be executed. Here we suppose that the Host is a Linux machine running on Fedora. This limitation is mainly to simplify the diagram, as it is totally possible to run the automatic installation procedure from a Windows Host, but with an intermediate temporary Fedora Linux Virtual Machine (the prototype tested was actually done using this configuration).   
2. FILES: the list of files used during the installation process, those files are supposed available in the Controller HOST or downloaded otherwise :
1. Oracle JRE : the Java Virtual Machine that will be used to start a ProActive Node.
2. ProActive Scheduler 3.3 : the ProActive archive.
3. guestfish scripts : the scripts that will be executed inside the guestfish environment (see 7).
4. RHSrvAny executable : the executable which will be used to start a ProActive Node as a service in Windows Guest VMs.
5. Linux Service definition : a text file which will be used to create a service in Linux Guest VMs.
3. Linux or Windows Vanilla Guest VM: we suppose clean installations of Windows or Linux Virtual Machines, without any specific software installed. The VMs are supposed to be turned off before the installation process starts, and we will do all the installation procedure using only the virtual machines VDI files.
4. Installation Process : The installation process is here described as a flowchart
5. Download Necessary Files : if the Files (2) are not available we download them from the network or the Internet.  
6. Install Libguestfs : The installation relies on the libguestfs library available on linux only and easier to install on Fedora. In this step we install libguestfs, for example using yum.
7. Connect Guestfish to the Linux or Windows VM File System : we will use the Guestfish interpreter to modify the VDI file. The first step of the procedure is to connect to the VDI file by using the guestfish command-line and thus starting a guestfish session. For example to connect to a Windows VM, we used the following command line : “guestfish --rw -a "/home/fviale/fviale/VirtualBox VMs/Windows 7 (32)/Windows 7 (32).vdi" -i”
8. Guestfish Session : the guestfish session describes a set of steps which will be done in the guestfish interpreter. Here we will describe those steps one-by-one but we can of course automate the session by using a script.
9. Unpack Java & ProActive archives : the first step of the installation will be to unpack the content of the ProActive archive and of the Java Virtual Machine archive in a dedicated directory inside the Windows or Linux VM. The following steps differ slightly when we operate a Linux or a Windows VM, but the general idea is the same. We will try to create a Linux or Windows service which we be launched automatically at startup. This service will start a ProActive Node that will connect automatically to a ProActive Resource Manager.
10. Copy proactive-node linux service definition : for linux VM, we will upload the definition of the proactive-node service. The place where this service definition must be uploaded can vary from one distribution to another. In this prototype, we used a Ubuntu VM and uploaded the service definition as /etc/init.d/proactive-node.
11. Add service to startup : we will add the proactive-node service as a startup service. We do this by editing the /etc/rc.local file inside the Linux VM.
12. Copy RHSrvAny executable: for windows VM, similarly we will create a windows service. This is done thanks to the windows program RHSrvAny#. The RHSrvAny program converts any windows command-line to a runnable windows service, simplifying greatly the configuration necessary.
13. Add executable as a service : the next step on the windows VM will be to edit the Windows Registry in order to add our service. This is done using a registry entry modification file service.reg and the libguestfs command virt-win-reg . For example, in our prototype, we executed the command : “virt-win-reg --merge "/home/fviale/fviale/VirtualBox VMs/Windows 7 (32)/Windows 7 (32).vdi" service.reg”
14. Modified Windows Guest VM : After all these installation steps the Linux and Windows Virtual Machines will now have a working ProActive installation and will be available as ProActive resources.
15. Start Linux/Windows VM : the next step will be start the Windows or Linux VM.
16. In the guest VM runtime, the linux or windows service will be automatically started on boot.
17. ProActive Node : the service will start a ProActive Node.
18. Connect to Resource Manager : the ProActive Node will connect to an existing Resource Manager

Resource Manager : The ProActive Resource Manager will receive the new node and will be able to deploy computations on it using ProActive.

Getting started with ProActive REST Clients.

REST APIs of the Scheduler and Resource Manager of ProActive Parallel Suite is one its most versatile features offered. They are as nearly as powerful as their Java counterparts, not bound to any programming language and accessible across heterogeneous networks and corporate firewalls.

REST Clients of ProActive Scheduler and Resource Manager are Java client applications which mimic the native clients and use REST APIs for communication. They persist session identifiers, cache user credentials when specified and use them to authenticate only when it required. They preserve command history and support both interactive and non-interactive modes.

Latest official ProActive REST Client distribution, ProActiveSchedulingREST-1.3.0_CLI, can be downloaded from here

Now lets have a look at how to submit a sample job and get the results when it finishes.

Submit a job file:

Command syntax:

 
$ sh scheduler-client.sh -u <https-rest-rul> -k -l <user> -s <job-file-pathname>
password:********

Eg:

 
bash-4.2$ sh scheduler-client.sh -u https://portal.cloud.sophia.inria.fr/SchedulingRest/rest -k -l ssamaran -s  ../../scheduling/samples/jobs_descriptors/Job_PI.xml
password:********
Job('../../scheduling/samples/jobs_descriptors/Job_PI.xml') successfully submitted: job('8354')

If the SSL server certificate is not installed into Java default trust store (e.g. /lib/security/cacerts), REST Clients will throw SSLPeerUnverifiedException type error. As a workaround, you can specify ‘-k’ option which disables SSL certificate verification.

Retrieve job state:

Command syntax:

 
$ sh scheduler-client.sh  -u <https-rest-url> -k -l ssamaran -js <job-id>

Eg:

 
bash-4.2$ sh scheduler-client.sh  -u https://portal.cloud.sophia.inria.fr/SchedulingRest/rest -k -l ssamaran -js 8354

job('8352')    NAME: job_PI    OWNER: ssamaran    STATUS: RUNNING    #TASKS: 9

     ID           NAME             ITER     DUP     STATUS       HOSTNAME                                            EXEC DURATION     TOT DURATION     #NODES USED     #EXECUTIONS     #NODES KILLED    

     83520008     Computation2                      RUNNING      node30.cloud.sophia.inria.fr (SSH-CNSlice1-143)     Not yet           Not yet          1               1/1             0/2              

     83520004     Computation4                      RUNNING      node7.cloud.sophia.inria.fr (SSH-CNSlice1-221)      Not yet           Not yet          1               1/1             0/2              

     83520005     Computation5                      RUNNING      node21.cloud.sophia.inria.fr (SSH-CNSlice1-228)     Not yet           Not yet          1               1/1             0/2              
..

Retrieve job result:

Command syntax:

 
$ sh scheduler-client.sh -u <https-rest-url> -k -l <user> -jr <job-id>

Eg:

 
bash-4.2$ sh ./scheduler-client.sh  -u https://portal.cloud.sophia.inria.fr/SchedulingRest/rest -k -l ssamaran -jr 8354

job('8354') result:
LastAverage : 3.1416703733333335
Computation2 : 3.14192544
Computation1 : 3.14179904
Computation4 : 3.14134824
Computation3 : 3.1417646
Computation6 : 3.14153496
Average1 : 3.1418296933333334
Average2 : 3.1415110533333332
Computation5 : 3.14164996

You can always find out other supported operations using their help option.

 
$ sh scheduler-client.sh  -h  

Devcloud Image Customization

Having recently worked on ProActive Cloudstack integration, I would like to share some challenges I faced and some tools that helped me. We present you a short video showing dynamic nodes deployment on a Cloudstack infrastructure.

Cloudstack is an open source cloud platform to build public, private or hybrid clouds. It offers a VirtualBox image, called DevCloud, that allows running a (virtual) cloud on a local machine without requiring a physical infrastructure. It basically consists of a Xen host virtual machine that will spawn Xen virtual machines as you create Cloudstack instances.

During the integration of Cloudstack with ProActive one of the challenges was to build a template allowing us to spawn instances that run as ProActive nodes. Consequently you are able to create elastic ProActive NodeSources that can grow more compute resources required. DevCloud comes with a lightweight ttyLinux template that is well suited to run several Cloudstack instances on your development machine. As ProActive requires only Java to run, I just needed to install it, unpack ProActive installation and create a boot script that would launch the ProActive node.

Being familiar with Debian based distribution, I first tried to create a template from an ISO image but then moved back on using the ttyLinux template as I hit issues regarding virtualization. It seems quite hard to create a new template using an ISO image on a paravirtualized Xen server. The ttyLinux template image comes with a disk of 50 MB only, so the JRE itself would not fit on it. The image is using VHD format and it seems quite hard to find useful information on the internet on how to work with. Here is how I managed to resize, mount and edit the image:

• All operations were performed on the DevCloud virtual machine as it has all the required tools installed.
• You can download the ttyLinux image from the Cloudstack web interface
• First, we start by expanding the VHD image
• vhd-util resize -n ttyLinux.vhd  -s 500 -j log.log
• size is in MB
• Now we need to expand the partition size otherwise the ttyLinux running with this template will not see the extra space
• It can be done by attaching the VHD image as a device with blktap

device=$(tap-ctl allocate)

deviceid=${device#/dev/xen/blktap-2/tapdev}

spawn=$(tap-ctl spawn)

pid=${spawn#tapdisk spawned with pid }

tap-ctl attach -p $pid -m $deviceid

tap-ctl open -p $pid -m $deviceid -a vhd:ABSOLUTE_PATH_TO_VHD_IMAGE

• A new device has been created under /dev/xen/blktap-2/, called tapdev (followed by the device id, let’s assume tapdev4), to check if it exists we can use fdisk
• fdisk -l /dev/xen/blktap-2/tapdev$deviceid
• resize2fs can now be used to resize the partition
• resize2fs /dev/xen/blktap-2/tapdev4 1G
• here we expand the partition to 1 gigabytes
• a filesystem check might be required (fsck), it is ok to run it
• Next are the commands to destroy the device properly
• $pid and $deviceid are the values use when creating the device

tap-ctl close -m $deviceid -p $pid

tap-ctl detach -m $deviceid -p $pid

tap-ctl free -m $deviceid

Since the VHD image is present as a device, we can mount it and directly edit files on it.

• To mount the device
• mount /dev/xen/blktap-2/tapdev4 mounted_image/
• For my needs, I copied Java and ProActive and created a boot script that runs a ProActive node
• To un mount the device
• umount mounted_image/

Installing the template in Cloudstack is just a matter of uploading the image file using the web client. As it requires downloading a file from a HTTP URL, "python -m SimpleHTTPServer 80" comes handy to expose local files on port 80.

We have now a custom made VHD image ready to use with Cloudstack!

 Sources: 

• http://lists.xen.org/archives/html/xen-api/2012-05/msg00145.html