Monday, September 16, 2013

Distributed Groovy with ProActive: GPars - Part 2

Since we leveraged ProActive to execute remote Groovy closure in the previous blog post, we will, in this one, integrate ProActive with GPars to allow the distribution of concurrent workloads.
GPars is a nice library written in Groovy to develop parallel applications. It leverages closures to provide a nice and concise API, something that we will “soon” benefit from in Java with the Lambda project. GPars covers many concepts of concurrent programming such as concurrent collections, actors, fork/join, agents, STM,... Here we will focus on concurrent collections processing.

GPars in its implementation details relies on Threads, plain old school Java synchronization and amongst other Fork/Join framework. To avoid changing GPars itself, I’ve chosen to implement the distribution as a decoration of the closure that we want to parallelize. The basic API should stay the same and the execution on the client side will keep the same concept, a thread pool that GPars uses to run the closure, except this time it will call a ProActive node to run the closure and get the result back. This simplistic approach is of course limited but it will be enough to illustrate the concept and the potential of mixing Groovy with ProActive.

Let’s start with an example of what GPars can do:
Here we ask Wally’s friends where they are and since they are not very disciplined they can all answer at the same time and will do! We see that they each live in a different thread, yet on the same JVM.
$> [Wally is here: ForkJoinPool-1-worker-1, Wilma is here: ForkJoinPool-1-worker-2, Woof is here: ForkJoinPool-1-worker-3]
Since Wally discovered worm holes (seriously?) his friends are able to travel to remote locations, such as a ProActive node!

Here we ask again where Wally’s friends are and again they answer as they wish. Now they are in different universes, aka different JVMs.

$> [Wally is here: RemoteClosure on rmi://jily.local:1099/PA_JVM1522417505_GCMNode-0, Wilma is here: RemoteClosure on rmi://jily.local:1099/PA_JVM139013877_GCMNode-0, Woof is here: RemoteClosure on rmi://jily.local:1099/PA_JVM1680423209_GCMNode-0]

If you want to get into the gory details of it, you can take a look at ProActivePool where it mimics GParsPool class to add the distributed call to a ProActive node.

With this simple example, we have demonstrated how GPars could be extended using ProActive to actually distribute the parallel computations. Groovy helps us to keep a simple API and to hide the implementation details to the user.

On a side note, we are adding Groovy as a script engine in ProActive latest release. Combined with the new script task, we hope it will help you to build ProActive application faster while still benefiting from a tight integration with Java.

Tuesday, July 30, 2013

Distributed Groovy with ProActive - Part 1

Being a new member of the ProActive team, I tried to teach to myself a bit of active objects, or at least create one. Active Objects are a core concept of ProActive, a distributed computing framework which is the foundation of the Scheduler and Resource Manager

I will not detail how the active objects work but illustrate how they can be used with concrete examples and especially showing how a language such as Groovy could integrate with to provide a nice and fluent distributed API. Groovy is a dynamic language running on the JVM providing scripting capacities and a transparent integration with Java. It also enables you to adopt functional programming in Java with closures for instance. To me, it is really a Java++ where you can increase your code readability. And if you are afraid of the performance of a dynamic language, well the latest version brought static typing so you can still use all the nice features and keep a coding style closer to Java with good performance. 

This is the first item of two blog posts where I will explain how to use Groovy with ProActive to execute code in a distributed manner and how GPars could be distributed with ProActive. 

The code presented here is available on GitHub and can be compiled and ran using Gradle wrapper (distributed with the sources). Useful commands are gradlew tasks, gradlew build. To build it you will also need the programming library (where the active objects live), available here and to set the project.ext.programmingLibDir property inside the build.gradle script. The programming library sources are available here

Let’s start with a very simple example showing the deployment of a remote execution node, the creation of a remote active object on this node and the remote execution of code.

This program finds Wally and his friend the Wizard whitebeard. Wally being a normal guy lives here in the present world (main thread) whereas the Wizard with its magic powers lives elsewhere (on a remote node). So when you run this program, you will see where each of them live.

$> gradle WhereIsWallyJava
Wally is here: main 
Wizard whitebeard is here: WallyFingerPointer on rmi:// 

Here we deployed the node locally using a GCM descriptor. WallyFingerPointer is the active object, created by the call PAActiveObject.newActive(). It is very simple class with one method, printing the current thread name from which we can infer Wally’s location. Now this is all Java code and we will now try to improve it with some Groovy.

Here we have the same program written in Groovy, the output is exactly the same. Let’s explain how it works. We create a WallyFingerPointer object and using the with() method we express that all the code inside the closure (inside the braces) will be invoked on the WallyFingerPointer object. It is a shortcut to avoid repeating the variable name when calling several methods on the same object. Then we have the remote code execution using the same closure block but with a call to a static method remote(). This methods takes care of creating the remote deployment context and cleans it when the closure is executed. The foundHim() calls inside the closure will be executed by the active object. Groovy enables us to hide the technical details around active objects and helps to provide a clean API to the user. Of course this a very limited example, a few technical limitations exist related to object serialization for instance.

Now what would be nice is to be able to execute remote closure because closures are core to Groovy, used everywhere for conciseness (and that will be useful for the second blog post of this series). A closure can be seen as an interface with one method so it is itself an object, with call() methods, eventually taking parameters. The closure has also access to variables outside its scope but we will not allow that in our example to facilitate serialization. 

We still use the same program with the first line printing out the location of Wally (local execution) and the second line using a remote closure. The code inside the braces will be executed on the remote node. The way it works is that we created an active object that has one method taking a closure as a parameter and executing it. The closure object will be serialized and sent to the remote node for execution. To serialize the closure we use the dehydrate() method that simply remove references to object out of the closure scopes (the script itself for example). It is now even simpler to execute remote code with Groovy!

We have shown a very simple example of remote code execution with ProActive, adapted this code to use Groovy and introduced a way to run remote closures. In the next blog post, we will use these remote closures to turn GPars, a concurrency framework, into a distributed concurrent framework!

Wednesday, June 12, 2013

3D Game Remote Rendering with ProActive

ActiveEon is again on the cloud, providing real solutions to real use cases.

Within the framework of CompatibleOne project, ActiveEon collaborated with INRIA Sophia Antipolis and Eureva to furnish a proof of concept that represents the future of game computing, or as we like to call it, cloud gaming. 

It does not really matter whether your workstation has a powerful GPU to stand heavy 3D rendering loads, because as long as your Internet connection bandwidth is good enough, the cloud will do the hard work for you. 
3D Game Rendering scenario 

Launching a game in the cloud has become very simple with ProActive framework. The Game Player simply chooses a game from Eureva's Cloud Gaming Client interface. At this point the Eureva Games Broker receives the request and processes it. When constraints are determined, a resources request is sent to CompatibleOne Broker which uses a ProActive connector to furnish ProActive enabled physical resources matching constraints, like CPU, physical memory, existence of GPU, among others. After ProActive has booked the correct resources from the ProActive Games Infrastructure, the game is launched together with a streaming server that allows your computer to receive streaming with heavy frames processed in the cloud. 

Take a look at our video at ActiveEon's Youtube Portal and let us know if you see future as we do. 

Wednesday, April 24, 2013

Dealing with a rogue DHCP server

While working with virtual machines running on XenServer, I had one issue with the new machines not picking up their addresses from the DHCP server I was running but instead getting some strange addresses from somewhere else. As it was on a network I had no control over, I had to find a solution on the software side. Provided that I could access the network, configuring the switch would have been a much easier journey!

By the way I would expect the DHCP client to find that the closest DHCP server was my own given that it was on the same switch. But it seems that the DHCP client was receiving a NAK packet from the rogue DHCP before finishing the handshake with the my DHCP server.

On the Web you will find plenty of forums and mailing lists where people explain how to block DHCP packets with iptables. It seems a good idea except that it does not work! In the end you will find people explaining that DHCP packets are raw packets and can’t be blocked by iptables. I also tried with ebtables but did not managed to block the DHCP packets. Tcpdump is very useful when troubleshooting these kind of problems to understand what is really going through the network interfaces.

In the end I switched the network configuration of XenServer to openvswitch instead of Linux bridging capability and managed to get it working by applying some specific rules. I basically dropped every traffic from and to the rogue DHCP server IP address. Finally! The rogue DHCP server was no more and the virtual machines were picking up the correct addresses!

Wednesday, March 13, 2013

How to Install a ProActive Node on a Virtual Machine only accessible via the VDI image file.

If we have access to the VDI (Virtual Disk Image) file, we will be able to install ProActive and configure the VM to automatically start a ProActive Node at next boot. In order to do that, we will use the libguestfs free library. Libguestfs is a set of tools for accessing and modifying virtual machine (VM) disk images.

The following diagram shows the steps required to prepare a VM image (Linux or Windows) using libguestfs :

Fig 1. Automatic Installation Diagram

We suppose that the libguestfs is installed (or can be installed) on a controller host OS Linux Fedora 16 that will be used to prepare images.

We will follow a similar procedure between Linux and Windows OS, we will detail now this procedure for both Linux and Windows VMs. We implemented and tested successfully this procedure for a few Windows and Linux virtual machines.

The naked image is a Linux OS

We will describe all the commands used and the scripts written in order to install ProActive on a naked Linux image. The image that we used here is a Linux Ubuntu 12.10 image.
We will use libguestfs via the command guestfish. This command connects to the file system of a Virtual Machine image and allow the user to execute bash-like commands which can for example, add directories and files inside the file system, edit existing files, listing directories, modifying files permissions, etc...
Guestfish is an interactive interpreter but commands can as well be executed all-at-once by providing to the guestfish command a script. Here is the command-line that we launched in order to connect to our ubuntu image, and execute our installation procedure :

guestfish --rw -a "/home/fviale/fviale/VirtualBox VMs/ubuntu-12.10-desktop-i386.vdi" -i -f guestfish_script_linux

The guestfish_script_linux definition with the explanation of each step is shown below :

# Insure that all created files are accessible
umask 000
# Equivalent of the Linux command mkdir -p
mkdir-p /opt/tools
# Upload Java packages in the image
tgz-in jdk-7u10-linux-i586.gz /opt/tools/
# Upload ProActive Scheduling packages
tgz-in ProActiveScheduling-3.3.2_bin.tgz /opt/tools/
# Create the logs directory
mkdir-p /opt/tools/ProActiveScheduling-3.3.2_bin/.logs
# Insure the logs directory is writable
chmod 0777 /opt/tools/ProActiveScheduling-3.3.2_bin/.logs
# Copy the service definition into the
copy-in proactive-node /etc/init.d/
# Backup previous services definitions
cp /etc/rc.local /etc/rc.local.old
# Copy the rc.local file from guest to host
download /etc/rc.local /tmp/rc.local
# Remove the exit 0 from last line
! head -n -1 /tmp/rc.local > /tmp/
# Add service startup at boot of the image
! echo /etc/init.d/proactive-node start >> /tmp/
# Add the mandatory exit 0
! echo exit 0 >> /tmp/
# Copy the new rc.local from host to guest
upload /tmp/ /etc/rc.local
chmod 0775 /etc/init.d/proactive-node
# Restrict service file to root only
chown 0 0 /etc/init.d/proactive-node

After executing our guestfish script, our naked linux Virtual Machine now contains ProActive. It has a new service called proactive-node which will create a ProActive Node when the VM will boot.

The proactive-node service definition is shown below :
# chkconfig: 98 02
# description: This is a daemon for automatically
# starting a proactive node
# processname: proactive-node

# Sanity checks.
# so we can rearrange this easily

start() {
echo "Starting my service "
echo " log location is:" ${START_UP_LOG}
export JAVA_HOME=/opt/tools/jdk1.7.0_10
nohup ${SCRIPT_BASE}/rm-start-node -r rmi:// >> ${START_UP_LOG} &
echo "Done starting my service! "

stop() {
echo "Stopping my service "
echo " log location is:" ${SHUT_DOWN_LOG}
killall -s 9 java>>${SHUT_DOWN_LOG}
echo "Done stopping my service! "

# See how we were called.
case "$1" in
status $pidfile $processname
if [ -f /var/lock/subsys/$servicename ]; then
echo $"Usage: $0 {start|stop|status|restart|condrestart}"
exit $RETVAL

At next boot, the ProActive node will be created and will connect to our resource manager : rmi://
The node will then be usable by the Resource Manager as a Linux computing resource.

The naked image is a Windows OS

Similarly to our Linux VM, we will upload ProActive binaries on the Windows VM by using guestfish. We used for the example a Windows 7 SP1 32 bits image. We run our installation procedure by the following command :

guestfish --rw -a "/home/fviale/fviale/VirtualBox VMs/Windows 7 (32)/Windows 7 (32).vdi" -i -f guestfish_script_windows

The guestfish_script_windows definition with the explanation of each step is shown below :

# creates the directory c:\tools
mkdir /tools
# upload to the VM and unpack Java Virtual Machine 1.6
tgz-in jre1.6.tgz /tools/
# upload to the VM and unpack the ProActive library
tgz-in ProActiveScheduling-3.3.2_bin.tgz /tools/
# upload the RHSrvAny program to convert any windows command to a service
upload rhsrvany.exe /tools/rhsrvany.exe
# upload a modified DOS batch file which will be used to start the node
upload start-node-with-jre.bat /tools/ProActiveScheduling-3.3.2_bin/bin/windows
# exit from the guestfish interpreter

In order to add the ProActive node as a service, the RHSrvAny  tool is used to wrap as a service the command that starts the ProActive node. After we upload the tool into the VM, we will have to edit the VM’s Windows Registry in order to install our service. This is done by using another libguestfs command called virt-win-reg :

virt-win-reg --merge "/home/fviale/fviale/VirtualBox VMs/Windows 7 (32)/Windows 7 (32).vdi" service.reg

The service.reg file must contain:


Since no JAVA_HOME variable is defined on the naked image, the start-node-with-jre.bat script is uploaded and used on the VM to start a node, the script sets the JAVA_HOME variable and the Resource Manager URL used :
set JAVA_HOME=c:\Program Files\Java\jdk1.6.0_32
rm-start-node.bat -r rmi://

After these steps, at next boot, the ProActive node will be created and will connect to our resource manager : rmi://
The node will then be usable by the Resource Manager as a Windows computing resource.


Automatic Installation Diagram Legend

  1. Controller HOST: a machine on which the installation procedure will be executed. Here we suppose that the Host is a Linux machine running on Fedora. This limitation is mainly to simplify the diagram, as it is totally possible to run the automatic installation procedure from a Windows Host, but with an intermediate temporary Fedora Linux Virtual Machine (the prototype tested was actually done using this configuration).   
  2. FILES: the list of files used during the installation process, those files are supposed available in the Controller HOST or downloaded otherwise :
    1. Oracle JRE : the Java Virtual Machine that will be used to start a ProActive Node.
    2. ProActive Scheduler 3.3 : the ProActive archive.
    3. guestfish scripts : the scripts that will be executed inside the guestfish environment (see 7).
    4. RHSrvAny executable : the executable which will be used to start a ProActive Node as a service in Windows Guest VMs.
    5. Linux Service definition : a text file which will be used to create a service in Linux Guest VMs.
  3. Linux or Windows Vanilla Guest VM: we suppose clean installations of Windows or Linux Virtual Machines, without any specific software installed. The VMs are supposed to be turned off before the installation process starts, and we will do all the installation procedure using only the virtual machines VDI files.
  4. Installation Process : The installation process is here described as a flowchart
  5. Download Necessary Files : if the Files (2) are not available we download them from the network or the Internet.  
  6. Install Libguestfs : The installation relies on the libguestfs library available on linux only and easier to install on Fedora. In this step we install libguestfs, for example using yum.
  7. Connect Guestfish to the Linux or Windows VM File System : we will use the Guestfish interpreter to modify the VDI file. The first step of the procedure is to connect to the VDI file by using the guestfish command-line and thus starting a guestfish session. For example to connect to a Windows VM, we used the following command line : “guestfish --rw -a "/home/fviale/fviale/VirtualBox VMs/Windows 7 (32)/Windows 7 (32).vdi" -i”
  8. Guestfish Session : the guestfish session describes a set of steps which will be done in the guestfish interpreter. Here we will describe those steps one-by-one but we can of course automate the session by using a script.
  9. Unpack Java & ProActive archives : the first step of the installation will be to unpack the content of the ProActive archive and of the Java Virtual Machine archive in a dedicated directory inside the Windows or Linux VM. The following steps differ slightly when we operate a Linux or a Windows VM, but the general idea is the same. We will try to create a Linux or Windows service which we be launched automatically at startup. This service will start a ProActive Node that will connect automatically to a ProActive Resource Manager.
  10. Copy proactive-node linux service definition : for linux VM, we will upload the definition of the proactive-node service. The place where this service definition must be uploaded can vary from one distribution to another. In this prototype, we used a Ubuntu VM and uploaded the service definition as /etc/init.d/proactive-node.
  11. Add service to startup : we will add the proactive-node service as a startup service. We do this by editing the /etc/rc.local file inside the Linux VM.
  12. Copy RHSrvAny executable: for windows VM, similarly we will create a windows service. This is done thanks to the windows program RHSrvAny#. The RHSrvAny program converts any windows command-line to a runnable windows service, simplifying greatly the configuration necessary.
  13. Add executable as a service : the next step on the windows VM will be to edit the Windows Registry in order to add our service. This is done using a registry entry modification file service.reg and the libguestfs command virt-win-reg . For example, in our prototype, we executed the command : “virt-win-reg --merge "/home/fviale/fviale/VirtualBox VMs/Windows 7 (32)/Windows 7 (32).vdi" service.reg”
  14. Modified Windows Guest VM : After all these installation steps the Linux and Windows Virtual Machines will now have a working ProActive installation and will be available as ProActive resources.
  15. Start Linux/Windows VM : the next step will be start the Windows or Linux VM.
  16. In the guest VM runtime, the linux or windows service will be automatically started on boot.
  17. ProActive Node : the service will start a ProActive Node.
  18. Connect to Resource Manager : the ProActive Node will connect to an existing Resource Manager
Resource Manager : The ProActive Resource Manager will receive the new node and will be able to deploy computations on it using ProActive.

Thursday, February 7, 2013

Getting started with ProActive REST Clients

Getting started with ProActive REST Clients.

REST APIs of the Scheduler and Resource Manager of ProActive Parallel Suite is one its most versatile features offered. They are as nearly as powerful as their Java counterparts, not bound to any programming language and accessible across heterogeneous networks and corporate firewalls.

REST Clients of ProActive Scheduler and Resource Manager are Java client applications which mimic the native clients and use REST APIs for communication. They persist session identifiers, cache user credentials when specified and use them to authenticate only when it required. They preserve command history and support both interactive and non-interactive modes.

Latest official ProActive REST Client distribution, ProActiveSchedulingREST-1.3.0_CLI, can be downloaded from here

Now lets have a look at how to submit a sample job and get the results when it finishes.

Submit a job file:

Command syntax:
$ sh -u <https-rest-rul> -k -l <user> -s <job-file-pathname>
bash-4.2$ sh -u -k -l ssamaran -s  ../../scheduling/samples/jobs_descriptors/Job_PI.xml
Job('../../scheduling/samples/jobs_descriptors/Job_PI.xml') successfully submitted: job('8354')
If the SSL server certificate is not installed into Java default trust store (e.g. /lib/security/cacerts), REST Clients will throw SSLPeerUnverifiedException type error. As a workaround, you can specify ‘-k’ option which disables SSL certificate verification.

Retrieve job state:

Command syntax:
$ sh  -u <https-rest-url> -k -l ssamaran -js <job-id>
bash-4.2$ sh  -u -k -l ssamaran -js 8354

job('8352')    NAME: job_PI    OWNER: ssamaran    STATUS: RUNNING    #TASKS: 9

     ID           NAME             ITER     DUP     STATUS       HOSTNAME                                            EXEC DURATION     TOT DURATION     #NODES USED     #EXECUTIONS     #NODES KILLED    

     83520008     Computation2                      RUNNING (SSH-CNSlice1-143)     Not yet           Not yet          1               1/1             0/2              

     83520004     Computation4                      RUNNING (SSH-CNSlice1-221)      Not yet           Not yet          1               1/1             0/2              

     83520005     Computation5                      RUNNING (SSH-CNSlice1-228)     Not yet           Not yet          1               1/1             0/2              
Retrieve job result:

Command syntax:
$ sh -u <https-rest-url> -k -l <user> -jr <job-id>
bash-4.2$ sh ./  -u -k -l ssamaran -jr 8354

job('8354') result:
LastAverage : 3.1416703733333335
Computation2 : 3.14192544
Computation1 : 3.14179904
Computation4 : 3.14134824
Computation3 : 3.1417646
Computation6 : 3.14153496
Average1 : 3.1418296933333334
Average2 : 3.1415110533333332
Computation5 : 3.14164996

You can always find out other supported operations using their help option.
$ sh  -h  

Wednesday, January 23, 2013

Devcloud Image Customization

Having recently worked on ProActive Cloudstack integration, I would like to share some challenges I faced and some tools that helped me. We present you a short video showing dynamic nodes deployment on a Cloudstack infrastructure.

Cloudstack is an open source cloud platform to build public, private or hybrid clouds. It offers a VirtualBox image, called DevCloud, that allows running a (virtual) cloud on a local machine without requiring a physical infrastructure. It basically consists of a Xen host virtual machine that will spawn Xen virtual machines as you create Cloudstack instances.

During the integration of Cloudstack with ProActive one of the challenges was to build a template allowing us to spawn instances that run as ProActive nodes. Consequently you are able to create elastic ProActive NodeSources that can grow more compute resources required. DevCloud comes with a lightweight ttyLinux template that is well suited to run several Cloudstack instances on your development machine. As ProActive requires only Java to run, I just needed to install it, unpack ProActive installation and create a boot script that would launch the ProActive node.

Being familiar with Debian based distribution, I first tried to create a template from an ISO image but then moved back on using the ttyLinux template as I hit issues regarding virtualization. It seems quite hard to create a new template using an ISO image on a paravirtualized Xen server. The ttyLinux template image comes with a disk of 50 MB only, so the JRE itself would not fit on it. The image is using VHD format and it seems quite hard to find useful information on the internet on how to work with. Here is how I managed to resize, mount and edit the image:

  • All operations were performed on the DevCloud virtual machine as it has all the required tools installed.
  • You can download the ttyLinux image from the Cloudstack web interface
  • First, we start by expanding the VHD image
    • vhd-util resize -n ttyLinux.vhd  -s 500 -j log.log
      • size is in MB
  • Now we need to expand the partition size otherwise the ttyLinux running with this template will not see the extra space
    • It can be done by attaching the VHD image as a device with blktap
device=$(tap-ctl allocate)

spawn=$(tap-ctl spawn)
pid=${spawn#tapdisk spawned with pid }

tap-ctl attach -p $pid -m $deviceid
tap-ctl open -p $pid -m $deviceid -a vhd:ABSOLUTE_PATH_TO_VHD_IMAGE

  • A new device has been created under /dev/xen/blktap-2/, called tapdev (followed by the device id, let’s assume tapdev4), to check if it exists we can use fdisk
    • fdisk -l /dev/xen/blktap-2/tapdev$deviceid
  • resize2fs can now be used to resize the partition
    • resize2fs /dev/xen/blktap-2/tapdev4 1G
      • here we expand the partition to 1 gigabytes
      • a filesystem check might be required (fsck), it is ok to run it
  • Next are the commands to destroy the device properly
    • $pid and $deviceid are the values use when creating the device
tap-ctl close -m $deviceid -p $pid
tap-ctl detach -m $deviceid -p $pid
tap-ctl free -m $deviceid

Since the VHD image is present as a device, we can mount it and directly edit files on it.
  • To mount the device
    • mount /dev/xen/blktap-2/tapdev4 mounted_image/
  • For my needs, I copied Java and ProActive and created a boot script that runs a ProActive node
  • To un mount the device
    • umount mounted_image/

Installing the template in Cloudstack is just a matter of uploading the image file using the web client. As it requires downloading a file from a HTTP URL, "python -m SimpleHTTPServer 80" comes handy to expose local files on port 80.

We have now a custom made VHD image ready to use with Cloudstack!