Wednesday, June 11, 2014

Docker & ProActive

Nowadays the cloud is all about Docker! Docker there, Docker here! And with the recent release of Docker 1.0, you just can’t miss it. My Twitter timeline is filled with #docker!

In this blog post, I will demonstrate how ProActive can play along with Docker. If you are new to this I recommend to go through the documentation, to try it online with the emulator and even better try it on your machine.

Docker looks really interesting when it comes to ProActive because it can provide good isolation and control over the ProActive nodes. It means you could share a powerful machine between multiple users and enforce certain rules (i.e resources available) while keeping it lightweight. You could also use it to provide customized and portable environments for your users. Docker relies on Linux containers and it is just fast! 

I’ll assume that the latest release is used, you can download it from our website (use version 5.4 of Server). We will start by running the ProActive Scheduler:

$> unzip
$> cd ProActiveScheduling-3.4.4_bin_full
$> ./bin/unix/scheduler-start-gui -Dproactive.useIPaddress=true

Here we explicitly tell the Scheduler to bind to docker0 interface, the interface used between your machine, the host and the containers. We also rely on IPs instead of host names for simplicity. The URLs of the Scheduler and the web portals are printed out.

Starting the scheduler...
Starting the resource manager...
The resource manager with 4 local nodes created on rmi://jily.local:1099/
The scheduler created on rmi://jily.local:1099/
Deployed application: http://localhost:8080/rest
Deployed application: http://localhost:8080/rm
Deployed application: http://localhost:8080/sched

Now we want to add ProActive nodes running in containers and connect them to the Scheduler.

$> docker run -ti activeeon/proactive:3.4.4 /opt/proactive/rm-start-docker-node rmi://

The address is the address of the network interface docker0 on the host when containers are running, the ProActive node in a container will connect to rmi://, the address of the Resource Manager. 

Here we run the docker run command interactively (-ti) so you should see the node’s output. You can check that a new ProActive node has been added in the Resource Manager web portal.

So what just happened there? Well Docker downloaded the image called activeeon/proactive:3.4.4 from the Docker registry and ran it with the command /opt/proactive/rm-start-docker-node rmi:// The image is available in the Docker hub that is used to share images, it already contains Java and ProActive. We also added a script called rm-start-docker-node to easily start a node (using default credentials and the id of the container in the node’s name).

The first launch probably took some time because the image was downloaded but if you relaunch the same command, it will be just blazing fast.

Below is a screenshot of the Resource Manager portal, showing 4 Docker nodes. You can see that a Docker node has only two processes: the shell script passed to the command line and the node itself (running with Java).

The Docker ProActive nodes can be used to schedule tasks, let’s see an example with the follow job:

The following output is produced:

[550000@b2a793c40183;12:26:49] PID TTY TIME CMD
[550000@b2a793c40183;12:26:49] 1 ? 00:00:00 rm-start-docker
[550000@b2a793c40183;12:26:49] 7 ? 00:00:08 java
[550000@b2a793c40183;12:26:49] 236 ? 00:00:06 java
[550000@b2a793c40183;12:26:49] 283 ? 00:00:00 ps

There is one more Java process, the one running the task.

The Docker image that was downloaded is fairly simple, here is the Dockerfile:

It just takes an official Docker image that comes with Java, copy ProActive to it and voilà! To make easy to test, the image has been pushed to the Docker registry.

Docker is well known as a devops tool to easily share environments from development to production. We think it can also be used for HPC use cases by allowing ProActive users to configure and test their computations on their machines and run them at scale on powerful grids. Let us know how you use it!