ProActive and Docker: schedule and orchestrate Docker with ProActive and Docker Compose + variable substitution

ProActive workflows and orchestration

The ProActive Scheduler is an open source software to orchestrate tasks in your hybrid cloud infrastructure. With an easy to use GUI and an easy setup on master and nodes (just executing one sh file or why not getting it in containers directly) it fits very well into the Docker ecosystem, which I find very user friendly.

Getting Docker Compose support into ProActive

The ProActive software uses the concept of JSR 223 script engines. I used that fact to implement a Docker Compose executing JSR 223 compatible script engine, which now drives the Docker Compose support. To add Docker Compose support simply add a jar file to your ProActive installation. You can find the Docker Compose script engine here, but don't worry it is already inside the containers we will use in this walk-through.

Running ProActive

You need to have docker 1.6  and Docker Compose installed, if not: follow the installation guides for docker and Docker Compose.

Run ProActive with Docker Compose! I prepared a yaml file (download it here):

image: tobwiens/proactive-scheduler
  ports:
    - "8080:8080"
  expose:
    - "64738"
proactiveNode:
  image: tobwiens/proactive-node
  command: -r pnp://proactiveScheduler:64738
  links:
    - proactiveScheduler
  volumes:
    - /var/run/docker.sock:/var/run/docker.sock

The ‘proactiveScheduler’ start the ProActive master with a web interface which will be accessible on port 8080. Additionally it exposes the port 64738, to which the ProActive node will connect. The scheduler manages the nodes and keeps track of the tasks.
The node will execute and monitor tasks, so we need at least one node to execute our Docker Compose tasks.

The ‘proactiveNode’ is linked to the proactiveScheduler and connects with the command ‘-r pnp://proactiveScheduler:64738’; check out the node's and scheduler's dockerfile.
The docker.sock is mounted inside the container to execute containers on your local docker daemon.

Now: Execute the yaml file!

Executing a Docker Compose script with the ProActive GUI

After executing the yaml file wait until the web interface is started and go to http://127.0.0.1:8080/studio and log in with the standard account (admin/admin). Click on create, to create a new workflow:

After creating your first workflow click on open (green button):

Now that the workflow is open: import the Docker Compose task by clicking import; download it here.
Note: The Docker Compose script engine is not selectable by the GUI, which will change soon, so for now, the xml files have to be edited in a text editor and imported.
More details: The GUI checks whether the script engine is valid as soon as the script is edited, that will remove the language="docker-compose" in the xml file.

After importing you will see a task which contains a Docker Compose script: 

ubuntuEcho:
  image: dockerfile/ubuntu
  command: /bin/bash -c 'sleep 5 && echo It worked!!!!'

It executes (in the dockerfile/ubuntu image), which will print "It worked!!!!" on the screen:

sleep 5 && echo It worked!!!!

 Note: I added the ->sleep 5<- because the echo command returned so fast that Docker Compose tried to attach to the container which had already stopped, that cause it do wait forever.

After clicking execute got to http://127.0.0.1:8080/scheduler/ to see the progress of the Docker Compose script. Select the submitted job and you will see:

After the job is finished you can access the output by clicking the ‘Fetch output’ button in the Output tab:

Not glamorous but we can see the ‘It worked!!!’ in the line ‘[36mubuntuEcho_1 | [0mIt worked!!!’

Variables in Docker Compose workflows 

The new yaml file inside the task is:

ubuntuEcho:
  image: dockerfile/ubuntu
  command: /bin/bash -c 'sleep 5 && echo "$output_variable" '

The output of the echo has been replace with a variable: $output_variable.

1. Create a new workflow 
2. import the xml file (download it here). 

You should see a variable called output_variable in the 'Job Variables' tab:

Execute the workflow and check the output in the http://127.0.0.1:8080/scheduler/ interface. The variable has been replaced and the the echo will now output the content of the variable.

Follow @tobwiens