Hot off the heels of ContainerCon/MesosCon we looked into how we could help support data persistence for the popular distributed systems manager. We trawled through a lot of documentation, and found that Marathon has a great Docker containerizer and also supports arbitrary options. Yay! Have we lost you yet?

Here’s how it works:

Mesos is a tool to abstract datacenter resources such as CPU, memory, storage and more. Marathon is used on top of Mesos to schedule applications on top of those resources, and those applications can be in Docker containers. As you might remember from previous posts, the EMC {code} team have worked on a Docker volume plugin called REX-Ray (new name suggestions welcome) to make sure that we can actually store data from our containerized applications. So, with all these pieces together, we should be able to create a volume using REX-Ray, tie it to a container, have that container scheduled by Marathon, and eventually started on a Mesos node by a Mesos master.

mesos

In our test environment we have:

  • 1 Mesos Master
  • 1 Zookeeper
  • 1 Marathon server
  • 3 Mesos Nodes

The entire setup can be replicated by using this repository branch: https://github.com/jonasrosland/vagrant-mesos/tree/mesos-rexray.

If you want to watch a demo of how it works before you dive into the details of the setup you can do so here:

In the automated setup above we make sure all the Mesos nodes have Docker 1.8.1 installed to take advantage of the Docker volume plugin functionality that was announced in 1.8, and also have the latest version of REX-ray installed.

We can verify that REX-ray is installed and working correctly:

[email protected]:/home/ubuntu# rexray get-instance
- providername: ec2
 instanceid: i-e11d784a
 region: us-east-1
 name: vagrant-mesos-slave2

Now we can create, attach, detach and delete volumes like this:

[email protected]:/home/ubuntu# rexray new-volume --size=10 --volumename=persistent-data
2015/08/27 15:58:50 Waiting for volume creation to complete
name: persistent-data
volumeid: vol-425e94af
availabilityzone: us-east-1a
status: available
volumetype: standard
iops: 0
size: "10"
networkname: ""
attachments: []

[email protected]:/home/ubuntu# rexray attach-volume --volumeid="vol-425e94af"
2015/08/27 16:03:40 Got next device name: /dev/xvdc
2015/08/27 16:03:40 Waiting for volume attachment to complete
2015/08/27 16:03:48 Attached volume vol-425e94af to instance i-e11d784a
- volumeid: vol-425e94af
 instanceid: i-e11d784a
 devicename: /dev/xvdc
 status: attached

[email protected]:/home/ubuntu# rexray get-volume
<snip>
- name: persistent-data
 volumeid: vol-425e94af
 availabilityzone: us-east-1a
 status: in-use
 volumetype: standard
 iops: 0
 size: "10"
 networkname: ""
 attachments:
 - volumeid: vol-425e94af
 instanceid: i-e11d784a
 devicename: /dev/xvdc
 status: attached

[email protected]:/home/ubuntu# rexray detach-volume --volumeid="vol-425e94af"
2015/08/27 16:04:00 Waiting for volume detachment to complete
2015/08/27 16:04:10 Detached volume vol-425e94af

[email protected]:/home/ubuntu# rexray remove-volume --volumeid="vol-425e94af"
2015/08/27 16:06:37 Deleted Volume: vol-425e94af

Ok cool, but that’s just for command line, how about we step it up a notch and use Marathon to do this for us?

Let’s create a JSON file with the following data:

{
 "id": "nginx",
 "container": {
 "docker": {
 "image": "million12/nginx",
 "network": "BRIDGE",
 "portMappings": [
 { "containerPort": 80, "hostPort": 0, "protocol": "tcp"}
 ],
 "parameters": [
 { "key": "volume-driver", "value": "rexray" },
 { "key": "volume", "value": "nginx-data:/data/www" }
 ]
 }
 },
 "cpus": 0.2,
 "mem": 32.0,
 "instances": 1
}

This JSON file contains an ID for the application (nginx), the Docker image we want to run (million12/nginx), a port mapping so we can access port 80, and some parameters to pass on to Docker when we run the container. Those parameters are what will create a volume using REX-ray, attach it to the container and let the container run with a persistent data store!

Let’s start it:

[email protected]:/home/ubuntu# curl -X POST https://your.marathon.ip.here:8080/v2/apps?force=true -d @nginx.json -H "Content-type: application/json"

{"id":"/nginx","cmd":null,"args":null,"user":null,"env":{},"instances":1,"cpus":0.2,"mem":32.0,"disk":0.0,"executor":"","constraints":[],"uris":[],"storeUrls":[],"ports":[0],"requirePorts":false,"backoffFactor":1.15,"container":{"type":"DOCKER","volumes":[],"docker":{"image":"million12/nginx","network":"BRIDGE","portMappings":[{"containerPort":80,"hostPort":0,"servicePort":0,"protocol":"tcp"}],"privileged":false,"parameters":[{"key":"volume-driver","value":"rexray"},{"key":"volume","value":"nginx-data:/data/www"}],"forcePullImage":false}},"healthChecks":[],"dependencies":[],"upgradeStrategy":{"minimumHealthCapacity":1.0,"maximumOverCapacity":1.0},"labels":{},"acceptedResourceRoles":null,"version":"2015-08-27T16:10:32.669Z","deployments":[{"id":"e1e62f4c-bab4-4dff-a126-efea074b29ae"}],"tasks":[],"tasksStaged":0,"tasksRunning":0,"tasksHealthy":0,"tasksUnhealthy":0,"backoffSeconds":1,"maxLaunchDelaySeconds":3600}

[email protected]:/home/ubuntu# rexray get-volume
<snip>
- name: nginx-data
 volumeid: vol-926fa57f
 availabilityzone: us-east-1a
 status: in-use
 volumetype: standard
 iops: 0
 size: "16"
 networkname: ""
 attachments:
 - volumeid: vol-926fa57f
 instanceid: i-e11d784a
 devicename: /dev/xvdc
 status: attached

[email protected]:/home/ubuntu# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
a40d9c17eba0 million12/nginx "/config/bootstrap.sh" About a minute ago Up 19 seconds 443/tcp, 0.0.0.0:31216->80/tcp mesos-20150820-201454-184623020-5050-3365-S1.20a739a9-8cf4-4943-9fdb-8593349e8b3f

[email protected]:/home/ubuntu# docker inspect a40d9c17eba0
<snip>
 {
 "Name": "nginx-data",
 "Source": "/var/lib/docker/volumes/nginx-data",
 "Destination": "/data/www",
 "Driver": "rexray",
 "Mode": "z",
 "RW": true
 },
<snip>

[email protected]:/home/ubuntu# ls /var/lib/docker/volumes/nginx-data
default lost+found
[email protected]:/home/ubuntu# ls /var/lib/docker/volumes/nginx-data/default/
index.html

And there you go! We now have a persistent data volume attached to the container, all through the wonders of Marathon and Mesos 🙂

But wait you say, what about failover scenarios? Our container has some really important data, what would happen if a container host goes down? Well, Mesos would actually handle that and automatically reschedule it onto another Mesos node. In this example I shut down the host that the container is currently running on (node2), and then the following happens on node3:

Screenshot 2015-08-27 12.48.29

This means that Mesos is actively trying to start the container on the new host, node3, so it’s downloading the container image, mapping the volume, and making sure everything looks right. After a short while we’re presented with this:

Screenshot 2015-08-27 12.50.21

And we can also log in to the Marathon GUI and see that our app is now available on another host, with another IP, another port, but still with the same persistent volume. That’s pretty cool if you ask me 🙂

Screenshot 2015-08-27 12.50.50

So again, if you want to try this out, use the repo branch here and edit the cluster.yml file to your liking.

Have fun!