Deploying services with Mesos, Marathon, Zookeeper and Docker

Datetime:2016-08-23 00:31:28          Topic: ZooKeeper  Docker           Share

In this post I'll demonstrate how to run a scalable and highly available service inside of Docker containers, managed by Marathon, Mesos and Zookeeper.

Here's a quick overview of the different components involved:

- Apache Zookeeper acts as the master election service, state store and high-performance coordination service for distributed applications such as Mesos. [1]

- Apache Mesos abstracts CPU, memory, storage, and other compute resources in a similar way to Open Stack Compute, effectively acting as a cluster manager and intelligent task distributor across a cluster of machines. The development community describes it as the kernel for distributed systems.[2]

- Marathon is a container orchestration platform and is one of the supported frameworks for Apache Mesos. It acts as a process and service manager. The development community describes it as systemd/upstart for the Mesos kernel.[3]

In a previouspost I showed how to deploy Apache Zookeeper from source, but in this tutorial I'll do it from a package.

The setup will consist of 3 nodes running the Zookeeper cluster, and 2 nodes in a master/slave setup running Mesos and Marathon.

First let's install and configure the zookeeper service on the 3 nodes:

Make sure that /var/zookeeper/data/myid has a unique id on each of the 3 servers.

To test if zookeeper is working properly lets insert and retreive a key/value (this is somewhat similar to etcd):


File: gistfile1.txt
-------------------

[server1]$ cd /usr/lib/zookeeper
[server1]$ java -cp zookeeper.jar:lib/slf4j-api-1.6.1.jar:lib/slf4j-log4j12-1.6.1.jar:lib/log4j-1.2.15.jar:conf:lib/jline-0.9.04.jar org.apache.zookeeper.ZooKeeperMain -server 127.0.0.1:2181
create /mytest 1
Created /mytest
get /mytest
1
cZxid = 0x50cfe
ctime = Fri Apr 08 15:39:21 UTC 2016
mZxid = 0x50cfe
mtime = Fri Apr 08 15:39:21 UTC 2016
pZxid = 0x50cfe
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 1
numChildren = 0
quit
Quitting...
2016-04-08 15:39:39,949 [myid:] - INFO  [main:ZooKeeper@684] - Session: 0x153f61f2f490000 closed

[server1]$ # Or use the provided script
[server1]$ /usr/lib/zookeeper/bin/zkCli.sh
[zk: localhost:2181(CONNECTED) 0] get /test
1
cZxid = 0x50cfe
ctime = Fri Apr 08 15:39:21 UTC 2016
mZxid = 0x50cfe
mtime = Fri Apr 08 15:39:21 UTC 2016
pZxid = 0x50cfe
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 1
numChildren = 0
[zk: localhost:2181(CONNECTED) 1] quit
Quitting...
2016-04-08 15:41:43,043 [myid:] - INFO  [main:ZooKeeper@684] - Session: 0x153f61f2f490001 closed
2016-04-08 15:41:43,128 [myid:] - INFO  [main-EventThread:ClientCnxn$EventThread@512] - EventThread shut down

Installing Apache Mesos on the master and slave nodes is next:By pointing both Mesos master and slave to the Zookeeper nodes, we delegate election to the Zookeeper service in case the Mesos master fails.

You should be able to access the Mesos GUI on port 5050:

To schedule a test task run:

For full list of the configuration parameters for Mesos refer to [4].

With Mesos deployed installing Marathon is next:Marathon uses --master to find the Mesos masters, and --zk to find ZooKeepers for storing state. The Marathon GUI is now accessible on port 8080.

For full list of the configuration parameters for Marathon refer to [5].

To run an application on Marathon you can either schedule it using the GUI on port 8080, or the REST API:

To list and delete the application through the API:


File: gistfile1.txt
-------------------

[server2]$ curl http://192.168.0.10:8080/v2/apps | python -mjson.tool
{
    "apps": [
        {
            "acceptedResourceRoles": null,
            "args": null,
            "backoffFactor": 1.15,
            "backoffSeconds": 1,
            "cmd": "python -m SimpleHTTPServer $PORT0",
            "constraints": [
                [
                    "hostname",
                    "UNIQUE",
                    ""
                ]
            ],
            "container": null,
            "cpus": 0.5,
            "dependencies": [],
            "deployments": [
                {
                    "id": "d9f6584e-7894-4515-9cfe-5b720af66082"
                }
            ],
            "disk": 0,
            "env": {},
            "executor": "",
            "fetch": [],
            "healthChecks": [
                {
                    "gracePeriodSeconds": 5,
                    "ignoreHttp1xx": false,
                    "intervalSeconds": 20,
                    "maxConsecutiveFailures": 3,
                    "path": "/",
                    "portIndex": 0,
                    "protocol": "HTTP",
                    "timeoutSeconds": 20
                }
            ],
            "id": "/my_app",
            "instances": 2,
            "ipAddress": null,
            "labels": {},
            "maxLaunchDelaySeconds": 3600,
            "mem": 32,
            "ports": [
                4000
            ],
            "requirePorts": false,
            "storeUrls": [],
            "tasksHealthy": 2,
            "tasksRunning": 2,
            "tasksStaged": 0,
            "tasksUnhealthy": 0,
            "upgradeStrategy": {
                "maximumOverCapacity": 1,
                "minimumHealthCapacity": 1
            },
            "uris": [],
            "user": null,
            "version": "2016-04-08T16:52:35.848Z",
            "versionInfo": {
                "lastConfigChangeAt": "2016-04-08T16:52:35.848Z",
                "lastScalingAt": "2016-04-08T16:52:35.848Z"
            }
        }
    ]
}

[server2]$ curl -X DELETE http://192.168.0.10:8080/v2/apps/my_app
{
    "deploymentId": "137f8969-97cb-4116-95a8-09901f6d3ad2",
    "version": "2016-04-08T17:06:04.442Z"
}

For a full list of available json config fields refer to [6].

And finally to run a web app from a Docker container:


File: gistfile1.txt
-------------------

[server1,2]$ sudo yum install -y golang git device-mapper-event-libs docker
[server1,2]$ sudo chkconfig docker on
[server1,2]$ sudo service docker start
[server1,2]$ echo 'docker,mesos' | sudo tee /etc/mesos-slave/containerizers
[server1,2]$ sudo service mesos-slave restart
[server1,2]$ docker pull docker.io/httpd
[server1]$ cat << EOF > http_docker_app.json
{
  "id": "http-docker-app",
  "cpus": 0.2,
  "mem": 32.0,
  "instances": 1,
  "constraints": [["hostname", "UNIQUE", ""]],
  "container": {
    "type": "DOCKER",
    "docker": {
      "image": "docker.io/httpd",
      "network": "BRIDGE",
      "portMappings": [
        { "containerPort": 8080, "hostPort": 0, "servicePort": 0, "protocol": "tcp" }
      ]
    }
  }
}
EOF

[server1]$ curl -X POST http://192.168.0.10:8080/v2/apps -d @http_docker_app.json -H "Content-type: application/json"
{
    "acceptedResourceRoles": null,
    "args": null,
    "backoffFactor": 1.15,
    "backoffSeconds": 1,
    "cmd": null,
    "constraints": [
        [
            "hostname",
            "UNIQUE",
            ""
        ]
    ],
    "container": {
        "docker": {
            "forcePullImage": false,
            "image": "docker.io/httpd",
            "network": "BRIDGE",
            "parameters": [],
            "portMappings": [
                {
                    "containerPort": 8080,
                    "hostPort": 0,
                    "protocol": "tcp",
                    "servicePort": 0
                }
            ],
            "privileged": false
        },
        "type": "DOCKER",
        "volumes": []
    },
    "cpus": 0.2,
    "dependencies": [],
    "deployments": [
        {
            "id": "c7d68104-98be-4055-8a59-88830c4a786c"
        }
    ],
    "disk": 0,
    "env": {},
    "executor": "",
    "fetch": [],
    "healthChecks": [],
    "id": "/http-docker-app",
    "instances": 1,
    "ipAddress": null,
    "labels": {},
    "maxLaunchDelaySeconds": 3600,
    "mem": 32,
    "ports": [
        0
    ],
    "requirePorts": false,
    "storeUrls": [],
    "tasks": [],
    "tasksHealthy": 0,
    "tasksRunning": 0,
    "tasksStaged": 0,
    "tasksUnhealthy": 0,
    "upgradeStrategy": {
        "maximumOverCapacity": 1,
        "minimumHealthCapacity": 1
    },
    "uris": [],
    "user": null,
    "version": "2016-04-08T17:18:30.387Z"
}

[server1]$ docker ps
CONTAINER ID        IMAGE               COMMAND              CREATED             STATUS              PORTS                             NAMES
5e3dc19cfa84        docker.io/httpd     "httpd-foreground"   3 minutes ago       Up 3 minutes        80/tcp, 0.0.0.0:31372->8080/tcp   mesos-cfccf3bb-3168-4ec5-99da-0bb9fa618a9c-S0.b5487c02-e054-4013-9bfc-eeb6c4387b15

Resources:

[1] https://zookeeper.apache.org/doc/r3.4.8/

[2] http://mesos.apache.org/documentation/latest/

[3] https://mesosphere.github.io/marathon/docs/

[4] http://mesos.apache.org/documentation/latest/configuration/

[5] https://mesosphere.github.io/marathon/docs/command-line-flags.html

[6] https://mesosphere.github.io/marathon/docs/rest-api.html#post-v2-apps





About List