DevOps for machine learning II

Datetime:2016-08-23 00:33:31          Topic: MongoDB  ZooKeeper  Docker           Share

Inpart I we set up a small virtual cluster using Vagrant and VirtualBox . It is now time to make it a coherent architecture with mesos , zookeeper , marathon and, optionally,  docker . As an exercice to demonstrate the cluster management features, we’ll use mesos and marathon to deploy a distributed mongoDB backend with sharding enabled.

Managing a cluster with Mesos

Mesos acts like a kernel for distributed systems and manages CPU, memory and disk at the cluster level. It provides a simplified API that can be used to ask for resources without specifying a particular instance. Even better, mesos is already able to run   a few frameworks and, inpart III, we’ll configure spark to run on mesos, allowing several users to run jobs concurrently.

In this part, let’s deploy mesos to our cluster.

If you haven’t configured the cluster as inpart I,   you can download the associated GitHub repo and run  vagrant up in the part2 directory (assuming you have installed VitualBox, Vagrant and ansible).

As we did inpart I, let’s use ansible to deploy the configuration to the cluster machines. This time, we’ll use roles in our playbook to have slightly different configurations on the master and slave nodes. I’ve just copied the content of the playbook directly below.

playbook-mesos.yml

YAML

- hosts: master
  remote_user: vagrant
  sudo: yes
  tasks:
      # Install mesos, zookeeper, marathon and docker
      - name: add mesosphere repo key
        apt_key: keyserver=keyserver.ubuntu.com id=E56151BF state=present
      - name: install the mesosphere apt repo
        apt_repository: repo='deb http://repos.mesosphere.com/ubuntu trusty main' state=present update_cache=yes
      - name: install mesos
        apt: pkg=mesos state=present update_cache=yes
      - name: install marathon
        apt: pkg=marathon state=present update_cache=yes

      # configure mesos and marathon
      - name: set the zookeeper master
        shell: sed -i -e 's/localhost/192.168.33.10/g' /etc/mesos/zk
      - name: configure zookeeper ID
        shell: echo 1 > /etc/zookeeper/conf/myid

      # Ensure the services are started on boot
      - name: restart zookeeper and enable it
        service: name=zookeeper state=restarted enabled=yes
      - name: enable mesos-master
        service: name=mesos-master state=started enabled=yes
      - name: enable mesos-slave
        service: name=mesos-slave state=started enabled=yes
      - name: start marathon
        service: name=marathon state=started enabled=yes


- hosts: slaves
  remote_user: vagrant
  sudo: yes
  tasks:
      # Install mesos and docker
      - name: add mesosphere repo key
        apt_key: keyserver=keyserver.ubuntu.com id=E56151BF state=present
      - name: install the mesosphere apt repo
        apt_repository: repo='deb http://repos.mesosphere.com/ubuntu trusty main' state=present update_cache=yes
      - name: install mesosphere
        apt: pkg=mesos state=present update_cache=yes

        # Configure zookeeper
      - name: set the zookeeper master
        shell: sed -i -e 's/localhost/192.168.33.10/g' /etc/mesos/zk

      # Ensure the services are started
      - name: make sure mesos-slave is running
        service: name=mesos-slave state=started enabled=yes
      - name: sisable mesos-master
        service: name=mesos-master state=stopped enabled=no


# Bonus: DOCKER
# - hosts: all
#   remote_user: vagrant
#   sudo: yes
#       - name: get docker
#         get_url: url=https://get.docker.com/ dest=/tmp/docker.sh force=no mode=777
#       - name: install docker
#         shell: /tmp/docker.sh -b executable=/bin/bash
#       - name: start docker
#         service: name=docker state=started enabled=yes
#       - name: enable docker containers 
#         template: src=templates/containerizers.j2 dest=/etc/mesos-slave/containerizers
- hosts: master
  remote_user: vagrant
  sudo: yes
  tasks:
      # Install mesos, zookeeper, marathon and docker
      - name: add mesosphere repo key
        apt_key: keyserver=keyserver.ubuntu.com id=E56151BF state=present
      - name: install the mesosphere apt repo
        apt_repository: repo='deb http://repos.mesosphere.com/ubuntu trusty main' state=present update_cache=yes
      - name: install mesos
        apt: pkg=mesos state=present update_cache=yes
      - name: install marathon
        apt: pkg=marathon state=present update_cache=yes
 
      # configure mesos and marathon
      - name: set the zookeeper master
        shell: sed -i -e 's/localhost/192.168.33.10/g' /etc/mesos/zk
      - name: configure zookeeper ID
        shell: echo 1> /etc/zookeeper/conf/myid
 
      # Ensure the services are started on boot
      - name: restart zookeeper and enable it
        service: name=zookeeper state=restarted enabled=yes
      - name: enable mesos-master
        service: name=mesos-master state=started enabled=yes
      - name: enable mesos-slave
        service: name=mesos-slave state=started enabled=yes
      - name: start marathon
        service: name=marathon state=started enabled=yes
 
 
- hosts: slaves
  remote_user: vagrant
  sudo: yes
  tasks:
      # Install mesos and docker
      - name: add mesosphere repo key
        apt_key: keyserver=keyserver.ubuntu.com id=E56151BF state=present
      - name: install the mesosphere apt repo
        apt_repository: repo='deb http://repos.mesosphere.com/ubuntu trusty main' state=present update_cache=yes
      - name: install mesosphere
        apt: pkg=mesos state=present update_cache=yes
 
        # Configure zookeeper
      - name: set the zookeeper master
        shell: sed -i -e 's/localhost/192.168.33.10/g' /etc/mesos/zk
 
      # Ensure the services are started
      - name: make sure mesos-slave is running
        service: name=mesos-slave state=started enabled=yes
      - name: sisable mesos-master
        service: name=mesos-master state=stopped enabled=no
 
 
# Bonus: DOCKER
# - hosts: all
#   remote_user: vagrant
#   sudo: yes
#       - name: get docker
#         get_url: url=https://get.docker.com/ dest=/tmp/docker.sh force=no mode=777
#       - name: install docker
#         shell: /tmp/docker.sh -b executable=/bin/bash
#       - name: start docker
#         service: name=docker state=started enabled=yes
#       - name: enable docker containers
#         template: src=templates/containerizers.j2 dest=/etc/mesos-slave/containerizers

Edit:  It seems that more recent versions require java 8. As these dependencies can be a bit sketchy on  ubuntu/trusty64 , I recommend using  ubuntu/vivid64  if you encounter any problem.

The script just installs ubuntu packages for mesosmarathon and optionally  docker (if you uncomment the right corresponding tasks). The only part that needs a bit of explaining is

# configure marathon, zookeeper and docker
- name: set the zookeeper master
  shell: sed -i -e 's/localhost/192.168.33.10/g' /etc/mesos/zk
- name: configure zookeeper ID  # Only on the master
  shell: echo 1 > /etc/zookeeper/conf/myid
- name: enable docker containers 
  template: source=./template/containerizers.j2 dest=/etc/mesos-slave/containerizers
# configure marathon, zookeeper and docker
- name: set the zookeeper master
  shell: sed -i -e 's/localhost/192.168.33.10/g' /etc/mesos/zk
- name: configure zookeeper ID  # Only on the master
  shell: echo 1> /etc/zookeeper/conf/myid
- name: enable docker containers
  template: source=./template/containerizers.j2 dest=/etc/mesos-slave/containerizers
  1. specify the ip address of the master instead of localhost in /etc/mesos/zk ,
  2. initialise the zookeeper ID to 1   (on the master node only) ,
  3. enable docker containers to run on mesos ( /etc/mesos-slaves/containerizers ) by adding the one line  mesos,docker   to the file  containerizers.j2 .

These three small configuration tasks should be sufficient to run a fully functional mesos cluster!

We can now save the file as playbook-mesos.yml,

/bin/sh

Shell

vagrant@node1:~$ tree
.
`-- ansible
    |-- ansible.cfg
    |-- inventory.ini
    |-- playbook-hosts.yml
    |-- playbook-mesos.yml
    `-- templates
        |-- containerizers.j2
        `-- hosts.j2
vagrant@node1:~$ tree
.
`-- ansible
    |-- ansible.cfg
    |-- inventory.ini
    |-- playbook-hosts.yml
    |-- playbook-mesos.yml
    `-- templates
        |-- containerizers.j2
        `-- hosts.j2

and run it from the master node.

/bin/sh

Shell

vagrant@node1:~$ cd ansible/ vagrant@node1:~/ansible$ ansible-playbook playbook-mesos.yml SSH password: PLAY [master] *

GATHERING FACTS *** ok: [node1]

... ... ...

vagrant@node1:~$ cd ansible/
vagrant@node1:~/ansible$ ansible-playbook playbook-mesos.yml 
SSH password: 
PLAY [master] ***************************************************************** 
 
GATHERING FACTS *************************************************************** 
ok: [node1]
 
...
...
...

If ansible complains about SSH HostKeys , try running the following command  ssh-keyscan node1 node2 node3 node4 >> ~/.ssh/known_hosts to add hosts fingerprints to the master.

After all tasks have completed, we can check the cluster state on mesos web UI at http://node1:5050 .

MongoDB with mesos and marathon

To demonstrate how easy service management is with mesos and marathon, we’ll install and start mongoDB database with sharding on the cluster.

Again, let’s use an ansible playbook to install mongoDB on all machines.

playbook-mongo.yml

YAML

- hosts: all
  remote_user: vagrant
  sudo: yes
  tasks:
      - name: add mongodb repo key
        apt_key: keyserver=keyserver.ubuntu.com id=7F0CEB10 state=present
      - name: install the mongodb apt repo
        apt_repository: repo='deb http://repo.mongodb.org/apt/ubuntu trusty/mongodb-org/3.0 multiverse' state=present update_cache=yes
      - name: install mongodb
        apt: pkg=mongodb-org state=present update_cache=yes
      - name: ensure mongodb is not running
        service: name=mongod state=stopped enabled=no
      - name: ensure db directory exists
        file: path=/data/db state=directory
      - name: ensure configdb directory exists
        file: path=/data/configdb state=directory
- hosts: all
  remote_user: vagrant
  sudo: yes
  tasks:
      - name: add mongodb repo key
        apt_key: keyserver=keyserver.ubuntu.com id=7F0CEB10 state=present
      - name: install the mongodb apt repo
        apt_repository: repo='deb http://repo.mongodb.org/apt/ubuntu trusty/mongodb-org/3.0 multiverse' state=present update_cache=yes
      - name: install mongodb
        apt: pkg=mongodb-org state=present update_cache=yes
      - name: ensure mongodb is not running
        service: name=mongod state=stopped enabled=no
      - name: ensure db directory exists
        file: path=/data/db state=directory
      - name: ensure configdb directory exists
        file: path=/data/configdb state=directory

An run it from the master node using ansible-playbook playbook-mongo.yml .

Now that mongo is installed on all nodes, let’s deploy it. We’ll configure the cluster as described in http://docs.mongodb.org/manual/tutorial/deploy-shard-cluster/ , but using marathon and mesos to manage the different processes instead. I’ll show you how to do it using marathon’s REST API and curl in the terminal but you could as well do this from the marathon web UI available at http://node1:8080 (I’ll let you figure it out as an exercice).

In your local terminal, you can enter the following commands:

curl -X POST -H "Content-type: application/json" node1:8080/v2/apps -d '{
    "id": "mongod",
    "cmd": "mongod --port 27017",
    "instances": 4,
    "constraints": [["hostname", "GROUP_BY"]]
  }'
curl -X POST -H "Content-type: application/json" node1:8080/v2/apps -d '{
    "id": "mongod",
    "cmd": "mongod --port 27017",
    "instances": 4,
    "constraints": [["hostname", "GROUP_BY"]]
  }'

This will start 4 mongod  instances listening on port 27017. The constraints  "constraints": [["hostname", "GROUP_BY"]] ensure that the instances are started on different machines.  mongod  instances background processes that will store the data. You can confirm that the instances are running by going to the web UI on http://node1:8080

In distributed mode, we also need 3 configserver instances that we can launch with the following command:

curl -X POST -H "Content-type: application/json" node1:8080/v2/apps -d '{
    "id": "mongod-configserver",
    "cmd": "mongod --configsvr --dbpath /data/configdb --port 27019",
    "instances": 3,
    "constraints": [["hostname", "GROUP_BY"], ["hostname", "LIKE", "node[1-3]"]]
  }'
  curl -X POST -H "Content-type: application/json" node1:8080/v2/apps -d '{
    "id": "mongod-configserver",
    "cmd": "mongod --configsvr --dbpath /data/configdb --port 27019",
    "instances": 3,
    "constraints": [["hostname", "GROUP_BY"], ["hostname", "LIKE", "node[1-3]"]]
  }'

This time we added the constraint that the three instances must be started on nodes 1 to 3. Each configserver manages all of the mongoDB cluster metadata and they are replicated to avoid having a single point of failure.

Finally, we’ll launch a mongos instance. mongos  act as entry points for applications and redirect to  mongod instances like load balancers. We’ll launch the mongos on a node chosen randomly in the cluster by mesos, without constraint.

curl -X POST -H "Content-type: application/json" node1:8080/v2/apps -d '{
    "id": "mongos",
    "cmd": "mongos --configdb node1:27019,node2:27019,node3:27019 --port 27018",
    "instances": 1,
    "constraints": []
  }'
curl -X POST -H "Content-type: application/json" node1:8080/v2/apps -d '{
    "id": "mongos",
    "cmd": "mongos --configdb node1:27019,node2:27019,node3:27019 --port 27018",
    "instances": 1,
    "constraints": []
  }'

And we’re finished. We can now check on which node the mongos instance is running using the marathon web UI,

and connect to it (node4 in my case) using

zer@localhost$ mongo --host node4:27018
zer@localhost$ mongo --host node4:27018

Conclusion

We’ve deployed mesos to our virtual cluster and used it to launch and manage a distributed version of mongoDB. Managing shards properly requires a bit more configuration as described in the distributed mongoDB tutorial . These commands are run inside the mongo shell and, as far as infrastructure is concerned, we should be good. If you want to go a bit further you can try launching a dockerized application using marathon. You could try a mongoDB UI for example?

Also remember that the web interfaces for mesos and marathon are available at http://node1:5050 and http://node1:8080 respectively.

This concludes part II. In part III, we’ll run spark on mesos and see how we can take advantage mesos’ resource management capabilities to run jobs concurrently. This allows several users to use spark on the same infrastructure or a single user to run multiple experiments for hyper parameter tuning)

The git repo associated with this tutorial is available at https://github.com/zermelozf/mlops in the part2 folder.

ansible distributed architectures marathon mesos





About List