Infrastructure Automation - Deploying ClusterControl and MySQL-based systems on AWS using A...

Datetime:2016-08-23 02:20:16          Topic: MySQL           Share

We recently made a number of enhancements to the ClusterControl Ansible Role, so it now also supports automatic deployment of MySQL-based systems (MySQL Replication, Galera Cluster, NDB Cluster). The updated role uses the awesome ClusterControl RPC interface to automate deployments. It is available at Ansible Galaxy and Github .

TLDR; It is now possible to define your database clusters directly in a playbook (see below example), and let Ansible and ClusterControl automate the entire deployment:

cc_cluster:
  - deployment: true
    operation: "create"
    cluster_type: "galera"
    mysql_hostnames:
      - "192.168.1.101"
      - "192.168.1.102"
      - "192.168.1.103"
    mysql_password: "MyPassword2016"
    mysql_port: 3306
    mysql_version: "5.6"
    ssh_keyfile: "/root/.ssh/id_rsa"
    ssh_user: "root"
    vendor: "percona"

What’s New?

The major improvement is that you can now automatically deploy a new database setup while deploying ClusterControl. Or you can also register already deployed databases.

Define your database cluster in the playbook, within the “cc_cluster” item, and ClusterControl will perform the deployment. We also introduced a bunch of new variables to simplify the initial setup of ClusterControl for default admin credentials, ClusterControl license and LDAP settings.

Along with these new improvements, we can leverage the Ansible built-in cloud module to automate the rest of the infrastructure that our database rely on - instance provisioning, resource allocation, network configuration and storage options. In simple words, write your infrastructure as code and let Ansible work together with ClusterControl to build the entire stack.

We also included example playbooks in the repository for reference. Check them out at our Github repository page.

Example Deployment on Amazon EC2

In this example we are going to deploy two clusters on Amazon EC2 using our new role:

  • 1 node for ClusterControl
  • 3 nodes Galera Cluster (Percona XtraDB Cluster 5.6)
  • 4 nodes MySQL Replication (Percona Server 5.7)

The following diagram illustrates our setup with Ansible:

First, let’s decide what our infrastructure in AWS will look like:

  • Region: us-west-1
  • Availability Zone: us-west-1a
  • AMI ID: ami-d1315fb1
    • AMI name: RHEL 7.2 HVM
    • SSH user: ec2-user
  • Instance size: t2.medium
  • Keypair: mykeypair
  • VPC subnet: subnet-9ecc2dfb
  • Security group: default

Preparing our Ansible Master

We are using Ubuntu 14.04 as the Ansible master host in a local data-center to deploy our cluster on AWS EC2.

  1. If you are already have Ansible installed, you may skip this step:

    $ apt-get install ansible python-setuptools
  2. Install boto (required by ec2 python script):

    $ pip install boto
  3. Download and configure ec2.py and ec2.ini under /etc/ansible. Ensure the Python script is executable:

    $ cd /etc/ansible
    $ wget https://raw.githubusercontent.com/ansible/ansible/devel/contrib/inventory/ec2.py
    $ wget https://raw.githubusercontent.com/ansible/ansible/devel/contrib/inventory/ec2.ini
    $ chmod 755 /etc/ansible/ec2.py
  4. Set up Secret and Access key environment variables. Get the AWS Secret and Access Key from Amazon EC2 Identity and Access Management (IAM) and configure them as per below:

    $ export AWS_ACCESS_KEY_ID='YOUR_AWS_API_KEY'
    $ export AWS_SECRET_ACCESS_KEY='YOUR_AWS_API_SECRET_KEY'
  5. Configure environment variables for ec2.py and ec2.ini.

    export ANSIBLE_HOSTS=/etc/ansible/ec2.py
    export EC2_INI_PATH=/etc/ansible/ec2.ini
  6. Configure AWS keypair. Ensure the keypair exists on the node. For example, if the keypair is located under /root/mykeypair.pem , use the following command to add it to the SSH agent:

    $ ssh-agent bash
    $ ssh-add /root/mykeypair.pem
  7. Verify if Ansible can see our cloud. If there are running EC2 instances, you should get a list of them in JSON by running this command:

    $ /etc/ansible/ec2.py --list

**Note that you can put steps 4 and 5 inside .bashrc or .bash_profile file to load the environment variables automatically.

Define our infrastructure inside the Ansible playbook

We are going to create 2 playbooks. The first one is the definition of our EC2 instances in AWS (ec2-instances.yml) and the second one to deploy ClusterControl and the database clusters (deploy-everything.yml).

Here is an example content of ec2-instances.yml:

- name: Create instances
  hosts: localhost
  gather_facts: False

  tasks:
    - name: Provision ClusterControl node
      ec2:
        count: 1
        region: us-east-1
        zone: us-east-1a
        key_name: mykeypair
        group: default
        instance_type: t2.medium
        image: ami-3f03c55c
        wait: yes
        wait_timeout: 500
        volumes:
          - device_name: /dev/sda1
            device_type: standard
            volume_size: 20
            delete_on_termination: true
        monitoring: no
        vpc_subnet_id: subnet-9ecc2dfb
        assign_public_ip: yes
        instance_tags:
          Name: clustercontrol
          set: ansible
          group: clustercontrol

    - name: Provision Galera nodes
      ec2:
        count: 3
        region: us-east-1
        zone: us-east-1a
        key_name: mykeypair
        group: default
        instance_type: t2.medium
        image: ami-3f03c55c
        wait: yes
        wait_timeout: 500
        volumes:
          - device_name: /dev/sdf
            device_type: standard
            volume_size: 20
            delete_on_termination: true
        monitoring: no
        vpc_subnet_id: subnet-9ecc2dfb
        assign_public_ip: yes
        instance_tags:
          Name: galeracluster
          set: ansible
          group: galeracluster

    - name: Provision MySQL Replication nodes
      ec2:
        count: 4
        region: us-east-1
        zone: us-east-1a
        key_name: mykeypair
        group: default
        instance_type: t2.medium
        image: ami-3f03c55c
        wait: yes
        wait_timeout: 500
        volumes:
          - device_name: /dev/sdf
            device_type: standard
            volume_size: 20
            delete_on_termination: true
        monitoring: no
        vpc_subnet_id: subnet-9ecc2dfb
        assign_public_ip: yes
        instance_tags:
          Name: replication
          set: ansible
          group: replication

There are three types of instances with different instance_tags (Name and set) in the playbook. The “set” tag distinguishes our host group accordingly so it can be called in our deployment playbook as part of Ansible host inventory. Since we are provisioning everything from a local data-center, we set assign_public_ip to “yes” so the instances are reachable inside a VPC under “subnet-9ecc2dfb”.

Next, we create the deployment playbook as per below (deploy-everything.yml):

- name: Configure ClusterControl instance.
  hosts: tag_group_clustercontrol
  become: true
  user: ec2-user
  gather_facts: true

  roles:
    - { role: severalnines.clustercontrol, tags: controller }

  vars:
    cc_admin:
      - email: "admin@email.com"
        password: "test123"

- name: Configure Galera Cluster instances.
  hosts: 
    - tag_group_galeracluster
    - tag_group_replication
  user: ec2-user
  become: true
  gather_facts: true

  roles:
    - { role: severalnines.clustercontrol, tags: dbnodes }

  vars:
    clustercontrol_ip_address: "{{ hostvars[groups['tag_group_clustercontrol'][0]]['ec2_ip_address'] }}"

- name: Create the database clusters.
  hosts: tag_group_clustercontrol
  become: true
  user: ec2-user

  roles:
    - { role: severalnines.clustercontrol, tags: deploy-database }

  vars:
    cc_cluster:
      - deployment: true
        operation: "create"
        cluster_type: "galera"
        mysql_cnf_template: "my.cnf.galera"
        mysql_datadir: "/var/lib/mysql"
        mysql_hostnames:
          - "{{ hostvars[groups['tag_group_galeracluster'][0]]['ec2_ip_address'] }}"
          - "{{ hostvars[groups['tag_group_galeracluster'][1]]['ec2_ip_address'] }}"
          - "{{ hostvars[groups['tag_group_galeracluster'][2]]['ec2_ip_address'] }}"
        mysql_password: "password"
        mysql_port: 3306
        mysql_version: "5.6"
        ssh_keyfile: "/root/.ssh/id_rsa"
        ssh_user: "root"
        sudo_password: ""
        vendor: "percona"
      - deployment: true
        operation: "create"
        cluster_type: "replication"
        mysql_cnf_template: "my.cnf.repl57"
        mysql_datadir: "/var/lib/mysql"
        mysql_hostnames:
          - "{{ hostvars[groups['tag_group_replication'][0]]['ec2_ip_address'] }}"
          - "{{ hostvars[groups['tag_group_replication'][1]]['ec2_ip_address'] }}"
          - "{{ hostvars[groups['tag_group_replication'][2]]['ec2_ip_address'] }}"
          - "{{ hostvars[groups['tag_group_replication'][3]]['ec2_ip_address'] }}"
        mysql_password: "password"
        mysql_port: 3306
        mysql_version: "5.7"
        ssh_keyfile: "/root/.ssh/id_rsa"
        ssh_user: "root"
        sudo_password: ""
        vendor: "oracle"

The ansible user is “ec2-user” for RHEL 7.2 image. The playbook shows the deployment flow as:

  1. Install and configure ClusterControl (tags: controller)
  2. Setup passwordless SSH from ClusterControl node to all database nodes (tags: dbnodes). In this section, we have to define clustercontrol_ip_address so we know which ClusterControl node is used to manage our nodes.
  3. Perform the database deployment. The database cluster item definition will be passed to the ClusterControl RPC interface listening on the EC2 instance that has “tag_group_clustercontrol”. For MySQL replication, the first node in the mysql_hostnames list is the master.

The above are the simplest variables used to get you started. For more customization options, you can refer to the documentation page of the role under Variables section.

Fire them up

You need have the Ansible role installed. Grab it from Ansible Galaxy or Github repository:

$ ansible-galaxy install severalnines.clustercontrol

Then, create the EC2 instances:

$ ansible-playbook -i /etc/ansible/ec2.py ec2-instances.yml

Refresh the inventory:

$ /etc/ansible/ec2.py --refresh-cache

Verify all EC2 instances are reachable before the deployment begins (you should get SUCCESS for all nodes):

$ ansible -m ping "tag_set_ansible" -u ec2-user

Install ClusterControl and deploy the database cluster:

$ ansible-playbook -i /etc/ansible/ec2.py deploy-everything.yml

Wait for a couple of minutes until the playbook completes. Then, login to ClusterControl using the default email address and password defined in the playbook and you should be inside the ClusterControl dashboard. Go to Settings -> Cluster Job, you should see the “Create Cluster” jobs scheduled and deployment is under progress.

This is our final result on ClusterControl dashboard:

The total deployment time from installing Ansible to the database deployment took about 50 minutes. This included waiting for the instances to be created and database clusters to be deployed. This is pretty good, considering we were spinning 8 nodes and deploying two database clusters from scratch. How long does it take you to deploy two clusters from scratch?

Future Plan

At the moment, the Ansible role only supports deployment of the following:

  • Create new Galera Cluster
    • Percona XtraDB Cluster (5.5/5.6)
    • MariaDB Galera Cluster (5.510.1)
    • MySQL Galera Cluster - Codership (5.5/5.6)
  • Create new MySQL Replication
    • Percona Server (5.7/5.6)
    • MariaDB Server (10.1)
    • MySQL Server - Oracle (5.7)
  • Add existing Galera Cluster
    • Percona/MariaDB/Codership (all stable version)

We’re in the process of adding support for other cluster types supported by ClusterControl.

We’d love to hear your feedback in the comments section below. Would you like to see integration with more cloud providers (Azure, Google Cloud Platform, Rackspace)? What about virtualization platforms like OpenStack, VMware, Vagrant and Docker? How about load balancers (HAProxy, MaxScale and ProxySQL)? And Galera arbitrator (garbd), asynchronous replication slaves to Galera clusters, and backup management right from the Ansible playbook? The list can be very long, so let us know what is important to you. Happy automation!





About List