Building a scalable Data Science Platform ( Luigi, Apache Spark, Pandas, Flask)
Fifth Elephant 2016
We use Vagrant along with Virtual Box to make our job easier
- To get started you will need Virtual Box. Go download it from here . We highly recommend using 5.0 but 5.1 should work just fine as well.
- Once Virtual Box is installed, install Vagrant from here .
By the end of these 2 steps, you should have the
executable in your path.
Once this is done, clone the repository into a location of your choice
git clone https://github.com/unnati-xyz/fifthel-2016-workshop.git
into the repository directory
From here, you want to bring up the vagrant box. Its quite simple
This will download the Unnati image and start up the virtual machine. Next SSH into the machine
Following this, if you see a prompt, then you're good to go :)
Setup without Vagrant
We thoroughly recommend that you use vagrant so that you have everything setup for you. However if you insist on not using it for whatsoever reason (corporate laptop, etc) then the following steps are for you.
The Vagrant box is created with precisely the same following steps. We use a Ubuntu 14.04 32 bit Operating System as the base for installation.
$ sudo apt-get update
Install the required packages
$ sudo apt-get install build-essential python3-dev python3-pip postgresql-9.3 postgresql-server-dev-9.3 openjdk-7-jdk openjdk-7-jre git-core
Install Apache Spark
$ cd $ wget http://d3kbcqa49mib13.cloudfront.net/spark-1.6.1-bin-hadoop2.6.tgz $ tar zxvf spark-1.6.1-bin-hadoop2.6.tgz $ rm spark-1.6.1-bin-hadoop2.6.tgz
Next, you need to set a few required environment variables for things to work. This step might change based on your installation
file in your home dir to be sourced
$ touch ~/.exports
to where the
$ echo "export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-i386" >> ~/.exports
$ echo "export PATH=/home/vagrant/spark-1.6.1-bin-hadoop2.6/bin:$PATH" >> ~/.exports
$ echo "export SPARK_HOME=/home/vagrant/spark-1.6.1-bin-hadoop2.6" >> ~/.exports
according to the
$ echo "export PYTHONPATH=\$SPARK_HOME/python:\$SPARK_HOME/python/lib/py4j-0.9-src.zip" >> ~/.exports
is for escaping the
so that the
variable isn't evaluated when being added into the file
You also need to add the repository location into
$ echo "export PYTHONPATH=$PYTHONPATH:/path/to/fifthel-2016-workshop/dir" >> ~/.exports
Finally we tell Spark to use Python 3 over 2
$ echo "export PYSPARK_PYTHON=/usr/bin/python3" >> ~/.exports
We need the
file to be sourced when the shell starts up, so lets do that
$ echo "source ~/.exports" >> ~/.bashrc
Next, install all the packages from
in the repository. Note
: Since we use vagrant, we install the packages globally. But you might not want to do that if you're installing this on your system. A Virtual environment with
is recommended. Make sure that you create a
if you are going down this path.
$ cd /path/to/fifthel-2016-workshop/dir $ sudo pip3 install -r requirements.txt
If your PostgreSQL is already configured, then you can skip the following step.
We need to set a password for the
user and allow login. In order to do this, first login via the
OS user and set the password using
$ sudo su - postgres $ psql postgres=# alter user postgres with password 'postgres'; postgres=# \q $ logout
Back as your regular user, Edit the
$ sudo vim /etc/postgresql/9.4/main/pg_hba.conf
and set change the following line
local all postgres peer
local all postgres md5
And restart PostgreSQL
$ sudo service postgresql restart
should ask you for the password
$ psql -U postgres Password for user postgres:
at the prompt and you should see the psql prompt.
After you've done all this, you should be setup for the workshop :)
"In theory, there is no difference between theory and practice. But in practice, there is." - Yogi Berra
Once the task of prototyping a data science solution has been accomplished on a local machine, the real challenge begins in how to make it work in production. To ensure that the plumbing of the data pipeline will work in production at scale is both an art and a science. The science involves understanding the different tools and technologies needed to make the data pipeline connect, while the art involves making the trade-offs needed to tune the data pipeline so that it flows.
In this workshop, you will learn how to build a scalable data science platform with set up and conduct data engineering using Pandas and Luigi, build a machine learning model with Apache Spark and deploy it as predictive api with Flask
The biggest challenge in building a data science platform is to glue all the moving pieces together. Typically, a data science platform consists of:
- Data engineering - involves a lot of ETL and feature engineering.
- Machine learning - involves writing a bunch of machine learning models and persistence of the model
- API - involves exposing end points to the outside world to invoke the predictive capabilities of the model
Over time the amount of data stored that needs to be processed increases and it necessitates the need to run the Data Science process frequently. But different technologies/stack solve different parts of the Data Science problem. Leaving it to respective teams introduces lag into the system. What is needed is an automated pipeline process - one that can be invoked based on business logic (real time, near-real-time etc) and a guarantee that it will maintain data integrity. Details of the workshop
We all know that 80% of the effort is spent on data engineering while the rest is spent in building the actual machine learning models. Data engineering starts with identifying the right data sources. Data sources can be databases, third party APIs, HTML documents which needs to be scrapped and so on. Acquiring data from databases is a straight forward job, while acquiring data from third party APIs and scrapping may come with its own complexities like page visit limits, API rate limiting etc. Once we manage to acquire data from all these sources, the next job is to clean the data.
We will be covering the following topics for data engineering:
- Identifying and working with 2 data sources.
- Writing ETL (Extraction, Transformation and Loading) with Pandas
- Building dependency management with Luigi
- Logging the process
- Adding notifications on success and failure
Building a robust and scalable machine learning platform is a hard job. As the data size increases, the need for more computational capabilities increase. So how do you build a system that can scale by just adding more hardware and not worrying about changing the code too much every time? The answer to that is to use Apache Spark ML. Apache Spark lets us build machine learning platforms by providing distributed computing capabilities out of the box.
We will be covering the following topics for Machine Learning:
- Feature Engineering
- Hypothesis to solve
- Configuration of environment variables for Apache Pyspark
- Build the Machine Learning code with Apache Spark
- Persisting the model
It ain’t over until the fat lady sings. Making a system API driven is very essential as it ensures the usage of the built machine learning model , thereby helping other systems integrate the capabilities with ease.
We will be covering the following topics for API:
- Building REST API with Flask
- Based on the input parameters, build respective methods to extract features to be fed into the model
- Send responses as a JSON
- Python - Knowledge of writing classes
Knowledge of data science:
- What is data science?
- Practical use cases for data science?
Knowledge of machine learning:
- Expect to know Linear regression and logistic regression
Knowledge of software engineering:
- Understanding scalability and high available systems