Here are some good examples to learn machine learning and data science using python pandas.

The following resources are from https://github.com/savarin/pyconuk-introtutorial

The tutorial will start with data manipulation using pandas - loading data, and cleaning data. We’ll then use scikit-learn to make predictions. By the end of the session, we would have worked on the Kaggle Titanic competition from start to finish, through a number of iterations in an increasing order of sophistication. We’ll also have a brief discussion on cross-validation and making visualisations.

• Section 1-0 - First Cut.ipynb
• Section 1-1 - Filling-in Missing Values.ipynb
• Section 1-2 - Creating Dummy Variables.ipynb
• Section 1-3 - Parameter Tuning.ipynb
• Appendix A - Cross-Validation.ipynb
• Appendix B - Visualisation.ipynb

Pandas and Scikit-learn

Pandas is a Python library that contains high-level data structures and manipulation tools designed for data analysis. Think of Pandas as a Python version of Excel. Scikit-learn, on the other hand, is an open-source machine learning library for Python.

While Scikit-learn does a lot of the heavy lifting, what’s equally important is ensuring that raw data is processed in such a way that we are able to ‘feed’ it to Scikit-learn. Hence, the ability to manipulate raw data with Pandas makes it an indispensible part of our toolkit.

Kaggle

Kaggle is the leading platform for data science competitions. Participants compete for cash prizes by submitting the best predictive model to problems posted on the competition website.

https://www.kaggle.com/competitions

Learning machine learning via Kaggle problems allows us to take a highly-directed approach because:

1. The problems are well-defined and the data is provided, allowing us to immediately focus on manipulating the data, and
2. The leaderboard allows us to keep track of how well we’re doing.

In the following set of exercises, we will be reviewing the data from the Kaggle Titanic competition. Our aim is to make predictions on whether or not specific passengers on the Titanic survived, based on characteristics such as age, sex and class.

Section 1-0 - First Cut

We will start by processing the training data, after which we will be able to use to ‘train’ (or ‘fit’) our model. With the trained model, we apply it to the test data to make the predictions. Finally, we output our predictions into a .csv file to make a submission to Kaggle and see how well they perform.

It is very common to encounter missing values in a data set. In this section, we will take the simplest (or perhaps, simplistic) approach of ignoring the whole row if any part of it contains an NaN value. We will build on this approach in later sections.

Pandas - Extracting data

First, we load the training data from a .csv file. This is the similar to the data found on the Kaggle website:

https://www.kaggle.com/c/titanic-gettingStarted/data

Pandas - Cleaning data

We then review a selection of the data.

We notice that the columns describe features of the Titanic passengers, such as age, sex, and class. Of particular interest is the column Survived, which describes whether or not the passenger survived. When training our model, what we are essentially doing is assessing how each feature impacts whether or not the passenger survived (or if the feature makes an impact at all).

Exercise:

• Write the code to review the tail-end section of the data.

We observe that the columns Name, Ticket and Cabin are, for our current purposes, irrelevant. We proceed to remove them from our data set.

Next, we review the type of data in the columns, and their respective counts.

We notice that the columns Age and Embarked have NaNs or missing values. As previously discussed, we take the approach of simply removing the rows with missing values.

Question

• If you were to fill in the missing values, with what values would you fill them with? Why?

Scikit-learn only takes numerical arrays as inputs. As such, we would need to convert the categorical columns Sex and Embarked into numerical ones. We first review the range of values for the column Sex, and create a new column that represents the data as numbers.

Similarly for Embarked, we review the range of values and create a new column called Port that represents, as a numerical value, where each passenger embarks from.

Question

• What problems might we encounter by mapping C, S, and Q in the column Embarked to the values 1, 2, and 3? In other words, what does the ordering imply? Does the same problem exist for the column Sex?

Now that we have numerical columns that encapsulate the information provided by the columns Sex and Embarked, we can proceed to drop them from our data set.

We review the columns our final, processed data set.

For convenience, we move the column Survived to the left-most column. We note that the left-most column is indexed as 0.

In our final review of our training data, we check that (1) the column Survived is the left-most column (2) there are no NaN values, and (3) all the values are in numerical form.

Finally, we convert the processed training data from a Pandas dataframe into a numerical (Numpy) array.

Scikit-learn - Training the model

In this section, we’ll simply use the model as a black box. We’ll review more sophisticated techniques in later sections.

Here we’ll be using the Random Forest model. The intuition is as follows: each feature is reviewed to see how much impact it makes to the outcome. The most prominent feature is segmented into a ‘branch’. A collection of branches is a ‘tree’. The Random Forest model, broadly speaking, creates a ‘forest’ of trees and aggregates the results.

http://en.wikipedia.org/wiki/Random_forest

We use the processed training data to ‘train’ (or ‘fit’) our model. The column Survived will be our second input, and the set of other features (with the column PassengerId omitted) as the first.

Scikit-learn - Making predictions

We first load the test data.

We then review a selection of the data.

We notice that test data has columns similar to our training data, but not the column Survived. We’ll use our trained model to predict values for the column Survived.

As before, we process the test data in a similar fashion to what we did to the training data.

We now apply the trained model to the test data (omitting the column PassengerId) to produce an output of predictions.

Pandas - Preparing for submission

We simply create a Pandas dataframe by combining the index from the test data with the output of predictions.

We briefly review our predictions.

Finally, we output our results to a .csv file.

However, it appears that we have a problem. The Kaggle submission website expects “the solution file to have 418 predictions.”

https://www.kaggle.com/c/titanic-gettingStarted/submissions/attach

We compare this to our result.

Since we eliminated the rows containing NaNs, we end up with a set of predictions with a smaller number of rows compared to the test data. As Kaggle requires all 418 predictions, we are unable to make a submission.

In this section, we took the simplest approach of ignoring missing values, but fail to produce a complete set of predictions. We look to build on this approach in Section 1-1.