Learn for Master

Using Azure ML to Build Clickthrough Prediction Models

• from: https://blogs.technet.microsoft.com/machinelearning/2015/11/03/using-azure-ml-to-build-clickthrough-prediction-models/
  

This blog post is by Girish Nathan, a Senior Data Scientist at Microsoft.

Ad click prediction is a multi-billion dollar industry, and one that is still growing rapidly. In this post, we build ML models on the largest publicly available ad click prediction dataset, from Criteo. The Criteo dataset consists of some 4.4 billion advertising feedback events. In Criteo’s words, “…this dataset contains feature values and click feedback for millions of display ads. Its purpose is to benchmark algorithms for clickthrough rate (CTR) prediction.”

Azure services provide the tools needed to build a predictive model using this data.

Data Transformation

One hot encoding transforms categorical features to a format that works better with classification and regression algorithms.

Let’s take the following example. I have seven sample inputs of categorical data belonging to four categories. Now, I could encode these to nominal values as I have done here, but that wouldn’t make sense from a machine learning perspective. We can’t say that the category of “Penguin” is greater or smaller than “Human”. Then they would be ordinal values, not nominal.

What we do instead is generate one boolean column for each category. Only one of these columns could take on the value 1 for each sample.

from: http://www.jianshu.com/p/32def2294ae6

最近挤出时间，用python在kaggle上试了几个project，有点体会，记录下。

Step1: Exploratory Data Analysis

EDA，也就是对数据进行探索性的分析，一般就用到pandas和matplotlib就够了。EDA一般包括：

1. 每个feature的意义，feature的类型,比较有用的代码如下
   <span class="hljs-selector-tag">df</span><span class="hljs-selector-class">.describe</span>() <span class="hljs-selector-tag">df</span><span class="hljs-selector-attr">['Category']</span><span class="hljs-selector-class">.unique</span>()

   1 2	<span class="hljs-selector-tag">df</span><span class="hljs-selector-class">.describe</span>() <span class="hljs-selector-tag">df</span><span class="hljs-selector-attr">['Category']</span><span class="hljs-selector-class">.unique</span>()

2. 看是否存在missing value
   df<span class="hljs-selector-class">.loc</span>[df<span class="hljs-selector-class">.Dates</span><span class="hljs-selector-class">.isnull</span>(),<span class="hljs-string">'Dates'</span>]

   1	df<span class="hljs-selector-class">.loc</span>[df<span class="hljs-selector-class">.Dates</span><span class="hljs-selector-class">.isnull</span>(),<span class="hljs-string">'Dates'</span>]

3. 每个特征下的数据分布，可以用boxplot或者hist来看
   %<span class="hljs-selector-tag">matplotlib</span> <span class="hljs-selector-tag">inline</span> <span class="hljs-selector-tag">import</span> <span class="hljs-selector-tag">matplotlib</span><span class="hljs-selector-class">.pyplot</span> <span class="hljs-selector-tag">as</span> <span class="hljs-selector-tag">plt</span> <span class="hljs-selector-tag">df</span><span class="hljs-selector-class">.boxplot</span>(column=<span class="hljs-string">'Fare'</span>, by = <span class="hljs-string">'Pclass'</span>) <span class="hljs-selector-tag">plt</span><span class="hljs-selector-class">.hist</span>(df[<span class="hljs-string">'Fare'</span>], bins = <span class="hljs-number">10</span>, range =(df[<span class="hljs-string">'Fare'</span>].min(),df[<span class="hljs-string">'Fare'</span>].max())) <span class="hljs-selector-tag">plt</span><span class="hljs-selector-class">.title</span>(<span class="hljs-string">'Fare >distribution'</span>) <span class="hljs-selector-tag">plt</span><span class="hljs-selector-class">.xlabel</span>(<span class="hljs-string">'Fare'</span>) <span class="hljs-selector-tag">plt</span><span class="hljs-selector-class">.ylabel</span>(<span class="hljs-string">'Count of Passengers'</span>) #如果变量是<span class="hljs-selector-tag">categorical</span>的，想看<span class="hljs-selector-tag">distribution</span>，则可以： <span class="hljs-selector-tag">df</span><span class="hljs-selector-class">.PdDistrict</span><span class="hljs-selector-class">.value_counts</span>()<span class="hljs-selector-class">.plot</span>(kind=<span class="hljs-string">'bar'</span>, figsize=(<span class="hljs-number">8</span>,<span class="hljs-number">10</span>))

   1 2 3 4 5 6 7 8 9 10

   %<span class="hljs-selector-tag">matplotlib</span> <span class="hljs-selector-tag">inline</span>   <span class="hljs-selector-tag">import</span> <span class="hljs-selector-tag">matplotlib</span><span class="hljs-selector-class">.pyplot</span> <span class="hljs-selector-tag">as</span> <span class="hljs-selector-tag">plt</span> <span class="hljs-selector-tag">df</span><span class="hljs-selector-class">.boxplot</span>(column=<span class="hljs-string">'Fare'</span>, by = <span class="hljs-string">'Pclass'</span>) <span class="hljs-selector-tag">plt</span><span class="hljs-selector-class">.hist</span>(df[<span class="hljs-string">'Fare'</span>], bins = <span class="hljs-number">10</span>, range =(df[<span class="hljs-string">'Fare'</span>].min(),df[<span class="hljs-string">'Fare'</span>].max())) <span class="hljs-selector-tag">plt</span><span class="hljs-selector-class">.title</span>(<span class="hljs-string">'Fare >distribution'</span>) <span class="hljs-selector-tag">plt</span><span class="hljs-selector-class">.xlabel</span>(<span class="hljs-string">'Fare'</span>) <span class="hljs-selector-tag">plt</span><span class="hljs-selector-class">.ylabel</span>(<span class="hljs-string">'Count of Passengers'</span>) #如果变量是<span class="hljs-selector-tag">categorical</span>的，想看<span class="hljs-selector-tag">distribution</span>，则可以： <span class="hljs-selector-tag">df</span><span class="hljs-selector-class">.PdDistrict</span><span class="hljs-selector-class">.value_counts</span>()<span class="hljs-selector-class">.plot</span>(kind=<span class="hljs-string">'bar'</span>, figsize=(<span class="hljs-number">8</span>,<span class="hljs-number">10</span>))

4. 如果想看几个feature之间的联立情况，则可以用pandas的groupby,
   temp = pd.crosstab([df.Pclass, df.Sex], df.Survived.astype(bool))
   temp.plot(kind=’bar’, stacked=True, color=[‘red’,’blue’], grid=False)

在这步完成之后，要对以下几点有大致了解

• 理解每个特征的意义
• 要知道哪些特征是有用的，这些特征哪些是直接可以用的，哪些需要经过变换才能用，为之后的特征工程做准备

Step2: Data Preprocessing

数据预处理，就是将数据处理下，为模型输入做准备，其中包括：

• 处理missing value：这里学问有点深，如果各位有好的经验可以跟我交流下。以我浅薄的经验来说我一般会分情况处理
  1. 如果missing value占总体的比例非常小，那么直接填入平均值或者众数
  2. 如果missing value所占比例不算小也不算大，那么可以考虑它跟其他特征的关系，如果关系明显，那么直接根据其他特征填入；也可以建立简单的模型，比如线性回归，随机森林等。
  3. 如果missing value所占比例大，那么直接将miss value当做一种特殊的情况，另取一个值填入
• 处理Outlier：这个就是之前EDA的作用了，通过画图，找出异常值
• 处理categorical feature：一般就是通过dummy variable的方式解决，也叫one hot encode，可以通过pandas.get_dummies()或者 sklearn中preprocessing.OneHotEncoder(), 我个人倾向于用pandas的get_dummies()
  看个例子吧，
  
  dummy variable

  将一列的month数据展开为了12列，用0、1代表类别。
  另外在处理categorical feature有两点值得注意：

  1. 如果特征中包含大量需要做dummy variable处理的，那么很可能导致得到一个稀疏的dataframe，这时候最好用下PCA做降维处理。
  2. 如果某个特征有好几万个取值，那么用dummy variable就并不现实了，这时候可以用Count-Based Learning.

Here are some best article for stock data analysis using python.
An Introduction to Stock Market Data Analysis with Python (Part 1)

from: https://ntguardian.wordpress.com/2016/09/19/introduction-stock-market-data-python-1/

This post is the first in a two-part series on stock data analysis using Python, based on a lecture I gave on the subject for MATH 3900 (Data Science) at the University of Utah. In these posts, I will discuss basics such as obtaining the data from Yahoo! Finance using pandas, visualizing stock data, moving averages, developing a moving-average crossover strategy, backtesting, and benchmarking. The final post will include practice problems.