Ipython notebook is a powerful tool to learn python programming. In this post, I demonstrate how to setup a ipython notebook to to spark program in python.

1. Install spark
   suppose spark is install at directory ~/spark, then execute:
   export SPARK_HOME="~/spark"

   1	export SPARK_HOME="~/spark"

2. Install anaconda at ~/anaconda
   cd ~/anaconda zip -r anaconda.zip .

   1 2 3

   cd ~/anaconda zip -r anaconda.zip .

   This will compress all the anaconda files to a zip file

   Run ipython notebook for pyspark using local model

3. Now you can start a ipython notebook server in local model:
   IPYTHON_OPTS=“notebook –notebook-dir=${WORKSPACE_DIR} –ip=* \ –config=${CONFIG_FILE} \ –port=${PORT}” pyspark –master local[2] \ –driver-memory 3G \ –jars /..../hcatalog-support.jar \ –conf spark.authenticate.secret=password \

   1 2 3 4 5 6 7

   IPYTHON_OPTS=“notebook –notebook-dir=${WORKSPACE_DIR} –ip=* \                –config=${CONFIG_FILE} \               –port=${PORT}” pyspark –master local[2] \    –driver-memory 3G \    –jars /..../hcatalog-support.jar \    –conf spark.authenticate.secret=password \

   WORKSPACE_DIR is the space where you want to save your codes.
   CONFIG_FILE is the location of the jupyter_notebook_config file.