How to setup ipython notebook server to run spark in local or yarn model

Ipython notebook is a powerful tool to learn python programming. In this post, I demonstrate how to setup a ipython notebook to to spark program in python.

1. Install spark
   suppose spark is install at directory ~/spark, then execute:
   export SPARK_HOME="~/spark"

   1	export SPARK_HOME="~/spark"

2. Install anaconda at ~/anaconda
   cd ~/anaconda zip -r anaconda.zip .

   1 2 3

   cd ~/anaconda zip -r anaconda.zip .

   This will compress all the anaconda files to a zip file

   Run ipython notebook for pyspark using local model

3. Now you can start a ipython notebook server in local model:
   IPYTHON_OPTS=“notebook –notebook-dir=${WORKSPACE_DIR} –ip=* \ –config=${CONFIG_FILE} \ –port=${PORT}” pyspark –master local[2] \ –driver-memory 3G \ –jars /..../hcatalog-support.jar \ –conf spark.authenticate.secret=password \

   1 2 3 4 5 6 7

   IPYTHON_OPTS=“notebook –notebook-dir=${WORKSPACE_DIR} –ip=* \                –config=${CONFIG_FILE} \               –port=${PORT}” pyspark –master local[2] \    –driver-memory 3G \    –jars /..../hcatalog-support.jar \    –conf spark.authenticate.secret=password \

   WORKSPACE_DIR is the space where you want to save your codes.
   CONFIG_FILE is the location of the jupyter_notebook_config file.
   

    

   
   Run ipython notebook for pyspark using yarn model
   

   You can use  the following code to start ipython notebook from yarn model. 

   IPYTHON_OPTS=“notebook –notebook-dir=${WORKSPACE_DIR} –ip=* \ –config=${CONFIG_FILE} \ –port=${PORT}” pyspark \ –verbose \ –master yarn \ –deploy-mode client \ –queue default \ –num-executors 3 \ –driver-memory 3g \ –executor-memory 3g \ –executor-cores 2 \ –jars /xxxxx/hcatalog-support.jar \ –archives hdfs://xxxxxx:8020/user/xxxx/anaconda.zip#anaconda_remote

   1 2 3 4 5 6 7 8 9 10 11 12 13

   IPYTHON_OPTS=“notebook –notebook-dir=${WORKSPACE_DIR} –ip=* \              –config=${CONFIG_FILE} \             –port=${PORT}” pyspark \ –verbose \ –master yarn \ –deploy-mode client \ –queue default \ –num-executors 3 \ –driver-memory 3g \ –executor-memory 3g \ –executor-cores 2 \ –jars /xxxxx/hcatalog-support.jar \ –archives hdfs://xxxxxx:8020/user/xxxx/anaconda.zip#anaconda_remote

   For the archives, you need to upload the anaconda.zip file to hdfs folder, then specify its location here.