How to setup ipython notebook server to run spark in local or yarn model

Tags: , ,

Ipython notebook is a powerful tool to learn python programming. In this post, I demonstrate how to setup a ipython notebook to to spark program in python.

  1. Install spark
    suppose spark is install at directory ~/spark, then execute:
  2. Install anaconda at ~/anaconda

    This will compress all the anaconda files to a zip file

    Run ipython notebook for pyspark using local model

  3. Now you can start a ipython notebook server in local model: WORKSPACE_DIR is the space where you want to save your codes.
    CONFIG_FILE is the location of the jupyter_notebook_config file.

     


    Run ipython notebook for pyspark using yarn model

    You can use  the following code to start ipython notebook from yarn model. 

    For the archives, you need to upload the anaconda.zip file to hdfs folder, then specify its location here.