Ipython notebook is a powerful tool to learn python programming. In this post, I demonstrate how to setup a ipython notebook to to spark program in python.

1. Install spark
   suppose spark is install at directory ~/spark, then execute:
   export SPARK_HOME="~/spark"

   1	export SPARK_HOME="~/spark"

2. Install anaconda at ~/anaconda
   cd ~/anaconda zip -r anaconda.zip .

   1 2 3

   cd ~/anaconda zip -r anaconda.zip .

   This will compress all the anaconda files to a zip file

   Run ipython notebook for pyspark using local model

3. Now you can start a ipython notebook server in local model:
   IPYTHON_OPTS=“notebook –notebook-dir=${WORKSPACE_DIR} –ip=* \ –config=${CONFIG_FILE} \ –port=${PORT}” pyspark –master local[2] \ –driver-memory 3G \ –jars /..../hcatalog-support.jar \ –conf spark.authenticate.secret=password \

   1 2 3 4 5 6 7

   IPYTHON_OPTS=“notebook –notebook-dir=${WORKSPACE_DIR} –ip=* \                –config=${CONFIG_FILE} \               –port=${PORT}” pyspark –master local[2] \    –driver-memory 3G \    –jars /..../hcatalog-support.jar \    –conf spark.authenticate.secret=password \

   WORKSPACE_DIR is the space where you want to save your codes.
   CONFIG_FILE is the location of the jupyter_notebook_config file.

When you run ipython notebook on a remote server using the following command

jupyter notebook –no-browser –port=8889

You will see:

The Jupyter Notebook is running at: http://localhost:8889/

In this case, you cann’t use the link:

public_IP_address:8889 or domain_name:8889

to get access to the Ipython Notebook Server.

You can do some config to allow the Ipython notebook server to run on the public IP address, but this will cause security problems.

A good way is to use ssh tunnel, so that We can get access to the remote server on a local machine.

This error happens when you use a different user from the login user to run jupyter notebook.

For example:

If you login in using user root, then switch to user_x, then run:

ipython notebook or jupyter notebook

you will get the following exception:

OSError: [Errno 13] Permission denied: ‘/run/user/1000/jupyter’

The solution is:

export XDG_RUNTIME_DIR=””