.
  • install spark kernel and pyspark kernel using toree

    Jupyter Notebook (formerly known as IPython Notebook) is an interactive notebook environment which supports various programming languages which allows you to interact with your data, combine code with markdown text and perform simple visualizations.

    Here are just a couple of reasons why using Jupyter Notebook with Spark is the best choice for users that wish to present their work to other team members or to the public in general:

    • Jupyter notebooks support tab autocompletion on class names, functions, methods and variables.
    • It offers more explicit and colour-highlighted error messages than the command line IPython console.
    [Read More...]
  • How to setup ipython notebook server to run spark in local or yarn model

    Ipython notebook is a powerful tool to learn python programming. In this post, I demonstrate how to setup a ipython notebook to to spark program in python.

    1. Install spark
      suppose spark is install at directory ~/spark, then execute:
    2. Install anaconda at ~/anaconda

      This will compress all the anaconda files to a zip file

      Run ipython notebook for pyspark using local model

    3. Now you can start a ipython notebook server in local model: WORKSPACE_DIR is the space where you want to save your codes.
      CONFIG_FILE is the location of the jupyter_notebook_config file.
    [Read More...]
  • How to Access Ipython Notebook Running on Remote Server

    When you run ipython notebook on a remote server using the following command

    jupyter notebook –no-browser –port=8889

    You will see:

    The Jupyter Notebook is running at: http://localhost:8889/

    In this case, you cann’t use the link:

    public_IP_address:8889 or domain_name:8889

    to get access to the Ipython Notebook Server.

    You can do some config to allow the Ipython notebook server to run on the public IP address, but this will cause security problems.

    A good way is to use ssh tunnel, so that We can get access to the remote server on a local machine.

    [Read More...]
  • Permission denied when running jupyter notebook after switch user

    This error happens when you use a different user from the login user to run jupyter notebook.

    For example:

    If you login in using user root, then switch to user_x, then run:

    ipython notebook or jupyter notebook

    you will get the following exception:

    OSError: [Errno 13] Permission denied: ‘/run/user/1000/jupyter’

    The solution is:

    export XDG_RUNTIME_DIR=””

    [Read More...]