To develop a Spark program, we often need to read arguments from the command line. Scopt is a popular and easy-to-use argument parser. In this post, I provide a workable example to show how to use the scopt parser to read arguments for a spark program in scala. Then I describe how to run the spark job in yarn-cluster mode.

The main contents of this post include:

1. Use scopt option parser to parse arguments for a scala program.
2. Use sbt to package the scala program
3. Run spark on yarn-cluster mode with third party libraries

Use Scopt to parse arguments in a scala program

In the following program,