To develop a Spark program, we often need to read arguments from the command line. Scopt is a popular and easy-to-use argument parser. In this post, I provide a workable example to show how to use the scopt parser to read arguments for a spark program in scala. Then I describe how to run the spark job in yarn-cluster mode.
The main contents of this post include:
- Use scopt option parser to parse arguments for a scala program.
- Use sbt to package the scala program
- Run spark on yarn-cluster mode with third party libraries
Use Scopt to parse arguments in a scala program
In the following program,[Read More...]