Scala | Learn for Master

Scala read file examples

August 1, 2016 Author: david

In this post, I show some of the best practices to read file in Scala. We can use the mkString method to read all the contents of a file into a variable. We can also use the getLines methods to iterator through the contents of a file.

Read all the data into a String

val source = scala.io.Source.fromFile("file.txt")
val lines = try source.mkString finally source.close()

// for file with utf-8 encoding
val lines = scala.io.Source.fromFile("file.txt", "utf-8").getLines.mkString

val source = scala.io.Source.fromFile("file.txt")

val lines = try source.mkString finally source.close()

// for file with utf-8 encoding

val lines = scala.io.Source.fromFile("file.txt", "utf-8").getLines.mkString

Scala loop a file

Sometimes we don’t want to load all the contents of a file into the memory, especially if the file is too large. What we want is to loop the file, and process one line each time. See the following example,

[Read More...]

Scala for loop

July 30, 2016 Author: david

No comment yet

In this post, we list the common ways of for loop in Scala programming.

Scala for loop with ranges

The simplest syntax of for scala loop with ranges is as follows:

for( var x <- i to j ){
   process(x);
}

for( var x <- i to j ){

process(x);

}

i to j can also be replaced with other generators such as i until j. See the following example:

object ScalaForLoop {
   def main(args: Array[String]) {
      var x = 0;
      // for loop execution with a range
      for( x <- 1 to 5){
         println( "Value of x: " + x );
      }
   }
}

object ScalaForLoop {

def main(args: Array[String]) {

var x = 0;

// for loop execution with a range

for( x <- 1 to 5){

println( "Value of x: " + x );

}

The output is:

value of a: 1
value of a: 2
value of a: 3
value of a: 4
value of a: 5

Scala for loop with multiple ranges

We can use multiple ranges separated by semicolon (;) within scala for loop.

[Read More...]

A Spark program using Scopt to Parse Arguments

July 29, 2016 Author: david
No comment yet
To develop a Spark program, we often need to read arguments from the command line. Scopt is a popular and easy-to-use argument parser. In this post, I provide a workable example to show how to use the scopt parser to read arguments for a spark program in scala. Then I describe how to run the spark job in yarn-cluster mode.

The main contents of this post include:
1. Use scopt option parser to parse arguments for a scala program.
2. Use sbt to package the scala program
3. Run spark on yarn-cluster mode with third party libraries
Use Scopt to parse arguments in a scala program

In the following program,
[Read More...]
Parse libsvm data for spark MLlib

July 12, 2016 Author: david
No comment yet
LibSVM data format is widely used in Machine Learning. Spark MLlib is a powerful tool to train large scale machine learning models. If your data is well formatted in LibSVM, it is straightforward to use the loadLibSVMFile method to transfer your data into an Rdd.
```
val data = MLUtils.loadLibSVMFile(sc, "data/mllib/sample_libsvm_data.txt")
```
However, in certain cases, your data is not well formatted in LibSVM. For example, you may have different models, and each model has its own labeled data. Suppose your data is stored into HDFS, and each line looks like this: (model_key, training_instance_in_livsvm_format).

In this case,
[Read More...]

Scala vs Java examples

July 2, 2016 Author: david

No comment yet

This tutorial gives a quick introduction to the Scala language by comparing Scala with Java using examples. It is for people who have learned Java and want to learn Scala.

Scala vs Java: The Hello Word Example
A hello word example for Scala:

object HelloWorld {
  def main(args: Array[String]) {
    println("Hello, world!")
  }
}

object HelloWorld {

def main(args: Array[String]) {

println("Hello, world!")

}

A hello world example for Java:

public class HelloWorld {

    public static void main(String[] args) {
        // Prints "Hello, World" to the terminal window.
        System.out.println("Hello, World");
    }
}

public class HelloWorld {

public static void main(String[] args) {

// Prints "Hello, World" to the terminal window.

System.out.println("Hello, World");

}

object Keyword in Scala

In scala, the object keyword is used to create a singleton object. The declaration above declares both a class called HelloWorld and an instance of that class, also called HelloWorld.

One difference between Scala and Java is that,

[Read More...]

How to package a Scala project to a Jar file with SBT

June 29, 2016 Author: david
No comment yet
When you develop a Spark project using Scala language, you have to package your project into a jar file. This tutorial describes how to use SBT
to compile and run a Scala project, and package the project as a Jar file. This will be helpful for you to create a spark project and package it to a jar file.

The directory structure of a typical SBT project

Here is an example to show a typical SBT project, which has the following directory structures.
```
.
|-- build.sbt
|-- lib
|-- project
|-- src
|   |-- main
|   |   |-- java (store main java files)
|   |   |-- resources (store include in main jar)
|   |   |-- scala (store main Scala source files)
|   |-- test
|       |-- java (store test java files)
|       |-- resources (store files include in test jar)
|       |-- scala (store test scala source files)
|-- target
```
You can use the following command to create this directory structures:
```
#!/bin/sh
cd ~/hello_world
mkdir -p src/{main,test}/{java,resources,scala}
mkdir lib project target

# create an initial build.sbt file
echo 'name := "MyProject"
version := "1.0"
scalaVersion := "2.10.0"' >
```
[Read More...]