In this post, I first give a workable example to run pySpark on oozie. Then I show how to run pyspark on oozie using your own python installation (e.g., anaconda). In this way, you can use numpy, pandas, other python libraries in your pyspark program. 

The syntax of creating a spark action on oozie workflow

As described in the document, here are the meanings of these elements.

The prepare element, if present, indicates a list of paths to delete or create before starting the job. Specified paths must start with hdfs://HOST:PORT .

Big Data is a really hot topic! There are millions of job openings. This post aims to introduce some of the most important techniques related to Big Data.  If you want to learn Big Data and find a job related to Big Data, please read the articles and books recommended here. 

The first key concept of Big Data is MapReduce, which is the core of Hadoop, a open sourced framework that becomes the foundation of the Big Data eco system. 

• MapReduce was invented by Google. It’s a paradigm for writing distributed systems inspired by some elements of functional programming.

When we use Hive, we can append data to the table easily, but when we use Pig (i.e., the HCatalog ) to insert data into Hive table, we are not allowed to append data to a partition if that partition already contains data. 

In this post, I describe a method that can help you append data to the existing partition using a dummy partition named run. It means  the run number you append some data to this partition. 

For example, we create the following partitioned hive table:

Then pig script looks like the following: 

Now we can run the pig script using the following command:

Then we have the following content in the table:

Each time when you want to append data to the partition DATE=20160605,

When we run hive scripts, such as Load data into Hive table, we often need to pass parameters to the hive scripts by defining our own variables. 

Here are some examples to show how to pass parameters or user defined variables to hive. 

Use hiveconf for variable subsititution

For example, you can define a variable DATE, then use it as ${hiveconf:DATE}

you can even pass the variable from command line:

Use env and system variables

You can also use env and system variables like this  ${env:USER}

You can run the following command to see all the available variables:

If you are o the hive prompt,

Partition is a very useful feature of Hive. Without partition, it is hard to reuse the Hive Table if you use HCatalog to store data to Hive table using Apache Pig, as you will get exceptions when you insert data to a non-partitioned Hive Table that is not empty. 

 In this post, I use an example to show how to create a partitioned table, and populate data into it. 

Let’s suppose you have a dataset for user impressions. For instance, a sample of the data set might be like this:

id
user_id
user_lang
user_device
time_stamp
url
date
country

1
u1
en
iphone
201503210011
http://xxx/xxx/1
20150321
US

2
u1
en
ipad
201503220111
http://xxx/xxx/2
20150322 
US

3
u2
en
desktop
201503210051
http://xxx/xxx/3
 20150321
CA

4
u3
en
iphone
201503230021
http://xxx/xxx/4
 20150323
HK

If you use Pig to analyze the data,

It’s straight forward to delete data from a traditional Relational table using SQL. However, delete rows from Hive Rows can cause several exceptions.

For examples, let see we have a imps_part table,  we want to delete the values in the Table.  You will get the exception:

When you run the simple delete command, we get: FAILED: SemanticException [Error 10294]: Attempt to do update or delete using transaction manager that does not support these operations

Some one suggest to use the following command:

This will result in the following exception:
FAILED: SemanticException 1:23 Need to specify partition columns because the destination table is partitioned.

We have described how to load data from Hive Table using Apache Pig, in this post, I will use an example to show how to save data to Hive table using Pig.

Before save data to Hive, you need to first create a Hive Table. Please refer to this post on how to create a Hive table. 

Suppose we use Apache Pig to Load some data from a text file, then we can save the data to the hive table using the following script. 

The store_student.pig script is like this:

Note: You must specify the table name in single quotes: STORE data into ‘tablename’.

When you have a hive table, you may want to check its delimiter or detailed information such as Schema. There are two solutions:

Get the delimiter of a Hive Table

To get the field delimiter of a hive table, we can use the following command:

Here is an example:

Get the schema of Hive Table

Another solution is to use: 

This will generate a competed information about the table. 

In this post, I describe how to insert data from a text file to a hive table. 

Suppose you have tab delimited file::

Create a Hive table stored as a text file.

Load the text file (stored locally) into the Hive table:

Create a Hive table stored as sequence file.

Now you can load into the sequence table from the text table: