Big Data | Learn for Master

How to load data from a text file to Hive table

June 24, 2016 Author: david
3 Comments

In this post, I describe how to insert data from a text file to a hive table.

Suppose you have tab delimited file::

% cat /tmp/input.txt k1 v1 k2 v2

1
2
3
4

% cat /tmp/input.txt

k1 v1
k2 v2

Create a Hive table stored as a text file.

create table test_table (k string, v string) </code><code>row format delimited </code><code>fields terminated by '\t' stored as textfile;

1
2
3
4

create table test_table (k string, v string)
</code><code>row format delimited
</code><code>fields terminated by '\t'
stored as textfile;

Load the text file (stored locally) into the Hive table:

hive(test_db)> load data local inpath 'file:/homes/user/hive/input.txt' into table test_table; Loading data to table test_db.test_table OK Time taken: 0.43 seconds

1
2
3
4
5
6
7

hive(test_db)> load data local inpath 'file:/homes/user/hive/input.txt' into table test_table;

Loading data to table test_db.test_table

OK

Time taken: 0.43 seconds

Create a Hive table stored as sequence file.

create table test_sq(k string, v string) stored as sequencefile;

1

create table test_sq(k string, v string) stored as sequencefile;

Now you can load into the sequence table from the text table:

insert into table test_sq select * from test_table;

1

insert into table test_sq select * from test_table;

[Read More...]

Apache Hive Usage Example – Create Hive Table

June 24, 2016 Author: david

In this post, I describe how to create a Hive Table.

Create Table Statement

The syntax and example to create a Hive table is as follows:

Syntax

CREATE [TEMPORARY] [EXTERNAL] TABLE [IF NOT EXISTS] [db_name.] table_name

[(col_name data_type [COMMENT col_comment], ...)]
[COMMENT table_comment]
[ROW FORMAT row_format]
[STORED AS file_format]

CREATE [TEMPORARY] [EXTERNAL] TABLE [IF NOT EXISTS] [db_name.] table_name

[(col_name data_type [COMMENT col_comment], ...)]

[COMMENT table_comment]

[ROW FORMAT row_format]

[STORED AS file_format]

For example, suppose you want to create a Table student, with the following attributes:

id, the student’s ID number, the type is int,
name, the type is String,
age, the type is int,
score, the type is Float,

hive (user_db)> CREATE TABLE IF NOT EXISTS students_db.student ( id int,
 > name String,
 > age int, score Float)
 > COMMENT 'Description of the table'
 > ROW FORMAT DELIMITED
 > FIELDS TERMINATED BY '\t'
 > LINES TERMINATED BY '\n'
 > STORED AS TEXTFILE;

OK
Time taken: 2.357 seconds

hive (user_db)> describe students_db.student;
OK
id int
name string
age int
score float
Time taken: 0.04 seconds, Fetched: 4 row(s)

hive (user_db)> CREATE TABLE IF NOT EXISTS students_db.student ( id int,

> name String,

> age int, score Float)

> COMMENT 'Description of the table'

> ROW FORMAT DELIMITED

> FIELDS TERMINATED BY '\t'

> LINES TERMINATED BY '\n'

> STORED AS TEXTFILE;

Time taken: 2.357 seconds

hive (user_db)> describe students_db.student;

id int

name string

age int

score float

Time taken: 0.04 seconds, Fetched: 4 row(s)

Note:

If you’re not currently working in the target database,

[Read More...]

Apache Hive Usage Example – How to Check the Current Hive Database

June 24, 2016 Author: david
No comment yet

To know which Hive database you are currently using, Use the following command:

set hive.cli.print.current.db=true

1

set hive.cli.print.current.db=true

then the prompt will display Hive(DB_name)

Another Method is to edit hive-site.xml by pasting this code:

<property> <name>hive.cli.print.current.db</name> <value>true</value> </property>

1
2
3
4

<property>
<name>hive.cli.print.current.db</name>
<value>true</value>
</property>

In second scenario, you can automatically display the Hive dabatabase name when you open terminal.
[Read More...]
Apache Hive Usage Example – Create and Use Database

June 24, 2016 Author: david
No comment yet
I this post, I describe how to Create a Hive Database, Create Database using JDBC, Describe and Show hive Database.

Create Database Statement

A database in Hive is a namespace or a collection of tables. In order to create a database in Hive, we need to use the Create Database statement. The syntax for this statement is as follows:
```
CREATE DATABASE|SCHEMA [IF NOT EXISTS] <database name>
```
Here, IF NOT EXISTS is an optional clause, which notifies the user that a database with the same name already exists. We can use SCHEMA in place of DATABASE in this command.
[Read More...]
What is MapReduce and how it works

June 22, 2016 Author: david
1 Comment

We usually develop programs based on open sourced MapReduce frameworks such as Hadoop, Apache Pig, Apache Hive, and Spark to solve Big Data problems. In this post, I will use an example to describe what MapReduce is and how it works. I hope this will help you learn those Big Data technologies such as Hadoop, Pig, Hive and Spark easier.

What is MapReduce?

MapReduce is the key of Big Data. It was invented by Google, and it is the heart of Hadoop.

It is a programming paradigm that allows engineers or scientist build scalable systems that can run on hundreds or thousands of servers.
[Read More...]

How to setup ipython notebook server to run spark in local or yarn model

June 20, 2016 Author: david

No comment yet

Ipython notebook is a powerful tool to learn python programming. In this post, I demonstrate how to setup a ipython notebook to to spark program in python.

Install spark
suppose spark is install at directory ~/spark, then execute:

export SPARK_HOME="~/spark"

1

export SPARK_HOME="~/spark"
Install anaconda at ~/anaconda

cd ~/anaconda zip -r anaconda.zip .

1
2
3

cd ~/anaconda
zip -r anaconda.zip .

This will compress all the anaconda files to a zip file

Run ipython notebook for pyspark using local model

Now you can start a ipython notebook server in local model:

IPYTHON_OPTS=“notebook –notebook-dir=${WORKSPACE_DIR} –ip=* \
               –config=${CONFIG_FILE} \
              –port=${PORT}” pyspark –master local[2] \
   –driver-memory 3G \
   –jars /..../hcatalog-support.jar \
   –conf spark.authenticate.secret=password \

IPYTHON_OPTS=“notebook –notebook-dir=${WORKSPACE_DIR} –ip=* \

–config=${CONFIG_FILE} \

–port=${PORT}” pyspark –master local[2] \

–driver-memory 3G \

–jars /..../hcatalog-support.jar \

–conf spark.authenticate.secret=password \

WORKSPACE_DIR is the space where you want to save your codes.
CONFIG_FILE is the location of the jupyter_notebook_config file.

[Read More...]

Spark MLlib Example

June 20, 2016 Author: david
No comment yet

In this post, I will use an example to describe how to use pyspark, and show how to train a Support Vector Machine, and use the model to make predications using Spark MLlib.

The following Program is developed using Ipython Notebook. Please refer to this article for how to set up in Ipython Notebook Server for PySpark, if you want to set up an ipython notebook server. You can also run the program use other python IDEs such Spyder or Pycharm.

A simple example to demonstrate how to use sc,
[Read More...]

Page 5 of 5«1 2 3 45

How to load data from a text file to Hive table

Apache Hive Usage Example – Create Hive Table

Apache Hive Usage Example – How to Check the Current Hive Database

Apache Hive Usage Example – Create and Use Database

What is MapReduce and how it works

How to setup ipython notebook server to run spark in local or yarn model

Spark MLlib Example

Get Two Free Audiobooks

Buy a phone

Best books to master algorithms

Best books to master machine learning

Search

Category List

Featured Categories

Rss Feeds

Categories

Subscribe

Contact US

Popular Posts

learn4master.com 2016 Accelerated by keyCDN

Get Two Free Audiobooks

Buy a phone

Best books to master algorithms

Best books to master machine learning

Search

Category List

Featured Categories

Tags

Categories

Subscribe

Categories

Contact US

Tags

Popular Posts

learn4master.com 2016 Accelerated by keyCDN