Shell | Learn for Master

shell script replace variable of linux path

March 25, 2017 Author: david
1 Comment

sed is a popular tool to be used to replace a string in a file.

For example, give a file, we can use the following command to replace all the string XXX into YY

sed -i s/XXX/YY/g fileName

or sed s/XXX/YY/g fileName > newFile

but when you want to replace linux path, the above method doesn’t work. You can try it your self.

here is a workable solution:

pathA=”/user/xx/zz/”

pathB=”/user/aa/zz/dd”

sed “s,$pathA,$pathB,g” fileName > newFileName

then all the path of /user/xx/zz/ will be replaced by /user/aa/zz/dd.
[Read More...]
Most popular hadoop commands

August 4, 2016 Author: david
No comment yet

Here is a list of the most popular hdfs or hadoop commands to manage your hdfs files.

List the files in a hdfs directory

hadoop fs -ls hdfs_path
hdfs dfs -ls hdfs_path

Hadoop Create Directory

hadoop fs -mkdir hdfs://user/new_foder/

Hadoop Create a directory tree

hadoop fs -mkdir -p hdfs://user/new_foder/new_subfolder/

Hadoop copy hdfs files

hadoop fs -cp source_hdfs_path target_hdfs_path

cp command usage detail:

hadoop fs -cp [-f] [-p | -p[topax]] URI [URI …] <dest>

This command allows multiple sources as well in which case the destination must be a directory.
[Read More...]
untar (decompress) a tgz or tbz file in linux

July 15, 2016 Author: david
No comment yet

We already know how to tar all the files under a directory into a tgz file. This posts use simple examples to show how to untar a tgz or tar.bz2 file .

untar a .tar file

tar -xvf file_name.tar

1

tar -xvf file_name.tar

x: tells tar to extract the files.

v: list all of the files one by one in the archive. The “v” stands for “verbose.”

f: tells tar that a file name is followed.

untar a .tgz or tar.gz file

If the file extension is .tgz or tar.gz,
[Read More...]
Test whether a file or directory exist in shell

July 15, 2016 Author: david
No comment yet

How to test whether a file exist in shell on Linux? We can use the test command to check file types and whether it exits.

How to test whether a path is a regular file in shell on Linux

We can use the following command to test whether a path is a regular file and exist.

test -f some_file if [[ $? == 0 ]] ; then echo "some_file is a file and exist" fi

1
2
3
4

test -f some_file
if [[ $? == 0 ]] ; then
echo "some_file is a file and exist"
fi

How to test whether a path is a folder in shell

Using the following command, we can test whether a path is a folder in shell on linux.

test -d some_folder if [[ $? == 0 ]] ; then echo "some_folder is a folder and exist" fi

1
2
3
4

test -d some_folder
if [[ $? == 0 ]] ; then
echo "some_folder is a folder and exist"
fi

How to test whether a file is a symbolic link on linux?
[Read More...]
tar all the files under a directory on linux

July 15, 2016 Author: david
No comment yet

We often need to tar all the files under a directory into a zip file. This post gives two examples to show how to tar all the contents under a directory into a tgz file.

Suppose we have directory structures like

my_folder --- file1 --- file2 --- file3

1
2
3
4

my_folder
   --- file1
   --- file2
   --- file3

By running the following command, we can tar all the files of my_folder into the tgz file.

tar -czvf my_folder.tgz my_folder

1

tar -czvf my_folder.tgz my_folder

However, you will find that you actually include the my_folder in the zip file. The structure of the tgz file will looks like this.

my_folder --- file1 --- file2 --- file3

1
2
3
4
5

my_folder
   --- file1
   --- file2
   --- file3

What if you want the file structures like this:

file1 file2 </code>file3

1
2
3

file1
file2
</code>file3

Two methods tar all the files under a directory into a tgz file

You can use the following command to include only the files under my_folder to the zip file:

cd my_folder tar -zcvf ../my_folder.tgz . cd .. or cd my_folder/ && tar -zcvf ../my_folder.tgz . && cd ..

1
2
3
4
5
6
7
8

cd my_folder
tar -zcvf ../my_folder.tgz .
cd ..

or

cd my_folder/ && tar -zcvf ../my_folder.tgz . && cd ..

Using the -C option to tar all the files under a directory into a tgz file

You can also using the -C option to include all the files under your directory into a zip file:

tar -czvf my_folder.tar.gz -C my_folder .

1
2

tar -czvf my_folder.tar.gz -C my_folder .

The -C my_folder tells tar to change the current directory to my_folder,
[Read More...]
check the size of directory or file on linux

July 15, 2016 Author: david
No comment yet

We often need to check the size of a file or subdirectories on linux. This post shows how to use du command to check the size of a file or directory on linux system. We also show how to sort folders or files by size.

The two most useful options of du commands are: -s and -h,The meaning of the options are:

-s, --summarize display only a total for each argument -h, --human-readable print sizes in human readable format (e.g., 1K 234M 2G)

1
2
3
4

   -s, --summarize
          display only a total for each argument
   -h, --human-readable
          print sizes in human readable format (e.g., 1K 234M 2G)

Get the human readable size of a file

We can simply use du -h command to get human readable size of a file:

du -h <file_name>

Get the size of a folder

To get the size of a folder only,
[Read More...]
Get shell output when calling shell from Python

July 15, 2016 Author: david
No comment yet

We have shown how to call shell in python using the subprocess communicate method. However, this method has some problems as the output is buffered into memory. We need to print out the shell output if the size is too large. Sometimes, when the shell scripts run for a long time, we may need to examine the output in real time to check the status of the problem. In order to get the real time output from shell from python, we can using the following method:

Suppose we have a shell script, long_shell.sh, like this:

#!/usr/bin/env bash for i in $(seq 1 5) do echo $i echo start sleep 1s sleep 1 done

1
2
3
4
5
6
7
8
9

#!/usr/bin/env bash

for i in $(seq 1 5)
do
echo $i
echo start sleep 1s
sleep 1

done

Then we have the following python program to call the shell command and print the real output in screen.
[Read More...]
Shell for loop: generate a sequence of numbers in Shell

July 15, 2016 Author: david
No comment yet

For loop is one of the most frequently used command in shell. In this post, we show how to generate a sequence of numbers in shell, and use for loop to print out the numbers.

How to generate a range of number in Shell or Bash.

The answer is using the seq command, here is the seq command syntax:

seq LAST
seq FIRST LAST
seq FIRST INCREMENT LAST

Generate a sequence from 1 to 10

To generate a sequence of numbers from 1 to 10, we can use both seq 10 or seq 1 10.
[Read More...]

Run external shell command in Python

July 15, 2016 Author: david

There is often a need to call shell command from python directly. In this post, I use examples to show how to run external shell commands in python.

The recommended way to call shell command from python is using the subprocess library. See the following example:

import subprocess
def run_cmd(args_list):
    print('Running system command: {0}'.format(' '.join(args_list)))
    proc = subprocess.Popen(args_list, stdout=subprocess.PIPE,
            stderr=subprocess.PIPE)
    (output, errors) = proc.communicate()
    if proc.returncode:
        raise RuntimeError(
            'Error running command: %s. Return code: %d, Error: %s' % (
                ' '.join(args_list), proc.returncode, errors))
    return (output, errors)


output, errors = run_cmd(['ls', '-al'])
print "run ls -al in python"
print output  # this will print out all the files under current folder

output, errors = run_cmd(['sh', 'test.sh'])
print "run sh test.sh command in python"
print output  # this will execute the test.sh and print the ouput


output, errors = run_cmd(['pwd'])
print "run pwd command in python"
print output  # this will print the current directory name

import subprocess

def run_cmd(args_list):

print('Running system command: {0}'.format(' '.join(args_list)))

proc = subprocess.Popen(args_list, stdout=subprocess.PIPE,

stderr=subprocess.PIPE)

(output, errors) = proc.communicate()

if proc.returncode:

raise RuntimeError(

'Error running command: %s. Return code: %d, Error: %s' % (

' '.join(args_list), proc.returncode, errors))

return (output, errors)

output, errors = run_cmd(['ls', '-al'])

print "run ls -al in python"

print output # this will print out all the files under current folder

output, errors = run_cmd(['sh', 'test.sh'])

print "run sh test.sh command in python"

print output # this will execute the test.sh and print the ouput

output, errors = run_cmd(['pwd'])

print "run pwd command in python"

print output # this will print the current directory name

In this post, I use example to show how to run shell command in a python program. However, this method has some problems as the output is buffered into memory.

We need to print out the shell output on the fly if the size is too large.

[Read More...]

Start, Restart and Stop Apache web server on Linux

July 14, 2016 Author: david
No comment yet

We often need to start, stop or restart apache web server on Linux system such as Debian and Ubuntu.

This post uses examples to show how to Start, Restart, or Stop apache web server by using service, systemctl or apache2ctl commands.

service command

This method works in most Linux distributions including Debian and Ubuntu.

To start Apache 2, run:
$ sudo service apache2 start

To restart Apache 2, run:
$ sudo service apache2 restart

To stop Apache 2, run:
$ sudo service apache2 stop

To gracefully reload Apache 2,
[Read More...]