Python | Learn for Master

Cool Python tricks and tips

August 31, 2016 Author: david
No comment yet

Here are some cool tricks to write better python code:

List comprehensions:

Instead of building a list with a loop:

Python

b = [] for x in a: b.append(10 * x) foo(b)

1
2
3
4

b = []
for x in a:
    b.append(10 * x)
foo(b)

We can often build it much more concisely with a list comprehension:

Python

b = [10 * x for x in a]

1

b = [10 * x for x in a]

Enumerate

We can use enumerate to do a for loop:

i = 0 for item in iterable: print i, item i += 1

1
2
3
4

i = 0
for item in iterable:
    print i, item
    i += 1

like this:

for i, item in enumerate(iterable): print i, item

1
2

for i, item in enumerate(iterable):
print i, item

Enumerate can also take a second argument. Here is an example:

>>> list(enumerate('abc')) [(0, 'a'), (1, 'b'), (2, 'c')] >>> list(enumerate('abc', 1)) [(1, 'a'), (2, 'b'), (3, 'c')]

1
2
3
4
5

>>> list(enumerate('abc'))
[(0, 'a'), (1, 'b'), (2, 'c')]

>>> list(enumerate('abc', 1))
[(1, 'a'), (2, 'b'), (3, 'c')]

Dict/Set comprehensions

dict/set comprehensions are simple to use and just as effective:

my_dict = {i: i * i for i in xrange(100)} my_set = {i * 15 for i in xrange(100)} # There is only a difference of ':' in both

1
2
3
4

my_dict = {i: i * i for i in xrange(100)}
my_set = {i * 15 for i in xrange(100)}

# There is only a difference of ':' in both

[Read More...]
python multi thread example

August 3, 2016 Author: david
No comment yet

In python, it is easy to start multiple threads using the Thread class in the threading module. The threading module is built on the low-level features of thread to make it easier to write multithreading program in python. If you want to run multiple operations concurrently in python, you need to master the Thread class.

Thread Objects
Create and start a Thread

We can easily make several threads run concurrently using the Thread class. The syntax to create and start a thread is as follows :

Python Thread Syntax
Python

t = threading.Thread(name= 'the thread name', target=my_function) t.start()

1
2

t = threading.Thread(name= 'the thread name', target=my_function)
t.start()

The jobs are defined in my_function,
[Read More...]
Python Queue examples

August 3, 2016 Author: david
No comment yet

In this post, I will discuss how to use the python Queue module. This module implements queues for multiple thread programming. Specifically, the python Queue object can be easily used to solve the multi-producer, multi-consumer problem, where messages must be exchanged safely between multiple threads. As the locking semantics have already been implemented in the Queue class, you don’t need to handle the low level lock, unlock operations, which can easily cause the dead lock problems.

Note

Tips: queue is one of the most widely used data structures in computer science.
[Read More...]

Count word frequency

July 30, 2016 Author: david

Count word frequency is a popular task for text analysis. In this post, I describe how to count word frequency using Java HashMap, python dictionary, and Spark.

Use Java HashMap to Count Word frequency

import java.util.Arrays;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

public class JavaHashMapExample {
    public Map<String, Integer> countFreq(List<String> list){
        Map<String, Integer> histogram = new HashMap<>();
        for(String word: list) {

            if (!histogram.containsKey(word)) {
                histogram.put(word, 1);
            } else {
                histogram.put(word, histogram.get(word) + 1);
            }
        }
        return histogram;
    }

    public static void main(String[] args) {
        String[] a = {"a", "b", "c", "d", "a", "b", "a", "d", "a", "c", "c", "d", "a", "c", "c", "c"};
        List<String> wordList = Arrays.asList(a);
        System.out.println(new JavaHashMapExample().countFreq(wordList));
     }
}

import java.util.Arrays;

import java.util.HashMap;

import java.util.List;

import java.util.Map;

public class JavaHashMapExample {

public Map<String, Integer> countFreq(List<String> list){

Map<String, Integer> histogram = new HashMap<>();

for(String word: list) {

if (!histogram.containsKey(word)) {

histogram.put(word, 1);

} else {

histogram.put(word, histogram.get(word) + 1);

}

return histogram;

}

public static void main(String[] args) {

String[] a = {"a", "b", "c", "d", "a", "b", "a", "d", "a", "c", "c", "d", "a", "c", "c", "c"};

List<String> wordList = Arrays.asList(a);

System.out.println(new JavaHashMapExample().countFreq(wordList));

}

{a=5, b=2, c=6, d=3}

Use Python Dict to count word frequency

words = ["a", "b", "c", "d", "a", "b", "a", "d", "a", "c", "c", "d", "a", "c", "c", "c"]

freq = {}

for w in words:
    if w in freq:
        freq[w] += 1
    else:
        freq[w] = 1

print freq

words = ["a", "b", "c", "d", "a", "b", "a", "d", "a", "c", "c", "d", "a", "c", "c", "c"]

freq = {}

for w in words:

if w in freq:

freq[w] += 1

else:

freq[w] = 1

print freq

The output:

{‘a’: 5, ‘c’: 6, ‘b’: 2, ‘d’: 3}

Use Spark to count word Frequency

The above method works well for small dataset. However, if you have a huge dataset, the hashTable based method will not work. You will need to develop a distributed program to accomplish this task.

[Read More...]

pyspark unit test based on python unittest library

July 27, 2016 Author: david
2 Comments

pyspark unit test

Pyspark is a powerful framework for large scale data analysis. Because of the easy-to-use API, you can easily develop pyspark programs if you are familiar with Python programming.

One problem is that it is a little hard to do unit test for pyspark. After some google search using “pyspark unit test”, I only get articles about using py.test or some other complicated libraries for pyspark unit test. However, I don’t want to install any other third party libraries . What I want is to set up the pyspark unit test environment just based on the unittest library,
[Read More...]

python UnicodeEncodeError, converting unicode to ascii

July 22, 2016 Author: david

No comment yet

In python, we often encounter the unicode convert issue. For instance, when you try to print a unicode string, you will get the following exception:

UnicodeEncodeError: 'ascii' codec can't encode character u'\xe6' in position 1535: ordinal not in range(128)

1	UnicodeEncodeError: 'ascii' codec can't encode character u'\xe6' in position 1535: ordinal not in range(128)

The reason is that the str() function tries to convert the unicode string using ascii, which doesn’t support the character u’\xe6′.

The solution is to convert the string into ‘utf-8’ encoding.

>>> x = (u'hello', u'how are you', u'xxxxxx yy \xe6r')
>>> print str(x[2])
 UnicodeEncodeError: 'ascii' codec can't encode character u'\xe6' in position 10: ordinal not in range(128)
>>> print str(x[2].encode('utf-8'))
 xxxxxx yy ær
>>> str(x[2].encode('utf-8'))
Out[121]: 'xxxxxx yy \xc3\xa6r'

>>> x = (u'hello', u'how are you', u'xxxxxx yy \xe6r')

>>> print str(x[2])

UnicodeEncodeError: 'ascii' codec can't encode character u'\xe6' in position 10: ordinal not in range(128)

>>> print str(x[2].encode('utf-8'))

xxxxxx yy ær

>>> str(x[2].encode('utf-8'))

Out[121]: 'xxxxxx yy \xc3\xa6r'

recommended conversion workflow: input (any cp) -> convert to unicode -> (process) -> output to utf-8

See the following two examples:

s = u'你好'
s.encode('utf-8')
Out[179]: '\xe4\xbd\xa0\xe5\xa5\xbd'
s.encode('utf-8').decode('utf-8') == u'你好'
Out[183]: True

s = u'你好'

s.encode('utf-8')

Out[179]: '\xe4\xbd\xa0\xe5\xa5\xbd'

s.encode('utf-8').decode('utf-8') == u'你好'

Out[183]: True

In[185]: z = '\xe4\xbd\xa0\xe5\xa5\xbd'
In[186]: print z
你好
In[187]: z
Out[186]: '\xe4\xbd\xa0\xe5\xa5\xbd'
In[189]: z.decode('utf-8')
Out[188]: u'\u4f60\u597d'
In[190]: z.decode('utf-8') == u'你好'
Out[189]: True

In[185]: z = '\xe4\xbd\xa0\xe5\xa5\xbd'

In[186]: print z

你好

In[187]: z

Out[186]: '\xe4\xbd\xa0\xe5\xa5\xbd'

In[189]: z.decode('utf-8')

Out[188]: u'\u4f60\u597d'

In[190]: z.decode('utf-8') == u'你好'

Out[189]: True

Best practice:

Always encode from unicode to bytes.

[Read More...]

Python error import: unable to open X server

July 18, 2016 Author: david
No comment yet
When you run python from shell, you may encounter the following error:
```
import: unable to open X server `' @ error/import.c/ImportImageCommand/368
```
Double check you have the proper shebang line in the beginning of your python script:
```
#!/usr/bin/env python
```
Once your set the shebang line, you can run your python script as :
```
./file.py
```
[Read More...]
Simple Json Manipulation using Python

July 15, 2016 Author: david
No comment yet

We list the top json related operations which include load, loads, dump, dumps and pretty-print json.

Create a json file from a python dictionary

We can easily store a python dictionary into a json file using the json dump method. In the following code, we first define a dictionary, then transfer that dictionary into a json file:

Python

import json mydict = {} mydict['a'] = 'aa' mydict['b'] = ['bb1', 'bb2'] mydict['c'] = {'cc':'cccc', 'dd': 'dddd'} f = open('my.json', 'w') json.dump(mydict, f) f.close()

1
2
3
4
5
6
7
8
9
10
11

import json

mydict = {}
mydict['a'] = 'aa'
mydict['b'] = ['bb1', 'bb2']
mydict['c'] = {'cc':'cccc', 'dd': 'dddd'}

f = open('my.json', 'w')
json.dump(mydict, f)
f.close()

The content of my.json file looks like this:

{"a": "aa", "c": {"cc": "cccc", "dd": "dddd"}, "b": ["bb1", "bb2"]}

1

{"a": "aa", "c": {"cc": "cccc", "dd": "dddd"}, "b": ["bb1", "bb2"]}

How to pretty-print JSON?

You can run python with the json.tool option to build a more readable json file: prettyprint json.
[Read More...]
Get shell output when calling shell from Python

July 15, 2016 Author: david
No comment yet

We have shown how to call shell in python using the subprocess communicate method. However, this method has some problems as the output is buffered into memory. We need to print out the shell output if the size is too large. Sometimes, when the shell scripts run for a long time, we may need to examine the output in real time to check the status of the problem. In order to get the real time output from shell from python, we can using the following method:

Suppose we have a shell script, long_shell.sh, like this:

#!/usr/bin/env bash for i in $(seq 1 5) do echo $i echo start sleep 1s sleep 1 done

1
2
3
4
5
6
7
8
9

#!/usr/bin/env bash

for i in $(seq 1 5)
do
echo $i
echo start sleep 1s
sleep 1

done

Then we have the following python program to call the shell command and print the real output in screen.
[Read More...]

Run hadoop command in Python

July 15, 2016 Author: david

No comment yet

Hadoop is the most widely used big data platform for big data analysis. It is easy to run Hadoop command in Shell or a shell script. However, there is often a need to run manipulate hdfs file directly from python. We use examples to describe how to run hadoop command in python to list, save hdfs files.

We already know how to call an extern shell command from python. We can simply call Hadoop command using the run_cmd method.

import subprocess
def run_cmd(args_list):
    print('Running system command: {0}'.format(' '.join(args_list)))
    proc = subprocess.Popen(args_list, stdout=subprocess.PIPE,
            stderr=subprocess.PIPE)
    (output, errors) = proc.communicate()
    if proc.returncode:
        raise RuntimeError(
            'Error running command: %s. Return code: %d, Error: %s' % (
                ' '.join(args_list), proc.returncode, errors))
    return (output, errors)

import subprocess

def run_cmd(args_list):

print('Running system command: {0}'.format(' '.join(args_list)))

proc = subprocess.Popen(args_list, stdout=subprocess.PIPE,

stderr=subprocess.PIPE)

(output, errors) = proc.communicate()

if proc.returncode:

raise RuntimeError(

'Error running command: %s. Return code: %d, Error: %s' % (

' '.join(args_list), proc.returncode, errors))

return (output, errors)

Run Hadoop ls command in Python

(out, errors)= run_cmd(['hadoop', 'fs', '-ls', 'hdfs_file_path']
lines = out.split('\n')

1 2	(out, errors)= run_cmd(['hadoop', 'fs', '-ls', 'hdfs_file_path'] lines = out.split('\n')

Run Hadoop get command in Python

(out, errors)= run_cmd(['hadoop', 'fs', '-get', 'hdfs_file_path', 'local_path']

1	(out, errors)= run_cmd(['hadoop', 'fs', '-get', 'hdfs_file_path', 'local_path']

Run Hadoop put command in Python

(out, errors)= run_cmd(['hadoop', 'fs', '-put', 'local_file', 'hdfs_file']

1	(out, errors)= run_cmd(['hadoop', 'fs', '-put', 'local_file', 'hdfs_file']

[Read More...]

Page 1 of 212 »

Cool Python tricks and tips

python multi thread example

Python Queue examples

Count word frequency

pyspark unit test based on python unittest library

python UnicodeEncodeError, converting unicode to ascii

Python error import: unable to open X server

Simple Json Manipulation using Python

Get shell output when calling shell from Python

Run hadoop command in Python

Get $35 Toward a New Fire TV 4K Device

Shop Amazon Gift Cards. Any Occasion. No Expiration

Best books to master algorithms

Best books to master machine learning

Recommended books to master big data

Search

Category List

Featured Categories

Rss Feeds

Categories

Subscribe

Contact US

Popular Posts

learn4master.com 2016 Accelerated by keyCDN

Get $35 Toward a New Fire TV 4K Device

Shop Amazon Gift Cards. Any Occasion. No Expiration

Best books to master algorithms

Best books to master machine learning

Recommended books to master big data

Search

Category List

Featured Categories

Tags

Categories

Subscribe

Categories

Contact US

Tags

Popular Posts

learn4master.com 2016 Accelerated by keyCDN