2.1   Programming Languages Timeline

3.1   Choosing IDE

Python provides interactive shell command python and also provides a graphical IDE called IDLE that ships with python on all platforms. That is good enough to start learning without having to use any heavy weight IDE.

There are many options available for using more powerful IDE. Here is a good summary at Python IDE wiki.

Since there are too many options, I will shortlist few good editors. These are opinionated choices, but rather good place to start.

My personal preference is vim editor with python-mode plugin.

Here is another interesting collection of information about Python IDEs from this stackoverflow question.

To follow the examples described in this book, we won't assume any specific IDE. We will just use command line python command and ipython shell.

ipython provides interactive shell similar to python shell with more powerful extended features. Go to IPython site to download it.

4.1   Hello World

Start your python interactive shell:

$ python

Python 2.7.3 (default, Aug  1 2012, 05:14:39)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>
>>> print 'Hello World!!!'
Hello World!!!
>>>

If you use ipython instead of python, it looks like below:

$ipython

Python 2.7.3 (default, Aug  1 2012, 05:14:39)
Type "copyright", "credits" or "license" for more information.

IPython 0.12.1 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: print 'Hello World!!!'
Hello World!!!

Congratulations! That was really simple.

As you can see, the print is a keyword for print statement. So, it was not necessary to type print('Hello World'). However, from Python 3.0+ print has been changed to function, not a keyword. It is better to start using print() since it works on both versions.

For reasons for the change See PEP-3105 (Python Enhancement Proposal)

All changes to Python follow PEP process similar to JSR process for Java community.

4.2   Supported Programming Styles

In any (procedural) language, you can usually expect very basic support for:

• Expressions involving operators
• Conditional Statement (if, else, elif)
• Looping (while, for)
• Built-in Types (bool, int, float, string, etc)
• Function definitions for reusable code

Python, ofcourse supports all these, and also supports both Functional and Object Oriented style of programming. We will look at each of these aspects later-- one after another.

Initial examples here focus on the simple and common procedural style programs so that you can readily map equivalent features in other languages which you may be already familiar with.

After the introduction of functional style programming, you will be encouraged to write programs in functional style which is more 'pythonic'.

4.3   Indentation

First thing you should be aware is that indentation matters in Python. In languages like C and Java, you have braces {} to define the blocks, but in Python you use the indentation. Enforcing indentation makes the program more readable in general. For some people who are already used to C-Style of braces, this can be very frustrating at first. But it is more likely that, if you accept this, you will find it is just natural after sometime. I am one example, who complained, whined and grudgingly started with Python and I can vouch that it no longer bothers me -- in fact, using indentation for blocks seems more natural to me now.

4.4   Example: Read and Print

Let us look at a simple exercise to do the following:

Write a program which prompts for your name and year of birth and prints your age. Do this in a loop until user inputs quit for the name. Type the following in a text editor and save it in name.py:

while True:
    name = raw_input('Enter Your Name ==> ')
    if name == 'quit':
        break
    year = raw_input('Enter your Year of Birth ==> ')
    age = 2013 - int(year)
    print('Hello, {}! You are {} years old!\n'.format(name, age))

Now run this program from shell:

$ python name.py

Enter Your Name ==> Johnny
Enter your Year of Birth ==> 1900
Hello, Johnny! You are 113 years old!

Enter Your Name ==> quit

There are several things to notice from this simple program:

• There is no braces {} for block. The while block was defined by the indentation. By convention, we use 4 spaces for single indentation. However technically any number of spaces will do.

• You don't declare variables. They spring into existence when you assign some value to them. Such languages are called dynamic languages. It provides convenience and generic programming capabilities (polymorphism) at the cost of some lack of compile time type checking safety. Python's approach is arguably superior.

• raw_input(str) is a built-in function, which prints and prompts for user input and reads the line from std input returning the result.

• True is a boolean of type bool. String comparisons like name == 'quit' evaluates to boolean result. Unlike C, you don't need special strcmp() function to compare strings.

• You convert string to integer by using int(year) function. Explicit type conversion is required. This is different behaviour from loosely typed languages like PHP. In PHP, a variable can act as either string or integer depending on context. Python is said to be strictly typed. This helps preventing silent errors arising out of ambiguous nature of loose typing.

• The break statement terminates the while loop when the condition is satisfied.

• The output format specifiers like %s (for string) and %d (for integer) works just like in C language.

• The '%' operator is used to format using the input

• The raw_input() function is a compact way of doing the following:

  import sys
  sys.stdout.write(“Enter your name :”)
  name = sys.stdin.readline()
  

4.5   Output Format

Suppose you want to format the output to align at specific column. Then you can use C-like format specifier as below:

>>> print( 'Name:  %15s; Age: %5d' % ('Johnny', 93) )
    Name:           Johnny; Age:    93

Note the % operator used as string format operator for the string. The print() function is unaware of the % operator. The % operator is also used as modulo operator:

>>> 9 % 2
    1

>>> 9 % 3
    0

>>> 'Number %d' % 5
    'Number 5'

>>> 'Numbers %d and %d' % (10, 20)
    'Numbers 10 and 20'

Note that the % operator takes tuple as second operand when input involves multiple components like (10, 20) in the above example. The tuple is a built-in type in Python which is immutable (means you can't change it once you create it).

When output does not involve format specifier, more readable version would be:

>>> 'Number {} and {}'.format(10, 20)
    'Number 10 and 20'

With format(), you can reuse the same argument:

>>> 'Number {0:2d} (in decimal), {0:x}, {0:#x} (in hex) and {1}'.format(10, 20)
    'Number 10 (in decimal), a, 0xa (in hex) and 20'

Even better, you can use named parameters as below:

>>> 'Number {first} and {second}'.format(first=10, second=20)
    'Number 10 and 20'

4.6   Input Format

Suppose you want to read the name and age from single line user input. In C language, you would write:

printf("Enter Name and Age: ");
scanf("%s %d", name, &age);

In Python, you can do:

(name, age) = raw_input('Enter Name and Age: ').split()
age = int(age)

The raw_input() function returns the input line after stripping the newline at the end. The split() splits the line into a list of words separated by whitespace. You can assign the list to a tuple as above. At the last, we convert the type of age from string to integer.

Suppose if you want to read the hexa-decimal input with "%x" (and such) ? For such more complicated input processing, there is no direct equivalent of C's scanf() in Python. You will have to use regular expression.

See also:

• Simple scanf recipe
• sscanf in Python

4.7   Example: Store List of Values

Let us say, you need to store all the (name, age) pairs in the first example in memory for later processing. The input may have any number of pairs.

How do you do that in Python? The answer is: we use -- list built-in datastructure. The list can dynamically grow in size as more elements are added. This is a powerful concept.

Just to compare this with other languages-- In C, you would write your implementation for linked list; In C++ you would probably use vector template; In Java, you would probably use java.util.Vector. However, you would notice that Python's style is much less verbose and easier to use:

nalist = []                          # Initialize empty list
while True:
    name = raw_input('Enter Your Name ==> ')
    if name == 'quit':
        break
    year = raw_input('Enter your Year of Birth ==> ')
    age = 2013 - int(year)
    print('Hello, {}! You are {} years old!\n'.format(name, age))
    nalist.append((name,age))       # Append (name, age) tuple.

print 'Input List is: ', nalist

We just added 3 lines to our first example: first at the beginning and the last 2 lines at the end!

Suppose you enter 3 pairs of names, an example output is:

Input List is:  [('john', 20), ('jack', 40), ('jill', 30)]

To see complete documentation about list just use pydoc command:

$ pydoc list

4.8   Example: Simple Function

Let us look at a simple function which returns a string to indicate if a given number is negative, zero or positive:

def numdesc(no):
    """Given number, return 'negative', 'zero' or 'positive'"""
    if no < 0:
        return 'negative'
    elif no == 0:
        return 'zero'
    else:
        return 'positive'

>>> numdesc(100)
    'positive'
>>> numdesc(0)
    'zero'
>>> numdesc(-20)
    'negative'

>>> numdesc.__doc__
    "Given number, return 'negative', 'zero' or 'positive'"

The program is self explanatory. The triple quote string right next to the definition serves as the documentation for the function. Actually triple quote string used here is a general string expression whose value is thrown away since it is not assigned to anything. It is the convention to use them for documentation. The documentation generation tool such as pydoc also recognizes such conventions.

Every block is prceded by ':' in the previous line and indented at additional level.

4.9   Example: Using File and Process

The following example illustrates how to write into file and how to invoke external command from the script:

import subprocess

def   writefile():
    """Example to illustrate file and process invoke operations"""

    outfp = open('/tmp/hello.txt', 'wb')
    outfp.write('Hello World!')
    outfp.close()
    subprocess.call(['/bin/cat', '/tmp/hello.txt'])

writefile()

The first import subprocess statement makes all the functions exported by the subprocess module available here. Then you open the file for writing ('wb' for write in binary mode). The 'wb' mode is same as 'w' in Unix, but different in windows. (due to how newline characters are translated before writing). It is recommended to use 'wb' in the interest of portability.

The subprocess.call() accepts the list of command line strings. If you prefer to pass single space separated command line, you can use either of the following:

* subprocess.call('/bin/cat  /tmp/hello.txt', shell=True)

* import os
  os.system('/bin/cat  /tmp/hello.txt')
  # Note: Use of os.system() is deprecated since new subprocess module
  # is more general and flexible.

See pydoc subprocess for more information about subprocess module.

4.10   Example: Your environment

Consider the following example which prints your os PATH directories and also python module search directories:

from pprint import pprint                                   # (1)
import os
import sys

def  ospath():
    """Print the list of directories as indicated by os PATH environment"""
    print('Your OS PATH directories :')
    pathlist = os.getenv('PATH').split(os.pathsep)          # (2)
    pprint(pathlist)

def  pythonpath():                                          # (3)
    """
    Print python module search path directories.
    Note that $PYTHONPATH directories are included in this list.
    """
    print('Your Python PATH module search directories :')
    pprint(sys.path)

if __name__ == '__main__':                                  # (4)
    ospath()
    pythonpath()

(#1) The pprint module provides pprint() function to pretty print list output.

(#2) os.getenv('PATH') returns system path string. It is split into a list using os dependant path separator character os.pathsep. Note that in Unix, it is ':' and in windows it is ';'

(#3) There is installation platform specific list of python module search path. User can add additional directories by setting PYTHONPATH environment variable. The sys.path reflects the final value of the module search path.

(#4) This program itself can be imported as module in another program. In such cases, it is useful to conditionally hide certain logic if it is not invoked as the main program.

4.11   Basic Introspection

The most effective way of introspecting any object is to use dir() command at the interactive shell. Continuing from the earlier example:

>>> dir(nalist)
....   # All the supported function and property names of the nalist object
       # are displayed.

>>> type(nalist)
    <type 'list'>

>>> help(list)
       # Displays the python doc for list.

Now that you have got little taste of the language, we will move forward with more fundamentals before looking at more complex examples.

4.12   Comments

Comments are used in following forms:

# Lines starting with hash are comments.
i = 10      # You can use partial line comments like this too.
"This is dummy string expression, but may serve like a comment"

def f():
   """The triple quoted strings is the standard for documenting
   functions and modules. It is a multi-line string expression,
   and used for documenting the above function by convention.
   The Python documentation tools look for this convention to
   auto-generate the documentation"""
   ....

4.13   Zen of Python

Try this at your prompt:

>>> import this

5.1   Built-in functions

Note that built-in types (such as int, float, etc) and built-in functions (such as len(), isinstance()) are not keywords. This is different from many other languages, such as C where built-in types are keywords.

Python's built-in functions include:

dir(),     id(),     len(),    isinstance(), issubclass(),  open(),
range(),   map(),    reduce(), filter(),     apply(),       locals(),
globals(), eval() ,  zip(),    enumerate(),  raw_input(),   and more ...

The built-in types are also functions since they are used as constructors:

int(), long(), float(), list(), tuple(), dict(), type(), and more ...

Since these are not keywords, you will be able to rebind them to different things:

>>> len = 10    # This essentially rebinds built-in function name 'len' to 10.
                # Not a good idea. Should avoid rebinding built-in function names.

You can pass built-in types as any other objects as function paramters which is very powerful concept useful for generic programming:

>>> def convert(val, sometype):    # This defines a simple function f
      return sometype(val)         # which just converts value to given type

>>> print(convert(10.5, int))      # Isn't that interesting ?
    10

5.2   Built-in DataTypes

Python defines a core set of built-in types designed for ease-of-use. The following list includes most of them. A few of them have been leftout intentionally since they are either rarely used or reserved for later advanced level discussions. Here is a summary:

>>> sys.float_info
>>> sys.maxsize
    9223372036854775807      # which is (2**63 - 1)

>>> sys.float_info
    sys.float_info(max=1.79e+308, min=2.2e-308, ... )

>>> complex(1.2, 3.4)
    (1.2+3.4j)

>>> s = 'Some ASCII string'

>>> s = u'San José'

>>> s = b'Some ASCII string'

>>> mylist = [1, 2, (3, 4), [5,6]]

>>> mytuple = (1, 2, (3, 4), [5,6])

>>> buf = bytearray(1024*1024)
>>> view = memoryview(buf)
>>> view = view[10:]
>>> view[:6] = bytes(b'Hello!')
>>> print(bytes(view[:6]))
    b'Hello!'
>>> print(bytes(buf[:16]))
    b'Hello!'

x | y                  Set union
x & y                  Set intersection
x – y                  Set difference
x ^ y                  Symmetric difference
len(x)                 Number of elements in the set
max(x)                 Maximum value
min(x)                 Minimum value

6.1   String Literals

Strings are immutable. Adjacent string literals are auto concatenated:

greet = 'Hello '   "World!"
greet = 'Hello ' + "World!"   # Result is same as above.

# Parentheses used to allow continuation:
greet = ('Hello ' +
             'This can be long greetings')

# Backslash can be used to continue long string.
greet = 'Hello  .... \
             This is long line ...!'

Best way to use multiline strings is to use triple quotes like below:

"""Triple quotes
   are good for multiline comments.
   because you don't have to escape single quotes inside"""

'''You can also use triple quote using
   single quote character'''

6.2   Common String Operations

Some common string operations:

>>> fruits = ['banana', 'apple', 'orange', 'tomato']

>>> ' '.join(fruits)
    'banana apple orange tomato'

>>> ' '.join(fruits).split(' ')
    ['banana', 'apple', 'orange', 'tomato']

>>> ',\n'.join(fruits)
    'banana,\napple,\norange,\ntomato'

>>> print ',\n'.join(fruits)
    banana,
    apple,
    orange,
    tomato

>>> s = 'orange'

>>> s[1:]
    range            # Array slice from 1st index

>>> s[0:3]
    ora

>>> s[1:3]    # Prints  string[start_index:end_index)
    'ra'      # Note: s[3] excluded.

>>> s[-1]     # Last character
    'e'

>>> s[2:-1]
    'ang'

>>> type(s)
    <type 'str'>

>>> type('e')            # Unlike C, single char type is also string
    <type 'str'>

>>> 'r'   in 'orange'    # Use  'in' Operator for substring check.
    True
>>> 'ang' in 'orange'
    True
>>> 'k'   in 'orange'
    False

6.3   bytes vs bytearray

In Python 3.0, the default string type str is unicode capable. For storing raw bytes, there is immutable string type called bytes type. There is bytearray type which is (kind of) list of mutable bytes. It is not exactly a mutable list of raw bytes, just similar but still different. The bytearray type has some unique characteristics:

>>> s = bytearray(b'The King !')
>>> s[4:9] = b'Queen'
>>> s
    bytearray(b'The Queen!')

>>> s[0]
    84

>>> type(s[0])
    builtins.int

>>> s[0] = b't'
    TypeError: an integer is required

>>> s[0] = ord(b't')
>>> s
    bytearray(b'the Queen!')

Hence it is more like a list of small-integers (with each element in range 0-255).

11.1   if-elif-else Statement

The if-elif-else statement has following structure:

if a>b :
   print 'a is greater'
elif a == b:
   print 'same'
else:
   print 'b is greater'

if x < y < z:
   print(' y is between x and z')
if x != y and y != z and x != z :
   print('x, y, z are all different')

Note that you can not use else if -- it must be elif. The conditional expression of the forms such as x < y < z is allowed. Note that the logical and operator is and; logical or operator is or.

11.2   The for Loop

The for loop has following structure:

for i in range(100):
   if (some_condition)
      continue              # Skip this iteration, go to next one.
   do_some_thing()
   if error_occured():
      break                 # Terminate loop, break the else too.
   do_some_more()
else:
   finishing_touches()

The i in range(100) is a common style for looping. The other common patterns are:

for i, v in enumerate(my_list):
   print(i, v)                    # If my_list == [10, 20, 30], it prints:
                                  #  (0, 10) (1, 20) (2, 30)

You iterate through all elements in dictionary as follows:

for key, val in locals():
   print(' %10s = %r ' % (key, val))  # Prints all variables in locals() dictionary.

The continue construct used to skip the current block and go to next iteration just similar to C language continue construct.

The break is used to break out of the loop just like C. It also breaks out of the else construct of the for loop.

The else block is executed if the for loop terminates normally without break. It can be thought of us the 'success path after the loop'. The break could be thought of as error path to break out of the whole statement which includes else part of the statement as well.

11.3   The while Loop

The while loop has following structure:

while not finished:
   do_something()
   if (some_condition)
      continue               # Skip current iteration, go to next.
   if problem:
       break                 # Break the while loop and else clause.
   some_more()
   if all_done:
      finished = True        # Let the loop terminate gracefully.
else:
   after_success_steps()     # Execute this if ``break`` was not called.

The continue and break statements behave just like same named C constructs. The else clause is unique to Python. For the purpose of understanding this clause-- imagine while construct acts like if for the final iteration:

if  (cond):             while (cond):
   ...                     ...
else:                   else:          # when cond is False ...
   ...                     ...

12.1   Simple function

A simple global level functions can be declared like below:

def  addxy(x, y):
   return x+y

def  mulxy(x, y):
   return x*y

Since functions are first class objects, they can be passed as argument:

def opxy(f, x, y):
    return f(x, y)

>>> opxy(addxy, 10, 5)
    15
>>> opxy(mulxy, 10, 5)
    50

12.2   Named arguments

Python allows you to pass parameters as named arguments when you call functions. This provides clarity when there are many parameters:

def subxy(x, y):
    return x - y

>>> subxy(y=10, x=20)        #  <===  Ok to change order of parameters.
10
>>> subxy(x=100)             #  <===  Not OK: Error: y is missing
...
TypeError: subxy() takes exactly 2 arguments (1 given)

We call x, y as positional arguments-- in the absence of parameter name with the call first parameter is x, second parameter is y. However these positional parameters becomes 'named arguments' when they are called in that style as specified above.

12.3   Optional arguments

Optional arguments allows to skip passing some arguments and assume default values for missing parameters:

def subxy(x=0, y=0):
   return x-y

>>> subxy()
0
>>> subxy(10)
10
>>> subxy(10, 5)
5

Even optional arguments can be called with 'named parameters' style:

>>> subxy(y=10)        # x is assumed to be 0
-10

12.4   Variable number of arguments

It is possible to support variable number of arguments.:

def p(*args):
     print(args)    # <== args is a tuple of all var args!

>>> p(10, 20, 30)
(10, 20, 30)

This is useful when you want to be flexible in accepting any number of arguments. Example:

# Following call may push any number of specified values into your stack.
>>> push_into_my_stack(10, 20, 30, 40)

12.5   Keyword Arguments

You can have keyword arguments. This allows passing arbitrary number of key, value pairs in named-parameter style:

def p(**kwargs):
     print(kwargs)        # <== kwargs is a dictionary!

>>> p(x=10, y=20, z=30)
{'y': 20, 'x': 10, 'z': 30}

Another alternative way of achieving the above would be:

def p(options={})
   print(options)

>>> p({'x':10, 'y':20, 'z':30})
{'y': 20, 'x': 10, 'z': 30}

However, as you can see, calling the function with dictionary argument is clumsy and not as elegant as the named parameters.

12.6   Combining Argument Styles

You can combine all the styles mentioned earlier like below. Some basic rules apply (as you may expect) to avoid ambiguity during function call:

• All optional parameters should appear after mandatory parameters
• Variable number of arguments *args should appear after any optional args.
• The keyword args **kwargs should appear at the last if present.

def p(msg, kind='INFO', *args, **kwargs):
    print(msg)
    print(kind)
    print(args)
    print(kwargs)

>>> p('Job Incomplete.', 'WARNING', 'Job100', 'Job200', batch=1, dest='remote')
Job Incomplete.
WARNING
('Job100', 'Job200')
{'dest': 'remote', 'batch': 1}

13.1   Static Scoping

Python mostly makes use of static scoping -- meaning the variables referenced from a function are bound to the module in which the function is defined-- not bound to the caller's module context. This enables robust modular design.

13.2   Dynamic and Strict Typing

Python is dynamically and strictly typed. Dynamic because you can just create variable by assigning something to it. And reassign to different variable of different type anytime

age = 90                 # No need to predeclare the type.
age = 'it is 90 years'   # Reassigned to another object of another type!

It is strictly typed, because every object behaves strictly according to associated type and you can not refer to an undefined variable:

#!python
age = 50
age = '40'         # Reassignment of different type OK.
age =  age - 10    # Error! Type mismatch: Expected numeric, found string

This is unlike C. In C, variables are statically and strictly typed. In PHP, variables are dynamically and loosely typed:

#!php
$age = 50
$age = '40';
$age = $age - 10;     // Perfectly fine. '40' becomes int 40.
echo $age;
// Prints 30

Also note that, Python variables can't be referenced before assignment-- You will get NameError. In PHP, such an action will generate warning but returns usually a default value of empty string. In that sense, Python is 'more strictly' typed.

13.3   Accessing Global vs Local Variables

The module level variables are referred as global variables. Function definition introduces a new level of scoping. A class declaration introduces a new level of scoping. A namespace in Python is currently implemented using dictionaries, but that is an implementation detail.

The available namespaces can be summarised as below:

Python rules for accessing global variables from inside function is not-obvious -- and often results in confusion for python beginners:

gvar = 10

# If you are only reading global variable, no need to declare them as global
def my_func():
   print('gvar is %d ' % gvar)       #  <====   Prints: gvar is 10     OK!

However, the moment you assign some value to a variable anywhere in the function, that variable becomes a local variable unless you declare it otherwise:

gvar = 10

def my_func():
   print('gvar is %d ' % gvar)
   gvar = 20

>>> my_func()
    UnboundLocalError: local variable 'gvar' referenced before assignment

To fix this error, declare gvar as global:

gvar = 10

def my_func():
   global gvar
   print('gvar is %d ' % gvar)
   gvar = 20

People usually expect the global variables to be always available inside function or always not-available inside function unless explicitly declared otherwise. 'Global variable is available for read, but not for write' rule is not obvious and can be confusing to Python beginners. However, once you understand this rule, there is no more confusions.

14.1   Various import forms

The import statement follows any one of this format:

import                 mod1              # Lets you access mod1.f()
import                 mod1 as m         # Lets you access m.f() instead of mod1.f()
from mod1      import  my_var            # Lets you access my_var directly.
from mod1      import  v1, v2            # Lets you access v1, v2 directly.
from mod1      import  my_var as v       # Lets you access my_var which is aliased to v.
from mod1      import  *                 # Imports all variables in mod1. Not recommended.

import                 pkg1.mod1         # Lets you access pkg1.mod1.f()
import                 pkg1.pkg2.mod1    # Lets you access pkg1.pkg2.mod1.f()
from pkg1.mod1 import  my_var            # You can access my_var directly.

If you use the statement of the form import X.Y, then X must be a package and Y may be a subpackage or a module-- However, note that X can not be a module and Y can not be just object. The use of dots is reserved for packages only:

import  mod1.my_var          # <==== This is not allowed!

14.2   Controlling Module Search Path

The sys module contains the path object which specifies the list of module search directories. To examine it's value, we must import the sys module first:

>> import sys
>> from pprint import pprint
>> pprint(sys.path)

 ['',
  '/home/user/myenv/bin',
  '/home/user/myenv/local/lib/python2.7/site-packages/distribute-0.6.24-py2.7.egg',
  '/home/user/myenv/local/lib/python2.7/site-packages/pip-1.1-py2.7.egg',
  '/home/user/myenv/lib/python2.7',
  '/home/user/myenv/lib/python2.7/plat-linux2',
  '/home/user/myenv/lib/python2.7/lib-tk',
  '/home/user/myenv/lib/python2.7/lib-old',
  '/home/user/myenv/lib/python2.7/lib-dynload',
  '/usr/lib/python2.7',
  '/usr/lib/python2.7/plat-linux2',
  '/usr/lib/python2.7/lib-tk',
  '/home/user/myenv/local/lib/python2.7/site-packages',
  '/home/user/myenv/local/lib/python2.7/site-packages/IPython/extensions']

The module search path (i.e. sys.path) includes certain platform dependent default directories. In addition, you can control the search path by one of the following methods:

• Set PYTHONPATH environment variable to include additional directories. They will be prefixed into your module search path.

• The program can dynamically modify the module search path. It can do the following first:

  >>> import sys
  >>> sys.path.insert(0, '/path/to/my/module')
  

• If you are running virtualenv, then you can choose to place your module in the local site-packages directory without affecting other installations.

14.3   The package directory layout

The packages are nothing but directories with a file named __init__.py. It may initialize the package or may just be empty. They are organized like this:

pkg1                             # Top level package
    __init__.py                  # Initializes the top level package
    pkg2                         # pkg2 is under pkg1.
       __init__.py               # Initializes pkg2.
       mod1.py                   # This is module-1 in package-2.
       mod2.py                   # This is module-2 in package-2.

14.4   Restricting exported variables

After importing module, you can access all global variables in that module. However, there are couple of mechanisms you can use to restrict the variables being exported:

• All variables starting with underscore (_) are excluded from the export list.

• A module can define a global variable __all__ like below to restrict only the specified variables to be exported. All other global variables will not be exported:

  __all__ = ['mysym1', 'mysym2', 'mysym3' ]
  

Similarly, when you import a package, all global variables in __init__.py file or the restricted symbols defined by __all__ are available for access.

See http://docs.python.org/ref/import.html for more information.

You can also import from future! i.e. It is helpful for gradual migration to start using the new features that will be introduced in near future in a later release.:

>>> from __future__ import print_function

Above statement enables the use of print() function in Python 2.7 instead of print statement. In Python 3, print() is available only as function. A future statement is special in the sense it is recognized and treated at compile time.

14.5   Relative Vs absolute imports

In Python2.7, the imports are by default relative. i.e. If a module imports another module, it would search in the same package first before searching other sys.path directories. This has the danger of hiding system modules in case some one creates a local module with same name as one of system modules.

X.py:

import  Y      # Looks into current pkg first in Python2.7, not in Python3

In Python3, all imports are absolute by default. So, module X in some package want to import module Y which is in the same package, then following will work:

from . import Y   # Looks into current pkg first in both Python2 and Python 3

Note: The empty string ('') in sys.path stands for current directory (same as '.')

which by default, included in interactive shell.

14.6   Circular Imports

If module X uses functions defined in module Y and vice versa, then following case of mutual import is possible:

# mod_x.py
import mod_y    # From module X

# mod_y.py
import mod_x    # From module Y

In general, it is good to avoid designing modules which depend on each other -- it is often a sign of poor design. However, there are cases where it is perfectly alright to do so. Python does necessary checks to prevent any infinite recursion as a result of mutual imports -- either direct or indirect.

If two modules access each other's objects at the global level initialization, try delaying the import as late as possible.

For example, let us consider the following files:

# mod_x.py

def f_x():
    mod_y.some_func()

def g_x(): pass

import mod_y                  # import of mod_y is delayed as much as possible.
mod_y.some_func()

The mod_y.py file is given below:

# mod_y.py

def some_func(): pass

import mod_x                  # import of mod_x is delayed as much as possible.
mod_x.g_x()

The delaying of import as mentioned above just works fine.

14.7   Importable, executable module

You can make a module importable and at the same time executable by including some code conditionally like below:

if __name__ == '__main__':
  import sys
  exit(main(sys.argv))    # or run some unit tests.

15.1   Collections Container Types

>>> Point = collections.namedtuple('Point', 'x y')
>>> p = Point(x=10, y=20)
    Point(x=10, y=20)
>>> p.y
    20

>>> fruits = ['apple','apple','orange','grapes','grapes','apple']
>>> c = Counter(fruits)
>>> c
    Counter({'apple': 3, 'grapes': 2, 'orange': 1})

>>> grades = [('Ram', 'B'), ('John', 'A'), ('Jack', 'B')]
>>> d = defaultdict(list)           # Default is empty list.
>>> for name, rating in grades:
...    d[rating].append(name)
>>> d.items()
    [('A', ['John']), ('B', ['Ram', 'Jack'])]

isinstance(myvar, collections.Sequence)

For more information about other datatypes defined by Python Standard Library, See http://docs.python.org/2/library/datatypes.html

15.2   Iterator

You can iterate over any container objects such as list, set, etc. A container object may or may not be a sequence. A set is a container but not a sequence.

Iterator is an object which helps with efficient iteration over any container. It is somewhat similar to C language's pointer pointing to an array element -- in the sense that it consumes minimal resources and just remembers the position in the container.

How is this concept helpful ? Once a container follows iterator protocol, it becomes an iterable object. For example:

for e in my_obj:
   print('e is %r' % e)

As long as the my_obj follows iterator protocol, above will work. This essentially means the following:

• Container must define __iter__() which returns a new instance of iterator.
• The iterator must define __next__() which returns next value.
• In addition, the iterator also must define __iter__() which returns itself. This helps the iterator to mimic the container only in the context of iterating.

Note the following methods:

container.__iter__() : Return a new iterator object.

        >>> x = [10, 20]
        >>> t1 = x.__iter__()
        >>> t2 = x.__iter__()      # or: t2 = iter(x)
        >>> t1 is t2               # They must be 2 separate instances.
            False

iterator.__next__() : Should return next value.
                      On reaching end, should raise StopIteration Exception.

        >>> t1.next()
            10
        >>> t1.next()
            20
        >>> t1.next()
            Traceback (most recent call last):
              File "<stdin>", line 1, in <module>
              StopIteration

 iterator.__iter__() : Should return itself.

        >>> t3 = t1.__iter__()
        >>> t3 is t1
            True

So, a container is iterable and can have multiple iterators pointing to it. The idea is that container does not preserve the state of iteration. If you want to iterate, get a new instance of iterator using the container, then do whatever you want with the iterator.

The iterator protocol is cool, because we can use for loop like below irrespective of whether obj is container object or iterator:

for e in obj:
  do_some_thing(e)

It calls __iter__() and __next__() underneath for you and stop the iteration once it gets StopIteration exception. You can easily define your own container classes as long as you follow this protocol.

16.1   Overview

Python's classes are somewhat similar to C++ and Modula-3. If you are already familiar with C++, the Python's data member variables are always public and all methods are virtual.

The method functions must be declared explicitly with the first argument representing the object-- the current object is often named as self by convention. This is similar to the this object of C++ or Java:

class Vehicle:
    """Generic Vehicle Class"""
    x = y = 0                        # Class data member variables
    def __init__(self, year):
        self.year = year

    def move(self, x, y):
        self.x = x
        self.y = y
        print('Vehicle moved to (%d, %d)' % (x, y))

To instantiate class instance, we use function notation-- there is no need to use new operator used in C++ or Java:

>>> v = Vehicle(2013)
>>> v.move(10,20)
Vehicle moved to (10, 20)

In above example, x, y are class variables which are inherited by the instances. In addition, you can create new instance variables anytime:

v.model = 'Honda'         # Created on assignment
...
del v.model               # The instance variable is undefined now.

Note that v.move(10, 20) is same as Vehicle.move(v, 10, 20).

16.2   Method Vs Static Method Vs Class Method

>>> v.move
    <bound method Vehicle.move of <__main__.Vehicle instance at 0x26f1cb0>>

>>> v.move.__func__
    <function move at 0x2746140>

class Test(object):

  @staticmethod                #  <=== Note the decorator!
  def setup_tests():
      print("Setup done for all tests")

>>> t = Test()
>>> t.setup_tests()
    Setup done for all tests

class Test(object):

  def setup_tests():
      print("Setup done for all tests")
  setup_tests = staticmethod(setup_tests)   # <== Same like using decorator.

class Test(object):

  def __init__(self):
      print("Common intialization steps done.")

  @classmethod
  def remote_test(cls):
      c = cls()
      print("Additional initialization for remote tests done.")
      return c

>>> t = Test()
    Common intialization steps done.

>>> t = Test.remote_test()
    Common intialization steps done.
    Additional initialization for remote tests done.

16.3   New-style vs old-style classes

There is a built-in object called object which every top level class must derive from. If the class extends any built-in type (such as list), then it already indirectly extends from the builtin object. The top level classes which do not derive from object are called old style classes.

The new-style classes were introduced in Python 2.2 in order to unify built-in types and classes . In terms of declaration, they look like below:

class NewStyleClass1(object):
    pass

class NewStyleClass2(Othernewstyleclass):
    pass

class OldStyleClass:
    pass

In Python 3, all classes are new-style. i.e. No need to explicitly derive from object-- they are all implicitly derived from object.

Only new-style classes can use new features like descriptors.

For more information, See http://www.python.org/doc/newstyle/

16.4   Constructors

class  Parent(object):
   def __init__(self):
       print('Parent initialized')

class  Child(Parent):
   def __init__(self):
       print('Child initialized')

c = Child()  # Prints just: Child initialized

class Child(Parent):
    def __init__(self):
       Parent.__init__(self)        # <== Works with even oldstyle classes.
       print('Child initialized')

class Child(Parent):
    def __init__(self):
       super(Child, self).__init()  # <== super available only with newstyle classes
       print('Child initialized')

class Child(Parent):
    def __init__(self):
       super().__init__()          # <== short form super() being used.
       print('Child initialized')

class A(object):

    def __new__(cls, x):
       print('A __new__() called')
       return  object.__new__(cls)

    def __init__(self, x):
       self.x = x
       print('A __init__() called')

>>> a = A()
    A __new__() called
    A __init__() called

16.5   Destructor

You can do the cleanup of the instance from the destructor method __del__():

class Test(object):

def __init__(self, outfilename):

self.fout = open(outfilename, 'w')

def __del__(self):

print('Deleting Test object ...') self.fout.close()

>>> t = Test('/tmp/mytest.out')
>>> del t                            # Explicit Delete!
    Deleting Test object ...
>>> t                                # t is undefined after delete!
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 't' is not defined

>>> t = Test('/tmp/mytest.out')
>>> t = 10                       # Destructor called by garbage collection!

The chaining of destructors and use of super() method is similar to that of constructors. However, you destruct the child resources first before calling the destructor of parent objects.

16.6   Multiple Inheritance

You can extend multiple classes if so desired -- i.e. Multiple inheritance is supported:

>>> class  Car(Vehicle, Product):
        no_of_wheels = 4

>>> Car.mro()                  # Displays method-resolution-order
    [__main__.Car, __main__.Vehicle, __main__.Product, builtins.object]

Multiple inheritance should be avoided if possible as it further complicates the use of super(). The super() really invokes the method in the next mro chain. This may lead to surprises. Either super() should be used in all base classes or should not be used at all favoring explicit invocation.

16.7   Everything is an Object

In Python, everything is an object. Even the fundamental built-in types are objects including int, float, etc. This is different from Java and C++ where primitive datatypes like int, float are not objects.

An int variable is bound to an integer object of type 'int'. The most fundamental thing in the type hierarchy is 'type'. Everything is directly or indirectly an instance of 'type'. The 'type' itself is an object of type 'type'. The buck stops there:

>>> a=10

>>> type(a)
    int

>>> type(int)
    type

>>> type(type)
    type

>>> type(list)
    type

Any class is also an object of some type -- which is a metaclass. The 'type' type is a metaclass since an instance of 'type' is another type. You can also define your own metaclass which is another topic for discussion.

17.1   Catching Exceptions

You can use try-except block to catch the exceptions generated in try-block:

try:
  something()
except ExceptionClass as exception_obj:
  process_exception(exception_obj)
  raise  exception_obj              # You can re-raise the exception like this.

The legacy syntax alternative for except is as follows:

except ExceptionClass, exception_obj:      # Works with Python 2, not with 3.
    ...

Since the above syntax is confusing, this has been deprecated in Python 3 in favour of the below form:

except ExceptionClass as exception_obj:    # Works with Python 2.6+ and 3+
                                           # Python 3 allows only this syntax.

You can catch multiple exceptions as given below. The following code works with Python 3:

>>> while True:
      s = input('Enter expr > ')      # Use raw_input() in Python 2 instead.
      try:
          eval(s)
      except  ZeroDivisionError as err:              #    1/0
          print('Got ZeroDivisionError:', err)
      except  ValueError as err:                     #    int('something')
          print('Got ValueError:', err)
      except  NameError  as err:                     #    undefined_var
          print('Got NameError:', err)
      except  Exception as err:                      #   e.g. 'some' + 3 (TypeError)
          print('Got Exception:', err)

Enter expr > x
Got NameError: name 'x' is not defined

Enter expr > 1/0
Got ZeroDivisionError: division by zero

Enter expr > int('some')
Got ValueError: invalid literal for int() with base 10: 'some'

Enter expr > 'some' + 3
Got Exception: Can't convert 'int' object to str implicitly

Enter expr > a++
Got Exception: unexpected EOF while parsing (<string>, line 1)

17.2   Exception Hierarchy

To examine the exception hierarchy, use the mro() method:

>>> NameError.mro()
    [builtins.NameError,
     builtins.Exception,
     builtins.BaseException,
     builtins.object]

>>> KeyboardInterrupt.mro()
    [builtins.KeyboardInterrupt,
     builtins.BaseException,
     builtins.object]

The order of except clause is important. The specific exceptions should appear first and more general base exceptions should appear later in that order. If the except Exception as err: had appeared first, it would catch other exceptions such as NameError, ValueError since those exceptions are derived from Exception. It is in general a dangerous practice to catch basic exception classes like Exception unless you really mean to-- because this can hide the real problem if you are not re-raising the exception that you really didn't intend to handle.

17.3   User Defined Exception

You can define your own exceptions for finer level of error handling:

>>> class MyAppError(Exception):
        def __init__(self, reason):
            self.reason = reason
...
...
if  value > max_value :
     raise MyAppError('Got a value which is too large.')

17.4   The finally clause

The try-finally clause is used to make sure the cleanup action specified in finally block is always executed in any case:

>>> try:
      some_function()
    except MyAppError as err:
      ...
    finally:
      print('closing any open files ...')
      ...

The code in finally block is always executed irrespective of whether there was exception generated or whether it was handled or not. Even if there was a return statement in try block, then also finally clause is executed. If the exception raised in try block was not handled, then it is re-raised later after the finally block is executed.

In versions prior to 2.5, try-except-finally was not supported-- however the try..except had to be nested inside try..finally to achieve the same result:

>>> try:
       try:
          some_function()
       except MyAppError as err:
          ...
    finally:
      print('closing any open files ...')
      ...

The finally can also appear in try-except-else-finally form where the else clause is meant to be executed if there were no exceptions generated in the try-block. However it is best to avoid the else clause, since the naming of this clause is often more confusing than helpful.

18.1   filter, map, reduce functions

The built-in functions filter, map, reduce are useful functional programming tools that can be applied on collection types such as list and set. Following example is given for Python 2:

>>> def my_filter(x):
        return (x % 3) != 0     # Use it to filter out 3's multiples.

>>> filter(my_filter,  range(20) )
    [1, 2, 4, 5, 7, 8, 10, 11, 13, 14, 16, 17, 19]

>>> def mul2(x): return 2*x
>>> map(mul2, range(10))                    # Applies func on the sequence
    [0, 2, 4, 6, 8, 10, 12, 14, 16, 18]

>>> def sumxy(x, y): return x + y           # Function can take many args.
>>> map(sumxy, range(5), range(100,105))    # Pass as many lists as func args.
    [100, 102, 104, 106, 108]

>>> map(None, range(5), range(100,105))     # None acts like identity function.
    [(0, 100), (1, 101), (2, 102), (3, 103), (4, 104)]

>>> reduce(sumxy, range(5))                 # Does cumulative sum.
    10

18.2   List comprehension

This feature is inspired from Haskell language. List comprehension provides easier way to construct lists without having to use map and filter. As from the previous example, to construct a list of integers excluding multiples of 3 in a specific range, we can do the following:

>>> [ i in range(20) if i % 3 != 0 ]
    [1, 2, 4, 5, 7, 8, 10, 11, 13, 14, 16, 17, 19]

# You can also generate a list of pairs like below ...
>>> [ (i, 2*i) for i in range(6) if i % 3 != 0 ]
    [(1, 2), (2, 4), (4, 8), (5, 10)]

18.3   Dictionary comprehension

The dictionary comprehension is now available in Python 2.7+ and 3. It is just like list comprehension applied to dictionary. For example, to invert a mapping of a given dictionary, you can do:

>>> reverse_map = { val:key for key, val in my_map.items() }

18.4   Anonymous Function Lambda

Lambda is used to create simple anonymous functions:

>>> f = lambda x: 2*x
>>> f(10)
    20
>>> map(f, range(4))
    [0, 2, 4, 6]

# lambda is mainly used for one-off use on-the-fly
>>> map(lambda x: 2*x, range(4))
    [0, 2, 4, 6]

Lambda function does not support multiple statements or explicit return statement-- It should just be used to calculate a simple expression. The lambda function also provides Closure support-- which essentially means that it remembers the variables of the (static compile-time) enclosing scope which is available for reading inside the function. The closure is a very powerful mechanism which provides ability to generate context specific dynamic functions.

It is less powerful than the more general inner functions, yet it is very useful since often simple-expression-functions are widely used.

20.1   Unicode Resources

• Unicode Howto

21.1   Summary

There are few things which are fairly unique to Python. These may not be so unique to all languages which exist today, but fairly unique considering other mainstream programming languages:

• Using indentation for blocks. The idea was mainly inspired from ABC programming language.
• The 'else' clause of for, while loops and try block. GvR has acknowledged the choice of the naming of 'else' was a bad idea as it may be confusing.
• Scoping rules. By default, a variable referenced in a function is decided wheter it is global or local depending on (1) if it is only read (2) or being written into. See Scoping section.
• There is no goto

• lambda function: It is an anonymous function with limited capabilities of allowing only one expression. Multiple statements are not allowed. It is no way more powerful than a nested named function. Still it is widely used for its convenience despite it's limitations.
• Generators: Though the idea is inspired from prior existing languages, none of the mainstream languages have similar feature.

• No automatic invocation of super classes.
• It makes use of Magic Attributes of the form __attributename__ of the objects.
• No new operator to instantiate class instance.

21.2   Magic Attributes

Python implementation adds various magic attributes of the form __attributename__ into the objects. They are used for special purposes. Following table includes some of those important ones.

22.1   Deep Vs Shallow copies

By default assignment such as a = b makes both variables point to the same object. If you want to make a copy of the object, you can use copy module:

>>> import copy
>>> a = [1, [2, 3], 4]
>>> b = copy.deepcopy(a)
>>> a[1].append(5)
>>> a
    [1, [2, 3, 5], 4]
>>> b
    [1, [2, 3], 4]

22.2   Linked List

The built-in lists are powerful enough for most cases. You can append or insert at arbitrary position. However inserting into list costs O(N) complexity. If that is a problem, consider using one of optional packages found in PyPI. For example, llist or blist.

22.3   Naming Conventions

• Use lower_case_name for functions, methods, attributes

• Use MyClassName for classes.

• Avoid using camelCase

• Module internal attributes: _mod_var

• Class private attributes: __private_var.

  The class private variable form __private_var is internally translated to classname_private_var to avoid name collisions with other classes in the inheritance hierarchy.

22.4   Keywords

To findout python's list of keywords:

>>> import keyword

>>> keyword.kwlist
    ['and',   'as',      'assert',  'break',  'class', 'continue', 'def',
     'del',   'elif',    'else',    'except', 'exec',  'finally', 'for',
     'from',  'global',  'if',      'import', 'in',    'is',      'lambda',
     'not',   'or',      'pass',    'print',  'raise', 'return',  'try',
     'while', 'with',    'yield']

The first statement import keyword makes the module named keyword available. The second statement prints the value of keyword.kwlist, that is, the object kwlist that has been exported from keyword module. The other famous modules include os, sys, re, network, etc.

As you can see, Python has a minimalistic approach when it comes to keywords.

22.5   Operators

Following table summarizes the operators and their precedence as documented from official python documentation.

The order displayed is the lowest precedence appearing first to highest precedence operators appearing at the last.

22.6   Singleton

The best way to implement singleton is by intercepting at the object creation time by overriding the __new__() method:

class Singleton(object):
    """Illustrates how to implement Singleton object"""
    _inst = None
    def __new__(cls, *args, **kwargs):
        if cls._inst is None:
            cls._inst = super(Singleton, cls).__new__(cls, *args, **kwargs)
        return cls._inst

Now instantiating the class any number of times will return the same instance:

>>> obj1 = Singleton()
>>> obj2 = Singleton()
>>> id(obj1) == id(obj2)
True

22.7   Extending immutable class

Extending immutable class is bit tricky since you can't modify the underlying base data members once it is created. You need to override the __new__() method as illustrated in the following example:

class Mylink(str):
    """Illustrates how to extend an immutable class.

    It extends basic string class. It prefixes the given string
    with 'http://' if not already present during intialization.
    """

    def __new__(cls, s):
      if (not s.startswith('http://')):
          s = 'http://' + s;
      return super(Mylink, cls).__new__(cls, s)

Note: This can only be done from __new__() method and can not be done from __init__() method since the base class is immutable. Following won't work since it is too late to modify base object:

class Mylink(str):
    def __init__(self, s):
        if (not s.startswith('http://')):
          s = 'http://' + s;
        str.__init__(self, s)      # <=== No effect! Does not work.