IPython – A Configurable Interactive Python Shell

Python is a popular interpretive object-orientated programming language that has been around since the early 1990s and comes with its own command line shell. To make Python more productive for himself, Fernando Perez, currently a research scientist at U.C. Berkeley, created an enhanced version of the standard interactive Python shell which he called IPython.

IPython evolved from a number of then existing projects:

  • An existing project by Fernando Pérez to add Mathematica-like prompts and a flexible configuration system to the standard Python interactive interpreter.
  • An enhanced interactive Python prompt by Janko Hauser, called IPP, which maintained a prompt history which could be incrementally searched. It also had tab completion like many shell interpreters and a help system which you could invoke using ?
  • A project called LazyPython developed by Nathan Gray, who also developed the deep_reload module, which provided automatic quoting and parens, verbose tracebacks and shell escapes amongst other features.

According to a talk Perez gave at a SciPy conference in 2003, while at CU Bolder he merged his code and theirs as an afternoon hack in 2001 and six weeks later (with little sleep or thesis progress) IPython version 0.1 was born.

Interestingly IPython started life under a GPL license but switched to a BSD license after version 0.6.3. All new code contributions must now be licensed under the BSD license or a similar open source license such as the MIT license. The current stable release is version 0.10.1. Officially, IPython 0.10.1 supports Python versions 2.5 and 2.6 but I use it with Python 2.7 without encountering any problems.

IPython is very popular within the scientific and research communities and is used in many scientific packages and computer algebra systems such as SAGE and Singular.

Nowadays IPython consists of two main components:

  • An enhanced interactive shell.
  • An architecture for interactive parallel computing

Starting with version 0.9, IPython supports a wxPython based IPython GUI. IPython also supports non-blocking interaction with Tkinter, GTK, Qt, WX and even your own custom graphics libraries whereas the standard Python shell only interacts with Tkinter which is is a thin object-oriented layer on top of Tcl/Tk.

This post focuses on providing a quick introduction to the interactive command-line shell. This shell is a replacement for the Python interactive shell which tries to incorporate the most common shell-like usage patterns in a natural way, while keeping 100% syntactic compatibility with the Python language. Some of the features are simple time savers like automatic parenthesizes and tab completion. Others add capabilities such as profiles and editor use from within the shell.

The first time you invoke the IPython shell, it creates a configuration directory (called .ipython) in your home directory.

$ ipython
Welcome to IPython. I will try to create a personal configuration directory
where you can customize many aspects of IPython's functionality in:

Initializing from configuration: /usr/lib/python2.7/site-packages/IPython/UserConfig

Successful installation!

Please read the sections 'Initial Configuration' and 'Quick Tips' in the
IPython manual (there are both HTML and PDF versions supplied with the
distribution) to make sure that your system environment is properly configured
to take advantage of IPython's features.

Important note: the configuration system has changed! The old system is
still in place, but its setting may be partly overridden by the settings in 
"~/.ipython/" config file. Please take a look at the file 
if some of the new settings bother you. 

Please press  to start IPython.
Python 2.7 (r27:82500, Sep 16 2010, 18:02:00) 
Type "copyright", "credits" or "license" for more information.

IPython 0.10.1 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object'. ?object also works, ?? prints more.

In [1]: 

To exit the IPython shell, type Ctrl-D (which will request confirmation before editing) or either Exit or Quit (note the case!), both of which exit without prompting.

Here is a listing of what is created in your $HOME/.ipython subdirectory:

[fpm@ultra ~]$ ls -al .ipython
total 80
drwxr-xr-x   3 fpm fpm  4096 Feb 12 15:40 .
drwx------. 53 fpm fpm  4096 Feb 12 15:40 ..
drwxrwxr-x   3 fpm fpm  4096 Feb 12 15:40 db
-rw-------   1 fpm fpm     4 Feb 12 15:40 history
-rw-r--r--   1 fpm fpm     0 Jun  2  2010
-rw-r--r--   1 fpm fpm   150 Nov 14 21:26 __init__.pyc
-rw-r--r--   1 fpm fpm   150 Nov 14 21:26 __init__.pyo
-rw-r--r--   1 fpm fpm 24116 Feb 12 15:40 ipythonrc
-rw-r--r--   1 fpm fpm  1345 Feb 12 15:40 ipythonrc-math
-rw-r--r--   1 fpm fpm  2138 Feb 12 15:40 ipythonrc-numeric
-rw-r--r--   1 fpm fpm  1692 Feb 12 15:40 ipythonrc-physics
-rw-r--r--   1 fpm fpm  3602 Feb 12 15:40 ipythonrc-pysh
-rw-r--r--   1 fpm fpm  1260 Feb 12 15:40 ipythonrc-tutorial
-rw-r--r--   1 fpm fpm  3474 Jun  2  2010
-rw-r--r--   1 fpm fpm  1742 Nov 14 21:26 ipy_user_conf.pyc
-rw-r--r--   1 fpm fpm  1742 Nov 14 21:26 ipy_user_conf.pyo
[fpm@ultra ~]$ 

I have to say that one problem with IPython is the number of different types of configuration files. Originally IPython used ipythonrc-style key-value files. Then it switched to using configuration files but still kept support for ipythonrc files. Starting with IPython v0.11, which is currently under development, the IPython developers have introduced yet another completely new configuration system that is designed from scratch to address the particular configuration needs of IPython.

One of the more interesting features of IPython is that it has a number of magic keywords (commands). It checks the command entered against its list of magic keywords. If the command is a magic keyword, IPython handles it. If it’s not a magic keyword, it passes the command off to Python to deal with. If automagic is on (the default), you do not need to prepend the % symbol to a magic keyword. If automagic is off, keywords must be prepended with the % symbol. Type magic at the shell prompt to see a list of all magic keywords together with descriptions of their usage. Type lsmagic at the shell prompt to see just a list of all the magic keywords.

In [1]: lsmagic
Available magic functions:
%Exit  %Pprint  %Quit  %alias  %autocall  %autoindent  %automagic  %bg  %bookmark  %cd  
%clear  %color_info  %colors  %cpaste  %debug  %dhist  %dirs  %doctest_mode  %ed  %edit 
%env  %exit  %hist  %history  %logoff  %logon  %logstart  %logstate  %logstop  %lsmagic  
%macro  %magic  %p  %page  %paste  %pdb  %pdef  %pdoc  %pfile  %pinfo  %popd  %profile 
%prun  %psearch  %psource  %pushd  %pwd  %pycat  %quickref  %quit  %r  %rehash  %rehashx 
%rep  %reset  %run  %runlog  %save  %sc  %store  %sx  %system_verbose  %time  %timeit
%unalias  %upgrade  %who  %who_ls  %whos  %xmode

Automagic is ON, % prefix NOT needed for magic functions.

In [2]: 

In IPython, commands like ‘cd’ or ‘ls’ do what you’d expect of them, while still allowing you to type normal Python code. For example:

In [2]: ls
100_2460.JPG*	 dia1.jpg			  hw*	       Music/	    pf5.c.1	      t2.js*
10.xml		 dia2.jpg			  hwaddr*      myimage/     pf5.c.2	      temp.dat
10.xsl		 dia3.jpg			  hwaddr1*     myimage.img  pfdemo*	      Templates/
1.ksh*		 Documents/			  hwaddr1.c    new.jpg	    pfdemo.c	      test
1.xml		 Downloads/			  hwaddr.c     newr.jpg     Pictures/	      t.js*
arps*		 edk5.png			  i686	       passwd	    Public/	      tl*
arps.c		 example1.js			  image/       perl/*	      tmp/

In [3]: 

You can define your own magic functions. The following example defines a new magic command called %stat:

def dostat(self, arg):
    """Demonstrate a custom magic function."""
    ip = self.api
    ip.ex('import os; print os.stat("%s")' % arg)

ip.expose_magic('stat', dostat)

If you add the above lines to your $HOME/.ipython/ (above main()), you can stat files directly from the command line.

In [1]: stat file1
posix.stat_result(st_mode=33204, st_ino=13895499, st_dev=64770L, st_nlink=1, st_uid=500, st_gid=500, st_size=0, st_atime=1297656251, st_mtime=1297656251, st_ctime=1297656251)

IPython supports custom modes called profiles. A profile is simply a configuration file that follows a specific naming convention and can be loaded using a specific syntax. The idea behind profiles is that a user may want to maintain different configurations for different purposes, e.g. one for doing numerical computing with NumPy and another for doing symbolic computing with SymPy.

Profiles (custom modes) make it easy to keep a separate configuration file for each of these purposes. The search path for profiles is the same as that of regular configuration files. The only difference is that profiles are named by simply adding _profilename to the end of the normal configuration file name. For example, if I wished to create a profile called fpm, I create a configuration file $HOME/.ipython/ A profile is loaded by means of the -p profilename command line option.

One such profile which ships with IPython is called the shell profile and activates even more defaults for shell-like behaviour. This mode can be invoked as follows:

$ ipython -p sh

System command list not initialized, probably the first run...
running %rehashx to refresh the command list. Run %rehashx
again to refresh command list (after installing new software etc.)

IPython 0.10.1   [on Py 2.7]


$ ipython -p sh
IPython 0.10.1   [on Py 2.7]

You can run any Python script and load all of its data directly into the interactive namespace using the %run magic command. This magic command supports flags for timing the execution of a Python script (-t) and for executing them under the control of either the Python debugger (-d) or profiler (-p).

In [1]: cat
def fibonacci(n):
    if n == 0:
        return 0
    elif n == 1:
        return 1
        return fibonacci(n-1) + fibonacci(n-2)

print fibonacci(8)

In [2]: run fibonacci

In [3]: run -d fibonacci
Breakpoint 1 at /home/fpm/
NOTE: Enter 'c' at the ipdb>  prompt to start your script.
> (1)()

ipdb> c
> /home/fpm/
1---> 1 def fibonacci(n):
      2     if n == 0:
      3         return 0

ipdb> q

In [4]: run -p fibonacci
         71 function calls (5 primitive calls) in 0.000 CPU seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    0.000    0.000 {execfile}
     67/1    0.000    0.000    0.000    0.000
        1    0.000    0.000    0.000    0.000
        1    0.000    0.000    0.000    0.000 :1()
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}

In [5]: 

Incidentally the %pdb magic command allows you to toggle on and off the automatic invocation of the IPython-enhanced pdb debugger.

To view your command line history (inputs), enter %hist at the prompt:

In [4]: hist
In [4]: hist
1: _ip.magic("lsmagic ")
2: x = 1 + 6
3: print x
4: _ip.magic("hist ")
In [5]:

To display the inputs without numbers, use %hist -n. %hist -r will show inputs exactly as you typed them. %history n1 n2 will print inputs between n1 and n2 (but excluding n2).

To access a single line in your history, you can use _i notation as shown below:

n [1]: 2+3
Out[1]: 5

In [2]: 4+5
Out[2]: 9

In [3]: 6+7
Out[3]: 13

In [4]: _i
Out[4]: u'6+7\n'

In [5]: _i3
Out[5]: u'6+7\n'

Output results are automatically stored in a global dictionary named Out and variables named _1, _2, and so on alias them. Additionally, the three variables named _, __ and ___ always contain the last three results.

In [1]: 4+5
Out[1]: 9

In [2]: 6*3
Out[2]: 18

In [3]: Out[1]
Out[3]: 9

In [4]: _1
Out[4]: 9

In [5]: _2
Out[5]: 18

In [6]: ___
Out[6]: 9

If a command begins with two exclamation marks, the command is executed but its output is captured and returned as a python list, split on newlines. The !! syntax is a shorthand for the %sx magic command.

In [1]: ls
file1  file2  file3  file4

In [2]: !!ls
Out[2]: SList (.p, .n, .l, .s, .grep(), .fields(), sort() available):
0: file1
1: file2
2: file3
3: file4

Another useful feature of IPython is that it can automatically add quotations to strings and parentheses to functions and methods.

In [5]: str=''
In [6]: ,str.lstrip cmowz.
-------> str.lstrip("cmowz.")
Out[6]: ''

In [7]: min 1,2
------> min(1,2)
Out[7]: 1

You can log your IPython session using the command line option -log which creates the file in the current directory or you can use the command line option -logfile to specify your own filename. To resume a session enter ipython -logplay [file].

Within a session the %logstart, %logoff and %logon keywords allow you to create session log and stop and start session logging.

In [1]: logstart
Activating auto-logging. Current session state plus future input saved.
Filename       :
Mode           : rotate
Output logging : False
Raw input log  : False
Timestamping   : False
State          : active

In [2]: logon
Logging is already ON

In [3]: logoff
Switching logging OFF

In [4]: hist
1: _ip.magic("logstart ")
2: _ip.magic("logon ")
3: _ip.magic("logoff ")
4: _ip.magic("hist ")

In [5]: logon
Switching logging ON

You can also customize your IPython shell prompt. The syntax for doing this is similar to that used in the Bash shell. The following are the default IPython prompts:

prompt_in1 'In [\#]:'
prompt_in2 '   .\D.:'
prompt_out 'Out[\#]:'

You can change the prompt to your liking by editing $HOME/.ipyton/ For example, here is my configuration:

 o.prompt_in1 = r'\W \#> '
 o.prompt_in2 = r'\W \D> '
 o.prompt_out = r'\W \#> '

which produces the following prompts when IPython is invoked:

$ ipython -p sh
IPython 0.10.1   [on Py 2.7]
.ipython 1> ls
db/   ipythonrc-math     ipythonrc-tutorial
history       __init__.pyc  ipythonrc-numeric
history-pysh  __init__.pyo  ipythonrc-physics  ipy_user_conf.pyc
history-sh    ipythonrc     ipythonrc-pysh     ipy_user_conf.pyo
.ipython 2> max 3,4
----------> max(3,4)
.ipython 2> 4
.ipython 3> Ctrl_D

Ipython makes it easier to work with objects. With Python the dir function is used to list all the attributes of an object. For example:

>>> import Crypto
>>> dir(Crypto)
['__all__', '__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__', '__revision__', '__version__', 'version_info']

IPython has a set of special functions for providing more information about an object. Typing object_name? will print all sorts of details about any object, including docstrings, function definitions and class constructor details. For example, Crypto? or ?Crypto (either way will work) outputs detailed information about the Crypto module.

In [12]: import Crypto

In [13]: ? Crypto
Type:		module
Base Class:	
String Form:	
Namespace:	Interactive
File:		/usr/lib64/python2.7/site-packages/Crypto/
    Python Cryptography Toolkit
    A collection of cryptographic modules implementing various algorithms
    and protocols.
    Crypto.Cipher             Secret-key encryption algorithms (AES, DES, ARC4)
    Crypto.Hash               Hashing algorithms (MD5, SHA, HMAC)
    Crypto.Protocol           Cryptographic protocols (Chaffing, all-or-nothing
                              transform).   This package does not contain any
                              network protocols.
    Crypto.PublicKey          Public-key encryption and signature algorithms
                              (RSA, DSA)
    Crypto.Util               Various useful modules and functions (long-to-string
                              conversion, random number generation, number
                              theoretic functions)
In [14]: 

There are also a number of magic commands for retrieving specific information as shown in the following example.

In [1]: import printer

In [2]: %psource printer
"""This function call print"""
def printer(x):
    print x
    printstatus = 1

In [3]: %pdoc printer
This function call print

In [4]: %pdef printer
Object is not callable.

Entering complex code or nested structures into the interpreter can be difficult with only single-line editing. The %edit magic keyword enables multiline editing by opening the editor defined by the environment variable $EDITOR or within your configuration files. If $EDITOR is not defined, IPython defaults to using vim on GNU/Linux.

When you call %edit (%ed is an alias) with no arguments, IPython opens an editor with a temporary empty file, executes the contents of the temporary file upon closing the editor and it returns the contents of the file as a string variable. If you do not want the contents of the temporary file to be executed upon editor exit, use %edit -x.

In [4] %ed
IPython will make a temporary file named: /tmp/
Editing... done. Executing edited code...
output is  4
Out[4]: 'print "output is ", 2 + 2\n'In [8]: %ed -x
In [5] %ed -x
IPython will make a temporary file named: /tmp/
Out[5]: 'print "output is ", 2 + 2\n'

On GNU/Linux the temporary file is created in the /tmp directory rather than the /var/tmp directory. So far, I have not found a way to change the pathname of the temporary file but there probably is some way of doing it.

By the way, IPython is not the only enhanced shell for Python. One of the more popular alternatives is bpython which features syntax highlighting, code completion, expected parameters, rewind, save work to file and pastebin support. Others include DreamPie and enhanced shells included in Python IDEs such as the SPE IDE.

Finally I am moved to once again point out that the Internet is becoming filled with rubbish and incorrect information. While researching IPython examples for this post, I came across more obsolete and incorrect information about IPython than correct information. This is because IPython has significantly changed since inception in areas such as configuration files and magic function definitions but documentation and examples relating to obsolete versions of IPython still litter the Internet. Perhaps there needs to be a mechanism for removing obsolete information from the Internet or at least clearly marking it as obsolete.

I have barely scratched the surface of all the features in IPython but I need to finish this post. If you are a Python developer, I strongly encourage you to switch to IPython and enjoy the resulting productivity gains.

Comments are closed.