Extending Python by reusing existing C++ codes

Zhuotong Nan ([email protected])

Why I do this?

First let me show you some codes wrote in Python:

(code snippet)

This class is to call an external executable, which will generate necessary data files in a proper file format for further work. When I come to port this class to Windows, I found it is unable to work since os.fork and os.wait do not exist in Windows, and much more worse, os.execvp cannot run as expected. My initial solution is to look for their equivelant under Windows but hard to solve it in a fast way. The alternative choice is to implement Read_xmrg2 as an module of Python. Fortunately, we have c source codes of Read_xmrg2 (which can be found at http://www.weather.gov/oh/hrl/dmip/2/xmrgformat.html). I start to look for some information regarding this implementation.

Extending using SIP

This solution is conventionally called extending Python. But the common implementation presented by Python help is hard to do. More efforts should be placed in doing the conversion between C/C++ data types and Python’s. Since I have background on Qt, a helpful tool called SIP (http://www.riverbankcomputing.co.uk/sip/) draws my attention. There are a number of tools to do the Python bindings. Some studies shows SIP is easier to do such kind of bindings. Although SIP comes with a nice reference file, it is a little bit hard for a newbie to start.

Step 1 Tuning existing c codes

Since I prefer C++ style, I changed existing Read_xmrg2 codes to a C++ class. Following is its simple header file (dmip2.h).

  (code snippet)

Note reverse_byte_order and its short equivalent are used by read_xmrg2 functions. I made them private. Also I declared read_xmrg2 as static. Header file should be generated since SIP will link this header file to generate conversion codes. One work we remains is to convert main function in original c codes to a common function.

(code snippet)

Keep in mind, we should add DMIP2 as a domain prefixing to every method (read_xmrg, constructor, reverse_byte_order, etc). Next, I commented some codes since as a function there is no necessity to check argument number. Also comment declarations of functions reverse_byte_order and reverse_byte_order_short since we declare them in the class definition. Replace original argv[1] to variable fn. Note I also add a fflush(stdout) to force the print information to show immediately, otherwise the caller (Python function) also prints messeges which will mess up the console output.

Then, make the class compiled as a shared library (with extension as DLL). In Eclipse C++ dev, we only need to switch Artifact type to Shared Library as shown below (figure 2), and then compile codes to generate dmip2_py.dll.

figure 1. DLL building settings in Eclipse c++

Step 2 Generate Python module using SIP

2.1 Install SIP

Get SIP package (sip-4.7.1zip) from http://www.riverbankcomputing.co.uk/sip/. SIP supports all versions of Python since v2.3. Version 2.4 was installed in my computer because some other Python modules depend only this version. Unpack SIP zip file to a directory (sip-4.7.1) under Python v2.4 directory (generally c:Python24). Open command window, enter the directory SIP located, and then type "c:python25python configure.py -p win32-g++" which will configure SIP with MinGW compiler I installed in my environment. Default settings of configuration are suitable for most cases. After its configuration, type "mingw32-make" to compile and "mingw32-make install" to install SIP module with Python. More information is available in the reference file (sipref.html) under doc directory of SIP installtion directory.

2.2 Generate a SIP specification file

(code snippet)

Note here if the header file of the wrapped class is not at the same directory as the sip files (dmip2.sip), we should specify its relative location with #include directive. I save this specifation file in sip diectory (figure 2). Since we are only interested in read_xmrg2 function, this sip specification file exposes only this declaration. Change %Module on your demand. In Python codes, we can call read_xmrg2 by DMIP2PY.DMIP2.read_xmrg2() if we import this module.

figure 2. sip folder contains sip files, while src folder c++ header files as wells as source files.

2.3 Built a configure file for faciliating building module

(code snippet)

The line os.system(…) will enable generated files put the same folder as sip files. You should change dmip2.sip according your naming. Other point should be mentioned is makefile.extra_libs where specifies the name of DLL files produced in Step 1. If the DLL is in different location, make corresponding change to makefile.extra_lib_dirs. I prefer copying DLL file(s) (for ex. dmip2_py.dll) to sip directory.

Save to configure.py in the sip directory, run it by typing "python configure.py" in command window. Be sure you have set python location (for ex. c:python24) and MinGW binary location (for ex. c:mingwbin) to your system path variable. Otherwise please specify the absolute path here.

This compilation will produce some new files. See figure 3 for my case.

figure 3. The compilation will produce ultimate python module which wraps DLL functionality.

Step 3 Using generated Python module in Python codes

We need to make some changes to runProgram to employ generated Python module.

(code snippet)

Here we first import DMIP2PY module, and enhance the function by enabling it runs on varying operating system. Please note if we use import directive as shown in the above case, we have to locate the function by MODULENAME.CLASSNAME.FUNCTIONNAME(). But if we import functions by "from DMIP2PY import *", we can refer to the function by CLASSNAME.FUNCTIONNAME().

Do not forget to copy the compiled Python module (dmip2py.pyd) and associated DLL file (dmip2_py.dll) to the same directory as the calling python file.

Conclusion

We walked through how to extend python by using SIP step by step. It’s not much hard to implement your real application. Efforts should be made to the SIP specification file. More details can be found in the reference manual located in the doc directory of sip source package. But for simple scenarios, simply do enhancement based on the above example. SIP also have special design for Qt (www.trolltech.com) application. In this context, it is the unique choice for Qt users.

Sources codes can be availabe on request. Send your comments and suggestions to [email protected].

All rights reversed.

IF YOU ARE INTESTED IN COMPLETE VERSION, PLEASE SEND ME EMAIL OR REQUEST BY FOLLOWING COMMENTS.

Download PDF version (174KB)