August 20, 2006 at 2:12 am · tags: Python SummerOfCode
I just commited the last lines to make mmap module compile as a shared library (DLL) on our beloved Windows platform.
I first tried porting my implementation of time module to Windows and it was a lot easier than I thought. Ok… easier… let’s go back a moment. I have to be fair to myself and you reader so I have to summarize what you should to achieve the same goal to make PyPy’s modules toolchain work in Windows:
- install MS .NET 1.1
- install MS .NET SDK 1.1
- install MS Platform SDK for Windows 2003 Server (I have XP in Parallels, but there’s only that available)
After this giga-byte-loola-palooza-taking-ages-to-install you go underground this way:
- ’search’ and download the MS Visual C++ 2003 Toolkit that MS kindly retired from its download center after telling everyone it was free for everybody. (I still don’t understand why the biggest company in the world can’t let somewhere in a repository its old packages. I can give you some webspace MS, call me!)
- hack (I mean H-A-C-K) distutils to make the dream come true. Thank you Mike Fletcher!
- inject basically every \bin, every \lib, every \include in my entire file system into the batch file. (I always hated DOS batch files, even when I had only DOS 3.0 on the computer. Come on MS, show me what you got!)
- last but not least: cross your 20 fingers. (remember my fellow Jedi, also the feet have fingers)
I forgot to mention that this easy procedure worked eventually and I was really happy the way things gone. I still hope Martin von Löwis will change its mind sometimes about sticking with the 2003 version of the MS C++ compiler but now that I mastered it, it’s not that urgent.
The implementation of time in Windows lacks altzone, tzname, daylight and timezone because of some limitations in PyPy (I can’t use in_dll() to export global variables) but everything else works well. The other module, mmap, should be equal to the POSIX version.
My Summer of Code project is basically finished and I’ll surely post a report of my experience after the official deadline.
August 7, 2006 at 3:15 pm · tags: Python SummerOfCode
I’ve finished porting the CPython’s bz2 module to ctypes plus PyPy and this was the last module in my SoC proposal 
It doesn’t compile yet, for the same reason _ssl doesn’t compile either.
PyPy’s rctypes has some limitations in structures and function pointers.
In those days I realized how nice is bzip2 library and how simulating pointers in Python can be a pain…
July 30, 2006 at 9:26 pm · tags: Python SummerOfCode
OpenSSL is a damn beast but I managed to handle it through the ctypes code generator written by Thomas Heller. This code is based upon GCCXML tool. Using the code generator is really straightforward:
rhymes@groove ~ % python2.4 h2xml.py ssl.h -o ssl.xml
It produces an XML representation of the header to feed to xml2py.py command this way:
rhymes@groove ~ % python2.4 xml2py.py ssl.xml -o ssl.py
Now you have a complete wrapping of the main OpenSSL header file to use in your ctypes based application.
Going back to the actual SoC work: I implemented the whole _ssl.c CPython file in rctypes. So you have ssl_socket.issuer(), .server(), .write() and .read().
By now is PyPy only (because the extension compiler doesn’t fully translate it).
July 27, 2006 at 6:43 pm · tags: SummerOfCode
I think I’ve found the most complicated library ever.
That library is really over-engineered. I spent a whole day trying to understand _ssl.c, wrapping ssl.h and the other headers I need with ctypes codegenerator (doing it manually is insane), writing 400+ lines of initialization code, dealing with the OpenSSL man pages and so on and I encountered one problem more:
Some functions like SSL_CTX_set_options are not exported by the shared library so I can’t use them with ctypes. But since the initialization protocol is not complete the SSL_connect fails miserably.
And if I can’t initialize the SSL protocol I can’t write the module. Crystal.
I think I’m cursed 
July 25, 2006 at 9:44 pm · tags: Python SummerOfCode
mmap module is complete. We (I and I, I guess) still have a problem with special
methods. Methods like __xyz__ are not supported by the PyPy’s extension compiler so
I had to implement stuff like len(mmap_object) as mmap_object.get_len().
That’s definitely ugly and breaks the compatibility with CPython so keep attention. It also lacks support of slice() objects.
strptime() function has been moved out the time module. There’s not support for it by now.
The matter is strptime() at application level (Python’s) works correctly and tests passes but it doesn’t compile for a bunch of reasons: first, TimeRE() class is a subclass of dict() objects (managed to fix that); second, it relies on enormous amount of modules being a single apparently harmless function (to name some: calendar, re, datetime, locale which in turn raaelies on the whole encodings package). So I tried to “backport” some of the stuff the ext. compiler didn’t like but I got stuck around the encodings package.
I decided to remove that function by now. I’ll see.
Now I gotta concentrate on bz2 and ssl to complete my tasks.
July 19, 2006 at 4:38 pm · tags: SummerOfCode
time module:
- asctime() has been implemented working around the *args limitation
- strftime() has been implemented
- strptime() is there but it’s the one which breaks the compilation step.
mmap module:
- mmap creation: ok.
- close(): ok.
- read_byte(), readline(), read(): ok.
- find(), seek(): ok.
- tell(), size(): ok.
I have problems with flush(), and writeXX() because some casting operations
are not supported in rctypes.
bz2 module is still in the void (there’s A LOT of scaffolding in that module)
fcntl module is completed (fcntl.ioctl() still lacks buffer() and array.array() support)
July 16, 2006 at 10:45 pm · tags: SummerOfCode
I just received my O’Reilly Safari-90-day-summerofcode-account!
Yay!
July 14, 2006 at 8:22 am · tags: SummerOfCode
A brief status update:
the time module is almost finished but:
- asctime() is masked because of a problem in the extension compiler to handle *args.
- strftime() and strptime() are not implemented for the same reason.
the fcntl module is basically finished:
- ioctl() does not support array.array() instances and buffer() instances but everything else is implemented.
the mmap module has been started but since the compiler lacks support for **kwargs (and *args too) I had to switch to another task in the meantime.
I’ve started bz2 porting. More on this in a later post when I’ll manage to pass some tests.
All is currently being tested on OSX, Ubuntu Linux and FreeBSD 6.1. Windows support is a tremendous mess because of the nature of Windows, problems with license of software and other stuff. I think I’ll dig into Windows AFTER completing the five modules (_ssl.c is the last one).
July 10, 2006 at 11:44 pm · tags: PyPy Python SummerOfCode
I’m making progresses to feed the rctime module to the extension compiler. He doesn’t like that much tuples so fixing a couple of tupled line to lists decreased the number of translation errors. Now the translation goes fine but I’m stuck in the compiler process. ctypes doesn’t have a mechanism to explicitly tell that one function returns void (nothing) and/or and by default all functions in ctypes return an integer.
The C generated source code from the translation process assigns the “hypothetic” return value of the function I’m calling (in this case tzset()) to a variable and GCC (he’s right ovbiously) does not allow that.
One step forward anyway!
UPDATE: I should think about the importance of None between one debugger session and the other. Thanks to Albert Strasheim. Better go to bed now.
I think I’ve found a bug in the rpython translation layer but more on this later.
July 9, 2006 at 3:52 pm · tags: PyPy Python SummerOfCode Switzerland
My last sprint day was good but I lost a lot of time around some limitations and a bug that’s actually there (anyway unit tests pass) but I’m fighting to win, so I’ll win
I had the chance to pair with Armin Rigo itself and I think that I learned a lot from him (and also from Anders of course). Pair programming with the guys like them it’s important to my knowledge of PyPy and Python.
We have been invited by Richard Jones, a pygame guru, to see a nice and funny presentation about pygame and its capabilities. He was using Bruce as the presentation tool and I’ll definitely try it because it seems awesome!
Other guys have started porting md5, random and zlib also so maybe we’ll managed to port all the Python standard library someday
In the evening Fabrizio and I played to Carcassonne with Armin, Richard and a really nice swedish summer student named Pauline. We italians didn’t know that game but Richard was very kind to teach us how to play. Obviously we lost but Richard, a master board game player, was defeated by Pauline
So that’s all from my first PyPy sprint.
If you want to follow my SoC/PyPy work you’ll find it in pypy.module.rctime (which probably will be renamed as time when completed).
Michael Hudson wrote a PyPy sprint report today to summarize all the work has been done in Geneva: PyPy Post-EuroPython 2006 Sprint Report
Next entries »
