inicio mail me! sindicaci;ón

Archive for June, 2006

Implementing Python with .NET

June 30, 2006 at 1:40 pm · tags:

This is the title of the just finished thesis by Antonio Cuni. He is one of the PyPy developers and the author of the exciting .NET backend named GenCLI.

The thesis is for people (hopefully the entire Python community :-)) who want to understand better the PyPy as a whole and its backend.

You can find it here: Implementing Python with .NET.

Comments

memory map your entire mind

June 29, 2006 at 3:08 pm · tags:

mmap porting is finished.

Here’s the latest changelog:

  • correctly raise error if used in concatenation and repeatition
  • responds to getitem, setitem
  • correctly raise error if called delitem
  • support slicing objects
  • add a big test ported from test_mmap.py

The only remaining issue is the lack of support of the Buffer object protocol because is not directly exposed in Python. You can’t pass a map to file.write() or regexp methods. Hope someday the Python guys expose this.

You can find everything at the usual place: http://codespeak.net/svn/user/rhymes/modules if you want to test it…

Now onto the fourth module: bz2.

Comments

mmap is almost finished but…

June 29, 2006 at 11:09 am · tags:

… I’m quite puzzled about a couple of things.

First, I don’t understand when PySequence_GetItem it’s been called. I explain myself:

in mmapmodule.c you can find the following function to implement getitem behavior:

  1. static PyObject *
  2. mmap_item(mmap_object *self, Py_ssize_t i)
  3. {
  4.         CHECK_VALID(NULL);
  5.         if (i < 0 || (size_t)i >= self->size) {
  6.                 PyErr_SetString(PyExc_IndexError, “mmap index out of range”);
  7.                 return NULL;
  8.         }
  9.         return PyString_FromStringAndSize(self->data + i, 1);
  10. }

What’s not clear from the C api documentation is that PySequence_GetItem is called automatically before passing the numeric index to that function. That means if you pass -1 to the map indexing mechanism in this way:

  1. m = mmap.mmap(f.fileno(), size)
  2. print m[-1]

you get it translated to size - 1 automatically.

Update: I dug into CPython more and the truth lies in abstract.c. PyObject_GetItem calls PySequence_GetItem which in turns converts the negative index before calling the actual mmap_item function.

As the title says the mmap porting is almost finished but it misses one thing: buffer-like behavior. If you write something like this with CPython’s mmap

  1. m = mmap(fd, size)
  2. print f.write(m)

it works. This triggers getsegcount and getcharbuffer in the mmapmodule.c. This is part of the Buffer Object protocol. How to simulate this in Python? I don’t find anything in the Python documentation.

Update: I know about buffer() existance but as the Python documentation says: Buffer objects are not directly supported by Python syntax, but can be created by calling the builtin function buffer(). They don’t support concatenation or repetition..

Comments

mmap methods are completed

June 28, 2006 at 3:28 pm · tags:

Changelog:

  • mmap.size() is done
  • mmap.tell() too
  • mmap.flush() is done
  • I had some problems with memmove() syscall from ctypes so I implemented mmap.move() with memcpy()
  • mmap.resize() works where supported (not supported on OSX and FreeBSD)

I discovered an ugly alignment problem caling msync() syscall but it has been fixed. The problem was in Linux only…

mmap methods are completed, now I have to make it behave as a string and a file like object for real.

Comments

Installing stuff in Parallels

June 27, 2006 at 5:10 pm · tags:

Parallels is damn cool and with 2.0Gb of RAM is even cooler. Why it takes 20 minutes to be up and running for the SoC with Ubuntu Dapper and after two hours of installation/configuration/reboot I’ve not completed the setup of the XP box yet?

I’m installing some SDK now…

posted with TextMate

Comments

some more mmap

June 25, 2006 at 6:39 pm · tags:

Today changelog:

  • mmap.find() is here
  • mmap.seek() is here
  • mmap.write() is here
  • many fixes to make it work under windows as well
  • big refactoring of tests
  • mmap.write_byte() is here

I do want mutable strings in Python!:-)

Comments

mmap is goin’ on

June 24, 2006 at 5:37 pm · tags:

Here’s the today changelog:

  • mmap frees resources correctly
  • close() is done and working
  • read_byte() is done and working
  • readline() is done and working
  • find() needs more work.

Comments

Do the bus a bus

June 23, 2006 at 11:15 pm · tags:

The bus error is gone but I’m not really comfortable with the reason why I don’t have that error anymore. Let’s explain a bit:

mmap() is a 6 argument function, all of them required.

Thomas Heller discovered under FreeBSD 6.0 with ktrace that it’s called with 8 parameters, not 6. The two argument in addition are all 0 if called with the 6-arg mmap from plain C.

If you call mmap() from ctypes the 7th parameter is zeroed, the 8th is a random hexadecimal. So it doesn’t work, because C calls it with 0.

A FreeBSD committer friend of mine told me instead that the syscall under mmap() is a 7 argument function, not 8.

Thomas Heller tried with 8 parameters and it worked. I, accordingly to my friend, tried with 7 instead and it worked well anyway.

So the mistery is still here but the error is gone.

You can discover odd things also under Unix, not only under Win32 :-D

Eventually, this is the correct code:

  1. from ctypes import *
  2. import ctypes.util
  3.  
  4. _libc = cdll.LoadLibrary(ctypes.util.find_library(“c”))
  5.  
  6. f = open(“foo”, “w+”)
  7. f.write(“foo\0“)
  8. f.flush()
  9.  
  10. _libc.mmap.restype = c_void_p
  11. m = _libc.mmap(0, 4, 3, 1, f.fileno(), 0, 0)
  12. print m
  13. if m != c_void_p(-1).value:
  14.     c = cast(m, POINTER(c_char))
  15.     print [c[i] for i in range(4)]
  16. f.close()

ps. the title is taken from a Busta Rhymes old song ;-)

Comments (1)

Bus error is still here

June 23, 2006 at 11:42 am · tags:

I don’t really know why the following example works well under Linux but crashes badly and with no mercy under MacOSX and FreeBSD:

  1. from ctypes import *
  2. import ctypes.util
  3.  
  4. _libc = cdll.LoadLibrary(ctypes.util.find_library(“c”))
  5.  
  6. f = open(“foo”, “w+”)
  7. f.write(“foo\0“)
  8. f.flush()
  9.  
  10. _libc.mmap.restype = c_void_p
  11. m = _libc.mmap(0, 4, 3, 1, f.fileno(), 0)
  12. print m
  13. if m != c_void_p(-1).value:
  14. c = cast(m, POINTER(c_char))
  15. print [c[i] for i in range(4)]
  16. f.close()

Anytime I try to cast the void * into a char * it crashes. Gdb still tells me a SIGBUS has occurred but I can’t fix it.

More info in the previous post: Bus error in FreeBSD/MacOSX

Any ideas?

Comments (3)

Bus error in FreeBSD/MacOSX

June 21, 2006 at 3:47 pm · tags:

I’m dealing with a nasty platform problem now. The following sample of code works well under linux and its POSIX implementation of mmap but under BSD/MacOSX it throws an ugly Bus Error:

  1. import cmmap
  2.  
  3. f = open(“foo”, “w+”)
  4. f.write(“c”)
  5. f.flush()
  6. m = cmmap.mmap(f.fileno(), 1)
  7. f.close()

gdb says:

Program received signal EXC_BAD_ACCESS, Could not access memory. Reason: 10 at address: 0×00045000 Bus error

And this is the ktrace output in MacOSX:

26481 Python   CALL  mmap(0,0x1,0x3,0x1,0x4,0)
26481 Python   RET   mmap 282624/0x45000
26481 Python   PSIG  SIGBUS SIG_DFL

I think I’m definitely missing something…

UPDATE: the mistery goes on…

This equivalent in C works well (obviously… the problem must be in my python code):

  1. #include
  2. #include
  3. #include
  4.  
  5. int main(void)
  6. {
  7. FILE* f;
  8. char * ret;
  9. int fd;
  10.  
  11. f = fopen(“foo”, “w+”);
  12. fd = fileno(f);
  13.  
  14. fprintf(f, “c”);
  15. fflush(f);
  16. ret = (char *) mmap(NULL, 1, 3, 1, fd, 0);
  17. printf(“%s\n“, ret);
  18.  
  19. return 0;
  20. }

If you want to try it by yourself: http://codespeak.net/svn/user/rhymes/modules/mmap/cmmap.py

Mmmm…

Comments

Next entries »