The reactor runs a giant loop in which events are processed in a non-blocking fashion. Sometimes, though, everything a man needs it’s just to make it stop. At least for a while, at least for the sake of getting data from the user.

The most prominent example of a command line client program that needs to stop and wait is a REPL. The following kind of REPL is a bit unorthodox, you’ll see.

The secret sauce (at least the one I found) to write such kind of interactive program in Twisted is twisted.internet.stdio.StandardIO. It connects your protocol to standard input and output:

class Repl(basic.LineReceiver):
    delimiter = '\n'
    prompt_string = 'cmd> '
 
    def prompt(self):
        self.transport.write(self.prompt_string)
 
    def connectionMade(self):
        self.sendLine('Welcome to Console')
        self.prompt()
 
    def lineReceived(self, line):
        # blank line
        if not line:
            self.prompt()
            return
 
        self.issueCommand(line)
 
    def issueCommand(self, command):
        # send the command to the server
        d = sendCmd("%s%s" % (command, self.delimiter))
        d.addCallback(self._checkResponse)
 
    def _checkResponse(self, args):
        success, num_lines, data = args
        if num_lines > 20:
            # use less to display the response
            self.lessify(data)
        else:
            self.sendLine(data)
        self.prompt()
 
    def lessify(self, data):
        p = subprocess.Popen(["less"], stdin=subprocess.PIPE)
        p.communicate(data)
 
    def connectionLost(self, reason):
        reactor.stop()

When the protocol it is connected to the standard output with

stdio.StandardIO(TaskConsole())
reactor.run()

the program displays the prompt, hence when a line is received from the standard input it is sent to the other protocol attached on the network and a callback is registered for the response. I also decided to use less to display the response if it’s more than some lines but that’s a detail.

The sendCmd function instantiate the networked protocol and its factory:

def sendCmd(cmd):
    factory = CFactory(cmd)
    reactor.connectTCP('127.0.0.1', 1234, factory)
    return factory.deferred

Then, when the server replies with some content we check to see if everything went ok and the reconstruct the whole response sending it back the REPL:

class Client(basic.LineReceiver):
    delimiter = '\n'
 
    def connectionMade(self):
        # send the command received by the cmdline to the server
        self.sendLine(self.factory.cmd)
        self.buffer = []
        self.cmd_success = True
 
    def lineReceived(self, line):
        # basic check error/success
        if line.startswith('OK'):
            return
        if line.startswith('ERR'):
            self.cmd_success = False
            return
 
        if line == 'END':
            # join the response at the end of it
            self.responseFinished(
                len(self.buffer), "\n".join(self.buffer))
            self.buffer = []
        else:
            self.buffer.append(line)
 
    def responseFinished(self, num_lines, data):
        # disconnect
        self.sendLine('quit')
        self.transport.loseConnection()
 
        # send back the response to the REPL
        self.factory.deferred.callback((
            self.cmd_success, num_lines, data))
 
 
class CFactory(protocol.ClientFactory):
    protocol = Client
 
    def __init__(self, cmd):
        self.cmd = cmd
        self.deferred = defer.Deferred()

How cool is that? Not really to be honest. It has a big gigantic fault: every time you input a line a connection to the server is opened and closed. That’s bad, really bad.

It didn’t take too long to create a version that use just one connection:

def connectionMade(self):
    self.sendLine('Welcome to Console')
    self.factory = CFactory()
    self.connector = reactor.connectTCP(
        '127.0.0.1', 1234, self.factory)
    self.prompt()

We store the connector and the factory. issueCommand does not open a connection anymore, just:

def issueCommand(self, command):
    self.connector.transport.write("%s%s" % (command, self.delimiter))
    self.factory.deferred.addCallback(self._checkResponse)

We write directly to the transport of the connector (and not the one connected to the stdout) and register the callback on the factory’s deferred.

That’s better in my opinion and a nice start. I know that within twisted.conch.stdio there’s something more evolved. I’ll try to look into it when I have more time.

You can find the first version (bad) and the second version (better) online.

What do you think about this try?

By the way, this is what I am doing/I have done recently:

• Patched httplib2 to support MD5-sess (at work)
• Patched soaplib to use httplib2 with the above patch (at work)
• All the above is because we are interfacing with a couple of SOAP servers (in Java and .NET)
• Reading The Ruby Way by Hal Fulton and JavaScript the Good Parts by Douglas Crockford (read) Dreaming In Code by Scott Rosenberg
• Planning my vacation: I’ll be staying in London for 3 weeks, from the 27th of July to the 17th of August. If you want to catch up and drink something contact me at l dot oluyede at gmail dot com
• Bought a Nokia E51 which is UMTS/HDSPA/WiFi ready Symbian phone (and also cheap)
• Still reading a lot of stuff in my backlog

There is by the way an incompatible change: I do not distribute anymore BeautifulSoup and httplib2 with the library.

Go take it!

http://twitter.com/pyconit

Now I can talk about Google AppEngine as well

It is simply a pastebin using the DataStore API, the webapp framework, the Users API, pygments and Django templates.

It is heavily inspired by http://dpaste.com

http://share11.appspot.com

The great thing? It took me less than an afternoon (mostly reading the framework docs) and it is ~200 LOC. Even greater? One shell command to upload and deploy

The savory new flavor of Py2.6 and Py3.0 is the title of the second day’s keynote. Who’s the speaker? Raymond Hettinger

The talk about callbacks and Python patterns have been replaced by a talk about the now utterly famous Google AppEngine.

The speakers of the PyPy talk are Antonio Cuni and Samuele Pedroni.

The title of Stallman’s keynote is Free Software in ethics and in practice.

PyCon Due is definitely taking shape.

The conference will take place on a three day span.

The first day venue (free of any kind of charge) will be Palazzo Vecchio, in Florence. The conference will be introduced by the spokeswoman of Arts of the city. The opening keynote will be given by Richard Stallman, a person who needs no introduction in the software world.

After that, which is definitely important, we plan to go out and eat all together at a restaurant. Social life is equally important

The actual conference will begin the next day, the 10th of May, divided in three parallels track: a tutorial track, an introduction one and one based on experiences and real life usage of the language.

We’ll have talks about Python for beginners, various incarnations of Python, Plone, information retrieval, PyQt, nginx and WSGI, Cython, advanced uses of Django, PyMaemo, Unicode, PyPy, Zope 3 and more on that day. There will be also a Skype sponsored talk about Skype4Py.

The second day will be full of talks about profiling and debugging, SQLAlchemy, Django again, callbacks and patterns, map sharing, concurrency, Twisted Matrix, IronPython, Ajax, compilers in Python, the Fedora Unified Network Controller, FlyPDF and component architectures and more.

As you can imagine will be a tough three days conference, one an Italian (Python) developer should not miss in my opinion.

A lot of well known speakers: Alex Martelli (another who needs no introduction), Federico di Gregorio (author of psycopg2), Arkadiusz Wahlig (author of Skype4Py), Manlio Perillo (of nginx’s mod_wsgi fame), Antonio Cuni (from the PyPy team), Giovanni Bajo (mantainer of PyInstaller and of GCC fame), Brian Fitzpatrick (Subversion anyone?), Menno Smits (from the Resolver Systems team), Michele Simionato (of metaclasses, decorators and mro fame) and more.

As part of the board of the conference I’m a little biased by I’m really looking forward for this one

See you there!

PyCon Due Italy is coming, here the press release (in Italian):

Firenze, 27 Marzo 2008: Sono finalmente aperte le iscrizioni a PyCon Due, la seconda conferenza italiana dedicata al linguaggio di programmazione Python, che si terrà a Firenze il 9, 10 e 11 Maggio 2008.

Dopo il largo successo di pubblico e critica ottenuto da PyCon Uno l’anno scorso, l’Associazione di Promozione Sociale “Python Italia” organizza quest’anno un evento ancora più ambizioso. Sono attesi infatti più di 300 tra professionisti, studenti e ricercatori, per una tre giorni intensa di appuntamenti e interventi imperdibili.

L’evento di apertura si svolgerà Venerdì 9 Maggio alle ore 15:00 nel prestigioso Salone De’ Cinquecento in Palazzo Vecchio dove, dopo il saluto da parte di Lucia De Siervo (Assessore all’Informatizzazione del Comune di Firenze, che ha donato all’evento il suo Patrocinio), Richard Stallman (fondatore del movimento del Free Software, inventore della licenza GPL e ideatore del progetto GNU) terrà un keynote sul tema “Free Software e Free Ethics”. Questo evento di apertura è ad ingresso libero (fino ad esaurimento posti, ma con priorità a chi è registrato al PyCon).

In seguito, nelle giornate di Sabato 10 e Domenica 11, all’Auditorium Al Duomo, si terranno numerose conferenze dedicate al linguaggo Python, su tre track parallele. Parteciperanno speaker internazionali di fama mondiale come Alex Martelli, Samuele Pedroni, Brian Fitzpatrick, e molti altri. Sono previsti anche interventi dedicati a programmatori che si avvicinano al linguaggio per la prima volta.

Inoltre, sono previste sessioni speciali come la sessione di recruiting (dedicate ai programmatori in cerca di nuova occupazione, e alla quale parteciperanno le aziende con posizioni lavorative aperte), momenti di svago e relax serale, e l’estrazione di regali offerti dagli sponsor (che includono aziende di caratura internazionale come Skype e Google).

Per maggiori informazioni, potete visitare il sito Internet ufficiale dedicato all’evento. Registrandosi in anticipo, inoltre, sono previsti forti sconti sul biglietto d’ingresso.

If you are at least midly interested in the Python word, come!

1 - Standard, cross-language, typeful serialization of data

REST is definitely standard and cross-language. Regarding serialization, you can transfer what you want where you want but if your goal is persisting to make a local resemblance of what you have remotely, I think that’s bad. What we, as developers, really care is having the possibility to use a service remotely with an eye on scalability and transparence. Hence, HTTP

2 - User-defined error codes and messages

Nobody forbids anyone to invent your own headers and place whatever meaning you want upon them. With HTTP you can surely do that. Even whole extensions have been written.

3 - “Boxcarring” of requests to reduce overhead

Box carring is not really an advantage, there are other ways to reduce overhead. See also 207 multi status

4 - Serialization of binary content

Who said REST protocols do not support binary data? You can send whatever you want from the client to server, place the right content type and you’re done. See also media resources in the Atom Publishing Protocol.

5 - Serialization of date-time values

Do I really have to answer that?

6 - Standardized parameter passing

And what if I want to pass a parameter that’s not on the list of the allowed types in the specification? Remember, SOAP has born also to address some of the limitations of XML-RPC.

7 - Introspection allowing for straightforward code generation

Let me say one thing once and for all. Code generation for this purposes is bad, and almost useless (if you are used to highly dynamic languages). Anyway, REST through hypertext is implicitly introspection enabled. Also, documentation of APIs is there for us

8 - High-level APIs for just about every language

I think there’s no need to say that each and every language has some sort of HTTP client.

9 - No manual parsing of XML, ever

This point does not really make sense to me. If one XML-RPC method does return XML as content, you have to parse it because is part of the application domain. The same for SOAP or REST. If, as Dave mentioned, the language has some sort of high-level API for the protocol you don’t have to parse what’s on the wire manually: no manual parsing of XML-RPC responses, or SOAP envelopes or HTTP responses.

10 - Only three lines to call a function in Python and several other languages

This is really pointless. Using a REST API from httplib2 or restclient has the same brevity.

I think Dave did not get REST at all, his conclusion speaks for him:

Not that the REST doesn’t have its benefits, but someone ought to be saying this. XML-RPC isn’t complicated like SOAP, it runs just about everywhere, and it lets you get on with your work rather than arguing about semicolons versus slashes or XML versus JSON or countless other things. Besides, when your goal is to support as many languages as possible, you want to minimize the amount of code you write for each language. As far as I’ve seen, nothing else accomplishes literally no-code binding like XML-RPC.

I suggest reading RESTful Web Services, it’s really a good book. I also suggest reading the latest article wrote by Stefan Tilkov addressing the most popular REST doubts.

• Added os.O_NOATIME constant. It serves the purpose to not update the access time within read operations.

• Added os.fchmod() and os.fchown().

• The new module has been deprecated. Use types instead.

• sys.py3kwarning flag has been exposed. True when python is started with -3 flag on command line. Also warning.warnpy3k() has been added to help the transition.

• Added PyFloat_GetMax, PyFloat_GetMin, PyFloat_GetInfo to the C API and float_info dictionary to the sys module which contains information about the internal floating point type.

• Added test suite for the cmd module.

• Added Python on Windows documentation.

• Implemented PEP 366 (Main module explicit relative imports).

• Support loading pickles of random.Random objects created on 32-bit systems on 64-bit systems, and vice versa. As a consequence of the change, Random pickles created by Python 2.6 cannot be loaded in Python 2.5.

• Changed GeneratorExit’s base class from Exception to BaseException.

• os.access now always returns True on Windows for any existing directory since there are no read only directories on Windows.

• Added msvcrt.getwch(), msvcrt.getwche(), msvcrt.putwch(), msvcrt.ungetwch() to handle wide chars.

• namedtuple.asdict and namedtuple.replace are now namedtuple._asdict() and namedtuple._replace()

• Dictionary construction had a speed-up of about 10%. There is a new opcode, STORE_MAP that saves the compiler from awkward stack manipulations and specializes for dicts using PyDict_SetItem instead of PyObject_SetItem. From the log message:

Old disassembly:
              0 BUILD_MAP                0
              3 DUP_TOP             
              4 LOAD_CONST               1 (1)
              7 ROT_TWO             
              8 LOAD_CONST               2 ('x')
             11 STORE_SUBSCR        
             12 DUP_TOP             
             13 LOAD_CONST               3 (2)
             16 ROT_TWO             
             17 LOAD_CONST               4 ('y')
             20 STORE_SUBSCR 

New disassembly:
              0 BUILD_MAP                0
              3 LOAD_CONST               1 (1)    
              6 LOAD_CONST               2 ('x')
              9 STORE_MAP                
             10 LOAD_CONST               3 (2)  
             13 LOAD_CONST               4 ('y')
             16 STORE_MAP

• Another optimization has been added: BUILD_MAP now has a meaning. It’s the estimated size of the dictionary. Allows dictionaries to be pre-sized (upto 255 elements) saving time lost to re-sizes with their attendant mallocs and re-insertions. Has zero effect on small dictionaries (5 elements or fewer), a slight benefit for dicts upto 22 elements (because they had to resize once anyway), and more benefit for dicts upto 255 elements (saving multiple resizes during the build-up and reducing the number of collisions on the first insertions). Beyond 255 elements, there is no addional benefit.

• Renamed Py_Size, Py_Type and Py_Refcnt to Py_SIZE, Py_TYPE and Py_REFCNT. Macros for compatibility are available.

• Added signal.set_wakeup_fd(). There is a correspondent C API as well: PySignal_SetWakeupFd.

• Slightly change in __ hash __ behavior and rich comparison: __ hash __ can be inherited when one __ lt __, __ le __, __ gt __, __ ge __ are overridden, as long as __ eq __ and __ ne __ aren’t.

• Improved performance of built-in any()/all() by avoiding PyIter_Next().