inicio mail me! sindicaci;ón

Producer/Consumer with multitask library

May 29, 2007 at 7:59 pm · tags:

Generators are great if you ask me and so are the power ups that Python 2.5 adds to them. Since the last version of Python you have:

  • yield as an expression so you can store a returned value
  • values can be sent to the generator with gen.send(value), so you can somewhat handle the generator from the outside easily
  • exceptions can be raised inside the generator from the outside with gen.throw()
  • the generation process can be stopped anytime with gen.close()

This way Python 2.3 simple generators turn into full fledged coroutines.

Recently a library called multitask appeared on PyPi index. With multitask you can do cooperative multitasking and async I/O only with bare generators.

So fifteen minutes later I rewrote the classic producer/consumer example with it.

The source code is not really different from the generators only example I demoed last month:

  1. import multitask
  2.  
  3. class Producer(object):
  4. # removed for brevity #
  5.  
  6. class Consumer(object):
  7. def __init__(self, buffer, name):
  8. self.buffer = buffer
  9. self.name = name
  10.  
  11. def run(self):
  12. while not self.buffer.empty():
  13. # task will be resumed after an element is available in the queue
  14. data = (yield self.buffer.get())
  15. print “%s has got %s from the buffer.” % (self.name, data)
  16. # the task will be resumed after 1.5 seconds
  17. yield multitask.sleep(1.5)
  18. print “%s is doing something with %s” % (self.name, data)
  19.  
  20. def main():
  21. buffer = multitask.Queue()
  22. prod = Producer(buffer, “Producer”)
  23. cons = Consumer(buffer, “Consumer”)
  24.  
  25. multitask.add(prod.run())
  26. multitask.add(cons.run())
  27. multitask.run()

This time, anyway, the scheduler is provided by multitask which basically iterates on the tasks and sends in the values getting the outputs. Generators can make your code really simple :-)

Here’s the full example: prodcons_multitask.py

Comments (1)

Producer/Consumer with greenlets

April 14, 2007 at 10:46 pm · tags:

This is an addition of the post Producer/Consumer in Python I wrote earlier.

It’s just the same example rewritten with py.magic.greenlet.

Greenlets, tasklets, micro-threads, coroutines are basically the same thing: suspending and resuming a function in multiple points is the core feature we need.

In the previous example we did it with plain old generators, now we try to use the greenlets in py library.

I won’t comment the code line by line because nothing really changes except the API of the library involved. In our case the key feature is greenlet.switch() which switches from one greenlet to another. That’s really it.

You can find the full example here: prodcons-greenlet.py

Comments

Producer/Consumer in Python

April 9, 2007 at 3:26 am · tags:

Like Hello World in imperative programming the Producer/Consumer problem is one of kind of program everybody has wrote once in a lifetime.

It’s one of the toy examples in concurrent programming demonstrating the basics of synchronization.

Basically there’s a shared buffer, a producer entity and a consumer one. The producer generates some data and puts it in the buffer (which can have one or multiple slots), then the consumer gets the data from the same buffer and does something with it.

Now we’re going to see a couple (and half) implementations in Python. I bet there are tons of ways to solve this little problem and hence I’ll show you a threaded implementation with locking around around data structures which are not thread safe, a slight modification using Queue objects and an alternative implementation using just simple generators.

Read the rest of this entry »

Comments (6)