How Do Python Coroutines Work?

*
Accepted Session
Short Form
Advanced
Scheduled: Wednesday, June 24, 2015 from 1:30 – 2:15pm in B201

Excerpt

Asynchronous I/O frameworks like Node, Twisted, Tornado, and Python 3.4’s new “asyncio” can efficiently scale past tens of thousands of concurrent connections. But async coding with callbacks is painful and error-prone. Programmers increasingly use coroutines in place of callbacks to get the best of both worlds: efficiency plus a natural and robust coding style. I’ll explain how asyncio’s coroutines work. They are built using Python generators, the “yield from” statement, and the Future and Task classes. You will gain a deep understanding of this miraculous new programming idiom in the Python standard library.

Description

Classical computer science focuses on CPU-bound problems. It is concerned with algorithms that complete computations quickly. But many networked programs spend their time not computing, but waiting for responses from remote machines. These I/O-bound programs present a very different challenge: wait for a huge number of network responses efficiently. The modern answer to this problem is asynchronous I/O, or “async”.

In the early days of async frameworks, programmers put their logic in callback functions that waited for network responses, made network requests, and then scheduled more callbacks to handle the next round of responses. Complex code descended into an unmanageable mess of spaghetti callbacks. Such code defeated traditional exception handling, and it was intimidating to debug.

Coroutines are a newly popular programming style. They are memory-efficient in I/O-bound applications, but they possess all the grace and simplicity callbacks lack. Best of all, we can keep using time-tested techniques for handling exceptions.

To begin to understand how coroutines work, we look at an example of coroutine code in action. It is a Python generator, but what does that mean? We explore the Python interpreter’s generator implementation, how it uses a stack frame and instruction pointer in an unorthodox way to pause and resume the generator at will.

Coroutines build upon generators: they can pause waiting for a network operation, and resume when the operation completes. I show a minimal implementation of the Future and Task classes used in asyncio to schedule coroutines.

Coroutines can be factored into subcoroutines, the same as regular functions can. We see how Python 3’s “yield from” statement allows us to factor coroutines. Most comfortingly, exception-handling works with coroutines the same as with functions. We get a sane stack trace that shows where our program failed.

Tags

async, python

Speaking experience

This is a sequel to “What Is Async, How Does It Work, and When Should I Use It?” which I gave at Open Source Bridge in 2013, and PyCon Montreal and OSCON in 2014:

http://emptysqua.re/blog/pycon-2014-video-what-is-async/

If you liked "What Is Async", you'll love "How Do Python Coroutines Work?" But if you didn't see the prior talk, you'll have no trouble keeping up with the material in this one.

Other recent talks:

* "The Weather of the Century" at MongoDB World in 2014.

* “Writing an Excellent Programming Blog” at Open Source Bridge in 2014.

* Coming up: “Python Profiling: The Guts And The Glory” and “Eventually Correct: Testing Async Apps” at PyCon Montreal in April 2015.

Videos of me speaking:

* MongoDB World 2014: http://www.mongodb.com/presentations/weather-century-part-3-visualization

* PyCon APAC 2014: https://www.youtube.com/watch?v=BOKcZjI5zME

Speaker

  • Profile

    Biography

    Staff Engineer at MongoDB in New York City. Author of Motor, an async MongoDB driver for Tornado, and of Toro, a library of locks and queues for Tornado coroutines. Contributor to Python, PyMongo, MongoDB, Tornado, and asyncio.

    Sessions

      • Title: Cat-herd's Crook: Enforcing Standards in 10 Programming Languages
      • Track: Cooking
      • Room: B201
      • Time: 2:303:15pm
      • Excerpt:

        At MongoDB we write open source database drivers in ten programming languages. Ideally, all behave the same. We also help developers in the MongoDB community replicate our libraries’ behavior in even more (and more exotic) languages. How can we herd these cats along the same track? For years we failed, but we’ve recently gained momentum on standardizing our libraries. Testable, machine-readable specs prove which code conforms and which does not.

      • Speakers: Samantha Ritter, A. Jesse Jiryu Davis
      • Title: How Do Python Coroutines Work?
      • Track: Chemistry
      • Room: B201
      • Time: 1:302:15pm
      • Excerpt:

        Asynchronous I/O frameworks like Node, Twisted, Tornado, and Python 3.4’s new “asyncio” can efficiently scale past tens of thousands of concurrent connections. But async coding with callbacks is painful and error-prone. Programmers increasingly use coroutines in place of callbacks to get the best of both worlds: efficiency plus a natural and robust coding style. I’ll explain how asyncio’s coroutines work. They are built using Python generators, the “yield from” statement, and the Future and Task classes. You will gain a deep understanding of this miraculous new programming idiom in the Python standard library.

      • Speakers: A. Jesse Jiryu Davis