Socket handoff: Concurrent fd sharing for performance and innovation

*

Excerpt

When different components want to use a shared resource in different ways--such as when they're implemented in different programming languages, or have APIs that aren't trivially compatible--the result is an API design challenge. X desktops today have both Xlib and XCB competing for access to the same network socket, and we needed a design that would let them share. We'll present this design, how we arrived at it, and why it's even more useful than we guessed.

Description

Xlib is over twenty years old, an impressive lifespan for any software project. However, Xlib has not been well-matched to the needs of X clients for some time now. We felt it had earned retirement, so we created a new X-protocol C Binding (XCB) library, providing a simple, small, thread-transparent interface to X. But to support all the applications that haven’t magically ported themselves to XCB yet, we needed a transition plan.

We first tried re-writing Xlib from scratch as a layer on top of XCB. That was hard. So we stopped.

Then we added some hooks to XCB to allow Xlib to remain mostly unchanged but send requests through XCB. That worked, mostly; however, the complexity of this approach led to bugs, ugly hacks in XCB, and far more hacking on Xlib than continued sanity permitted. Adding special code in our clean new XCB library just to keep the old Xlib way working made us sad, and the looming prospect of becoming the Xlib maintainers made us lose sleep. So we stopped that too.

The current versions of Xlib and XCB now use a technique that we call “socket handoff”, cooperatively alternating exclusive access to the X connection instead of dumping Xlib’s requests through XCB. Socket handoff allowed us to delete a pile of buggy, hackish code, while improving performance, and that made us happy. So we didn’t stop.

The socket handoff interface has proven to be useful not just for Xlib, but also for new language bindings to XCB and for experimental X protocol-level hacks. We believe other network protocol implementations would also benefit from this approach.

By the way, you may be running our socket handoff code on your system right now, so why not come hear about the code used by every application on your desktop?

Speaking experience

Speakers

  • Jamey is climbing the halls

    Biography

    Jamey Sharp was placed on Ritalin, briefly, in fifth grade. His interests and activities have been varied ever since. Today his day job involves a computer test for attention deficit disorder, but his biggest projects have been the Portland State Aerospace Society, a student rocketry club at Portland State University; XCB, a new low-level binding to the X protocol, in the process of replacing Xlib; and Serialist, because his other projects didn’t leave him enough time to read his favorite webcomics without tool support.

    Jamey’s interests span computer science fields including cryptography, combinatorial search, compilers, and computational complexity; systems-level programming, such as file format and network protocol implementations, Linux kernel development, and boot-loader hacking; computer architecture and its impact on software design; and functional programming, preferably in Haskell.

    Sessions

      • Title: Serialist: lazy web-crawling in Haskell
      • Track: Hacks
      • Room: Fremont
      • Time: 3:454:30pm
      • Excerpt:

        Serialist (http://serialist.net/) provides a way to find, track and read serialized content (e.g., web comics). It’s implemented entirely in Haskell and demonstrates functional web application development, crawling, scraping and distributed architecture. Serialist uses interesting graph algorithms to add and step through content lazily.

      • Speakers: <a href="/users/432">Jamey Sharp</a>, <a href="/users/434">Josh Triplett</a>
      • Title: Unlikely tools for pair programming
      • Track: Cooking
      • Room: Steel
      • Time: 3:454:30pm
      • Excerpt:

        Co-conspirators Jamey Sharp and Josh Triplett get up to a lot of miscellaneous hacking mischief together. Much of this hacking occurs while staring at the same screen, and tag-teaming the keyboard. Sometimes this happens with the two of them in different places. We’ll demo our favorite tools and invite audience contributions to the discussion.

      • Speakers: <a href="/users/432">Jamey Sharp</a>, <a href="/users/434">Josh Triplett</a>
  • Face 100x100

    Biography

    Josh Triplett is a PhD student at Portland State University and a Free and Open Source Software hacker. Josh is involved in research on relativistic programming and advanced synchronization techniques for highly parallel systems. Josh builds and launches Linux-powered rockets with the Portland State Aerospace Society, and hacks on numerous other projects . Lately, Josh does a lot of his hacking in Haskell.

    Sessions

      • Title: Unlikely tools for pair programming
      • Track: Cooking
      • Room: Steel
      • Time: 3:454:30pm
      • Excerpt:

        Co-conspirators Jamey Sharp and Josh Triplett get up to a lot of miscellaneous hacking mischief together. Much of this hacking occurs while staring at the same screen, and tag-teaming the keyboard. Sometimes this happens with the two of them in different places. We’ll demo our favorite tools and invite audience contributions to the discussion.

      • Speakers: <a href="/users/432">Jamey Sharp</a>, <a href="/users/434">Josh Triplett</a>
      • Title: Serialist: lazy web-crawling in Haskell
      • Track: Hacks
      • Room: Fremont
      • Time: 3:454:30pm
      • Excerpt:

        Serialist (http://serialist.net/) provides a way to find, track and read serialized content (e.g., web comics). It’s implemented entirely in Haskell and demonstrates functional web application development, crawling, scraping and distributed architecture. Serialist uses interesting graph algorithms to add and step through content lazily.

      • Speakers: <a href="/users/432">Jamey Sharp</a>, <a href="/users/434">Josh Triplett</a>