Black Pipe Testing, or "@#$! Up Your App by Impersonating a Database"

*
Accepted Session
Short Form
Intermediate
Scheduled: Tuesday, June 21, 2016 from 4:45 – 5:30pm in B302/303

Excerpt

A “black box” test sends input to your program and tests the output. But a networked application has I/O at two ends: the API and the network. A black box test can’t validate it, especially its error-handling. But a “black pipe” test can! Such a test talks to your code over the network at the same time as it tests the API. I’ll present a handy library for Black Pipe tests of MongoDB apps and advise you when to use it. I want you to write a library like it for your favorite DB, so we can all test our programs better!

Description

Black box tests depend only on the inputs and outputs of the program under test. But network applications have two interfaces: the public API, and the messages they send and receive on the network. Let’s call the validation of both interfaces a “black pipe” test.

Black pipe tests are a mid-point between unit tests and integration tests: the system under test is really talking to a server, but the server is a simulation. Such tests are best for two situations. The first is when correct network behavior cannot be proven by observing the API behavior. I’ll show an example of a block-box testing problem like this that arose in my work on a Python driver for MongoDB.

I wrote a black-pipe testing toolkit for MongoDB applications called MockupDB. It’s a MongoDB simulator. I’ll demonstrate how MockupDB’s expressive and convenient features let us easily write black-pipe tests. Now we can know for certain whether PyMongo sends the right messages over the network or not!

The second situation that requires black pipe tests is simulating network conditions like timeouts, hangups, or rejected connections. How confident are you that your app handles these errors correctly? Can you reliably trigger a given error-handling path? I’ll demonstrate how a black-pipe test exercises a web app’s error-handling code with detailed, reproducible network errors.

Any networked application benefits from this method! Go forth and write more black pipe frameworks. Simulate MySQL, Redis, RESTful HTTP APIs, or any other widely-used network server, so we can all test our programs better.

Speaking experience

I have given 5- and 10-minute versions of this talk, I think it would be valuable to share in a longer format. It's based on a series of articles I wrote:

https://emptysqua.re/blog/black-pipe-testing-series/

Here are some videos of past talks:

* MongoDB World 2015: https://www.mongodb.com/presentations/mongodb-drivers-and-high-availability-deep-dive
* Open Source Bridge 2015: https://www.youtube.com/watch?v=GSk0tIjDT10
* PyCon 2015: http://emptysqua.re/blog/pycon-video-eventually-correct-async-testing/
* MongoDB World 2014: http://www.mongodb.com/presentations/weather-century-part-3-visualization
* PyCon Montreal 2014: http://emptysqua.re/blog/pycon-2014-video-what-is-async/
* PyCon APAC 2014: https://www.youtube.com/watch?v=BOKcZjI5zME

Speaker

Leave a private comment to organizers about this proposal