Constructing a Next Generation Open Source Web Querying System with Node.js

*
Proposal
Short Form
Intermediate

Excerpt

Raw API data from different sites and services is the lifeblood that powers most web and mobile applications in the market. With this power comes increased network roundtrips, bandwidth consumption, and reduced product reliability from dealing with inconsistent and volatile APIs.

ql.io, a new open source querying system built on top of Node.js, is an answer to these development woes. Providing a mechanism for mashing up raw data sources in an easily consumed package, as well as the ability to deploy the service from your own servers, ql.io is working towards taking away many of the sharp pains that we have all endured within our development careers.

Description

In most modern web and mobile applications, much of the raw data source backbone comes from the mashing up of HTTP based APIs from multiple sites and services, both internally and externally. These sources are used for everything from localization to personalization and product querying. As more services are added to provide this functionality, the number of network roundtrips increases along with bandwidth consumption, and due to interdependency problems and inconsistent APIs, product reliability suffers.

These were the main developer pains that lead to the development of ql.io, an open source querying system built on top of Node.js. Using a SQL and JSON based language to declare API calls, interdependencies, forks and joins, filtering and projections, app developers are able to cut down the amount of effort required to integrate multiple APIs from hundreds of lines of code down to a few. Besides the language itself, ql.io can be deployed as a stand-alone server, allowing you to deploy it as an HTTP gateway between client apps and API servers to have ql.io process and condense the data to just the fields that the client needs.

In this talk we will explore the underpinnings of ql.io and how Node.js powers this next generation system. We will look into how to query multiple HTTP API data sources simply and efficiently, and explore how the service can be deployed on your own systems to handle raw mashups of your own and external data, all while decreasing overall bandwidth usage, network roundtrips, and increasing product reliability.

Speaking experience

I have been a developer evangelist for 5 years, giving talks at dozens of conferences with audiences ranging from a few people to 700+, within formats ranging from lightning talks to all day workshops.

Here is an audio track from SXSWi 2012 where I cover the concepts of human identity and social grouping: http://www.archive.org/details/ProgrammingSocialApplicationsUnderstandingHumanIdentity

Here are links to some of my talks / past work:
Slides: http://www.slideshare.net/jcleblanc
Events: http://lanyrd.com/profile/jcleblanc/past/speaking/
My book: http://shop.oreilly.com/product/0636920014201.do

A few conference session videos:
http://www.youtube.com/watch?v=eAGLpx6k340
http://www.youtube.com/watch?v=1PDaDMV5Gp8
http://www.youtube.com/watch?v=GeOkJeCGWAA
http://vimeo.com/17459184

Speaker

  • Pic jon2

    Jonathan LeBlanc

    X.commerce (eBay)

    Biography

    Jonathan is an Emmy award-winning software engineer and the author of the O’Reilly book “Programming Social Applications.” He specializes in open source initiatives around the implementation of social engagement services. He also works with and promotes emerging technologies to aid in the adoption and utilization of new social development techniques, such as his work on the OpenSocial foundation board. As a software engineer, Jonathan works extensively with social interaction development, engaging in new methods for targeting the social footprint of users to drive the ideal of an open web.