Nice 'Gestures' via FOSS and ML

Short Form


While movies like Harry Potter and Iron Man taught us many things, the only lesson that stood out for me was "there are really cool ways to interact with things around us". A missing piece of the puzzle to convert our hands into wands is gestures. There has been an incredible rise in the number and types of proprietary products for virtual reality, augmented reality and smart home systems in the last couple of years. However, the corporate atmosphere and market competition has stymied the true potential of such devices and they often restrict users, developers and hackers to a particular platform. I believe FOSS, a place where collaboration over competition is the driving force, is the key to 'alohomora' the doors to the next level of HCI. I hope you'll join me as I talk about one step in this direction!


The talk will kick off with hacking the Thalimic Myo, a popular proprietary hand gesture interface, to recognize gestures beyond those defined by their API’s (5 to be precise). Through this example, we will develop some basic concepts about gesture definition and gesture recognition using gyro sensors and accelerometers and any other information that may be available to us. Then we’ll see how open source ML libraries can help us define and recognize custom gestures. We can then extend these concepts to everyday devices like smart watches and phones and get them to work as wands in our hands. If time permits, there will also be a demonstration of the sorcery we just unveiled. Sneak peak of an old version demonstrating what we can do with these newly defined gestures is here:


machine learning, HCI, Natural User Interface, OpenNI, harry potter

Speaking experience

1) OpenMRS Radio (September 2016) :

2) Explorer of Grid Load (CERN, Geneva) : Best project presentation at CERN Openlab Summer Student Lightning Talks :

3) HackerspaceSG Awesome Talks, (Singapore, 2016) "The value of open source healthcare IT solutions" :

4) Become an OAuth2 provider using Spring Security (FOSSASIA 2016, Singapore) :

5) OpenMRS Update : FOSSASIA special :

6) OpenMRS Summit 2015, Lightning Talks :

I will be presenting this talk at a seminar at National Institute of Technology Delhi in May. I will utilize the experience and feedback to improve upon it.


  • Pc110188large

    Mayank Sharma



    I am a motivated and inquisitive Electronics and Communication Engineering Senior at National Institute of Technology, Delhi. I have a strong tech stack(Java, Web, Android, Game Engine, Hardware development) and rich project experience including CERN Openlab, Google Summer of Code 2014 & 2015, FOSS contributions, engineering/designing API’s for software firms/startups and a couple of major hackathons (Education Category winner at HackHarvard 2015). I like to learn new tech and build projects that show why technology is awesome. I love contributing to FOSS. I was the Release Manager for OpenMRS Platform 2.0 and am currently working with their infrastructure team as the developer tools manager.