New Tech Helps Users Point And Seamlessly Connect With Multiple Surfaces

New Tech Helps Users Point And Seamlessly Connect With Multiple Surfaces

PSFK spoke about the Future Of Gaming with the co-founder of teehan+lax, whose product let you control screens you encounter with your smartphone.

Timothy Ryan, PSFK Labs
  • 27 october 2011

Touch Vision Interface from Teehan+Lax Labs on Vimeo.

The PSFK consulting team has noticed that enhanced connectivity across devices is enabling seamless interactions between tablets, smartphones and other screens, creating new opportunities for planned and impromptu gaming experiences and offering users new ways to engage with their surroundings.

One company operating in this space is teehan+lax — a user experience design company. Its Touch Vision Interface is a tool that enables users to interact seamlessly with different connected surfaces through a mobile phones’ camera view. Users simply point their viewfinders to a connected screen and are able to ‘touch’ the distant screen virtually. The technology creates the feel of being able to directly manipulate a distant surface, resulting in an interaction that is natural and almost invisible. The client server application additionally works over IP based systems and doesn’t require any complex pairing for the device to take control of the screen. PSFK spoke with co-founder Jon Lax about his work.

Please provide a brief introduction about yourself and your company.

My name is Jon Lax. I am the co-founder of teehan+lax, a user experience design company. We help clients define and design customer experiences in the digital channel. We do this across a variety of devices and screen types. An important part of being able to come up with new ideas of users will interact with devices and screens we have a Labs group who’s mission is to help us understand what is possible in the future.

Please tell us about your Touch Vision Interface. What is the idea and goal behind the product and its platform?

TVI is a client server application where, the camera on your phone, recognizes controllable screens around you and lets you control the interactions on that screen with your phone acting as a touchscreen input device. This works over IP based systems and doesn’t require any complex pairing for the device to take control of the screen.

TVI comes out of our Labs group and its genesis and creation is due to Peter Nitsch and his team. Peter was experimenting with computer vision libraries. There was no product or idea when he started, just a curiosity to learn the tech. We meet weekly to discuss what the Lab is working on and Pete came in and said… “So you know how in AR you point your phone at the world and then this layer appears and you interact with it but all your interaction is contained within a layer on the phone? What if you could point your phone at the world and control it?”

That was the idea. Pete and his team spent the next two weeks trying to do that. When he first showed Geoff Teehan a prototype, he had figured out how to point the phone at a computer monitor and control the screen interaction with the phone. Geoff said “This idea is only powerful if it  works across multiple screens.” Peter and his team were able to get that working.

On a technical level this is what is happening. The phone has the TVI app on it. The app is able to look at the world around it through the camera and is trained to recognize specific borders on screens. When it discovers a screen it can control it “pairs” with it giving it control.

The screen is running an openFrameworks app that receives touch data from the phone over TCP. We like the idea that you are walking around with this controller (your phone) you should be able to point it at screens out in the world and take control of that screen.

Our goal for the project was just a Lab experiment. We wanted to see if we could accomplish what Pete had imagined. We don’t have specific applications for the tech although we’re exploring its practical applications for everything from retail experiences to gaming.


What has been the audience reaction? Can you share any stats around user engagement?

We got very positive reaction with posts in Engadget and Creative Applications. We’ve had inquiries from several companies (including some gaming companies) about licensing the technology. Since it was only meant to be a Lab experiment it is still in a very formative state. We’re going to be open sourcing parts of TVI  on Git and hopefully others will do some cool things with it.

We have noticed that enhanced connectivity across devices is enabling seamless interactions between tablets, smartphones and other screens, creating new opportunities for planned and impromptu gaming experiences and offering users new ways to engage with their surroundings. Do you see this trend manifesting on a larger scale?

As a company we identified this trend of seamless interactions late last year and have shifted our company’s focus to designing customer experiences in this world. This trend applies very broadly and is not confined to gaming.

TVI is a good demonstration of this trend. Why should I be tied to my game console and controller to play a game. We increasingly have these screens around us in our daily life, we should be able to use them.

There are some barriers to enhanced connectivity and seamless interactions. Discovery is a really complex tricky problem. How your phone or tablet discovers another device or screen is not simple for the user. TVI relies on an app because its the only way we can do discovery, this is not optimal.

While we can imagine going to Times Square pointing your phone at several screens and then playing a game with several hundred people standing around you and even online players, we’re not sure what that game looks like or if it would be an enjoyable game experience.

We need to go through some exploration and even some failures before this trend is realized or even manifesting on a larger scale.





Machine Printer Uses Coffee Drips To Create Intricate Portraits

Arts & Culture
Technology december 2, 2016

Why Nest Doesn't Get The Holidays

PSFK founder reacts to the damaging effects of poor email marketing

Children december 2, 2016

Robots Could Be Joining Dubai’s Police Force In 2017

The real-life RoboCops can salute, shake hands and collect traffic fines


Get PSFK's Related Report: Future of Retail 2017

See All
Travel december 2, 2016

Parka Hides And Charges Portable Devices

Bolt is a jacket that lets people carry and charge their various electronics without the need for an outlet

Related Expert

Benedetta Piantella

Sustainable Solutions

Food december 2, 2016

Yelp's New 'Yelfie' Feature Lets Diners Take Selfies

The update is designed to encourage people to attach a selfie when they share their experiences

Design & Architecture december 2, 2016

Build Your Own Savory Cheese Advent Calendar

A British food blogger has created a guide to building a different kind of holiday surprise

Fitness & Sport december 2, 2016

Floating Gym Concept In Paris Is Powered By Your Workout

The proposed design from Carlo Ratti Associati lets passengers ride a stationary bike as they travel through Paris along the Seine River


Future Of Retail 2017
Transformation Strategies For Customer-First Business

PSFK Op-Ed december 1, 2016

Communications Officer: What This Holiday Season Means For Millennial Shoppers

Dallas Lawrence of the Rubicon Project shares why holiday cheer is all about likes, views and retweets

PSFK Labs december 1, 2016

Retail Spotlight: Home Depot Reimagines How Employees Conduct Tasks

The home improvement retailer puts the customer first by initiating local fulfillment centers and simplifying freight-to-shelf inventory management

Syndicated december 2, 2016

What Does The Future Of Android Look Like In A World With The Pixel?

Google’s decision to make its own phone might have looked like a blow to the likes of Samsung but the reality is much more interesting

Retail december 2, 2016

Customer Service Expert: Why Offline Retail Has Better Data Than Online Retail

Healey Cypher, Founder and CEO of Oak Labs, shares why we should be thinking about the physical store as an e-commerce site

Fashion december 2, 2016

Alexander McQueen Designs A 3D-Printed Umbrella

3D-printed fashion arrives in time for the winter season

Work december 2, 2016

Why Training Associates To Be Advocates Is Key To Retail Success

In our Future of Retail 2017 report, PSFK Labs discusses strategies to prioritize customer service, which begins with associate advocates

Media & Publishing december 2, 2016

Netflix Creates Binge Candle To Celebrate A New Season Of Gilmore Girls

The streaming service developed a special layered candle that creates candle with episode-specific smells


Future Of Work
Cultivating The Next Generation Of Leaders

Arts & Culture december 2, 2016

Interactive Film Tells A Story About Living With Cancer

A moving song written by a father of a cancer patient comes alive in a 3D environment

Automotive december 2, 2016

Audi And LEGO Exhibit Autonomous Vehicle Installation

The installation at Design Miami explores the 25th hour, which represents bonus productive work or play time

No search results found.