New Tech Helps Users Point And Seamlessly Connect With Multiple Surfaces

New Tech Helps Users Point And Seamlessly Connect With Multiple Surfaces

PSFK spoke about the Future Of Gaming with the co-founder of teehan+lax, whose product let you control screens you encounter with your smartphone.

Timothy Ryan, PSFK Labs
  • 27 october 2011

Touch Vision Interface from Teehan+Lax Labs on Vimeo.

The PSFK consulting team has noticed that enhanced connectivity across devices is enabling seamless interactions between tablets, smartphones and other screens, creating new opportunities for planned and impromptu gaming experiences and offering users new ways to engage with their surroundings.

One company operating in this space is teehan+lax — a user experience design company. Its Touch Vision Interface is a tool that enables users to interact seamlessly with different connected surfaces through a mobile phones’ camera view. Users simply point their viewfinders to a connected screen and are able to ‘touch’ the distant screen virtually. The technology creates the feel of being able to directly manipulate a distant surface, resulting in an interaction that is natural and almost invisible. The client server application additionally works over IP based systems and doesn’t require any complex pairing for the device to take control of the screen. PSFK spoke with co-founder Jon Lax about his work.

Please provide a brief introduction about yourself and your company.

My name is Jon Lax. I am the co-founder of teehan+lax, a user experience design company. We help clients define and design customer experiences in the digital channel. We do this across a variety of devices and screen types. An important part of being able to come up with new ideas of users will interact with devices and screens we have a Labs group who’s mission is to help us understand what is possible in the future.

Please tell us about your Touch Vision Interface. What is the idea and goal behind the product and its platform?

TVI is a client server application where, the camera on your phone, recognizes controllable screens around you and lets you control the interactions on that screen with your phone acting as a touchscreen input device. This works over IP based systems and doesn’t require any complex pairing for the device to take control of the screen.

TVI comes out of our Labs group and its genesis and creation is due to Peter Nitsch and his team. Peter was experimenting with computer vision libraries. There was no product or idea when he started, just a curiosity to learn the tech. We meet weekly to discuss what the Lab is working on and Pete came in and said… “So you know how in AR you point your phone at the world and then this layer appears and you interact with it but all your interaction is contained within a layer on the phone? What if you could point your phone at the world and control it?”

That was the idea. Pete and his team spent the next two weeks trying to do that. When he first showed Geoff Teehan a prototype, he had figured out how to point the phone at a computer monitor and control the screen interaction with the phone. Geoff said “This idea is only powerful if it  works across multiple screens.” Peter and his team were able to get that working.

On a technical level this is what is happening. The phone has the TVI app on it. The app is able to look at the world around it through the camera and is trained to recognize specific borders on screens. When it discovers a screen it can control it “pairs” with it giving it control.

The screen is running an openFrameworks app that receives touch data from the phone over TCP. We like the idea that you are walking around with this controller (your phone) you should be able to point it at screens out in the world and take control of that screen.

Our goal for the project was just a Lab experiment. We wanted to see if we could accomplish what Pete had imagined. We don’t have specific applications for the tech although we’re exploring its practical applications for everything from retail experiences to gaming.


What has been the audience reaction? Can you share any stats around user engagement?

We got very positive reaction with posts in Engadget and Creative Applications. We’ve had inquiries from several companies (including some gaming companies) about licensing the technology. Since it was only meant to be a Lab experiment it is still in a very formative state. We’re going to be open sourcing parts of TVI  on Git and hopefully others will do some cool things with it.

We have noticed that enhanced connectivity across devices is enabling seamless interactions between tablets, smartphones and other screens, creating new opportunities for planned and impromptu gaming experiences and offering users new ways to engage with their surroundings. Do you see this trend manifesting on a larger scale?

As a company we identified this trend of seamless interactions late last year and have shifted our company’s focus to designing customer experiences in this world. This trend applies very broadly and is not confined to gaming.

TVI is a good demonstration of this trend. Why should I be tied to my game console and controller to play a game. We increasingly have these screens around us in our daily life, we should be able to use them.

There are some barriers to enhanced connectivity and seamless interactions. Discovery is a really complex tricky problem. How your phone or tablet discovers another device or screen is not simple for the user. TVI relies on an app because its the only way we can do discovery, this is not optimal.

While we can imagine going to Times Square pointing your phone at several screens and then playing a game with several hundred people standing around you and even online players, we’re not sure what that game looks like or if it would be an enjoyable game experience.

We need to go through some exploration and even some failures before this trend is realized or even manifesting on a larger scale.





How Indiegogo Is Becoming An Adult Product Destination

Home september 23, 2016

Watch This Fire Pit Dance Along To Your Favorite Song

Music City Fire is a system that is designed to flicker in time to ambient music

Automotive september 23, 2016

Slick GPS Navigator Gives Directions To Moped Riders

This small, round device attaches to a sideview mirror to display maps for safer traveling


Get PSFK's Related Report: Future of Automotive

See All
Home september 23, 2016

A Clock That Beautifully Manages Your Information Overload

The wall-mounted timekeeper is made to help people maintain focus and stay up to date with their appointments

Arts & Culture september 23, 2016

Performance Piece Blends Dancers Into Folds Of Light

The work provides commentary on the increasing connection between programmed and analog dimensions

Design & Architecture september 23, 2016

Design Firm Adapts Childhood Homes For Unemployed Young Adults

The studio has unveiled three prototypes of transformed living spaces for people forced to move back in with their families

Health september 23, 2016

These Chocolate Squares Claim To Reverse The Aging Process

A group of researchers from Cambridge University have developed a candy bar that promises to give you a youthful glow

Cities september 23, 2016

Food-Producing Architecture Competition Seeks To Better Feed Cities

A design challenge in Copenhagen highlights the need and beauty of urban farming


Future Of Automotive
Scenarios Driving The Digital Transformation Of An Industry

PSFK Op-Ed september 23, 2016

Productivity Expert: The Magic Of The Five-Hour Workday

Stephan Aarstol, Founder of Tower Paddle Boards, explains why the modern notion of office hours needs to evolve

PSFK Labs september 22, 2016

The Future Of Work: Why Innovation Is Every Employee’s Job

PSFK Labs sits down with management at Johnson & Johnson to learn how the company comes up with their next ‘big idea’

Health september 23, 2016

Wearable Monitors Sun Exposure To Prevent Sunburn

The clip monitors UV rays to make sure you're not receiving too much sunlight

Culture september 23, 2016

This Exhibition’s Crowning Jewel Is An 18k Gold-Plated Toilet

The piece, titled 'America,' is meant to raise questions about the country's wealth inequality

Culture september 23, 2016

Match Up With Dates On Tinder Based On Your Music Preferences

A new partnership with Spotify lets people pair up based on shared tastes in artists

Entertainment september 23, 2016

Capture, Cut Up, And Configure Your World In 360 Degrees

To meet a booming 360 and VR video-capturing industry, a multimedia software company bolstered its media editing suite for 360-video creation


Future Of Work
Cultivating The Next Generation Of Leaders

Advertising september 23, 2016

Buy Movie Tickets Directly From Your Facebook Profile

The new feature is part of a campaign from Fandango to further imbed social media into the ticket vendor's digital presence

Asia september 23, 2016

Co-Working Space Brings The Calming Atmosphere Of Nature Indoors

An architectural firm in China has designed a new type of shared office that prioritizes the natural environment

Retail september 23, 2016

Live Beacon Simplifies Cloud Sharing For Businesses

The battery-powered device and smartphone app allow anyone to send location-specific content without the hassle of code or hardware

No search results found.