New Tech Helps Users Point And Seamlessly Connect With Multiple Surfaces

New Tech Helps Users Point And Seamlessly Connect With Multiple Surfaces

PSFK spoke about the Future Of Gaming with the co-founder of teehan+lax, whose product let you control screens you encounter with your smartphone.

Timothy Ryan, PSFK Labs
  • 27 october 2011

Touch Vision Interface from Teehan+Lax Labs on Vimeo.

The PSFK consulting team has noticed that enhanced connectivity across devices is enabling seamless interactions between tablets, smartphones and other screens, creating new opportunities for planned and impromptu gaming experiences and offering users new ways to engage with their surroundings.

One company operating in this space is teehan+lax — a user experience design company. Its Touch Vision Interface is a tool that enables users to interact seamlessly with different connected surfaces through a mobile phones’ camera view. Users simply point their viewfinders to a connected screen and are able to ‘touch’ the distant screen virtually. The technology creates the feel of being able to directly manipulate a distant surface, resulting in an interaction that is natural and almost invisible. The client server application additionally works over IP based systems and doesn’t require any complex pairing for the device to take control of the screen. PSFK spoke with co-founder Jon Lax about his work.

Please provide a brief introduction about yourself and your company.

My name is Jon Lax. I am the co-founder of teehan+lax, a user experience design company. We help clients define and design customer experiences in the digital channel. We do this across a variety of devices and screen types. An important part of being able to come up with new ideas of users will interact with devices and screens we have a Labs group who’s mission is to help us understand what is possible in the future.

Please tell us about your Touch Vision Interface. What is the idea and goal behind the product and its platform?

TVI is a client server application where, the camera on your phone, recognizes controllable screens around you and lets you control the interactions on that screen with your phone acting as a touchscreen input device. This works over IP based systems and doesn’t require any complex pairing for the device to take control of the screen.

TVI comes out of our Labs group and its genesis and creation is due to Peter Nitsch and his team. Peter was experimenting with computer vision libraries. There was no product or idea when he started, just a curiosity to learn the tech. We meet weekly to discuss what the Lab is working on and Pete came in and said… “So you know how in AR you point your phone at the world and then this layer appears and you interact with it but all your interaction is contained within a layer on the phone? What if you could point your phone at the world and control it?”

That was the idea. Pete and his team spent the next two weeks trying to do that. When he first showed Geoff Teehan a prototype, he had figured out how to point the phone at a computer monitor and control the screen interaction with the phone. Geoff said “This idea is only powerful if it  works across multiple screens.” Peter and his team were able to get that working.

On a technical level this is what is happening. The phone has the TVI app on it. The app is able to look at the world around it through the camera and is trained to recognize specific borders on screens. When it discovers a screen it can control it “pairs” with it giving it control.

The screen is running an openFrameworks app that receives touch data from the phone over TCP. We like the idea that you are walking around with this controller (your phone) you should be able to point it at screens out in the world and take control of that screen.

Our goal for the project was just a Lab experiment. We wanted to see if we could accomplish what Pete had imagined. We don’t have specific applications for the tech although we’re exploring its practical applications for everything from retail experiences to gaming.


What has been the audience reaction? Can you share any stats around user engagement?

We got very positive reaction with posts in Engadget and Creative Applications. We’ve had inquiries from several companies (including some gaming companies) about licensing the technology. Since it was only meant to be a Lab experiment it is still in a very formative state. We’re going to be open sourcing parts of TVI  on Git and hopefully others will do some cool things with it.

We have noticed that enhanced connectivity across devices is enabling seamless interactions between tablets, smartphones and other screens, creating new opportunities for planned and impromptu gaming experiences and offering users new ways to engage with their surroundings. Do you see this trend manifesting on a larger scale?

As a company we identified this trend of seamless interactions late last year and have shifted our company’s focus to designing customer experiences in this world. This trend applies very broadly and is not confined to gaming.

TVI is a good demonstration of this trend. Why should I be tied to my game console and controller to play a game. We increasingly have these screens around us in our daily life, we should be able to use them.

There are some barriers to enhanced connectivity and seamless interactions. Discovery is a really complex tricky problem. How your phone or tablet discovers another device or screen is not simple for the user. TVI relies on an app because its the only way we can do discovery, this is not optimal.

While we can imagine going to Times Square pointing your phone at several screens and then playing a game with several hundred people standing around you and even online players, we’re not sure what that game looks like or if it would be an enjoyable game experience.

We need to go through some exploration and even some failures before this trend is realized or even manifesting on a larger scale.





Japanese Face Wash Creates A Perfect Rose Every Time

Arts & Culture
Mobile Yesterday

Get A Better Idea Of How You Are Wasting Your Time

The TouchTime app is trying to revolutionize personal task management by providing detailed insight on how to be more efficient

Culture Yesterday

London Telephone Box Repurposed As A Tiny Mobile Repair Shop

Tools and supplies to replace broken screens or damage are neatly stowed away in these micro-workrooms


Get PSFK's Latest Report: Future of Work

See All
Design Yesterday

Conceptual Sportswear Created Out Of Futuristic Condom Material

A Dutch fashion designer is experimenting with new methods and fabrics to make high performance clothing

Fashion Yesterday

Fashionable Tassel Will Ensure You Never Lose Your Valuables Again

The device is fashion meets connected tech, that will help you keep track of your belongings at all times


Kane Sarhan

Entrepreneurship, Apprenticeship, Education

Syndicated Yesterday

Would You Wear Wool Shoes To Save The Environment?

As demand for wool shoes grows, a number of US footwear brands are heading directly to the source: the sheep pastures of New Zealand

Sustainability Yesterday

Self-Healing Material Is Fashioned Out Of Squid Teeth

Penn State researchers have devised a new textile that uses organic proteins

Arts & Culture Yesterday

Search Engine Turns Your Own Drawings Into Photos

This image-matching software accepts hand-made sketches instead of keywords


Future Of Work
Cultivating The Next Generation Of Leaders

PSFK Op-Ed august 24, 2016

Why Building Better Offices Is The Key To Employee Engagement

Interaction Designer and Audio-visual Technologist at ESI Design illustrates the value in creating environments filled with surprise and delight

PSFK Labs august 25, 2016

PSFK’s Workplace Vision: How The Nurturing Of Seeds Will Come To Define The Onboarding Process

Our Future of Work vision is a service that allows companies to assemble and deliver welcome packets that are uniquely focused on the concept of growth

Arts & Culture Yesterday

Illustrator Interprets The Experiences Of Blind Travelers

Artist Alby Letoy creates drawings of poignant travel memories for the visually impaired

Advertising Yesterday

Clickbait Titles Used For The Good Of Charity

An agency devised an unlikely campaign that uses clickbait as a positive force to drive awareness to nonprofit initiatives

Advertising Yesterday

The Best In Eye-Catching Olympics Campaigns

PSFK rounds out the Rio Games with our picks for the best advertising moments off the field

Work Yesterday

Editorial Roundtable: The Arrival Of The People-First Workplace

Managed By Q, Soma, Workbar, Primary and thinkPARALLAX enumerate the reasons why companies need an employee-embracing workforce in order to exist

Arts & Culture Yesterday

Transforming Light Waves Into A New Art Form

An artist uses glass treated with layers of metallic coatings to create a unique installation called lightpaintings


Rio Olympics
Innovation Coverage From The Rio Games

Design Yesterday

This Windbreaker Lets You Explore The Outdoors While Charging Your Phone

The apparel includes solar panels that allow the wearer to stay connected through the power of renewable energy

Asia Yesterday

The Goal Of This Game Is To Not Get Laid Off From Your Job

A hit mobile app has you working really, really hard to not get fired as you climb the corporate ladder

Advertising Yesterday

Movie Critic Bot Guides Viewers Through Festival Offerings

The Toronto International Film Festival has created a Facebook Messenger chatbot to help attendants curate their schedule

No search results found.