New Tech Helps Users Point And Seamlessly Connect With Multiple Surfaces
PSFK spoke about the Future Of Gaming with the co-founder of teehan+lax, whose product let you control screens you encounter with your smartphone.
The PSFK consulting team has noticed that enhanced connectivity across devices is enabling seamless interactions between tablets, smartphones and other screens, creating new opportunities for planned and impromptu gaming experiences and offering users new ways to engage with their surroundings.
One company operating in this space is teehan+lax — a user experience design company. Its Touch Vision Interface is a tool that enables users to interact seamlessly with different connected surfaces through a mobile phones’ camera view. Users simply point their viewfinders to a connected screen and are able to ‘touch’ the distant screen virtually. The technology creates the feel of being able to directly manipulate a distant surface, resulting in an interaction that is natural and almost invisible. The client server application additionally works over IP based systems and doesn’t require any complex pairing for the device to take control of the screen. PSFK spoke with co-founder Jon Lax about his work.
Please provide a brief introduction about yourself and your company.
My name is Jon Lax. I am the co-founder of teehan+lax, a user experience design company. We help clients define and design customer experiences in the digital channel. We do this across a variety of devices and screen types. An important part of being able to come up with new ideas of users will interact with devices and screens we have a Labs group who’s mission is to help us understand what is possible in the future.
Please tell us about your Touch Vision Interface. What is the idea and goal behind the product and its platform?
TVI is a client server application where, the camera on your phone, recognizes controllable screens around you and lets you control the interactions on that screen with your phone acting as a touchscreen input device. This works over IP based systems and doesn’t require any complex pairing for the device to take control of the screen.
TVI comes out of our Labs group and its genesis and creation is due to Peter Nitsch and his team. Peter was experimenting with computer vision libraries. There was no product or idea when he started, just a curiosity to learn the tech. We meet weekly to discuss what the Lab is working on and Pete came in and said… “So you know how in AR you point your phone at the world and then this layer appears and you interact with it but all your interaction is contained within a layer on the phone? What if you could point your phone at the world and control it?”
That was the idea. Pete and his team spent the next two weeks trying to do that. When he first showed Geoff Teehan a prototype, he had figured out how to point the phone at a computer monitor and control the screen interaction with the phone. Geoff said “This idea is only powerful if it works across multiple screens.” Peter and his team were able to get that working.
On a technical level this is what is happening. The phone has the TVI app on it. The app is able to look at the world around it through the camera and is trained to recognize specific borders on screens. When it discovers a screen it can control it “pairs” with it giving it control.
The screen is running an openFrameworks app that receives touch data from the phone over TCP. We like the idea that you are walking around with this controller (your phone) you should be able to point it at screens out in the world and take control of that screen.
Our goal for the project was just a Lab experiment. We wanted to see if we could accomplish what Pete had imagined. We don’t have specific applications for the tech although we’re exploring its practical applications for everything from retail experiences to gaming.
What has been the audience reaction? Can you share any stats around user engagement?
We got very positive reaction with posts in Engadget and Creative Applications. We’ve had inquiries from several companies (including some gaming companies) about licensing the technology. Since it was only meant to be a Lab experiment it is still in a very formative state. We’re going to be open sourcing parts of TVI on Git and hopefully others will do some cool things with it.
We have noticed that enhanced connectivity across devices is enabling seamless interactions between tablets, smartphones and other screens, creating new opportunities for planned and impromptu gaming experiences and offering users new ways to engage with their surroundings. Do you see this trend manifesting on a larger scale?
As a company we identified this trend of seamless interactions late last year and have shifted our company’s focus to designing customer experiences in this world. This trend applies very broadly and is not confined to gaming.
TVI is a good demonstration of this trend. Why should I be tied to my game console and controller to play a game. We increasingly have these screens around us in our daily life, we should be able to use them.
There are some barriers to enhanced connectivity and seamless interactions. Discovery is a really complex tricky problem. How your phone or tablet discovers another device or screen is not simple for the user. TVI relies on an app because its the only way we can do discovery, this is not optimal.
While we can imagine going to Times Square pointing your phone at several screens and then playing a game with several hundred people standing around you and even online players, we’re not sure what that game looks like or if it would be an enjoyable game experience.
We need to go through some exploration and even some failures before this trend is realized or even manifesting on a larger scale.