Apple Introduces Powerful Suite of Image-Based Tools For Collaboration

Apple has released powerful new tools that make use of recent advances in image recognition technology and computer vision are part of a larger trend noticed by PSFK researchers around an increase in image-driven interfaces. The camera is becoming a more and more integral component of the mobile experience, allowing users to find, capture, and activate information from the physical world.

One of the more technically advanced features in the new software update is Live Text, which allows iOS users to pull textual information from images, including email addresses, phone numbers, and both handwritten and typed text. The feature recognizes useful information and text within photographs and images hosted online, and, using its advanced OCR capabilities, is able to separate those text-based signals from the noise of the rest of the image. The Live Text capability is currently able to identify and copy English, Chinese, Portuguese, French, Italian, German, and Spanish text from pictures, and can capture both typed and handwritten content. 

For Apple consumers with an iPhone XS or newer, it is possible to use the Live Text solution in real-time, directly from the camera app, by pointing the camera lens at the desired text to capture and tapping the Live Text indicator icon, which provides an easy interface to other actions such as composing an email to the identified email address, making a phone call to the scanned number, or copying and pasting the captured text into a message or note, as well as even translating it into another supported language. For owners of older iPhones who are running the latest iOS 15 software,  the feature can be accessed once an image has been taken or selected. The new feature’s use-cases are as endless as the variety of text that can be contained within an image, and because it is a system-wide implementation that will appear in all other apps when users attempt to copy and paste text, Apple consumers across all the brand’s products will be able to enjoy the added seamless convenience of no longer having to physically copy down and transfer relevant text contained in an image to make use of it. 

Another Optical Character Recognition-based feature included in the iOS 15 software update is Apple’s new Visual Look Up tool. The tool allows users to learn more about identified objects in photos, including popular landmarks, art, plants, animals, and other objects that are either on the web or appear in users’ own captured photos. By tapping an icon that appears on the selected image, icons will appear around each identified object or category with a click-through option to learn more about it. For example, if a user takes a picture of a flower on a walk, by engaging the Visual Look Up tool they can learn more about the name, biology, and details of the plant. 

These powerful new tools that make use of recent advances in image recognition technology and computer vision are part of a larger trend noticed by PSFK researchers around an increase in image-driven interfaces. The camera is becoming a more and more integral component of the mobile experience, allowing users to find, capture, and activate information from the physical world. 

+++

This article originally appeared in PSFK’s research paper, Emerging Interface Experiences