Facebook Is Using AI To Write Captions For Blind Users
Phrase "Image may contain" brings joy to users as they discover whole new side of Facebook
With 2 billion photos shared across Facebook‘s apps every day, it is clear that posting images is a quick and fun way to keep your friends up to date. But, until now, 39 million blind and 246 million visually impaired users could not fully engage in this conversation. When they scrolled through their feeds, the alternative text reader would only point out that a photo was posted by their friends, without giving any indication on what that photo could represent.
In order to ameliorate their user experience, the Facebook Accessibility team developed a new feature that utilizes artificial intelligence and object recognition technology to “read” the photo to them. To recreate and communicate to the users what the “image may contain,” the software divides every visual into three groups: people, objects and scenes. For each photo, the software first identifies the number of people (or more accurately, the number of faces) and whether they are smiling or not. Then it proceeds to list the objects present in the photo, ordered by how confident the program is in its guess. Finally, the AI gives users broader overview of the image’s setting, such as whether it’s outdoors, indoors, a selfie or a meme.