Twitter is harnessing the power of machine learning to offer a subtle improvement to the user experience when users tweet photos. By using neural networks, Twitter is able to automatically crop picture previews to show the most interesting parts as a preview.
They have been working on this feature for quite a while now, prior to this, they trained the system using facial recognition to crop the images but found there is a problem. The tool did not work very well when it comes to pictures of scenery, objects and animals. The second solution was to use “cropping using saliency, which means the program will crop the image to show what is the most interesting part of the phot. This is done by using data based on eye-tracking which records what people look at first in a picture. Based on that data, it was used to train the neural network to predict what people might look at.
Top: before machine learning implementation
Bottom: after machine learning implementation
The outcome is ten times faster than its original design and allows them to perform saliency detection on all the images as soon as it gets uploaded. The feature is being rolled out on desktop, iOS and Android in the next couple of days.
