Visual search on the web has been around for some time
In 2008, TinEye became the first image search engine to use image identification technology, and in 2010, the Google Goggles app allowed users to search the physical world with their phone cameras.
But in the last couple of years, visual search has come into new prominence, with companies like Pinterest and Bing developing into serious contenders in the visual search space, and search engines like Splash conceptualising new ways to search the web visually.
We now have an impressive range of visual search methods available to us: we can search with images, with part of an image, with our cameras, with paint on a digital canvas. And combined with applications in ecommerce, and recent advances in augmented reality, visual search is a powerful tool with huge potential.
So what can it do currently, and where might it develop in the future?
Then and now: The evolution of visual search
Although the technology behind image search has come on in leaps and bounds in the past few years, it’s as a result of developments that have taken place over a much longer time period.
Image search on the web was around even before the launch of reverse image search engine TinEye in 2008. But TinEye claims that it was the first such search engine to use image identification technology rather than keywords, watermarks or metadata. In 2011, Google introduced its own version of the technology, which allowed users to perform reverse image searches on Google.
Both reverse image searches were able to identify famous landmarks, find other versions of the same image elsewhere on the web, and locate ‘visually similar’ images with similar composition of shapes and colour. Neither used facial recognition technology, and TinEye was (and still is) unable to recognise outlines of objects.
Google reverse image search in 2011. Source: Search Engine Land
Meanwhile, Google Goggles allowed users of Android smartphones (and later in 2010, iPhones and iPads) to identify labels and landmarks in the physical world, as well as identifying product labels and barcodes that would allow users to search online for similar products. This was probably the first iteration of what seems to be a natural marriage between visual search and ecommerce, something I’ll explore a bit more later on.
The Google Goggles app is still around on Android, although the technology hasn’t advanced all that much in the last few years (tellingly, it was removed as a feature from Google Mobile for iOS due to being “of no clear use to too many people”), and it tends to pale in comparison to a more modern ‘object search’ app like CamFind.
CamFind is a visual search and image recognition mobile app that was launched in 2013, and while it doesn’t appear to be able to solve Sudoku puzzles for you, it does have an impressive rate of accuracy.
Back when Google Glass was still a thing, Image Searcher, the startup behind CamFind, developed a version of the app to bring accurate visual search to Google Glass, activated by the command “OK Glass, what do you see?” This is the kind of futuristic application of visual search that many people imagined for a technology like Google Glass, and could have had great potential if Google Glass had caught on.
The CamFind mobile app has an impressive accuracy rate, even down to identifying the brand of an object.
When the ‘pinboard’-style social network Pinterest launched in 2012, it was a bit of a dark horse, gaining huge popularity with a demographic of young-to-middle-aged women but remaining obscure in most conventional tech circles. Even those who recognised its potential as a social network probably wouldn’t have guessed that it would also shape up into a force to be reckoned with in visual search.
But for Pinterest, accurate visual search just makes sense, as it allows Pinterest to serve relevant Pin recommendations to users who might be looking for something visually similar (say, the perfect copper lamp to light their living room) or hone in on the specific part of a Pinned image that interests them.
In 2014, Pinterest acquired VisualGraph, a two-person startup which was cofounded by one of Google’s first computer vision engineers, bringing the company’s visual search know-how into the fold. In the same year, it introduced and began refining a function that allowed users to highlight a specific part of a Pin and find other Pins that are visually similar to the highlighted area – two years ahead of Bing, who only introduced that functionality to its mobile image search in July 2016.
Bing has pipped Pinterest to the post by introducing visual searching with a smartphone camera to its native iOS app (I can’t comment on how accurate it is, as the Bing iOS app is only available in the US), something that Pinterest is still working on launching. But it’s clear that the two companies are at the vanguard of visual search technology, and it’s worth paying attention to both to see what developments they announce next.
Meanwhile, Google is yet to offer any advance on Google Goggles for more accurate searching in the physical world, but you can bet that Google isn’t going to let Pinterest and Bing stay ahead of it for too long. In July, Google announced the acquisition of French startup Moodstocks, which specialises in machine learning-based image recognition technology for smartphones.
And at Google I/O in May, Google’s Engineering Director Erik Kay revealed some pretty impressive image recognition capabilities for Google’s new messaging app, Allo.
“Allo even offers smart replies when people send photos to you. This works because in addition to understanding text, Allo builds on Google’s computer vision capabilities to understand of the content and the context of images. In this case, Allo understood that the picture was of a dog, that it was a cute dog, and even the breed of the dog. In our internal testing, we found that Allo is 90% accurate in determining whether a dog deserves the ”cute dog” response.”
Visual search and ecommerce: A natural partnership
How many times have you been out and about and wished you could find out where that person bought their cool shoes, or their awesome bag, without the awkwardness of having to approach a stranger and ask?
What if you could just use your phone camera to secretly take a snap (though that’s still potentially quite awkward if you get caught, let’s be honest) and shop for visually similar search results online?
Ecommerce is a natural application for visual search, something which almost all companies behind visual search have realised, and made an integral part of their offering. CamFind, for example, will take you straight to shopping results for any object that you search, creating a seamless link between seeing an item and being able to buy it (or something like it) online.
Pinterest’s advances in visual search also serve the ecommerce side of the platform, by helping users to isolate products that they might be interested in and smoothly browse similar items. An ‘object search’ function for its mobile app would also be designed to help people find items similar to ones they like in the physical world on Pinterest, with a view to buying them.
With the myriad possibilities that visual search holds for ecommerce, it’s no surprise that Amazon has also thrown its hat into the ring. In 2014, it integrated a shopping-by-camera functionality into its main iOS app (and has since released the function on Android), and also launched Firefly, a visual recognition and search app for the Amazon Fire Phone.
Even after the Fire Phone flopped, Amazon refused to give up on Firefly, and introduced the app to the more affordable Kindle Fire HD. The visual search function on its mobile app works best with books, DVDs and recognisably branded objects, but it otherwise has a good rate of accuracy.
Amazon’s visual search in action.
Other companies operating in the cross-section of visual search and ecommerce which have emerged in the past few years include Slyce, whose slogan is “Give your customer’s camera a buy button”, and Catchoom, which creates image recognition and augmented reality tools for retail, publishing and other sectors.
Although searching the physical world has yet to cross over into the mainstream (most people I’ve talked to about it aren’t even aware that the technology exists), that could easily change as the technology becomes more accurate and increasingly widespread.
But ecommerce is only one possible application for visual search. What other uses and innovations could we see spring up around visual search in the future?
Source : searchenginewatch
