Share if you like it!
And especially if you don't!

Image Similarity Search — from Research to Production

Let’s say you sit down in a cozy cafe with a coffee and spot a lovely lamp on the corner table. It perfectly matches the look you’re going for at home, but how can you find one just like it? Sure, there are hundreds of lamps with funny names on Ikea’s website, but scrolling through them takes ages. There’s got to be a faster, easier way.

image-similarity-search.png

💡 What if you could just take a photo of the lamp and immediately find similar ones for sale online? One of our clients had the same idea.

This is how we used computer vision to take it from idea to reality.

Overview — what is image similarity search?

In this blog post we will share our experience of creating an image similarity search application from research to production stages. We will try to explain it from a high level with the focus on Machine Learning (ML) / Computer Vision (CV) concepts. However, we will have some “optional sections” showing some lower level technical details time to time.

Let’s start, our main task was to find and show similar furniture images (chair, lamp, table, bed or sofa) for an input image uploaded by the user. The technical challenge here was to implement a fairly accurate image similarity search with very small reference image sets (this is the pre-indexed image set which is used to find similar images for a given input image) ~200 images per category.

For comparison, Google image search probably has millions or billions of reference images which makes it more likely to find similar images for an input image.

Let’s see a visual example of our task, here’s how Google image similarity search looks:

google-image-search.png

Machine learning library for image classification

We researched several machine learning libraries and decided to use the DeepDetect library. DeepDetect is an open source machine learning API and server written in C++.

First off, we deployed an image similarity search with the pre-trained model “ResNet50” (a 50 layer-deep convolutional neural network) and started testing. After some initial testing, we realized that our solution worked better when an “image classification” service is also used.

So, the final solution was to first find the class of the image (e.g. chair, lamp, table etc...) and then find similar images of that class (e.g. find similar lamp images).

Here’s a simple view of the final architecture:

Image-Search-Architecture.png

More about the Resnet50 convolutional neural network:

What is a Convolutional Neural Network?

A convolutional neural network (CNN or ConvNet for short) is a class of deep neural networks, most commonly used to analyze images. Let’s briefly outline the key concepts:

  • Convolution layer: is repeated application of small image filters (aka features) on an input image to find a pattern or feature.
  • Max-Pooling layer: is downsizing an input image with similar patterns
  • Fully-Connected layer: connects every neuron in one layer to every neuron in another layer.
  • Output: is the final layer which returns the prediction(s) for an input image

Here's an example CNN architecture to classify handwritten digits:

CNN-example.png

Image Similarity Search prototype

We implemented the first working version with 30 reference images (this is the pre-indexed image set which is used to find similar images for a given input image) which was a super small image set.

We knew that we would have better results once we increased the number of reference images and used different layers or even different convolutional neural network models.

Steps to achieve the first working version:

  1. Create image classification service
  2. Verify image predictions work
  3. Create image similarity search service
  4. Index reference images for image similarity search service
  5. Verify image similarity search works

Let’s look at our very first results now. The problem here is that the reference image set is too small, so it is very unlikely that the service will find a similar image.

first-results.png

Here are some key technical details about the creation and configuration of image similarity search service:

Key parameters explained

How to create an image similarity search service

How to index images

This is for one image, we created a script to index all different classes and images.

Image similarity search query

Image similarity search response

Getting image search results with ResNet50

At this stage, we focused on improving our existing solution. We have gradually increased the reference image count, tried different layers of existing working ResNet50 models, tried different models with different extraction layers.

In the end, we saw that the result with 200 images and using the ResNet50 model (with extract_layer “pool5/7x7_s1”) was much better than others.

This image, showing initial results for the 10 most similar images from a random input image, is from development:

For those who are even more curious, let's briefly compare different convolutional neural network model architectures. We experimented with these models in order to get better results.

For instance, we have tried and compared ResNet50 last max pooling layer “7x7 avg pool” (kernel size: 7, stride: 1) versus VGG16 last max pooling layer “pool5” layer (kernel size: 3, stride: 2).

We saw that the results with Resnet50 were better. We have also tried and compared different layers of the ResNet50 model. And all of these attempts are not quick or simple tasks — for every try, a new service needs to be created and the reference image set must be re-indexed.

Source

The Final Product

The final product has become a beautiful web application* where the users can upload furniture images to find similar products. This is, of course, a result of teamwork so I want to mention and thank our great team here:

  • ML Lead & Project Manager - Vladimir Panteleev
  • Designer - Elizaveta Gaiduk
  • ML Engineer - Gökhan Şimşek (myself)
  • Software Engineer - Denis Zhidis

*The product itself is still in very early stages and functions with a very limited image database.

Let’s see how the final product and image search looks

Thanks for reading 🙏

We hope you enjoy our series on AI/ML topics. Specifically, we explore how to integrate proven technologies into products in a way that matches the needs of the end-user and the budget of the project.

Hungry for more? Check out how we applied transfer learning to improve our image classification results with a small dataset.

Citations:

A Comprehensive Guide to Convolutional Neural Networks — the ELI5 way by Sumit Saha

Deep Feature-Based Classifiers for Fruit Fly Identification Leonardo, Matheus & Carvalho, Tiago & Rezende, Edmar & Zucchi, Roberto & Faria, Fabio. (2018)

Previous Article
Most common challenges faced by startups
Next Article
A Quick Guide to Product vs Project Management

Get the useful tips on building startups to your email

Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.