Visual Recognition with IBM Watson, HERE and Python - Part 2

This is the second blog post in a two-part blog series about application of visual recognition using Python. In part one, we acquired credentials from IBM Cloud and detected food images using visual recognition service of IBM Watson. Now we will learn how to integrate location services to find the places which offer the type of food we recognized.

Architecture

Here’s the big picture for how the different components interact:

First the user passes an image and location (latitude and longitude information) to the Python application
Watson Visual Recognition’s food detection model returns its predicted food name as a string
Python application queries the HERE Discover API with the food name and location

Prerequisites

The requirements are same as for the Part 1:

An IBM Cloud account (you can sign up here)
Python 3.8 and pip
A Freemium account on the HERE Developer Portal

When it comes to testing your application, you will also need a food image. If you’d like to use the image below, feel free to download and save it as ‘test.png’ for this blog we will be using below given image, save it as ‘test.png’.

Integrating Flask in the visual recognition Python code

With the help of code created in part 1, we are able to recognize an image of food and get a match as ‘Pizza’. Save the below code as ‘my_code.py’ in a folder with ‘test.png’

Copied

        
import json 
from ibm_watson import VisualRecognitionV3 
from ibm_cloud_sdk_core.authenticators import IAMAuthenticator 

# Passing API KEY and URL to the Visual Recognition 
authenticator = IAMAuthenticator('IBM_API_KEY') 

visual_recognition = VisualRecognitionV3( 
    version='2018-03-19', 
    authenticator=authenticator) 

visual_recognition.set_service_url('IBM_URL')  

# Running the Visual Recognition on test.img file 
with open('./test.jpg', 'rb') as image:  
    classes = visual_recognition.classify(images_file=image,threshold='0.6',classifier_ids='food').get_result() 

output_query = classes['images'][0]['classifiers'][0]['classes'][1]['class'] 
print(output_query)

When the above code is executed, it will recognize the image and print ‘pizza’ (for details on how this code works check out Part 1)

Next, we have to pass the acquired string (pizza) to a Flask code along with the location (latitude and longitude) data. To work with Flask, which is a micro web framework for creating web applications, run an installation command:

pip install Flask

Copied

        
from flask import Flask,render_template  

#Somewhere in Bangalore, India 
latitude = 12.959111 
longitude = 77.732022 

app = Flask(__name__) 

@app.route('/') 

def map_func(): 
    return render_template('map.html', 
                            latitude = latitude, 
                            longitude = longitude, 
                            output_query=output_query 
                            ) 

if __name__ == '__main__': 
    app.run(debug = True)

The above code represents a simple Flask template in which we have imported two sub-packages of Flask, viz., Flask and render_template. Also, we have created a function called the “map_func”. This function returns a jinga2 html page and passes the output_query(pizza), latitude and longitude to our HTML code (yes, we will need HTML too).

Showing your location on a map as an icon

Now let’s create an HTML file and call it “map.html” to display the results on a map.

Copied

        
<html>   
<head>   
<meta name="viewport" charset="UTF-8" content="initial-scale=1.0, width=device-width" />
<script type="text/javascript" src="https://js.api.here.com/v3/3.1/mapsjs-core.js"></script> 
<script type="text/javascript" src="https://js.api.here.com/v3/3.1/mapsjs-service.js"></script> 
<script type="text/javascript" src="https://js.api.here.com/v3/3.1/mapsjs-ui.js"></script> 
<script type="text/javascript" src="https://js.api.here.com/v3/3.1/mapsjs-mapevents.js"></script> 
<link rel="stylesheet" type="text/css" href="https://js.api.here.com/v3/3.1/mapsjs-ui.css"/> 
</head> 

<body style='margin: 0'> 
<div id="mapContainer" style="width: 90vw; height: 80vh; display: block; margin: 0 auto; border: solid 2px black; margin-top: 10px;" >
</div> 
 <div style="width: 100vw; height: 40px; margin-top: 30px;"> 
 <input type="button" onclick="showRestaurants()" value = "Show Restaurants" style="width: 200px; height: 30px; border: 2px solid black; display: block; margin: 0 auto; margin-top: 20px;"> 
 </div>
</body> 

<script>    
    const lat = {{latitude}}; 
    const lng = {{longitude}}; 
    var query = "{{output_query}}"; 

    //Initialise communication with backend services 
     var platform = new H.service.Platform({ 
            apikey: "JS_API_KEY"    
        }); 

    // Obtain the default map types from the platform object: 
    var defaultLayers = platform.createDefaultLayers(); 

    // Get your current position from wego.here.com 
    var myPosition = {lat: lat, lng: lng}; 

    // Instantiate (and display) a map object: 
    var map = new H.Map( 
        document.getElementById('mapContainer'), 
        defaultLayers.vector.normal.map, 
        { 
            zoom: 15, 
            center: myPosition 
        }); 

    //zoom and view controls 
    var ui = H.ui.UI.createDefault(map, defaultLayers, 'en-US'); 

    //Move around the map 
    var mapEvents = new H.mapevents.MapEvents(map); 
    var behavior = new H.mapevents.Behavior(mapEvents); 

    // create an icon for the marker. Choose any image you want. 
    var homeIcon = new H.map.Icon('/static/YOUR_IMAGE');  

    // Create a marker using the previously instantiated icon: 
    var posMarker = new H.map.Marker(myPosition,{icon:homeIcon}); 

    // Add the marker to the map  
    map.addObject(posMarker);    
</script>
</html>

The above code when executed will show a map with an icon at the center showing your location and a button, which will not work right now since we have not written code for it.

You will need to replace JS_API_KEY with the JavaScript API Key value acquired from HERE. You can get the value by signing up on Freemium account.

Place an image you want to show as your location in a folder named “static”. Replace the name of image in code from YOUR_IMAGE to the name of image you pasted (for example home.png)

Showing pizza serving places on map

Next, we want to fetch the places which serves pizza around you and display them on the map. All this with a click of a button (Show Restaurants)

Firstly, we will have to fetch all the pizza places near the provided location, for that we need an instance of the Geocoding and Search service.

Copied

        
var service = platform.getSearchService();

Then, we need to put a logic in button click method

Copied

        
    function showRestaurants(){ 
            let param = { 
                at : myPosition.lat+','+myPosition.lng, 
                q: query, 
                limit:10 
            };  
            service.browse(param,displayRestaurants,alert); 
        }

In the above code, ‘param’ is storing all the parameter we require to pass to our Discover endpoint of Geocoding and Search API.

at – Your location
query – Value passed from Python code (pizza)
limit – Limiting our result to 10 values

Note: If you run the code now it will show an error since we have not defined ‘displayRestaurants’

Lastly, we need to show the pizza places as a clickable icon on the map.

Copied

        
function displayRestaurants(response){ 
            var restaurantIcon = new H.map.Icon('/static/PIZZA_IMAGE'); 

            // A group that can hold map objects: 
            var restGroup = new H.map.Group(); 

            for(let i = 0; i<response.items.length; i++){ 
                let restPosition = response.items[i].position; 
                let address = response.items[i].address.label; 
                 
                let restMarker = new H.map.Marker(restPosition,{icon: restaurantIcon} ); 
                 
                restMarker.setData("<p>" + address + "</p>"); 
                 
                restMarker.addEventListener('tap', function(evt){ 
                    var bubble =  new H.ui.InfoBubble(evt.target.getGeometry(), { 

                    // read custom data 
                    content: evt.target.getData() 
                }); 

                ui.addBubble(bubble); 
                }, false); 

                // Add the marker to the group (which causes it to be displayed on the map) 
                restGroup.addObject(restMarker); 
                } 

            // Add the group to the map object 
            map.addObject(restGroup); 
    }

In the above code, you will have to place an image for pizza icon in the ‘static’ folder and replace the name PIZZA_IMAGE with your image name (like pizzaIcon.png).

Complete HTML code

Copied

        
<html>   
<head>   
<meta name="viewport" charset="UTF-8" content="initial-scale=1.0, width=device-width" />
<script type="text/javascript" src="https://js.api.here.com/v3/3.1/mapsjs-core.js"></script> 
<script type="text/javascript" src="https://js.api.here.com/v3/3.1/mapsjs-service.js"></script> 
<script type="text/javascript" src="https://js.api.here.com/v3/3.1/mapsjs-ui.js"></script> 
<script type="text/javascript" src="https://js.api.here.com/v3/3.1/mapsjs-mapevents.js"></script> 
<link rel="stylesheet" type="text/css" href="https://js.api.here.com/v3/3.1/mapsjs-ui.css"/> 
</head> 

<body style='margin: 0'> 
<div id="mapContainer" style="width: 90vw; height: 80vh; display: block; margin: 0 auto; border: solid 2px black; margin-top: 10px;" >
</div> 
 <div style="width: 100vw; height: 40px; margin-top: 30px;"> 
 <input type="button" onclick="showRestaurants()" value = "Show Restaurants" style="width: 200px; height: 30px; border: 2px solid black; display: block; margin: 0 auto; margin-top: 20px;"> 
 </div>
</body> 

<script>
	const lat = {{latitude}};
	const lng = {{longitude}};
	var query = "{{output_query}}";
	
	//Initialise communication with backend services
	 var platform = new H.service.Platform({
            apikey: "JS_HERE_API"   
        });

	// Obtain the default map types from the platform object:
	var defaultLayers = platform.createDefaultLayers();

	// Get your current position from wego.here.com
	var myPosition = {lat: lat, lng: lng};

	// Instantiate (and display) a map object:
	var map = new H.Map(
		document.getElementById('mapContainer'),
		defaultLayers.vector.normal.map,
		{
			zoom: 15,
			center: myPosition
		});

	//zoom and view controls
	var ui = H.ui.UI.createDefault(map, defaultLayers, 'en-US');

	//Move around the map
	var mapEvents = new H.mapevents.MapEvents(map);
	var behavior = new H.mapevents.Behavior(mapEvents);

	// Get an instance of the geocoding and search service:
	var service = platform.getSearchService();
	
	// create an icon for the marker. Choose any image you want.
	var homeIcon = new H.map.Icon('/static/home.png'); 
		
	// Create a marker using the previously instantiated icon:
	var posMarker = new H.map.Marker(myPosition,{icon:homeIcon});
			
	// Add the marker to the map 
	map.addObject(posMarker);
	
	function showRestaurants(){
			let param = {
				at : myPosition.lat+','+myPosition.lng,
				q: query,
				limit:10
			}; 

			service.browse(param,displayRestaurants,alert);
		}
	function displayRestaurants(response){
			var restaurantIcon = new H.map.Icon('/static/pizzaIcon.png');

			// A group that can hold map objects:
			var restGroup = new H.map.Group();

			for(let i = 0; i<response.items.length; i++){
				let restPosition = response.items[i].position;
				let address = response.items[i].address.label;
				
				let restMarker = new H.map.Marker(restPosition,{icon: restaurantIcon} );
				
				restMarker.setData("<p>" + address + "</p>");
				
				restMarker.addEventListener('tap', function(evt){
					var bubble =  new H.ui.InfoBubble(evt.target.getGeometry(), {
                      // read custom data
      				content: evt.target.getData()
				});
				ui.addBubble(bubble);
				}, false);
              
				// Add the marker to the group (which causes it to be displayed on the map)
				restGroup.addObject(restMarker);
				}

			// Add the group to the map object
			map.addObject(restGroup);
	}		
</script>
</html>

Running the code

Open command prompt or terminal and run the Python code (as shown in image below).

An IP address will appear, http://127.0.0.1:5000 (see in the above image).

Copy this address and paste it in your browser. In response you will get the map image showing a marker and a button. Click on the button and you will get all the Pizza places near you!

Visual Recognition is a subset of Artificial Intelligence meant to make machine intelligent like humans and there cant be anything more human than recognizing food by an image, and craving for it.

For more details on how make this code work on IBM Cloud and code related resources, visit GitHub link.

Happy Learning!

Visual Recognition with IBM Watson, HERE and Python - Part 2

Vidhan Bhonsle

Have your say

Sign up for our newsletter

HERE