About

Large numbers of images uploaded to platforms such as Flickr do not contain GPS-based latitude/longitude coordinates. Obtaining such geographic information is beneficial for a variety of applications including the E-learning domain. While most works in this area only exploit an image's textual meta-data (tags, title, etc.) to estimate at what geographic location the image was taken, we consider an additional textual dimension: the image owner's traces on the social Web. Specifically, we hypothesize that information extracted from a person's microblog stream(s) can be utilized to improve the accuracy with which the geographic location of the images is estimated. We investigate this hypothesis on the example of Twitter streams and find it to be confirmed. The median error distance in kilometres decreases by up to 67% in comparison to existing state-of-the-art. The best results are achieved when tweets that were posted up to two days before and after an image was taken are considered. Moreover, we also find another type of additional information useful: population density data.

 

The implemented service infers the locations a user has traveled to and the amount of time the user has spent at each location based on the set of images the user has uploaded to the photo sharing platform Flickr and - if known - based on the additional information gained from crawling the user's Twitter stream.

 

Given the learner's Flickr & Twitter IDs, we extract the meta-data of the publicly available images that were uploaded by the user to Flickr. We also crawl all available tweets on Twitter. The vast majority of images are accompanied by the date, the image was taken at, and thus creating a time line of locations visited is straight forward. What is most often missing though, is the particular geographic location (the latitude and longitude), the image was taken at. If the latitude/longitude of all images would be known - because for instance the user used a GPS-enabled camera - we would be able to create an overview of the locations visited. Our service implements an algorithm that estimates the location of each image that is not accompanied by latitude/longitude coordinates. The image location estimation works as follows: for each region of the world (a region may be the city of Paris, France, the area of Lake Baikal, etc.) we collect a set of training images. Training images contain latitude/longitude values (like the image above) which allow us to place them in their corresponding region. We build models for each region based on the tags the users assign to the images. Given an image without latitude/longitude, we assign it the region whose model is most similar to the image's tags and the terms in the temporally closest Twitter messages.

 

Below is an example of an image and some of the available Flickr meta-data which is important to us. For this image, the geo-location is known (blue box), it was taken at Enschede, the Netherlands. The date the image was taken at is available (turquoise) as well as a very short description provided by the user (green). We are most interested in the tags (red), also provided by the user.

Shown is also a tweet the user posted the same day.

 

 

To top

Objective

The initial investigation can only be considered as a stepping stone leaves a lot of potential for future work. One possibility is to not only consider the traces a user leaves on Twitter to enhance the location estimation accuracy of Flickr images, but also on other social Web platform such as Facebook or LinkedIn. Another area of future work is to classify users' tweets as either being about personal experiences and activities or about local vs. global news: tweets discussing global news (e.g. Japan announced meltdown yesterday; situation grim.) are often adding noise for our purposes, while tweets discussing local news (e.g. Here in Toronto the police made multiple arrests today) are useful as they can serve as indicators of the user's home location. Lastly, we will also consider an investigation into the similarities/differences between (estimated) travel patterns of different users and user groups.

To top

Example scenarios

In the ImREAL project, estimating the location at which an image was taken is a preliminary step for creating a location-based user profile. Location-based profiles are utilized in a number of use-cases:

  • In the medical use case, which focuses on training medial students to conduct doctor-patient interviews. A location-based user profile can give an indication to the simulator about the user's familiarity with a particular country, which in turn can be used inside the simulation to adapt the feedback the learner receives. A learner who made a mistake in the interviewing strategy can receive more guidance on how to resolve the mistake when the learner has been found to not be familiar with the country the patient is from.
  • In the business use case, a similar observation holds. The training a learner receives can be adapted according to the learner's intercultural awareness. A learner whose location profile indicates that she has been to a range of countries for prolonged periods of time may receive a training with a different focus than a learner who has been found to not have visited a lot of different countries.

To top

Services

A number of Web services were created that provide functionality related to location detection.

  • PhotoLocationDetection: The service takes a Flickr ID - username or NSID - and a Twitter ID (if available) as input and collects the most recent 1500
    images. For each image, the geographic location is estimated based on the title and tag of each image (as well as the tweets at the time). If available, the true location (e.g. when the photo was taken with a GPS-enabled device) and the estimation error in kilometers are also emitted.
  • PhotoCountryDetection: The service takes a Flickr ID and Twitter ID (if available) as input and determines the country each photo was taken in - it is either an estimate (if the photo contains no latitude/longitude) or, if available, the true location is used.
  • hasUserVisitedCountry: The service takes a Flickr ID and a Twitter ID (if available) as well as a country. The countries of the most recent 1500 images are determined and if one of them is taken in the input country, true is returned, and false otherwise.
  • CulturalCompetencyImg: The service computes the cultural competency based on the number of countries visited - one or two countries yield a score of 1.0, up to five countries yields a score of 2.0 and above the score is 3.0. 

To top

Datasets

For the experiments reported in the listed publications, we relied on a public data set in order to increase the reproducibility of our results: Placing task at MediaEval 2011.

To top

Publications

  1. Claudia Hauff and Geert-Jan Houben, Geo-Location Estimation of Flickr Images: Social Web Based Enrichment, ECIR 2012, pp. 85-96, 2012. [PDF] [Slides]
  2. Claudia Hauff and Geert-Jan Houben, Placing images on the world map: a microblog-based enrichment approach, SIGIR 2012, pp. 691-700, 2012. [PDF] [Slides]
  3. Claudia Hauff and Geert-Jan Houben, WISTUD at MediaEval 2011: Placing Task at MediaEval 2011 (Benchmark) [PDF]

To top