Evaluation of Information Access with Smartphones

Abstract

NTCIR 1CLICK and MobileClick are the earliest attempts toward test-collection-based evaluation for information access with smartphones. Those campaigns aimed to develop an IR system that outputs a short text summary for a given query, which is expected to fit a small screen and to satisfy users’ information needs without requiring much interaction. The textual output was evaluated on the basis of iUnits, pieces of relevant text for a given query, with consideration of users’ reading behaviors. This chapter begins with an introduction to NTCIR 1CLICK and MobileClick, explains the evaluation methodology and metrics such as S-measure and M-measure, and finally discusses the potential impacts of those evaluation campaigns.

Publication
Evaluating Information Retrieval and Access Tasks: NTCIR’s Legacy of Research Impact