Evaluating Entity Relationship Recommenders in a Complex Information Retrieval Context

Abstract

Information Retrieval, as a field, has long subscribed to an orthodox evaluation approach known as the Cranfield paradigm. This approach and the assumptions that underpin it have been essential to building the traditional search engine infrastructure that drives today’s modern information economy. In order to build the information economy of tomorrow, however, we must be prepared to reexamine these assumptions and create new, more sophisticated standards of evaluation to match the more complex information retrieval systems on the horizon.

In this thesis, we begin this introspective process and launch our own evaluation method for one of these complex IR systems, entity-relationship recommenders. We will begin building a new user model adapted to the needs of a different user experience. To support these endeavors, we will also conduct a study with a mockup of our complex system to collect real behavior data and evaluation results. By the end of this work, we shall present a new evaluative approach for one kind of entity-relationship system and point the way for other advanced systems to come.