Anne Diekema

Unifying PIM Research: Fostering a Connection Between Descriptive PIM Studies and Prescriptive Outcomes

Anne R. Diekema — Wed, 15 Feb 2012 12:36:42 PST

Descriptive Personal Information Management (PIM) studies inform us about PIM behavior and their findings should guide the design and development of PIM tools to support the behavior under study. Unfortunately, judging from the literature, descriptive studies do not always provide useful recommendations and PIM tool research is often carried out separately. This paper discusses what appears to be a possible research dichotomy and ways to bring the research back together. Three solutions are suggested: 1) PIM workshops where both types of studies are presented and researchers meet should be important venues for dissemination of results, cross-fertilization between different research areas, and collaboration between researchers; 2) A bridging methodology to translate research findings explicitly into design criteria could bring research and practice closer together; and 3) A general PIM framework based on the three essential PIM activities (finding/refinding activities, keeping activities, and meta-level activities).

The Holistic Cognitive Framework for Personal Information Management Research

Anne R. Diekema — Wed, 15 Feb 2012 12:36:36 PST

Seeking a Framework to Study and Understand Personal Information Management

Anne R. Diekema — Wed, 15 Feb 2012 12:36:30 PST

Question Answering: CNLP at the TREC-10 Question Answering Track

Jiangping Chen et al. — Tue, 29 Nov 2011 15:50:47 PST

This paper describes the retrieval experiments for the main task and list task of the TREC-10 question answering track. The question answering system described automatically finds answers to questions in a large document collection. The system uses a two-stage retrieval approach to answer finding based on matching of named entities, linguistic patterns, and keywords. In answering a question, the system carries out a detailed query analysis that produces a logical query representation, an indication of the question focus, and answer clue words.

Evaluation of restricted domain Question-Answering systems

Anne R. Diekema et al. — Tue, 29 Nov 2011 15:50:37 PST

Question-Answering (QA) evaluation efforts have largely been tailored to open-domain systems. The TREC QA test collections contain newswire articles and the accompanying queries cover a wide variety of topics. While some apprehension about the limitations of restricted-domain systems is no doubt justified, the strict promotion of unlimited domain QA evaluations may have some unintended consequences. Simply applying the open domain QA evaluation paradigm to a restricted-domain system poses problems in the areas of test question development, answer key creation, and test collection construction. This paper examines the evaluation requirements of restricted domain systems. It incorporates evaluation criteria identified by users of an operational QA system in the aerospace engineering domain. While the paper demonstrates that user-centered task-based evaluations are required for restricted domain systems, these evaluations are found to be equally applicable to open domain systems.

Resource Discoverability at the Crossroads

Cheryl D. Walters et al. — Tue, 29 Nov 2011 15:50:29 PST

The Internet is changing how people find and use information. Users increasingly are discovering relevant resources outside traditional library systems and expect immediate access to resources at the point of discovery. These changes are challenging traditional thinking on how we should "catalog" and provide access to resources.

Panel discussion engaged the audience in a lively discussion of discoverability, trends in user information seeking behavior, the role of librarians in information seeking and finding, and implications for library system design.

What do You Mean? Finding Answers to Complex Questions

Anne R. Diekema et al. — Tue, 29 Nov 2011 15:50:22 PST

This paper illustrates ongoing research and issues faced when dealing with real-time questions in the domain of Reusable Launch Vehicles (aerospace engineering). The question- answering system described in this paper is used in a collaborative learning environment with real users and live questions. The paper describes an analysis of these more complex questions as well as research to include the user in the question-answering process by implementing a question negotiation module based on the traditional reference interview.

Re-framing information literacy: Problem-based learning as informed learning

Wendy Holliday et al. — Tue, 29 Nov 2011 15:50:13 PST

This study explores an online information literacy module that uses problem-based learning (PBL). The goal was to enable students to experience information literacy in a richer way, by moving away from a focus on locating information sources to one of information use in the construction of knowledge. A content analysis of the research journals and reflection papers of students (N = 15) in a distance education school library media administration endorsement program suggests that PBL was an effective approach for some students, but not others. Some students were motivated by working on authentic problems, and at least half the students engaged deeply with information and discovered new questions and angles for research during the process. These students applied more sophisticated evaluation strategies and were more metacognitive in their thinking, assessing their progress and shifting strategies as they progressed through the module.

Question Answering: CNLP at the TREC-9 Question Answering Track

Anne R. Diekema et al. — Tue, 29 Nov 2011 15:50:02 PST

This paper describes a question answering system that automatically finds answers to questions in a large collection of documents. The prototype CNLP question answering system was developed for participation in the TREC-9 question answering track. The system uses a two-stage retrieval approach to answer finding based on keyword and named entity matching. Results indicate that the system ranks correct answers high (mostly rank 1), provided that an answer to the question was found. Performance figures and further analyses are included.

Modeling Teacher Ratings of Online Resources: A Human-Machine Approach to Quality

Mimi Recker et al. — Tue, 15 Mar 2011 12:56:00 PDT

In education, the scalable deployment of media-rich online resources supports peer production in ways that promise to radically transform teaching and learning (CRA, 2005; Pea et al., 2008). Online educational repositories such as the Digital Library for Earth Systems Education (DLESE.org) and the National Science Digital Library (NSDL.org) collect and curate online learning resources created for a wide range of educational audiences and subject areas (McArthur & Zia, 2008). Through a simple, web-based authoring tool, called the Instructional Architect (IA.usu.edu) teachers locate and share educational resources and activities in an IA project. These IA projects can then be viewed, copied, and adapted by other IA users, in ways that support innovative teacher peer production. A vexing problem for such initiatives remains the elusive notion of quality. In peer production environments, how does one identify quality online content? Moreover, how does one do so in sustainable, cost-effective, and scalable ways? Previous work (Bethard, et al, 2009) presented an innovative approach for using machine learning models to automatically assess the quality and pedagogic utility of educational digital library resources. They demonstrated the feasibility and accuracy of automatic quality assessments for a single STEM domain and audience-level: high school Earth science. This work reports recent efforts to extend these models to support a broader range of STEM topics and grade levels, specifically applied to IA projects and compared model outputs to quality assessments made by K-12 teachers. Since the nature of the resources being compared in the IA (peer) versus DLESE (expert) are different, results of this study provide insights on the generalizability of this machine learning approach and its potential for facilitating teacher peer production.

Teaching Use of Digital Primary Sources for K-12 Settings

Anne R. Diekema et al. — Tue, 15 Mar 2011 12:55:56 PDT

This paper describes learning outcomes of a three-day workshop on integrating primary sources into K-12 teaching. The short curriculum — intended for teams of teachers and school librarians — combined visits to a museum and a library's special collections with an introduction to significant national and local digital collections of primary sources. The paper draws on focus group data, reflection papers, and a conference presentation by the workshop participants as well as curricular artifacts presented to the workshop instructors. Using their workshop experience, teachers integrated digitized primary sources into their curricula thereby creating quality instructional content that engaged students' interest. School librarians and teachers worked together during the workshop, establishing a model for future collaboration. They were exposed to readily accessible digital sources they can draw upon for scholastic projects and lifelong learning. Primary source sets created by workshop participants were added to local and national educational websites for others to use.

A Retrospective on Generating & Evaluating Automatic Metadata for Educational Resources: A Holistic Program

E. D. Liddy et al. — Sat, 04 Sep 2010 16:53:40 PDT

CINDOR Conceptual Interlingua Document Retrieval: TREC-8 Evaluation

M. Ruiz et al. — Sat, 04 Sep 2010 16:53:40 PDT

The TREC-8 evaluation of the CINDOR system was based on English and French data from the cross-language retrieval track. Our objective was to continue our investigation of our conceptual interlingua approach to cross-language retrieval, specifically by measuring the contribution of conceptual retrieval over and above a baseline cross-language retrieval approach based on machine translation of queries. In both of the cross-language runs that were submitted for evaluation, corresponding to English-French and French-English retrieval, performance was measured at 75% of the equivalent monolingual searches. We noted however that absolute average precision values achieved were somewhat lower than many other systems in the cross- language track. Our hypothesis, that the underlying retrieval engine used in CINDOR was employing a simple retrieval function that was impacting performance, was confirmed through experiments with the SMART system configured with several different retrieval settings. Taken together, our TREC-8 experiments point to the value of our conceptual interlingua approach to retrieval, but indicate that our retrieval algorithm must be brought up to date so that valid comparisons may be made to other approaches used in other cross-language systems.

Text Mining Educational Metadata

Anne R. Diekema et al. — Sat, 04 Sep 2010 16:53:39 PDT

Modeling Reference Interviews as a Basis for Improving Automatic QA Systems

N. J. McCracken et al. — Sat, 04 Sep 2010 16:53:39 PDT

The automatic QA system described in this paper uses a reference interview model to allow the user to guide and contribute to the QA process. A set of system capabilities was designed and implemented that defines how the user’s contributions can help improve the system. These include tools, called the Query Template Builder and the Knowledge Base Builder, that tailor the document processing and QA system to a particular domain by allowing a Subject Matter Expert to contribute to the query representation and to the domain knowledge. During the QA process, the system can interact with the user to improve query terminology by using Spell Checking, Answer Type verification, Expansions and Acronym Clarifications. The system also has capabilities that depend upon, and expand the user’s history of interaction with the system, including a User Profile, Reference Resolution, and Question Similarity modules.

Examining Perception of Digital Information Space

J. A. D'Ignazio et al. — Sat, 04 Sep 2010 16:53:38 PDT

A study using a modified think aloud protocol of University of Rochester undergraduate students' interactions with a general, humanities scholarly database helped a research team gain insight into their information-seeking behavior and thus the impact of the digital library.

Automatic Mapping of Resources to Content Standards

J. Chen et al. — Sat, 04 Sep 2010 16:53:38 PDT

Using Digital Primary Sources for Teaching K-12

Cheryl D. Walters et al. — Sat, 04 Sep 2010 16:53:37 PDT

USU faculty and northern Utah K-12 teachers and library media specialists talk about how to identify and integrate digital primary sources such as photographs, letters, diaries, interviews and more into the curriculum to create engaging lessons. Digital resources such as the Library of Congress' American Memory, Mountain West Digital Library, and Utah Digital Newspapers, as well as tools for organizing the digital objects will be discussed.

Analysis of User Image Descriptions and Automatic Image Indexing Vocabularies: An Exploratory Study

N. Balasubramanian et al. — Sat, 04 Sep 2010 16:53:37 PDT

This study explores the terms assigned by users to index, manage, and describe images and compares them to indexing terms derived automatically by systems for image retrieval. Results of this study indicate that userderived indexing vocabulary largely reflects what users see in the image or what they perceive as the overall topic of an image. This is in contrast to system-derived indexing wherein terms are extracted from existing text surrounding the image. In many cases, the surrounding text does not describe the image, rather, the image is used to illustrate or expand upon the text. Systemderived vocabulary may describe higher level concepts, for example, industrial pollution rather than smoke. The paper concludes with suggestions for the use of natural language processing techniques to provide vocabulary alignment in image retrieval.

Automatic Metadata Generation & Evaluation

E. D. Liddy et al. — Sat, 04 Sep 2010 16:53:36 PDT

The poster reports on a project in which we are investigating methods for breaking the human metadata-generation bottleneck that plagues Digital Libraries. The research question is whether metadata elements and values can be automatically generated from the content of educational resources, and correctly assigned to mathematics and science educational materials. Natural Language Processing and Machine Learning techniques were implemented to automatically assign values of the GEMgenerate metadata element set tofor learning resources provided by the Gateway for Education (GEM), a service that offers web access to a wide range of educational materials. In a user study, education professionals evaluated the metadata assigned to learning resources by either automatic tagging or manual assignment. Results show minimal difference in the eyes of the evaluators between automatically generated metadata and manually assigned metadata.