Manas Hardas

A Proposed Approach for Semantic Evaluation of Test Problems and Information Extraction from Course Ontology

Date: Monday, October 23, 2006
Time: 1:00
Place: 274 MSB

Committee:
Dr. Javed Khan, Computer Science (Advisor)
Dr. Paul Wang, Computer Science
Dr. Ruoming Jin, Computer Science
Dr. Anne Reynolds, Teaching, Leadership and Curriculum Studies

Abstract:
Evaluation is an integral part of any teaching and learning process. One of the fundamental types of educational resource is test problems. This research investigates the properties of test problems by following a purely knowledge based approach for assessment of knowledge content and complexity using course ontologies. The knowledge content of a test problem can give insights into the composition and knowledge required to answer that particular question while the evaluation for complexity can give a basis for grading of the answer to that question and extracting information from answers.

Ontologies, used to represent the backend context for the course, derive their roots from philosophy where they are used to represent the account of what exists. In computer science ontologies are generally defined as the specification of a conceptualization in a particular domain of knowledge. Ontologies progressively are used to represent structured information in a hierarchical format. Course ontology, particularly, can be roughly defined as a hierarchical representation of the topics involved in the course, connected by relationships with specific semantic significance. We present a novel approach for representing course ontologies in an expressible and computable format using “has-prerequisite” holonymy relationships. It differs from traditional ontologies most significantly in that, it is not IS-A relation based and it is not a directed acyclic graph (DAG) as most traditional ontologies. Recent advances in metadata representation standards like OWL and RDF now provide a convenient platform for metadata representation for such resources. This presents real opportunities to create systems that can design and reuse complex educational resources such as test problems.

We isolate the main pedagogical challenge as finding measurable quantities that can provide guidance in the process of automatic evaluation and propose an assessment system based on evaluation parameters which might offer some insight into the internal composition of the test problems. These parameters are tested in one of the real world courses being offered. The performance of the students is observed and used in determining the effectiveness of these parameters in predicting the knowledge content and difficulty of a problem.