Articles & Papers

Ontology evaluation poses a number of difficult challenges requiring different evaluation methodologies, particularly for a "dynamic ontology" generated by a combination of automatic and semi-automatic methods. We review evaluation methods that focus solely on syntactic (formal) correctness, on the preservation of semantic structure, or on pragmatic utility. We propose two novel methods for dynamic ontology evaluation and describe the use of these methods for evaluating the different taxonomic representations that are generated at different times or with different amounts of expert feedback. These methods are then applied to the Indiana Philosophy Ontology (InPhO), and used to guide the ontology enrichment process.

Jaimie Murdock, Cameron Buckner, Colin Allen. Evaluating Dynamic Ontologies. Springer-Verlag, 2012.

The unique convergence of humanities scholars, computer scientists, librarians, and information scientists in digital humanities projects highlights the collaborative opportunities such research entails. The digital humanities aspire to create, maintain, and deploy high integrity metadata that are derived from the activities and feedback of domain experts in the humanities, to support scholarly activities in the humanities which meet the high standards of academic peer review. Unfortunately, the relatively limited human resources committed to many digital projects for the humanities have led to unwieldy initial implementations and underutilization of semantic web technology, with the result that most projects in the digital humanities are standalone projects whose data cannot easily be integrated with others. In addition to the barriers arising from idiosyncratic implementations, the difficulties of integrating data from multiple sources are compounded by the use of standards that serve one particular purpose well but do not facilitate other kinds of scholarly activities, often making the combination of resources from different projects laborious and expensive. Thus, much of the potential for collaboration in the digital humanities still remains to be unlocked.

Jaimie Murdock, Colin Allen. InPhO for All: Why APIs Matter. Journal of the Chicago Colloquium of Digital Humanities and Computer Science, Evanston, Illinois, 2011.

Ontology evaluation poses a number of difficult challenges requiring different evaluation methodologies, particularly for a "dynamic ontology" representing a complex set of concepts and generated by a combination of automatic and semi-automatic methods. We review evaluation methods that focus solely on syntactic (formal) correctness, on the preservation of semantic structure, or on pragmatic utility. We propose two novel methods for dynamic ontology evaluation and describe the use of these methods for evaluating the different taxonomic representations that are generated at different times or with different amounts of expert feedback. The proposed "volatility" and "violation" scores represent an attempt to merge syntactic and semantic considerations. Volatility calculates the stability of the methods for ontology generation and extension. Violation measures the degree of "ontological fit" to a text corpus representative of the domain. Combined, they support estimation of convergence towards a stable representation of the domain. No method of evaluation can avoid making substantive normative assumptions about what constitutes "correct" representation but rendering those assumptions explicit can help with the decision about which methods are appropriate for selecting amongst a set of available ontologies or for tuning the design of methods used to generate a hierarchically organized representation of a domain.

Jaimie Murdock, Cameron Buckner, Colin Allen. Two Methods for Evaluating Dynamic Ontologies. Proceedings of the 2nd International Conference on Knowledge Engineering and Ontology Development (KEOD), Valencia, Spain, 2010.

The application of digital humanities techniques to philosophy is changing the way scholars approach the discipline. This paper seeks to open a discussion about the difficulties, methods, opportunities, and dangers of creating and utilizing a formal representation of the discipline of philosophy. We review our current project, the Indiana Philosophy Ontology (InPhO) project, which uses a combination of automated methods and expert feedback to create a dynamic computational ontology for the discipline of philosophy. We argue that our distributed, expert-based approach to modeling the discipline carries substantial practical and philosophical benefits over alternatives. We also discuss challenges facing our project (and any other similar project) as well as the future directions for digital philosophy afforded by formal modeling.

Cameron Buckner, Mathias Niepert, Colin Allen. From Encyclopedia to Ontology: Toward Dynamic Representation of the Discipline of Philosophy. Synthese, Springer-Verlag, 2010.

The "wisdom of crowds" is accomplishing tasks that are cumbersome for individuals yet cannot be fully automated by means of specialized computer algorithms. One such task is the construction of thesauri and other types of concept hierarchies. Human expert feedback on the relatedness and relative generality of terms, however, can be aggregated to dynamically construct evolving concept hierarchies. The InPhO (Indiana Philosophy Ontology) project bootstraps feedback from volunteer users unskilled in ontology design into a precise representation of a specific domain. The approach combines statistical text processing methods with expert feedback and logic programming to create a dynamic semantic representation of the discipline of philosophy. In this paper, we show that results of comparable quality can be achieved by leveraging the workforce of crowdsourcing services such as the Amazon Mechanical Turk (AMT). In an extensive empirical study, we compare the feedback obtained from AMT's workers with that from the InPhO volunteer users providing an insight into qualitative differences of the two groups. Furthermore, we present a set of strategies for assessing the quality of different users when gold standards are missing. We finally use these methods to construct a concept hierarchy based on the feedback acquired from AMT workers.

Kai Eckert, Mathias Niepert, Christof Niemann, Cameron Buckner, Colin Allen, and Heiner Stuckenschmidt. Crowdsourcing the Assembly of Concept Heirarchies. Proceedings of the 10th ACM/IEEE Joint Conference on Digital Libraries (JCDL), Gold Coast, Australia, ACM Press, 2010.

The Indiana Philosophy Ontology (InPhO) project is presented as one of the first social-semantic web endeavors which aims to bootstrap feedback from users unskilled in ontology design into a precise representation of a specific domain. Our approach combines statistical text processing methods with expert feedback and logic programming approaches to create a dynamic semantic representation of the discipline of philosophy. We describe the basic principles and initial experimental results of our system.

Mathias Niepert, Cameron Buckner, Colin Allen. Working the Crowd: Design Principles and Early Lessons from the Social-Semantic Web. Proceedings of the Workshop on Web 3.0: Merging Semantic Web and Social Web - (SW)^2 at ACM Hypertext, Turin, Italy, 2009.

The Indiana Philosophy Ontology (InPhO) is a "dynamic ontology" for the domain of philosophy derived from human input and software analysis. The structured nature of the ontology supports machine reasoning about philosophers and their ideas. It is dynamic because it tracks changes in the content of the online Stanford Encyclopedia of Philosophy. This paper discusses ways of managing the varying expertise of people who supply input to the InPhO and provide feedback on the automated methods.

Colin Allen, Cameron Buckner, Mathias Niepert. The World is Not Flat: Expertise and InPhO. Selected papers from the Ninth Annual WebWise Conference. First Monday, Volume 13, Number 8, 2008.

The next generation of online reference works will require structured representations of their contents in order to support scholarly functions such as semantic search, automated generation of cross-references, tables of contents, and ontology-driven conceptual navigation. Many of these works can be expected to contain massive amounts of data and be updated dynamically, which limits the feasibility of "manually" coded ontologies to keep up with changes in content. However, relationships relevant to inferring an ontology can be recovered from statistical text processing, and these estimates can be verified with carefully-solicited expert feedback. In this paper, we explain a method by which we have used answer set programming on such expert feedback to dynamically populate and partially infer an ontology for a well-established, open-access reference work, the Stanford Encyclopedia of Philosophy.

Mathias Niepert, Cameron Buckner, Colin Allen. Answer Set Programming on Expert Feedback to Populate and Extend Dynamic Ontologies. Proceedings of the 21st International FLAIRS Conference, Coconut Grove, Florida, pages 500-505, AAAI Press, 2008.

The successful deployment of digital technologies by humanities scholars presents computer scientists with a number of unique sci- entific and technological challenges. The task seems particularly daunting because issues in the humanities are presented in abstract language demanding the kind of subtle interpretation often thought to be beyond the scope of artificial intelligence, and humanities scholars themselves often disagree about the structure of their disciplines. The future of humanities computing depends on having tools for automatically discovering complex semantic relationships among different parts of a corpus. Digital library tools for the humanities will need to be capable of dynamically tracking the introduction of new ideas and interpretations and applying them to older texts in ways that support the needs of scholars and students.

This paper describes the design of new algorithms and the adjustment of existing algorithms to support the automated and semi-automated management of domain-rich metadata for an established digital humanities project, the Stanford Encyclopedia of Philosophy. Our approach starts with a "hand-built" formal ontology that is modified and extended by a combination of automated and semi-automated methods, thus becoming a "dynamic ontology". We assess the suitability of current information retrieval and information extraction methods for the task of automatically maintaining the ontology. We describe a novel measure of term-relatedness that appears to be particularly helpful for predicting hierarchical relationships in the ontology. We believe that our project makes a further contribution to information science by being the first to harness the collaboration inherent in a expert-maintained dynamic reference work to the task of maintaining and extending a formal ontology. We place special emphasis on the task of bringing domain expertise to bear on all phases of the development and deployment of the system, from the initial design of the software and ontology to its dynamic use in a fully operational digital reference work.

Mathias Niepert, Cameron Buckner, Colin Allen. A Dynamic Ontology for a Dynamic Reference Work. Proceedings of the 7th ACM/IEEE Joint Conference on Digital Libraries (JCDL), Vancouver, British Columbia, pages 288-297, ACM Press, 2007.

The successful deployment of digital technologies by humanities scholars presents computer scientists with a number of unique sci- entific and technological challenges. The task seems particularly daunting because issues in the humanities are presented in abstract language demanding the kind of subtle interpretation often thought to be beyond the scope of artificial intelligence, and humanities scholars themselves often disagree about the structure of their disciplines. The future of humanities computing depends on having tools for automatically discovering complex semantic relationships among different parts of a corpus. Digital library tools for the humanities will need to be capable of dynamically tracking the introduction of new ideas and interpretations and applying them to older texts in ways that support the needs of scholars and students.

This paper describes the design of new algorithms and the adjustment of existing algorithms to support the automated and semi-automated management of domain-rich metadata for an established digital humanities project, the Stanford Encyclopedia of Philosophy. Our approach starts with a "hand-built" formal ontology that is modified and extended by a combination of automated and semi-automated methods, thus becoming a "dynamic ontology". We assess the suitability of current information retrieval and information extraction methods for the task of automatically maintaining the ontology. We describe a novel measure of term-relatedness that appears to be particularly helpful for predicting hierarchical relationships in the ontology. We believe that our project makes a further contribution to information science by being the first to harness the collaboration inherent in a expert-maintained dynamic reference work to the task of maintaining and extending a formal ontology. We place special emphasis on the task of bringing domain expertise to bear on all phases of the development and deployment of the system, from the initial design of the software and ontology to its dynamic use in a fully operational digital reference work.

Cameron Buckner, Mathias Niepert Colin Allen. InPhO: The Indiana Philosophy Ontology. APA Newsletter on Philosophy and Computers, Volume 7, Number 1, pages 26-28, 2007, 2007.