3^rd PASCAL Textual Entailment Challenge and Resources Pool

(www.pascal-network.org/Challenges/RTE3)

New!Submission Deadline

New!Development Data Set Re-released

New!Test Data Set Released

New!RTE3 OPTIONAL PILOT TASK: Extending the Evaluation of Inferences from Texts

Introduction

Encouraged by the success of the two previous rounds of the Recognizing Textual Entailment (RTE) challenge[1], the RTE organizing committee is glad to launch the 3^rd PASCAL Recognizing Textual Entailment (RTE) Challenge.

The goal of the first RTE challenge was to provide the NLP community with a new benchmark to test progress in recognizing textual entailment, and to compare the achievements of different groups. This goal has proven to be of great interest, and the community's response has encouraged us to gradually extend the scope of the original task to address some novel sub-problems.

The 2^nd RTE Challenge built on the success of the first, with 23 groups from around the world (as compared to 17 for the first challenge) submitting the results of their systems. Representatives of participating groups presented their work at the PASCAL Challenges Workshop in April 2006 in Venice, Italy. The event was successful and the number of participants and their contributions to the discussion demonstrated that Textual Entailment is a quickly growing field of NLP research. Already, the workshops have spawned an impressive number of publications in major conferences, with more work in progress (see RTE-2 website for a comprehensive reference list). Another encouraging sign of the growing interest in the RTE challenge comes from the increase in the number of downloads of the challenge datasets, with 120 registered downloads for the RTE2 development set.

RTE 3 Highlights: What is new in the next challenge?

RTE 3 follows the same basic structure of the previous campaign, in order to facilitate the participation of newcomers and to allow "veterans" to assess the improvements of their systems in a comparable test exercise. Nevertheless, the following innovations are introduced to make the challenge more stimulating and, at the same time, to encourage collaboration between system developers:

• a limited number of longer texts, i.e. up to a paragraph- in order to move toward more comprehensive scenarios which incorporate the need for discourse analysis. However, the majority of examples will remain similar to those in the previous challenges, providing pairs with relatively short texts.
• an RTE Resource Pool has been created where contributors have the possibility to share the resources they use.
• an optional task, "Extending the Evaluation of Inferences from Texts", which explores two other tasks closely related to textual entailment: differentiating unknown from false/contradicts and providing justifications for answers.

In order to avoid copyright problems, data is limited to either what has already been publicly released by official competitions or else is drawn from freely available sources such as Wikinews and Wikipedia.

Optional Pilot Task: Extending the Evaluation of Inferences from Texts

This optional pilot explores two other tasks closely related to textual entailment: differentiating unknown from false/contradicts and providing justifications for answers. This task will piggyback on the existing RTE3 Challenge infrastructure and evaluation process by using the same test set but with a later submission deadline for answers than the primary task.

The goal of making a three-way decision of "YES", "NO" and "UNKNOWN" is to drive systems to make more precise informational distinctions. A hypothesis being unknown on the basis of a text should be distinguished from a hypothesis being shown false/contradicted by a text. The goal for providing justifications for decisions is to explore how eventual users of tools incorporating entailment can be made to understand how decisions were reached by a system. Users are unlikely to trust a system that gives no explanation for its decisions.

The pilot task seeks participation from all interested parties, and we hope that it will be of interest to many PASCAL RTE participants, and can help inform the design of the main task for future RTE Challenges. The US National Institute of Standards and Technology will perform evaluation, using human assessors for the inference task.

This pilot task is being organized by Christopher Manning   < manning at cs . stanford . edu>, Dan Moldovan  < moldovan at languagecomputer . com>, and Ellen Voorhees   < ellen . voorhees at nist . gov>, with input from the PASCAL RTE3 Organizers. Please direct any questions to the pilot organizers, putting "[RTE3]" in the subject header.

Extended Task Description: http://nlp.stanford.edu/RTE3-pilot/

Schedule

• Guidelines distributed: Feb 23, 2007
• A 3-way answer key for the RTE-3 development data is available: Feb 28, 2007.
• Sample justifications (8-10 illustrative examples and how they might be judged) available: Mar 30, 2007.
• Submissions for the extended task are due: April 30, 2007.
• Results for both parts of the extended task returned to participants no later than: June 7, 2007.

The RTE Resource Pool at NLPZone.org

One of the key conclusions at the 2^nd RTE Challenge Workshop was that entailment modeling requires vast knowledge resources that correspond to different types of entailment reasoning. Examples of useful knowledge include ontological and lexical relationships, paraphrases and entailment rules, meaning entailing syntactic transformations and certain types of world knowledge. Textual entailment systems also utilize general NLP tools such as POS taggers, parsers and named-entity recognizers, sometimes posing specialized requirements to such tools. With so many resources being continuously released and improved, it can be difficult to know which particular resource to use when developing a system.

In response, RTE-3 will include a new activity for building an RTE Resource Pool, which will serve as a portal and forum for publicizing and tracking resources, and reporting on their use. We actively solicit both RTE participants and other members of the NLP community who develop or use relevant resources to contribute to the RTE Resource Pool. Contributions include links and descriptions of relevant resources as well as informational postings regarding resource use and accumulated experience. Utilized resources will be cited and evaluated by the RTE-3 participants and their impact will be reviewed in the RTE-3 organizers paper, which we hope will reward contributors of useful resources.

Now NLPZone.org has joined forces with ACLWiki under the auspices of the The Association for Computational Linguistics.

The RTE Resource Pool can be accessed from: http://aclweb.org/aclwiki/index.php?title=Textual_Entailment_Resource_Pool and the Textual Entailment Subzone from: http:http://aclweb.org/aclwiki/index.php?title=Textual_Entailment_Portal.

A growing bibliography of articles related to Textual Entailment is being collected at the aforementioned portal. See References on Textual Entailment.

Organizing Committee

Danilo Giampiccolo, CELCT (Trento), Italy (coordinator)

Bernardo Magnini, ITC-irst (Trento), Italy (advisor)

Ido Dagan, Bar Ilan University, Israel (supervisor and scientific advisor)

Bill Dolan, Microsoft Research, USA

Patrick Pantel, ISI, USA (RTE Resources Pool)

 

Schedule