Comparison with Alvey

The evaluation in this section was done with the earlier 1995 release of the grammar. This section compares XTAG to the Alvey Natural Language Tools (ANLT) Grammar. We parsed the set of LDOCE Noun Phrases presented in Appendix B of the technical report ([#!Carroll93!#]) using XTAG. Table  summarizes the results of this experiment. A total of 143 noun phrases were parsed. The NPs which did not have a correct parse in the top three derivations were considered failures for either system. The maximum and average number of derivations columns show the highest and the average number of derivations produced for the NPs that have a correct derivation in the top three. We show the performance of XTAG both with and without the tagger since the performance of the POS tagger is significantly degraded on the NPs because the NPs are usually shorter than the sentences on which it was trained. It would be interesting to see if the two systems performed similarly on a wider range of data.

 

Table: Comparison of XTAG and ANLT Parser

System	# of	# parsed	% parsed	Maximum	Average
	NPs			derivations	derivations
ANLT Parser	143	127	88.81%	32	4.57
XTAG Parser with	143	93	65.03%	28	3.45
POS tagger
XTAG Parser without	143	120	83.91%	28	4.14
POS tagger