Below you will find the results of my experiments on Maximum Entropy parse disambiguation. All of the experiments are performed on the WeScience treebank. The reported scores are all 10-fold crossvalidations of exact match accuracies on only the ambiguous sentences. The random choice baseline is 8.1%. The purpose of these experiments is twofold: To determine the relative merits of a variety of feature types; To establish a strong baseline for more exotic future experiments. For each configuration of feature templates studied, a series of 50 (sometimes more) crossvalidation runs with different regularization parameters was undertaken. The highest accuracy score and corresponding regularization parameter is reported in each case; graphs are included showing accuracy as a function of regularization parameter, which help to highlight which differences are significant and which are not.

Following are descriptions of the feature templates employed:

The results of the experiments: Here are links to a few more performance curves, showing head-to-head accuracies of several classes of features: grandparenting levels, n-gram feature curves, assorted unuseful feature curves, and everything.

These conclusions are only initial; there are still many more feature template types to try. Grandparenting is the most effective feature template, with probably a maximum useful level of three grandparents. Unparenting and punctuation show little improvement. N-grams and fragmented contexts may offer small gains, but are at best secondary information sources. It is likely that the information in some of these feature sets is independent of the others, and hence that further gains in accuracy are possible by combining some of the best performers, e.g. GP[3] plus lpp2gram. It is also conceivable that certain combinations of these features would offer more information together than individually, i.e. the whole can be more than the sum of the parts. However, in general it appears that 45% or so is a difficult accuracy to beat.

Berger 1996, Johnson et al 1999, Toutanova et al 2002, 2004, 2005