E- Type of tokens in the tagging.
public class NBestTaggerEvaluator<E> extends Object implements ObjectHandler<Tagging<E>>
NBestTaggerEvaluatorprovides an evaluation framework for n-best taggers.
Test cases may be added directly using the
addCase(Tagging,Iterator) method, which accepts a reference
gold-standard tagging and a system response consisting of an
iterator such as the result produced by an n-best tagger.
A specific tagger may be supplied to the constructor or set
setTagger(NBestTagger) method. Test cases may
be supplied to the tagger through the object handler method
handle(Tagging), which accepts a gold-standard reference tagging.
The tagger is then used to produce the system response which is
then added as a test case.
The main n-best evaluation is the histogram returned
nBestHistogram(), which provides counts for
the number of times the correct reference result was
found in the response at a particular rank.
recallAtN() returns an array of
recall values indexed by the rank of results. For instance,
recallAtN() is the percentage of cases for which
the first-best result was correct,
the percentage of cases where the first-best result was
returned as the first or second result (ranks 0 or 1).
A string-based representation of the last case that was
evaluated is available through
The report is based on a set of known tokens, which is up to
the evaluation client to provide; null values evaluate without
previously known tokens being used.
An n-best tagger evaluator must be read-write synchronized.
The write methods are
|Constructor and Description|
Construct an n-best tagger evaluator using the specified tagger, restricting the response taggings to the maximum number of outputs specified, and writing the specified number of outputs to the last case string.
|Modifier and Type||Method and Description|
Add a test case consisting of the specified reference tagging and iterator over responses.
Add the specified reference tagging as a test case, with a response tagging computed by the contained n-best tagger.
Return a string-based representation of the last case evaluated.
Returns the maximum number of results examined in the response for each test case.
Return the histogram of results mapping ranks to the number of test cases where the correct result was at that rank.
Return the number of test cases in this evaluation.
Return the total number of tokens in all test cases for this evaluator.
Return an array of recall values indexed by rank.
Set the tagger to the specified value.
Return the n-best tagger used for this evaluator.
public NBestTaggerEvaluator(NBestTagger<E> tagger, int maxNBest, int maxNBestToString)
tagger- Tagger to evaluate, or
nullif cases are added directly or the tagger is set later.
maxNBest- Maximum number of n-best results in the system response to evaluate.
maxNBestToString- Maximum number of n-best results to write to the string output for the last case.
public int maxNBest()
public void setTagger(NBestTagger<E> tagger)
tagger- Tagger to use for evaluation.
public NBestTagger<E> tagger()
public void addCase(Tagging<E> referenceTagging, Iterator<ScoredTagging<E>> responseTaggingIterator)
referenceTagging- Reference gold-standard tagging.
responseTaggingIterator- System response as an iterator over taggings.
public ObjectToCounterMap<Integer> nBestHistogram()
public double recallAtN()
public int numCases()
public long numTokens()
public String lastCaseToString(int maxNBestReport)
maxNBestReport- Maximum number of n-best results to report.