com.aliasi.classify
Class ScoredClassifierEvaluator<E>

java.lang.Object
  com.aliasi.classify.BaseClassifierEvaluator<E>
      com.aliasi.classify.RankedClassifierEvaluator<E>
          com.aliasi.classify.ScoredClassifierEvaluator<E>

Type Parameters:

E - The type of objects being classified by the evaluated classifier.

All Implemented Interfaces:

Handler, ObjectHandler<Classified<E>>

Direct Known Subclasses:

ConditionalClassifierEvaluator

public class ScoredClassifierEvaluator<E>
extends RankedClassifierEvaluator<E>

A ScoredClassifierEvaluator provides an evaluation harness for score-based classifiers. It extends the ranked classifier evaluator with score-specific evaluation metrics.

Thread Safety

This class must be read-write synchronized externally for use in multiple threads.

Since:

LingPipe3.9.1

Version:

3.9.1

Author:

Bob Carpenter

Constructor Summary

ScoredClassifierEvaluator(ScoredClassifier<E> classifier, String[] categories, boolean storeInputs)
Construct a scored classifier evaluator with the specified classifier, categories and flag indicating whether or not to store inputs.

Method Summary

double	averageScore(String refCategory, String responseCategory) Returns the average score of the specified response category for test cases with the specified reference category.
double	averageScoreReference() Returns the average over all test cases of the score of the response that matches the reference category.
ScoredClassifier<E>	classifier() Return the classifier being evaluated.
void	handle(Classified<E> classified) Add the specified classified object to this evaluator.
ScoredPrecisionRecallEvaluation	scoredOneVersusAll(String refCategory) Returns a scored precision-recall evaluation of the classification of the specified reference category versus all other categories using the classification scores.
void	setClassifier(ScoredClassifier<E> classifier) Set the classifier being evaluated to the specified value.

Methods inherited from class com.aliasi.classify.RankedClassifierEvaluator

averageRank, averageRankReference, meanReciprocalRank, rankCount, setClassifier

Methods inherited from class com.aliasi.classify.BaseClassifierEvaluator

addClassification, categories, confusionMatrix, falseNegatives, falsePositives, numCases, numCategories, oneVersusAll, setClassifier, toString, trueNegatives, truePositives

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

ScoredClassifierEvaluator

public ScoredClassifierEvaluator(ScoredClassifier<E> classifier,
                                 String[] categories,
                                 boolean storeInputs)

Construct a scored classifier evaluator with the specified classifier, categories and flag indicating whether or not to store inputs.

Parameters:

classifier - Classifier to evaluate.

categories - Complete list of categories.

storeInputs - Set to true to store input objects.

Method Detail

setClassifier

public void setClassifier(ScoredClassifier<E> classifier)

Set the classifier being evaluated to the specified value. This method is useful to evaluate multiple classifiers with the same evaluator, for instance for use in cross-validation.

Parameters:

classifier - New classifier for this evaluation.

Throws:

IllegalArgumentException - If called from an evaluator with a runtime type other than ScoredClassifierEvaluator.

classifier

public ScoredClassifier<E> classifier()

Return the classifier being evaluated.

Overrides:

classifier in class RankedClassifierEvaluator<E>

Returns:

The classifier for this evaluator.

handle

public void handle(Classified<E> classified)

Description copied from class: BaseClassifierEvaluator

Add the specified classified object to this evaluator.

Specified by:

handle in interface ObjectHandler<Classified<E>>

Overrides:

handle in class RankedClassifierEvaluator<E>

Parameters:

classified - Classified object to add to evaluation.

scoredOneVersusAll

public ScoredPrecisionRecallEvaluation scoredOneVersusAll(String refCategory)

Returns a scored precision-recall evaluation of the classification of the specified reference category versus all other categories using the classification scores.

Parameters:

refCategory - Reference category.

Returns:

The scored one-versus-all precision-recall evaluatuion.

Throws:

IllegalArgumentException - If the specified category is unknown.

averageScore

public double averageScore(String refCategory,
                           String responseCategory)

Returns the average score of the specified response category for test cases with the specified reference category. If there are no cases matching the reference category, the result is Double.NaN.

Better classifiers return high values when the reference and response categories are the same and lower values when they are different. Depending on the classifier, the scores may or may not be meaningful as an average.

Parameters:

refCategory - Reference category.

responseCategory - Response category.

Returns:

Average score of response category in test cases for specified reference category.

Throws:

IllegalArgumentException - If the either category is unknown.

averageScoreReference

public double averageScoreReference()

Returns the average over all test cases of the score of the response that matches the reference category. Better classifiers return higher values for this average.

Whether average scores make sense across training instances depends on the classifier.

Returns:

The average score of the reference category in the response.