com.aliasi.classify
Class BaseClassifierEvaluator<E>

java.lang.Object
  extended by com.aliasi.classify.BaseClassifierEvaluator<E>
Type Parameters:
E - The type of objects being classified by the evaluated classifier.
All Implemented Interfaces:
Handler, ObjectHandler<Classified<E>>
Direct Known Subclasses:
RankedClassifierEvaluator

public class BaseClassifierEvaluator<E>
extends Object
implements ObjectHandler<Classified<E>>

A BaseClassifierEvaluator provides an evaluation harness for first-best classifiers. An evaluator is constructed from a classifier and a complete list of the categories returned by the classifier.

Test cases are then added using the handle(Classified) which accepts a string-based category and object to classify. The evaluator will run the classifier over the input object and collect results over multiple cases.

There are subtypes of this classifier that evaluate richer classifiers.

An exhaustive set of evaluation metrics for first-best classification results is accessbile as a confusion matrix through the confusionMatrix() method. Confusion matrices provide dozens of statistics on classification which can be computed from first-best results; see ConfusionMatrix for more information.

Thread Safety

This class requires concurrent read and synchronous write synchronization. Reads are any of the statistics gathering methods and write is just adding new test cases.

Storing Cases

This class always stores the classification results and true category of an cases. There is a flag in the constructor that additionally allows the inputs for cases to be stored as part of an evaluation. If the flag is set to true, all input cases are stored. This enables the output of true positives, false positives, false negatives, and true negatives through the methods of the same names.

Since:
LingPipe3.9.1
Version:
3.9.1
Author:
Bob Carpenter

Constructor Summary
BaseClassifierEvaluator(BaseClassifier<E> classifier, String[] categories, boolean storeInputs)
          Construct a classifier evaluator for the specified classifier that records results for the specified set of categories, storing cases or not based on the specified flag.
 
Method Summary
 void addClassification(String referenceCategory, Classification classification, E input)
          Adds the specified classification as a response for the specified reference category.
 String[] categories()
          Returns a copy of the the categories for which this evaluator stores results.
 BaseClassifier<E> classifier()
          Returns the classifier for this evaluator.
 ConfusionMatrix confusionMatrix()
          Returns the confusion matrix of first-best classification result statistics for this evaluator.
 List<Classified<E>> falseNegatives(String category)
          Returns the list of false negative cases along with their classifications for items of the specified category.
 List<Classified<E>> falsePositives(String category)
          Returns the list of false positive cases along with their classifications for items of the specified category.
 void handle(Classified<E> classified)
          Add the specified classified object to this evaluator.
 int numCases()
          Returns the number of test cases which have been provided to this evaluator.
 int numCategories()
          Returns the number of categories for the classifier being evaluated.
 PrecisionRecallEvaluation oneVersusAll(String refCategory)
          Returns the first-best one-versus-all precision-recall evaluation of the classification of the specified reference category versus all other categories.
 void setClassifier(BaseClassifier<E> classifier)
          Set the classfier for this evaluator to the specified value.
 String toString()
          Return a string-based representation of the results of the evaluation.
 List<Classified<E>> trueNegatives(String category)
          Returns the list of true negative cases along with their classifications for items of the specified category.
 List<Classified<E>> truePositives(String category)
          Returns the list of true positive cases along with their classifications for items of the specified category.
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait
 

Constructor Detail

BaseClassifierEvaluator

public BaseClassifierEvaluator(BaseClassifier<E> classifier,
                               String[] categories,
                               boolean storeInputs)
Construct a classifier evaluator for the specified classifier that records results for the specified set of categories, storing cases or not based on the specified flag.

If the classifier evaluator is only going to be populated using the addClassification(String,Classification,Object) method, then the classifier may be null.

Parameters:
classifier - Classifier to evaluate.
categories - Categories of the classifier.
storeInputs - Flag indicating whether input cases should be stored.
Method Detail

numCategories

public int numCategories()
Returns the number of categories for the classifier being evaluated.

Returns:
Number of categories being evaluated.

categories

public String[] categories()
Returns a copy of the the categories for which this evaluator stores results.

Returns:
The categories for which this evaluator stores results.

classifier

public BaseClassifier<E> classifier()
Returns the classifier for this evaluator.

Returns:
The classifier for this evaluator.

setClassifier

public void setClassifier(BaseClassifier<E> classifier)
Set the classfier for this evaluator to the specified value. This method allows the results of evaluating several different classifiers to be aggregated into a single evaluation. The primary use case is cross-validation, where a single evaluator may be used to evaluate classifiers trained on different folds.

This method will throw an exception if called from an evaluator with a more specific runtime type.

Parameters:
classifier - New classifier for this evaluator.
Throws:
IllegalArgumentException - If called from a class with a runtime type other than BaseClassifierEvaluator.

truePositives

public List<Classified<E>> truePositives(String category)
Returns the list of true positive cases along with their classifications for items of the specified category.

The cases will be returned in decreasing order of conditional probability if applicable, decreasing order of score otherwise, and if not scored, in the order in which they were processed.

A true positive case for the specified category has reference category equal to the specified category and first-best classification result equal to the specified category.

Parameters:
category - Category whose cases are returned.
Returns:
True positives for specified category.
Throws:
UnsupportedOperationException - If this class does not store its cases.

falsePositives

public List<Classified<E>> falsePositives(String category)
Returns the list of false positive cases along with their classifications for items of the specified category.

A false positive case for the specified category has reference category unequal to the specified category and first-best classification result equal to the specified category.

Parameters:
category - Category whose cases are returned.
Returns:
False positives for specified category.
Throws:
UnsupportedOperationException - If this class does not store its cases.

falseNegatives

public List<Classified<E>> falseNegatives(String category)
Returns the list of false negative cases along with their classifications for items of the specified category.

A false negative case for the specified category has reference category equal to the specified category and first-best classification result unequal to the specified category.

Parameters:
category - Category whose cases are returned.
Returns:
False negatives for specified category.
Throws:
UnsupportedOperationException - If this class does not store its cases.

trueNegatives

public List<Classified<E>> trueNegatives(String category)
Returns the list of true negative cases along with their classifications for items of the specified category.

A true negative case for the specified category has reference category unequal to the specified category and first-best classification result unequal to the specified category.

Parameters:
category - Category whose cases are returned.
Returns:
True positives for specified category.
Throws:
UnsupportedOperationException - If this class does not store its cases.

handle

public void handle(Classified<E> classified)
Add the specified classified object to this evaluator.

Specified by:
handle in interface ObjectHandler<Classified<E>>
Parameters:
classified - Classified object to add to evaluation.

addClassification

public void addClassification(String referenceCategory,
                              Classification classification,
                              E input)
Adds the specified classification as a response for the specified reference category. If this evaluator stores cases, the input will be stored.

Parameters:
referenceCategory - Reference category for case.
classification - Response classification for case.
input - Input for the specified classification.

numCases

public int numCases()
Returns the number of test cases which have been provided to this evaluator.

Returns:
The number of test cases which have been provided to this evaluator.

confusionMatrix

public ConfusionMatrix confusionMatrix()
Returns the confusion matrix of first-best classification result statistics for this evaluator. See ConfusionMatrix for details of the numerous available evaluation metrics provided by confusion matrices.

Returns:
The confusion matrix for the test cases evaluated so far.

oneVersusAll

public PrecisionRecallEvaluation oneVersusAll(String refCategory)
Returns the first-best one-versus-all precision-recall evaluation of the classification of the specified reference category versus all other categories. This method may be called for any evaluation.

Parameters:
refCategory - Reference category.
Returns:
The first-best one-versus-all precision-recall evaluatuion.
Throws:
IllegalArgumentException - If the specified category is unknown.

toString

public String toString()
Return a string-based representation of the results of the evaluation.

Overrides:
toString in class Object
Returns:
String-based representation of the results.