com.aliasi.classify
Class RankedClassifierEvaluator<E>

java.lang.Object
  com.aliasi.classify.BaseClassifierEvaluator<E>
      com.aliasi.classify.RankedClassifierEvaluator<E>

Type Parameters:

E - The type of objects being classified by the evaluated classifier.

All Implemented Interfaces:

Handler, ObjectHandler<Classified<E>>

Direct Known Subclasses:

ScoredClassifierEvaluator

public class RankedClassifierEvaluator<E>
extends BaseClassifierEvaluator<E>

A RankedClassifierEvaluator provides an evaluation harness for ranked classifiers. It extends the base classifier evaluator with ranking-specific evaluation metrics.

Thread Safety

This class must be read-write synchronized externally for use in multiple threads.

Since:

LingPipe3.9.1

Version:

3.9.1

Author:

Bob Carpenter

Constructor Summary

RankedClassifierEvaluator(RankedClassifier<E> classifier, String[] categories, boolean storeInputs)
Construct a ranked classifier evaluator with the specified classifier, categories and flag indicating whether or not to store inputs.

Method Summary

double	averageRank(String refCategory, String responseCategory) Returns the average rank of the specified response category for test cases with the specified reference category.
double	averageRankReference() Returns the average over all test samples of the rank of the the response that matches the reference category.
RankedClassifier<E>	classifier() Return the classifier being evaluated.
void	handle(Classified<E> classified) Add the specified classified object to this evaluator.
double	meanReciprocalRank() Returns the average over all test samples of the reciprocal of one plus the rank of the reference category in the response.
int	rankCount(String referenceCategory, int rank) Returns the number of times that the reference category's rank was the specified rank.
void	setClassifier(RankedClassifier<E> classifier) Set the classifier being evaluated to the specified value.

Methods inherited from class com.aliasi.classify.BaseClassifierEvaluator

addClassification, categories, confusionMatrix, falseNegatives, falsePositives, numCases, numCategories, oneVersusAll, setClassifier, toString, trueNegatives, truePositives

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

RankedClassifierEvaluator

public RankedClassifierEvaluator(RankedClassifier<E> classifier,
                                 String[] categories,
                                 boolean storeInputs)

Construct a ranked classifier evaluator with the specified classifier, categories and flag indicating whether or not to store inputs.

Parameters:

classifier - Classifier to evaluate.

categories - Complete list of categories.

storeInputs - Set to true to store input objects.

Method Detail

setClassifier

public void setClassifier(RankedClassifier<E> classifier)

Set the classifier being evaluated to the specified value. This method is useful to evaluate multiple classifiers with the same evaluator, for instance for use in cross-validation.

Parameters:

classifier - New classifier for this evaluation.

Throws:

IllegalArgumentException - If called from an evaluator with a runtime type other than RankedClassifierEvaluator.

classifier

public RankedClassifier<E> classifier()

Return the classifier being evaluated.

Overrides:

classifier in class BaseClassifierEvaluator<E>

Returns:

The classifier for this evaluator.

handle

public void handle(Classified<E> classified)

Description copied from class: BaseClassifierEvaluator

Add the specified classified object to this evaluator.

Specified by:

handle in interface ObjectHandler<Classified<E>>

Overrides:

handle in class BaseClassifierEvaluator<E>

Parameters:

classified - Classified object to add to evaluation.

rankCount

public int rankCount(String referenceCategory,
                     int rank)

Returns the number of times that the reference category's rank was the specified rank.

For example, in the set of training samples and results described in the method documentation for averageRank(String,String), sample rank counts are as follows:

rankCount("a",0) = 3
rankCount("a",1) = 1
rankCount("a",2) = 0

rankCount("b",0) = 1
rankCount("b",1) = 0
rankCount("b",2) = 1

rankCount("c",0) = 1
rankCount("c",1) = 0
rankCount("c",2) = 0

These results are typically presented as a bar graph histogram per category.

Parameters:

referenceCategory - Reference category.

rank - Rank of count.

Returns:

Number of times the reference category's ranking was the specified rank.

Throws:

IllegalArgumentException - If the category is unknown.

averageRankReference

public double averageRankReference()

Returns the average over all test samples of the rank of the the response that matches the reference category.

Using the example classifications shown in the method documentation of averageRank(String,String):

averageRankReference()
= (0 + 0 + 0 + 1 + 0 + 2 + 0)/7 ~ 0.43

Returns:

The average rank of the reference category in all classification results.

meanReciprocalRank

public double meanReciprocalRank()

Returns the average over all test samples of the reciprocal of one plus the rank of the reference category in the response. This represents counting from one, so if the first-best answer is correct, the reciprocal rank is 1/1; if the second is correct, 1/2; if the third, 1/3; and so on. These individual recirpocals are then averaged over cases.

Using the example classifications shown in the method documentation of averageRank(String,String):

averageRankReference()
= (1/1 + 1/1 + 1/1 + 1/2 + 1/1 + 1/3 + 1/1)/7 ~ 0.83

Returns:

The mean reciprocal rank of the reference category in the result ranking.

averageRank

public double averageRank(String refCategory,
                          String responseCategory)

Returns the average rank of the specified response category for test cases with the specified reference category. If there are no cases matching the reference category, the result is Double.NaN.

Better classifiers return lower values when the reference and response categories are the same and higher values when they are different.

For example, suppose there are three categories, a, b and c. Consider the following seven test cases, with the specified ranked results:

Test Case	Reference	Rank 0	Rank 1	Rank 2
0	a	a	b	c
1	a	a	c	b
2	a	a	b	c
3	a	b	a	c
4	b	b	a	c
5	b	a	c	b
6	c	c	b	a

for which:

averageRank("a","a") = (0 + 0 + 0 + 1)/4 = 0.25
averageRank("a","b") = (1 + 2 + 1 + 0)/4 = 1.00
averageRank("a","c") = (2 + 1 + 2 + 2)/4 = 1.75

averageRank("b","a") = (1 + 0)/2 = 0.50
averageRank("b","b") = (0 + 2)/2 = 1.0
averageRank("b","c") = (2 + 1)/2 = 1.5

averageRank("c","a") = (2)/1 = 2.0
averageRank("c","b") = (1)/1 = 1.0
averageRank("c","c") = (0)/1 = 0.0

If every ranked result is complete in assigning every category to a rank, the sum of the average ranks will be one less than the number of cases with the specified reference value. If categories are missing from ranked results, the sums may possible be larger than one minus the number of test cases.

Note that the confusion matrix is computed using only the reference and first column of this matrix of results.

Parameters:

refCategory - Reference category.

responseCategory - Response category.

Returns:

Average rank of response category in test cases for specified reference category.

Throws:

IllegalArgumentException - If either category is unknown.