com.aliasi.lm
Interface LanguageModel.Tokenized

All Superinterfaces:
LanguageModel
All Known Implementing Classes:
CompiledTokenizedLM, TokenizedLM
Enclosing interface:
LanguageModel

public static interface LanguageModel.Tokenized
extends LanguageModel

A LanguageModel.Tokenized provides a means of estimating the probability of a sequence of tokens. These may be returned in either linear or log form.

Since:
LingPipe2.2
Version:
2.2
Author:
Bob Carpenter

Nested Class Summary
 
Nested classes/interfaces inherited from interface com.aliasi.lm.LanguageModel
LanguageModel.Conditional, LanguageModel.Dynamic, LanguageModel.Process, LanguageModel.Sequence, LanguageModel.Tokenized
 
Method Summary
 double tokenLog2Probability(String[] tokens, int start, int end)
          Returns the log (base 2) probability of the specified token slice in the underlying token n-gram distribution.
 double tokenProbability(String[] tokens, int start, int end)
          Returns the probability of the specified token slice in the token n-gram distribution.
 
Methods inherited from interface com.aliasi.lm.LanguageModel
log2Estimate, log2Estimate
 

Method Detail

tokenLog2Probability

double tokenLog2Probability(String[] tokens,
                            int start,
                            int end)
Returns the log (base 2) probability of the specified token slice in the underlying token n-gram distribution. This includes the estimation of the actual token for unknown tokens.

Parameters:
tokens - Underlying array of tokens.
start - Index of first token in slice.
end - Index of one past the last token in the slice.
Returns:
The log (base 2) probability of the token slice.

tokenProbability

double tokenProbability(String[] tokens,
                        int start,
                        int end)
Returns the probability of the specified token slice in the token n-gram distribution. This estimate includes the estimates of the actual token for unknown tokens.

Parameters:
tokens - Underlying array of tokens.
start - Index of first token in slice.
end - Index of one past the last token in the slice.
Returns:
The probability of the token slice.