com.aliasi.tokenizer
Class TokenFilterTokenizer

java.lang.Object
  extended by com.aliasi.tokenizer.Tokenizer
      extended by com.aliasi.tokenizer.FilterTokenizer
          extended by com.aliasi.tokenizer.TokenFilterTokenizer
All Implemented Interfaces:
Iterable<String>
Direct Known Subclasses:
LowerCaseFilterTokenizer, PorterStemmerFilterTokenizer

Deprecated. Use ModifyTokenTokenizerFactory instead.

@Deprecated
public abstract class TokenFilterTokenizer
extends FilterTokenizer

A TokenFilterTokenizer allows a sequence of tokens to be filtered a token at a time. Each token in the input corresponds to one token in the output, with the transform being implemented in a subclass with the method filter(String).

Since:
LingPipe1.0
Version:
3.8
Author:
Bob Carpenter

Field Summary
 
Fields inherited from class com.aliasi.tokenizer.FilterTokenizer
mTokenizer
 
Constructor Summary
TokenFilterTokenizer(Tokenizer tokenizer)
          Deprecated. Construct a token filter tokenizer that filters the specified tokenizer.
 
Method Summary
abstract  String filter(String token)
          Deprecated. Returns a filtered version of the specified token.
 String nextToken()
          Deprecated. Returns the filtered form of the next token produced by the contained tokenizer.
 
Methods inherited from class com.aliasi.tokenizer.FilterTokenizer
baseTokenizer, lastTokenEndPosition, lastTokenStartPosition, nextWhitespace, setTokenizer, toString
 
Methods inherited from class com.aliasi.tokenizer.Tokenizer
iterator, tokenize, tokenize
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait
 

Constructor Detail

TokenFilterTokenizer

public TokenFilterTokenizer(Tokenizer tokenizer)
Deprecated. 
Construct a token filter tokenizer that filters the specified tokenizer.

Parameters:
tokenizer - Underlying tokenizer to filter.
Method Detail

nextToken

public String nextToken()
Deprecated. 
Returns the filtered form of the next token produced by the contained tokenizer. If the next token returned by the tokenizer is null, it is not passed through the filter, but returned as-is.

Overrides:
nextToken in class FilterTokenizer
Returns:
Output of filter applied to next token from filtered tokenizer.

filter

public abstract String filter(String token)
Deprecated. 
Returns a filtered version of the specified token.

Parameters:
token - Input token.
Returns:
Output token after filtering.