com.aliasi.tokenizer
Class TokenFilterTokenizer

java.lang.Object
  com.aliasi.tokenizer.Tokenizer
      com.aliasi.tokenizer.FilterTokenizer
          com.aliasi.tokenizer.TokenFilterTokenizer

All Implemented Interfaces:

Iterable<String>

Direct Known Subclasses:

LowerCaseFilterTokenizer, PorterStemmerFilterTokenizer

public abstract class TokenFilterTokenizer
extends FilterTokenizer

A TokenFilterTokenizer allows a sequence of tokens to be filtered a token at a time. Each token in the input corresponds to one token in the output, with the transform being implemented in a subclass with the method filter(String).

Since:

LingPipe1.0

Version:

1.0.4

Author:

Bob Carpenter

Field Summary

Fields inherited from class com.aliasi.tokenizer.FilterTokenizer

mTokenizer

Constructor Summary

TokenFilterTokenizer(Tokenizer tokenizer)
Construct a token filter tokenizer that filters the specified tokenizer.

Method Summary

abstract String	filter(String token) Returns a filtered version of the specified token.
String	nextToken() Returns the filtered form of the next token produced by the contained tokenizer.

Methods inherited from class com.aliasi.tokenizer.FilterTokenizer

lastTokenStartPosition, nextWhitespace, setTokenizer

Methods inherited from class com.aliasi.tokenizer.Tokenizer

iterator, tokenize, tokenize

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TokenFilterTokenizer

public TokenFilterTokenizer(Tokenizer tokenizer)

Construct a token filter tokenizer that filters the specified tokenizer.

Parameters:

tokenizer - Underlying tokenizer to filter.

Method Detail

nextToken

public String nextToken()

Returns the filtered form of the next token produced by the contained tokenizer.

Overrides:

nextToken in class FilterTokenizer

Returns:

Output of filter applied to next token from filtered tokenizer.

filter

public abstract String filter(String token)

Returns a filtered version of the specified token.

Parameters:

token - Input token.

Returns:

Output token after filtering.