com.aliasi.tokenizer
Class FilterTokenizer

java.lang.Object
  com.aliasi.tokenizer.Tokenizer
      com.aliasi.tokenizer.FilterTokenizer

All Implemented Interfaces:

Iterable<String>

Direct Known Subclasses:

NormalizeWhiteSpaceFilterTokenizer, SoundexFilterTokenizer, StopFilterTokenizer, TokenFilterTokenizer

public class FilterTokenizer
extends Tokenizer

A FilterTokenizer contains a tokenizer to which it delegates the tokenizer methods.

Since:

LingPipe1.0

Version:

1.0.3

Author:

Bob Carpenter

Field Summary

protected Tokenizer

mTokenizer The contained tokenizer.

Constructor Summary

FilterTokenizer(Tokenizer tokenizer)
Construct a filter tokenizer that contains the specified tokenizer.

Method Summary

int	lastTokenStartPosition() Returns the starting index of the last token returned.
String	nextToken() Returns the next token from this tokenizer.
String	nextWhitespace() Returns the next white space from this tokenizer.
void	setTokenizer(Tokenizer tokenizer) Sets the contained tokenizer to the specified tokenizer.

Methods inherited from class com.aliasi.tokenizer.Tokenizer

iterator, tokenize, tokenize

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

mTokenizer

protected Tokenizer mTokenizer

The contained tokenizer.

Constructor Detail

FilterTokenizer

public FilterTokenizer(Tokenizer tokenizer)

Construct a filter tokenizer that contains the specified tokenizer.

Parameters:

tokenizer - Contained tokenizer.

Method Detail

setTokenizer

public void setTokenizer(Tokenizer tokenizer)

Sets the contained tokenizer to the specified tokenizer.

Parameters:

tokenizer - New contained tokenizer.

nextToken

public String nextToken()

Returns the next token from this tokenizer. The method is delegated to the contained tokenizer.

Specified by:

nextToken in class Tokenizer

Returns:

Next token from this tokenizer.

nextWhitespace

public String nextWhitespace()

Returns the next white space from this tokenizer. The method is delegated to the contained tokenizer.

Overrides:

nextWhitespace in class Tokenizer

Returns:

Next white space from this tokenizer.

lastTokenStartPosition

public int lastTokenStartPosition()

Returns the starting index of the last token returned. The method is delegated to the contained tokenizer.

Overrides:

lastTokenStartPosition in class Tokenizer

Returns:

Starting index of last token in sequence.