com.aliasi.tokenizer
Class FilterTokenizer

java.lang.Object
  extended by com.aliasi.tokenizer.Tokenizer
      extended by com.aliasi.tokenizer.FilterTokenizer
All Implemented Interfaces:
Iterable<String>
Direct Known Subclasses:
NormalizeWhiteSpaceFilterTokenizer, SoundexFilterTokenizer, StopFilterTokenizer, TokenFilterTokenizer

Deprecated. Use ModifiedTokenizerFactory instead.

@Deprecated
public class FilterTokenizer
extends Tokenizer

A FilterTokenizer contains a tokenizer to which it delegates the tokenizer methods.

Since:
LingPipe1.0
Version:
3.8.1
Author:
Bob Carpenter

Field Summary
protected  Tokenizer mTokenizer
          Deprecated. Use the method baseTokenizer() for reads and create a new instance of FilterTokenizer itself for different values.
 
Constructor Summary
FilterTokenizer(Tokenizer tokenizer)
          Deprecated. Construct a filter tokenizer that contains the specified tokenizer.
 
Method Summary
 Tokenizer baseTokenizer()
          Deprecated. Returns the base tokenizer underlying this filtered tokenizer.
 int lastTokenEndPosition()
          Deprecated. Returns the end position of the last token returned.
 int lastTokenStartPosition()
          Deprecated. Returns the starting index of the last token returned.
 String nextToken()
          Deprecated. Returns the next token from this tokenizer.
 String nextWhitespace()
          Deprecated. Returns the next white space from this tokenizer.
 void setTokenizer(Tokenizer tokenizer)
          Deprecated. Create a new immutable instance of FilterTokenizer instead.
 String toString()
          Deprecated. Returns a string representation of this tokenizer including the class name and representation of the contained tokenizer.
 
Methods inherited from class com.aliasi.tokenizer.Tokenizer
iterator, tokenize, tokenize
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait
 

Field Detail

mTokenizer

@Deprecated
protected Tokenizer mTokenizer
Deprecated. Use the method baseTokenizer() for reads and create a new instance of FilterTokenizer itself for different values.
The contained tokenizer.

Constructor Detail

FilterTokenizer

public FilterTokenizer(Tokenizer tokenizer)
Deprecated. 
Construct a filter tokenizer that contains the specified tokenizer.

Parameters:
tokenizer - Contained tokenizer.
Method Detail

baseTokenizer

public Tokenizer baseTokenizer()
Deprecated. 
Returns the base tokenizer underlying this filtered tokenizer.

Returns:
The base tokenizer.

setTokenizer

@Deprecated
public void setTokenizer(Tokenizer tokenizer)
Deprecated. Create a new immutable instance of FilterTokenizer instead.

Sets the contained tokenizer to the specified tokenizer.

Parameters:
tokenizer - New contained tokenizer.

nextToken

public String nextToken()
Deprecated. 
Returns the next token from this tokenizer. The method is delegated to the contained tokenizer.

Specified by:
nextToken in class Tokenizer
Returns:
Next token from this tokenizer.

nextWhitespace

public String nextWhitespace()
Deprecated. 
Returns the next white space from this tokenizer. The method is delegated to the contained tokenizer.

Overrides:
nextWhitespace in class Tokenizer
Returns:
Next white space from this tokenizer.

lastTokenStartPosition

public int lastTokenStartPosition()
Deprecated. 
Returns the starting index of the last token returned. The method is delegated to the contained tokenizer.

Overrides:
lastTokenStartPosition in class Tokenizer
Returns:
Starting index of last token in sequence.

lastTokenEndPosition

public int lastTokenEndPosition()
Deprecated. 
Returns the end position of the last token returned. The end is one past the last character in the token. The method is delegated to the contained tokenizer.

Overrides:
lastTokenEndPosition in class Tokenizer
Returns:
Starting index of last token in sequence.

toString

public String toString()
Deprecated. 
Returns a string representation of this tokenizer including the class name and representation of the contained tokenizer. This does not include the characters or current position.

Overrides:
toString in class Object
Returns:
A string representation of this tokenizer.