com.aliasi.tokenizer
Class StopFilterTokenizer

java.lang.Object
  extended by com.aliasi.tokenizer.Tokenizer
      extended by com.aliasi.tokenizer.FilterTokenizer
          extended by com.aliasi.tokenizer.StopFilterTokenizer
All Implemented Interfaces:
Iterable<String>
Direct Known Subclasses:
LengthStopFilterTokenizer, PunctuationStopListTokenizer, StopListFilterTokenizer

Deprecated. Use ModifyTokenTokenizerFactory instead.

@Deprecated
public abstract class StopFilterTokenizer
extends FilterTokenizer

A StopFilterTokenizer removes tokens from the token stream if they meet conditions specified by concrete subclasses. Subclasses must implement the stop(String) method, which determines whether a token should be removed from the token stream.

If a token is removed, so is the whitespace immediately following it.

Since:
LingPipe1.0
Version:
3.8
Author:
Bob Carpenter

Field Summary
 
Fields inherited from class com.aliasi.tokenizer.FilterTokenizer
mTokenizer
 
Constructor Summary
StopFilterTokenizer(Tokenizer tokenizer)
          Deprecated. Construct a stop filter tokenizer from the specified tokenizer.
 
Method Summary
 String nextToken()
          Deprecated. Returns the next token that does not satisfy stop(String), or null if there are no more underlying tokens.
abstract  boolean stop(String token)
          Deprecated. Returns true if the specified token should be filtered out.
 
Methods inherited from class com.aliasi.tokenizer.FilterTokenizer
baseTokenizer, lastTokenEndPosition, lastTokenStartPosition, nextWhitespace, setTokenizer, toString
 
Methods inherited from class com.aliasi.tokenizer.Tokenizer
iterator, tokenize, tokenize
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait
 

Constructor Detail

StopFilterTokenizer

public StopFilterTokenizer(Tokenizer tokenizer)
Deprecated. 
Construct a stop filter tokenizer from the specified tokenizer.

Parameters:
tokenizer - Tokenizer from which to read tokens.
Method Detail

nextToken

public String nextToken()
Deprecated. 
Returns the next token that does not satisfy stop(String), or null if there are no more underlying tokens.

Overrides:
nextToken in class FilterTokenizer
Returns:
Next token that is not stopped.

stop

public abstract boolean stop(String token)
Deprecated. 
Returns true if the specified token should be filtered out. The argument token will never be null.

Parameters:
token - Token to test for removal.
Returns:
true if the token should be removed.