com.aliasi.tokenizer
Class StopFilterTokenizer

java.lang.Object
  com.aliasi.tokenizer.Tokenizer
      com.aliasi.tokenizer.FilterTokenizer
          com.aliasi.tokenizer.StopFilterTokenizer

All Implemented Interfaces:

Iterable<String>

Direct Known Subclasses:

LengthStopFilterTokenizer, PunctuationStopListTokenizer, StopListFilterTokenizer

Deprecated. Use ModifyTokenTokenizerFactory instead.

@Deprecated
public abstract class StopFilterTokenizer
extends FilterTokenizer

A StopFilterTokenizer removes tokens from the token stream if they meet conditions specified by concrete subclasses. Subclasses must implement the stop(String) method, which determines whether a token should be removed from the token stream.

If a token is removed, so is the whitespace immediately following it.

Since:

LingPipe1.0

Version:

3.8

Author:

Bob Carpenter

Field Summary

Fields inherited from class com.aliasi.tokenizer.FilterTokenizer

mTokenizer

Constructor Summary

StopFilterTokenizer(Tokenizer tokenizer)
Deprecated. Construct a stop filter tokenizer from the specified tokenizer.

Method Summary

String	nextToken() Deprecated. Returns the next token that does not satisfy stop(String), or null if there are no more underlying tokens.
abstract boolean	stop(String token) Deprecated. Returns true if the specified token should be filtered out.

Methods inherited from class com.aliasi.tokenizer.FilterTokenizer

baseTokenizer, lastTokenEndPosition, lastTokenStartPosition, nextWhitespace, setTokenizer, toString

Methods inherited from class com.aliasi.tokenizer.Tokenizer

iterator, tokenize, tokenize

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

StopFilterTokenizer

public StopFilterTokenizer(Tokenizer tokenizer)

Deprecated.

Construct a stop filter tokenizer from the specified tokenizer.

Parameters:

tokenizer - Tokenizer from which to read tokens.

Method Detail

nextToken

public String nextToken()

Deprecated.

Returns the next token that does not satisfy stop(String), or null if there are no more underlying tokens.

Overrides:

nextToken in class FilterTokenizer

Returns:

Next token that is not stopped.

stop

public abstract boolean stop(String token)

Deprecated.

Returns true if the specified token should be filtered out. The argument token will never be null.

Parameters:

token - Token to test for removal.

Returns:

true if the token should be removed.