com.aliasi.tokenizer
Class StopListFilterTokenizer

java.lang.Object
  extended by com.aliasi.tokenizer.Tokenizer
      extended by com.aliasi.tokenizer.FilterTokenizer
          extended by com.aliasi.tokenizer.StopFilterTokenizer
              extended by com.aliasi.tokenizer.StopListFilterTokenizer
All Implemented Interfaces:
Iterable<String>
Direct Known Subclasses:
EnglishStopListFilterTokenizer

Deprecated. Use StopTokenizerFactory instead.

@Deprecated
public class StopListFilterTokenizer
extends StopFilterTokenizer

A StopListFilterTokenizer is a stop-list-based stop filter tokenizer that removes tokens from a tokenizer stream if they are on a specified list of so-called ``stop'' tokens.

Since:
LingPipe1.0
Version:
3.8
Author:
Bob Carpenter

Field Summary
 
Fields inherited from class com.aliasi.tokenizer.FilterTokenizer
mTokenizer
 
Constructor Summary
StopListFilterTokenizer(Tokenizer tokenizer, Set<String> stopList)
          Deprecated. Use StopTokenizerFactory instead.
 
Method Summary
 boolean stop(String token)
          Deprecated. Returns true if the specified token should be ignored.
 
Methods inherited from class com.aliasi.tokenizer.StopFilterTokenizer
nextToken
 
Methods inherited from class com.aliasi.tokenizer.FilterTokenizer
baseTokenizer, lastTokenEndPosition, lastTokenStartPosition, nextWhitespace, setTokenizer, toString
 
Methods inherited from class com.aliasi.tokenizer.Tokenizer
iterator, tokenize, tokenize
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait
 

Constructor Detail

StopListFilterTokenizer

@Deprecated
public StopListFilterTokenizer(Tokenizer tokenizer,
                                          Set<String> stopList)
Deprecated. Use StopTokenizerFactory instead.

Construct a stop-list-based stop filter tokenizer based on the specified set of stop tokens.

Keeps a handle on the set, so that changes to the set will affect this stop list.

Parameters:
stopList - Set of tokens to remove from token streams.
Method Detail

stop

public boolean stop(String token)
Deprecated. 
Returns true if the specified token should be ignored.

Specified by:
stop in class StopFilterTokenizer
Parameters:
token - Token to test for removal.
Returns:
true if the token should be removed from the stream of tokens.