com.aliasi.tokenizer
Class EnglishStopListFilterTokenizer

java.lang.Object
  extended by com.aliasi.tokenizer.Tokenizer
      extended by com.aliasi.tokenizer.FilterTokenizer
          extended by com.aliasi.tokenizer.StopFilterTokenizer
              extended by com.aliasi.tokenizer.StopListFilterTokenizer
                  extended by com.aliasi.tokenizer.EnglishStopListFilterTokenizer
All Implemented Interfaces:
Iterable<String>

Deprecated. Use EnglishStopTokenizerFactory instead.

@Deprecated
public class EnglishStopListFilterTokenizer
extends StopListFilterTokenizer

An EnglishStopListFilterTokenizer filters its input by removing words on the English stop list. The stoplist is:

a, be, had, it, only, she, was, about, because, has, its, of, some, we, after, been, have, last, on, such, were, all, but, he, more, one, than, when, also, by, her, most, or, that, which, an, can, his, mr, other, the, who, any, co, if, mrs, out, their, will, and, corp, in, ms, over, there, with, are, could, inc, mz, s, they, would, as, for, into, no, so, this, up, at, from, is, not, says, to
Note that the stoplist entries are all lowercase. The input should first be filtered by a LowerCaseFilterTokenizer.

Since:
LingPipe1.0
Version:
3.8
Author:
Bob Carpenter

Field Summary
 
Fields inherited from class com.aliasi.tokenizer.FilterTokenizer
mTokenizer
 
Constructor Summary
EnglishStopListFilterTokenizer(Tokenizer tokenizer)
          Deprecated. Use EnglishStopTokenizerFactory or ModifyTokenTokenizerFactory.modify(Tokenizer) instead.
 
Method Summary
 
Methods inherited from class com.aliasi.tokenizer.StopListFilterTokenizer
stop
 
Methods inherited from class com.aliasi.tokenizer.StopFilterTokenizer
nextToken
 
Methods inherited from class com.aliasi.tokenizer.FilterTokenizer
baseTokenizer, lastTokenEndPosition, lastTokenStartPosition, nextWhitespace, setTokenizer, toString
 
Methods inherited from class com.aliasi.tokenizer.Tokenizer
iterator, tokenize, tokenize
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait
 

Constructor Detail

EnglishStopListFilterTokenizer

@Deprecated
public EnglishStopListFilterTokenizer(Tokenizer tokenizer)
Deprecated. Use EnglishStopTokenizerFactory or ModifyTokenTokenizerFactory.modify(Tokenizer) instead.

Construct an English stoplist filter tokenizer.