com.aliasi.tokenizer
Class PunctuationStopListTokenizer

java.lang.Object
  extended by com.aliasi.tokenizer.Tokenizer
      extended by com.aliasi.tokenizer.FilterTokenizer
          extended by com.aliasi.tokenizer.StopFilterTokenizer
              extended by com.aliasi.tokenizer.PunctuationStopListTokenizer
All Implemented Interfaces:
Iterable<String>

Deprecated. Use RegExFilteredTokenizerFactory with a pattern matching the characters specified in Strings.allPunctuation(String).

@Deprecated
public class PunctuationStopListTokenizer
extends StopFilterTokenizer

A PunctuationStopListTokenizer removes tokens consisting entirely of punctuation. Punctuation is defined by the method Strings.allPunctuation(String).

Since:
LingPipe3.3
Version:
3.8
Author:
Bob Carpenter

Field Summary
 
Fields inherited from class com.aliasi.tokenizer.FilterTokenizer
mTokenizer
 
Constructor Summary
PunctuationStopListTokenizer(Tokenizer tokenizer)
          Deprecated. Use RegExFilteredTokenizerFactory.RegExFilteredTokenizerFactory(TokenizerFactory,java.util.regex.Pattern) instead.
 
Method Summary
 boolean stop(String token)
          Deprecated. Returns true if the specified token consists entirely of punctuation.
 
Methods inherited from class com.aliasi.tokenizer.StopFilterTokenizer
nextToken
 
Methods inherited from class com.aliasi.tokenizer.FilterTokenizer
baseTokenizer, lastTokenEndPosition, lastTokenStartPosition, nextWhitespace, setTokenizer, toString
 
Methods inherited from class com.aliasi.tokenizer.Tokenizer
iterator, tokenize, tokenize
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait
 

Constructor Detail

PunctuationStopListTokenizer

@Deprecated
public PunctuationStopListTokenizer(Tokenizer tokenizer)
Deprecated. Use RegExFilteredTokenizerFactory.RegExFilteredTokenizerFactory(TokenizerFactory,java.util.regex.Pattern) instead.

Construct a punctuation stop-list tokenizer.

Parameters:
tokenizer - Tokenizer to filter.
Method Detail

stop

public boolean stop(String token)
Deprecated. 
Returns true if the specified token consists entirely of punctuation.

Specified by:
stop in class StopFilterTokenizer
Parameters:
token - Token to test for removal.
Returns:
true if the token is all punctuation.