com.aliasi.tokenizer
Class NormalizeWhiteSpaceFilterTokenizer

java.lang.Object
  extended by com.aliasi.tokenizer.Tokenizer
      extended by com.aliasi.tokenizer.FilterTokenizer
          extended by com.aliasi.tokenizer.NormalizeWhiteSpaceFilterTokenizer
All Implemented Interfaces:
Iterable<String>

Deprecated. Use WhitespaceNormTokenizerFactory instead.

@Deprecated
public class NormalizeWhiteSpaceFilterTokenizer
extends FilterTokenizer

A NormalizeWhiteSpaceFilterTokenizer reduces each non-empty whitespace to a single space, leaving empty whitespaces alone.

Since:
LingPipe1.0
Version:
3.8
Author:
Bob Carpenter

Field Summary
 
Fields inherited from class com.aliasi.tokenizer.FilterTokenizer
mTokenizer
 
Constructor Summary
NormalizeWhiteSpaceFilterTokenizer(Tokenizer tokenizer)
          Deprecated. Use WhitespaceNormTokenizerFactory or ModifyTokenTokenizerFactory.modify(Tokenizer) instead.
 
Method Summary
 String nextWhitespace()
          Deprecated. Returns the next whitespace, which will either be the single space string Strings.SINGLE_SPACE_STRING or the empty string Strings.EMPTY_STRING.
 
Methods inherited from class com.aliasi.tokenizer.FilterTokenizer
baseTokenizer, lastTokenEndPosition, lastTokenStartPosition, nextToken, setTokenizer, toString
 
Methods inherited from class com.aliasi.tokenizer.Tokenizer
iterator, tokenize, tokenize
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait
 

Constructor Detail

NormalizeWhiteSpaceFilterTokenizer

@Deprecated
public NormalizeWhiteSpaceFilterTokenizer(Tokenizer tokenizer)
Deprecated. Use WhitespaceNormTokenizerFactory or ModifyTokenTokenizerFactory.modify(Tokenizer) instead.

Construct a filter tokenizer that normalizes whitespace, using the specified contained tokenizer.

Parameters:
tokenizer - Contained tokenizer.
Method Detail

nextWhitespace

public String nextWhitespace()
Deprecated. 
Returns the next whitespace, which will either be the single space string Strings.SINGLE_SPACE_STRING or the empty string Strings.EMPTY_STRING.

Overrides:
nextWhitespace in class FilterTokenizer
Returns:
Next whitespace.