com.aliasi.tokenizer
Class NormalizeWhiteSpaceFilterTokenizer

java.lang.Object
  com.aliasi.tokenizer.Tokenizer
      com.aliasi.tokenizer.FilterTokenizer
          com.aliasi.tokenizer.NormalizeWhiteSpaceFilterTokenizer

All Implemented Interfaces:

Iterable<String>

Deprecated. Use WhitespaceNormTokenizerFactory instead.

@Deprecated
public class NormalizeWhiteSpaceFilterTokenizer
extends FilterTokenizer

A NormalizeWhiteSpaceFilterTokenizer reduces each non-empty whitespace to a single space, leaving empty whitespaces alone.

Since:

LingPipe1.0

Version:

3.8

Author:

Bob Carpenter

Field Summary

Fields inherited from class com.aliasi.tokenizer.FilterTokenizer

mTokenizer

Constructor Summary

NormalizeWhiteSpaceFilterTokenizer(Tokenizer tokenizer)
Deprecated. Use WhitespaceNormTokenizerFactory or ModifyTokenTokenizerFactory.modify(Tokenizer) instead.

Method Summary

String

nextWhitespace() Deprecated. Returns the next whitespace, which will either be the single space string Strings.SINGLE_SPACE_STRING or the empty string Strings.EMPTY_STRING.

Methods inherited from class com.aliasi.tokenizer.FilterTokenizer

baseTokenizer, lastTokenEndPosition, lastTokenStartPosition, nextToken, setTokenizer, toString

Methods inherited from class com.aliasi.tokenizer.Tokenizer

iterator, tokenize, tokenize

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

NormalizeWhiteSpaceFilterTokenizer

@Deprecated
public NormalizeWhiteSpaceFilterTokenizer(Tokenizer tokenizer)

Deprecated. Use WhitespaceNormTokenizerFactory or ModifyTokenTokenizerFactory.modify(Tokenizer) instead.

Construct a filter tokenizer that normalizes whitespace, using the specified contained tokenizer.

Parameters:

tokenizer - Contained tokenizer.

Method Detail

nextWhitespace

public String nextWhitespace()

Deprecated.

Returns the next whitespace, which will either be the single space string Strings.SINGLE_SPACE_STRING or the empty string Strings.EMPTY_STRING.

Overrides:

nextWhitespace in class FilterTokenizer

Returns:

Next whitespace.