com.aliasi.tokenizer
Class LengthStopFilterTokenizer

java.lang.Object
  com.aliasi.tokenizer.Tokenizer
      com.aliasi.tokenizer.FilterTokenizer
          com.aliasi.tokenizer.StopFilterTokenizer
              com.aliasi.tokenizer.LengthStopFilterTokenizer

All Implemented Interfaces:

Iterable<String>

public class LengthStopFilterTokenizer
extends StopFilterTokenizer

A StopFilterTokenizer removes tokens that exceed a specified length.

Since:

LingPipe3.6

Version:

3.6

Author:

Bob Carpenter

Field Summary

Fields inherited from class com.aliasi.tokenizer.FilterTokenizer

mTokenizer

Constructor Summary

LengthStopFilterTokenizer(Tokenizer tokenizer, int maxTokenLength)
Construct a length filtering tokenizer with the specified maximum length.

Method Summary

int	maxTokenLength() Returns the maximum token length for this filter.
boolean	stop(String token) Returns true if the specified token exceeds the maximum length.

Methods inherited from class com.aliasi.tokenizer.StopFilterTokenizer

nextToken

Methods inherited from class com.aliasi.tokenizer.FilterTokenizer

lastTokenStartPosition, nextWhitespace, setTokenizer

Methods inherited from class com.aliasi.tokenizer.Tokenizer

iterator, tokenize, tokenize

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

LengthStopFilterTokenizer

public LengthStopFilterTokenizer(Tokenizer tokenizer,
                                 int maxTokenLength)

Construct a length filtering tokenizer with the specified maximum length.

Parameters:

maxTokenLength - Maximum token length that is accepted by the filter.

Method Detail

maxTokenLength

public int maxTokenLength()

Returns the maximum token length for this filter.

Returns:

The maximum token length for this filter.

stop

public boolean stop(String token)

Returns true if the specified token exceeds the maximum length.

Specified by:

stop in class StopFilterTokenizer

Parameters:

token - Token to filter.

Returns:

true if the specified token exceeds the maximum length.