LingPipe Home

What is LingPipe?

Jargon Key

Text Data Mining / Text Analytics: Extracting and analyzing information patterns in text collections.
Discovery: Uncovering facts in text data and analyzing their connections.
Database Linkage: Matching extracted entities and relations to databases or ontologies.
Unstructured Information Management: Storage, analysis, and retrieval of human language data.
Natural Language Processing (NLP): Handling human language by computer.
Computational Linguistics: Science of the computational properties of language.
N-Best and Confidence: The ability to return multiple hypotheses ranked by statistical confidence.
Corpus: A collection of human language, often annotated with linguistic and/or content features.
API: Application Programmer's Interface; the language to communicate with LingPipe.
Concurrency: Doing more than one thing simultaneously.

LingPipe is a suite of Java libraries for the linguistic analysis of human language.

Feature Overview

LingPipe's information extraction and data mining tools:

track mentions of entities (e.g. people or proteins);
link entity mentions to database entries;
uncover relations between entities and actions;
classify text passages by language, character encoding, genre, topic, or sentiment;
correct spelling with respect to a text collection;
cluster documents by implicit topic and discover significant trends over time; and
provide part-of-speech tagging and phrase chunking.

Architecture

LingPipe's architecture is designed to be efficient, scalable, reusable, and robust. Highlights include:

Java API with source code and unit tests;
multi-lingual, multi-domain, multi-genre models;
training with new data for new tasks;
n-best output with statistical confidence estimates;
online training (learn-a-little, tag-a-little);
thread-safe models and decoders for concurrent-read exclusive-write (CREW) synchronization; and
character encoding-sensitive I/O.

Latest Release: LingPipe 3.5.1

Tarball = Web Site = CD

This web site is simply the image of the unpacked tarball. So are the CDs we periodically burn, if you happen to see one of those.

It took us a while to figure this out, but now everything works the same way everywhere.

This means you can start with the jar and download everything else you need a piece at a time. And you can also look at all of our source, build files, etc., on the web.

For instance, the rescoring named entity decoder is in:

src/com/aliasi/chunk/ RescoringChunker.java

and the top-level build file is in:

build.xml

and the model for the part-of-speech tagger for general English is in:

demos/models/ pos-en-general-brown.HiddenMarkovModel

Patch Release

The latest release of LingPipe is LingPipe 3.5.1. This release replaces LingPipe 3.5.0, with which it is fully backward compatible.

Bug Fixes

Fast Cache

The performance (speed and size) bug in util.FastCache has been patched. In addition, there's a new cache implementation util.HardFastCache that does not use soft references to reduce load on the garbage collector.

XML Element Stack Filter

The class xml.ElementStackFilter was patched to deal with implementations of SAX that reuse the same attributes element on each callback. Now the element stack filter copies the attributes to a local version for later access.

Missing Demo Files and Libraries

A demo input file for evaluating spell checking was missing and is now included.

Generic Demo XML Dependency

The NekoHTML package we use in our demos to parse HTML input depends on the XercesJ XML libraries, so we included them along with the appropriate classpaths on all of the commands and in the web service demos.

Convenience Methods and Generic Specifications

Version 3.5.1 includes a few new convenience methods in some classes such as util.Streams, util.AbstractCommand, util.Arrays, xml.TextAccumulatorHandler, stats.LogisticRegression. and all vector.Vector implementations.

The hmm.HmmDecoder cache specification was given generic specifications. (This change is fully backward compatible; the implementation did not change and generic specifications are optional.)

Generic Model Interface

Version 3.5.1 introduces a generic stats.Model interface. The language model implementations have been retrofitted to implement this interface.

LingPipe

Home