About the Part-of-Speech Demo

Part-of-Speech Tutorial

There is a tutorial covering part of speech tagging at the API level at:

Part-of-Speech Demo Implementation

The part-of-speech demo implementation is in:

Like every demo, this implements:

The demos and framework are in:

Part-of-speech tagging assigns classical grammatical categories to tokens. Although the tag sets vary between the demos, they all include singular and plural nouns, proper nouns, adjectives, determiners, verbs of various forms, prepositions, and so on.

Genre-Specific Models

There are three different part-of-speech models included as demos. The first is a general English tagger derived from the Brown corpus, which was curated for "balance". The second and third are for biomedical journal articles in English, and were trained from two different corpora, the GENIA corpus and the MedPost corpus.

Sentence Annotation Included

The demos use the appropriate sentence models. See the Sentence Demo for more information.

Part-of-Speech XML Markup

First-best output

Each token is marked with an element token, with an attribute pos providing the part-of-speech; e.g. <token pos="np">John</token>.

N-best output

Each analysis is marked with a tag analysis, with attribute jointLog2Prob providing the joint log (base 2) probability of the analysis, and rank providing the rank on the n-best list (numering from zero), e.g. <analysis jointLog2Prob="-39.9" rank="5">. Within each analysis, tokens are tagged as for first-best output.

Per tag confidence output

Each token and its analyses are wrapped with an nBestTags element. Following the text content is a sequence of tag elements, marking tags, conditional probabilities, and several location identifiers: sentence position, token position and confidence rank.

Part-of-Speech Demo on the Web

The demos are hosted on the web at the following URLs:

English General Text: Brown Corpus

http://lingpipe-demos.com:8080/lingpipe-demos/pos_en_general_brown/textInput.html

English Biomedical Text: MedPost Corpus

http://lingpipe-demos.com:8080/lingpipe-demos/pos_en_bio_medpost/textInput.html

English Biomedical Text: GENIA Corpus

http://lingpipe-demos.com:8080/lingpipe-demos/pos_en_bio_genia/textInput.html

For detailed information about using web demos, including web form, file upload and web service instructions, see the web demo instructions

Part-of-Speech Demo via GUI

To launch the demo in a GUI, first change directories to the command directory and then invoke the demo batch script. Note: Parameters are set in the GUI, not as arguments to the launch script.

Windows Operating System

English General: Brown Corpus

> cd %LINGPIPE_HOME%\demos\generic\bin
> gui_pos_en_general_brown.bat

English Biomedical: MedPost Corpus

> cd %LINGPIPE_HOME%\demos\generic\bin
> gui_pos_en_bio_medpost.bat

English Biomedical: GENIA Corpus

> cd %LINGPIPE_HOME%\demos\generic\bin
> gui_pos_en_bio_genia.bat

Unix-like Operating Systems

English General: Brown Corpus

> cd %LINGPIPE_HOME%\demos\generic\bin
> sh gui_pos_en_general_brown.sh

English Biomedical: MedPost Corpus

> cd %LINGPIPE_HOME%\demos\generic\bin
> sh gui_pos_en_bio_medpost.sh

English Biomedical: GENIA Corpus

> cd %LINGPIPE_HOME%\demos\generic\bin
> sh gui_pos_en_bio_genia.sh

For detailed information about running demos in a GUI, see the GUI demo instructions

Part-of-Speech Demo via Shell Command

Shell commands may be run over single files, all of the files in a directory, or using standard input/output.

Running over a Directory

English General: Brown Corpus

> cd $LINGPIPE/demos/generic/bin
> cmd_pos_en_general_brown.bat -inDir=../../data/testdir -outDir=/testout

English Biomedical: MedPost

> cd $LINGPIPE/demos/generic/bin
> cmd_pos_en_bio_medpost.bat -inDir=../../data/testdir -outDir=/testout

English Biomedical: GENIA

> cd $LINGPIPE/demos/generic/bin
> cmd_pos_en_bio_genia.bat -inDir=../../data/testdir -outDir=/testout

Running a Single File

English General: Brown Corpus

> cd $LINGPIPE/demos/generic/bin
> cmd_pos_en_general_brown.bat -inFile=../../data/testdir/foo.txt -outFile=foo.out.xml

The other genres are handled the same way, with different suffixes in place of general_brown.

Running through a Pipe (Standard input/output)

English General: Brown Corpus

> cd demos/generic/bin
> echo See Spot. See Spot run. | cmd_pos_en_general_brown.bat

The other genres are handled the same way, with different suffixes in place of general_brown.

Running in Unix-like Operating Systems

For unix-like operating systems such as Unix, Solaris, Linux, or Macintosh OS X:

For detailed information about running demos from the command line, see the command line demo instructions

Part-of-Speech Demo Scripts

The following scripts are available in $LINGIPE/demos/generic/bin for running the demo. Note that each script comes in four flavors, distinguishing command line from GUI, and the Windows DOS shell from the Unix shell.

Language Genre Corpus Mode Windows DOS Unix/Linux/Mac sh
English General Brown Command cmd_pos_en_general_brown.bat cmd_pos_en_general_brown.sh
GUI gui_pos_en_general_brown.bat gui_pos_en_general_brown.sh
English Biomedical MedPost Command cmd_pos_en_bio_medpost.bat cmd_pos_en_bio_medpost.sh
GUI gui_pos_en_bio_medpost.bat gui_pos_en_bio_medpost.sh
English Biomedical GENIA Command cmd_pos_en_bio_genia.bat cmd_pos_en_bio_genia.sh
GUI gui_pos_en_bio_genia.bat gui_pos_en_bio_genia.sh

Part-of-Speech Demo Parameters

The following is a complete list of parameters for the demo.

Demo-Specific Parameters

The following parameter is specific to the part-of-speech demo (though also found in the named-entity demo).

Parameter Description Usage Constraints
resultType Form of results Values determine output:
  • firstBest
  • nBest
  • conf

General Demo Parameters

These parameters apply to every version (web/GUI/command) of every demo.

Parameter Description Usage Constraints
inCharset Input character set Optional. Defaults to platform default.
outCharset Output character set
contentType Input content type May be one of:
  • text/plain
  • text/html
  • text/xml
Defaults to text/plain.
removeElts Element tags to remove Optional. May only be used with contentType=text/html or contentType=text/xml. Each value may be comma-separated list. If neither of these are specified, all text content is processed.
includeElts Elements to annotate

Command-Line Only Parameters

These parameters apply to every command-line demo, but are not relevant for the GUI or web versions of the demos.

Parameter Description Usage Constraints
inFile Readable input file May not be used with inDir. If either is not specified, defaults to standard input or output.
outFile Writeable output file
inDir Readable input directory May not be used with inFile or outFile. If used, inDir and outDir must both be specified.
outDir Writeable output directory