Twitter NLP and Part-of-Speech Tagging

We provide a fast and robust Java-based tokenizer and part-of-speech tagger for Twitter, its training data of manually labeled POS annotated tweets, a web-based annotation tool, and hierarchical word clusters from unlabeled tweets.

These were created by Olutobi Owoputi, Brendan O'Connor, Kevin Gimpel, Nathan Schneider, Chris Dyer, Dipanjan Das, Daniel Mills, Jacob Eisenstein, Michael Heilman, Dani Yogatama, Jeffrey Flanigan, and Noah Smith.

What the tagger does

./runTagger.sh --output-format conll examples/casual.txt

ikr smh he asked fir yo last name so he can add u on fb lololol

word    tag     confidence
ikr     !       0.8143
smh     G       0.9406
he      O       0.9963
asked   V       0.9979
fir     P       0.5545
yo      D       0.6272
last    A       0.9871
name    N       0.9998
so      P       0.9838
he      O       0.9981
can     V       0.9997
add     V       0.9997
u       O       0.9978
on      P       0.9426
fb      ^       0.9453
lololol !       0.9664

:o :/ :'( >:o (: :) >.< XD -__- o.O ;D :-) @_@ :P 8D :1 >:( :D =| ") :> ....

word    tag     confidence
:o      E       0.9387
:/      E       0.9983
:'(     E       0.9975
>:o     E       0.9964
(:      E       0.9994
:)      E       0.9997
>.<     E       0.9952
XD      E       0.9938
-__-    E       0.9956
o.O     E       0.9899
;D      E       0.9995
:-)     E       0.9992
@_@     E       0.9964
:P      E       0.9996
8D      E       0.9961
:       E       0.6925
1       $       0.9194
>:(     E       0.9715
:D      E       0.9996
=|      E       0.9963
"       ,       0.6125
)       ,       0.9078
:       ,       0.7460
>       G       0.7490
...     ,       0.5223
.       ,       0.9946

Download

• Releases of the tagger (and tokenizer), data, and annotation tool are available here on Google Code.
• The tagger source code (plus annotated data and web tool) is on GitHub.
• The tokenizer code is self-contained in Twokenize.java; or use twokenize.sh in the tagger download.

To receive announcements about updates, join the ARK-tools mailing list.

Further Reading

• The newest paper (for version 0.3), is:
  Improved Part-of-Speech Tagging for Online Conversational Text with Word Clusters
  Olutobi Owoputi, Brendan O’Connor, Chris Dyer, Kevin Gimpel, Nathan Schneider and Noah A. Smith.
  In Proceedings of NAACL 2013.
  • Tech report version, with a few more and few less details: Owoputi et al. (2012). Technical Report, Machine Learning Department. CMU-ML-12-107. September 2012.
• The original paper:
  Part-of-Speech Tagging for Twitter: Annotation, Features, and Experiments
  Kevin Gimpel, Nathan Schneider, Brendan O'Connor, Dipanjan Das, Daniel Mills, Jacob Eisenstein, Michael Heilman, Dani Yogatama, Jeffrey Flanigan, and Noah A. Smith.
  In Proceedings of ACL 2011.
• See also:
  The Annotation Guidelines, extensively revamped for 0.3.

Resources

Twitter Word Clusters

Here is an HTML viewer of the word clusters. Produced by an unsupervised HMM: Percy Liang's Brown clustering implementation on Lui and Baldwin's langid.py-identified English tweets; see Owoputi et al. (2012) for details.

We recommend the largest one:

filename	#wordtypes	#tweets	#tokens	#clusters	min count	tweet source
50mpaths2	216,856	56,345,753	847,372,038	1000	40	100k tweet/day sample, 9/10/08 to 8/14/12

Also, here are the smaller ones used in the experiments:

filename	#wordtypes	#tweets	#tokens	#clusters	min count	tweet source
6mpaths	111,844	~6,000,000	1,575,589	800	10	10k tweet/day sample, 9/10/08 to 7/18/12
3mpaths	124,731	3,000,000	1,006,324	800	5	subsample
750kpaths	50,780	750,000	?	800	5	subsample
100kpaths	21,345	100,000	?	800	3	subsample
10kpaths	6,944	10,000	?	800	2	subsample
1kpaths	4,142	1000	15,159	800	1	subsample

Tagger Models

• model.20120919 (2MB, plaintext) -- the Twitter POS model with a 25-tag tagset. This is included with the tagger release and used by default.
• model.irc.20121211 (3MB, plaintext) -- a model trained on the NPSChat IRC corpus, with a PTB-style tagset.

See also ARK Social Media Research.

Acknowledgments