Big Multilinguality for Data-Driven Lexical Semantics

Project Personnel

• David Bamman, Carnegie Mellon University School of Computer Science
• Chris Dyer (PI), Carnegie Mellon University School of Computer Science
• Manaal Faruqui, Carnegie Mellon University School of Computer Science
• Noah Smith (PI), Carnegie Mellon University School of Computer Science

Summary of Research Findings

• Faruqui and Dyer (2014) introduced a technique based on canonical correlation analysis that incorporates multilingual evidence into word vectors. The key idea is to use word-aligned parallel corpora and project word vectors learned separately for the two languages into the same space. The resulting word vectors are shown to be of higher quality in standard benchmark evaluations than vectors learned from monolingual text alone.
• Bamman et al. (2014) introduced a model for incorporating geographical contextual information in learning vector-space representations of situated language. Using geolocated social media messages, this technique allows us to see how word meanings can depend on place. We developed a new quantitative evaluation that judges geographically informed semantic similarity, and used it to demonstrated the ability of the new method to make reasonable inferences about word meaning.
• Faruqui et al. (2015) introduced a technique, "retrofitting," that uses semantic lexicons such as WordNet, FrameNet, and the Paraphrase Database to improve word vectors. Given such a lexicon of similarity relationships and a set of learned word vectors, the technique adapts word vectors, encouraging neighbors to be more similar. We find the new vectors show substantial improvement on benchmark tasks.
• Yogatama et al. (2015) introduced a new technique for learning word representations using hierarchical regularization in sparse coding. This required significant algorithmic development. The resulting vectors were found to perform competitively on standard evaluation tasks, and the sparsity patterns are suggestive that greater interpretability in these representations may be possible.
• Word vectors from this project were incorporated into several synergistic research projects. These include: (1) tests leading up to an entry in the SemEval 2014 competition on broad-coverage semantic dependency parsing (Thomson et al., 2014), which achieved second place; (2) developing a cross-lingual model of metaphor that discrimnates between syntactic constructions with literal or metaphorical meaning in several languages, supporting the hypothesis that metaphors are conceptual rather than lexical (Tsvetkov et al., 2014); and (3) the development of a supersense taxonomy for adjectives (Tsvetkov et al., 2014).
• Current distributed representations of words show little resemblance to theories of lexical semantics. The former are dense and uninterpretable, the latter largely based on familiar, discrete classes (e.g., supersenses) and relations (e.g., synonymy and hypernymy). Faruqui et al. (2015) proposed methods that transform word vectors into sparse (and optionally binary) vectors. The resulting representations are more similar to the interpretable features typically used in NLP, though they are discovered automatically from raw corpora. Because the vectors are highly sparse, they are computationally easy to work with. Most importantly, we find that they outperform the original vectors on benchmark tasks.

Downloads and Links

• wordvectors.org, a site for word vector evaluation
• code for geographically situated word vectors (Bamman et al., 2014)
• code for cross-lingual vectors (Faruqui and Dyer, 2014)
• code for sparse word vectors (Faruqui et al., 2015)
• code for retrofitting (Faruqui et al., 2015)

Publications

• Distributed Representations of Geographically Situated Language. David Bamman, Chris Dyer and Noah A. Smith. In Proceedings of ACL 2014.
• Retrofitting Word Vectors to Semantic Lexicons. Manaal Faruqui, Jesse Dodge, Sujay Kumar Jauhar, Chris Dyer, Eduard Hovy, and Noah A. Smith. In Proceedings of NAACL 2015. Best student paper award.
• Sparse Binary Word Vector Representations. Manaal Faruqui, Yulia Tsvetkov, Dani Yogatama, Chris Dyer, and Noah A. Smith. In Proceedings of ACL 2015.
• Improving Vector Space Word Representations Using Multilingual Correlation. Manaal Faruqui and Chris Dyer. In Proceedings of EACL 2014.
• Community Evaluation and Exchange of Word Vectors at wordvectors.org. Manaal Faruqui and Chris Dyer. In Proceedings of ACL 2014 (Demonstrations).
• CMU: Arc-Factored, Discriminative Semantic Dependency Parsing. Sam Thomson, Brendan O'Connor, Jeffrey Flanigan, David Bamman, Jesse Dodge, Swabha Swayamdipta, Nathan Schneider, Chris Dyer, and Noah A. Smith. In Proceedings of SemEval 2014.
• Augmenting English Adjective Senses with Supersenses. Yulia Tsvetkov, Nathan Schneider, Dirk Hovy, Archna Bhatia, Manaal Faruqui and Chris Dyer. In Proceedings of LREC 2014.
• Metaphor Detection with Cross-Lingual Model Transfer. Yulia Tsvetkov, Leonid Boytsov, Anatole Gershman, Eric Nyberg and Chris Dyer. In Proceedings of ACL 2014.
• Learning Word Representations with Hierarchical Sparse Coding. Dani Yogatama, Manaal Faruqui, Chris Dyer, and Noah A. Smith. In Proceedings of ICML 2015.

Acknowledgments

This project is supported by the National Science Foundation (IIS-1251131).