2020 NLP Wish List

All the cool kids on twitter are talking about their wishlists for NLP in 2020, and it got me thinking. Many of these overlap with stuff mentioned on twitter, but in my defense, Yoav had a very long list.

these are some of my nlp wishlists for 2020. what are yours? https://t.co/Vnquwuad3D

— (((ل()(ل() 'yoav)))) (@yoavgo) December 14, 2019

Learning from small data

If I decided to learn a new language, and further decided that I would do it by examining only parallel texts, I wouldn’t need 100M sentence pairs. I could make good headway by studying, say, parallel translations of the New Testament, which has less than 10,000 sentences. Current MT models can learn very little from 10,000 sentence pairs. What’s the difference?

Similarly, I could take a person off the street and teach them to annotate NER in less than an hour. Why do our models need 200K+ tokens of annotated data?

Understanding of multilingual contextual models

One of the most interesting things to come from the contextualization of NLP are the effortlessly multilingual models. Multilingual BERT just… works as a cross-lingual representation, even with no cross-lingual signal during pretraining. We saw this in the BSNLP workshop, and in our LORELEI work, and there have been several other interesting works since. I’d love to see more understanding of why this works well, and how to exploit the mechanism for better multilingual representations.

Reality checks against end users

Significance tests are important, sure, but I often wonder if end users would notice the difference between 93.0 and 93.5 on CoNLL2003. Similarly, at what point does a metric cross from being not useful to useful?

Better definition of summarization

Summarization is an important and interesting problem, but it is also ill-defined. It seems to me that a summary depends on the goal or the audience of the summarization.

There’s a neat example of this in a paper from Salesforce at EMNLP 2019, where they ask humans to create summaries in two conditions, 1) unconstrained (“just write a summary”) and 2) constrained, in which each summary is required to answer a small set of questions, created by annotators in a separate task. They write: “Constrained summaries were more succinct and targeted, without sacrificing the natural flow of sentences. Unconstrained writers tended to write more verbose summaries that did not add information.” This suggests that some kind of target-focused or goal-oriented summarization would make for a better task.

Green AI – with a laptop

I’ve been intrigued for a while by a statement in the abstract of the #GreenAI paper from Roy Schwartz et al. They write: “Our goal is to make AI both greener and more inclusive — enabling any inspired undergraduate with a laptop to write high-quality research papers.” This seems almost ridiculous in this age of quadrillion parameter models and hellabyte-scale corpora, but I think it could be a productive constraint. If you only had a laptop, you’d be forced to do something thoughtful and precise. Maybe we need a “laptops only” workshop.