Research highlights

• Implicit representations of meaning in neural language models.
  

  Belinda Z. Li, Maxwell Nye and Jacob Andreas. ACL 2021.

  

  Language models (trained to predict missing words in sentences and paragraphs) learn to build structured representations of the state of the world. These representations support model-internal reasoning about the consequences of actions (like mixing a beaker or unlocking a door with a key). The organization of semantic information in language model embeddings bears striking resemblance to formal meaning representations like DRT and file change semantics from the linguistics literature, suggesting that it's possible to discover a little bit about how meaning works with nothing but text as training data.

• Leveraging language to learn program abstractions and search heuristics.
  

  Catherine Wong, Kevin Ellis, Joshua B. Tenenbaum and Jacob Andreas. ICML 2021.

  

  Can natural language enable better automated reasoning about programming languages? We describe how to generalize a recent Bayesian model for program synthesis to incorporate text-based hints and heuristics, and show that even small amounts of side information from language dramatically improve this model's ability to discover reusable abstractions for visual reasoning and programming by demonstration. Natural language is a useful source of supervision even for learning problems that don't involve language as input or output.

• Compositional explanations of neurons.

  Jesse Mu and Jacob Andreas. NeurIPS 2020.

  

  We explain the behavior of deep network models by approximating each neuron with an executable program or logical form. These compositional explanations let us automatically quantify (aspects of) the complexity, interpretability, and naturalness of learned abstractions in vision and language models. They surface a number of surprising successes (image classifiers discover abstract functional categories like "sports facility" without supervision) but also sources of brittleness (vision models easily confuse washing machines with viaducts and cribs with fire escapes, and language processing models are easily tricked by surprising combinations of pronouns).

Full paper list

2021

• How do neural sequence models generalize? Local and global context cues for out-of-distribution prediction.

  Anthony Bau and Jacob Andreas. EMNLP 2021.

• The low-dimensional linear geometry of contextualized word representations.

  Evan Hernandez and Jacob Andreas. CoNLL 2021.

• Leveraging language to learn program abstractions and search heuristics.

  Catherine Wong, Kevin Ellis, Joshua B. Tenenbaum and Jacob Andreas. ICML 2021.

• Implicit representations of meaning in neural language models.

  Belinda Z. Li, Maxwell Nye and Jacob Andreas. ACL 2021.

• What context features can transformer language models use?

  Joe O’Connor and Jacob Andreas. ACL 2021.

• Lexicon learning for few-shot sequence modeling.

  Ekin Akyürek and Jacob Andreas. ACL 2021.

• Multitasking inhibits semantic drift.

  Athul Paul Jacob, Mike Lewis and Jacob Andreas. NAACL 2021.

• Representing partial programs with blended abstract semantics.

  Maxwell Nye, Yewen Pu, Matthew Bowers, Jacob Andreas, Joshua B. Tenenbaum, Armando Solar-Lezama. ICLR 2021.

• Learning to recombine and resample data for compositional generalization.

  Ekin Akyürek, Afra Feyza Akyürek and Jacob Andreas. ICLR 2021.

2020

• Compositional explanations of neurons.

  Jesse Mu and Jacob Andreas. NeurIPS 2020.

• A benchmark for systematic generalization in grounded language understanding.

  Laura Ruis, Jacob Andreas, Marco Baroni, Diane Bouchacourt and Brenden Lake. NeurIPS 2020.

• Good-Enough Compositional Data Augmentation.

  Jacob Andreas. ACL 2020.

• Unnatural language processing: bridging the gap between synthetic and natural language data.

  Alana Marzoev, Sam Madden, Frans Kaashoek, Mike Cafarella and Jacob Andreas. NeurIPS workshop on Emergent Communication.

2019

• A survey of reinforcement learning informed by natural language.

  Jelena Luketina, Nantas Nardelli, Gregory Farquhar, Jakob Foerster, Jacob Andreas, Edward Grefenstette, Shimon Whiteson and Tim Rocktäschel. IJCAI 2019.

• Measuring compositionality in representation learning.

  Jacob Andreas. ICLR 2019.