★ [August 2021] MIT Press version available now.
★ [August 2021] To be presented at ACL 2021 (Oral Session 5B).
★ [Spring 2021] A pre-print version of the paper can be found [here].

Abstract

We present a new conjunctivist framework, neural event semantics (NES), for compositional grounded language understanding. Our approach treats all words as classifiers that compose to form a sentence meaning by multiplying output scores. These classifiers apply to spatial regions (events) and NES derives its semantic structure from language by routing events to different classifier argument inputs via soft attention. NES is trainable end-to-end by gradient descent with minimal supervision. We evaluate our method on compositional grounded language tasks in controlled synthetic and real-world settings. NES offers stronger generalization capability than standard function-based compositional frameworks, while improving accuracy over state-of-the-art neural methods on real-world language tasks.

pull figure

Video Overview

Spotlight short videos:

(ACL21 Oral recorded talk coming soon!)

Citation

@inproceedings{buch2021nes,
  title={Neural Event Semantics for Grounded Language Understanding},
  author={Shyamal Buch and Li Fei-Fei and Noah D. Goodman},
  booktitle={Transactions of the Association for Computational Linguistics (TACL)},
  year={2021}
}