[Vanderbilt Machine Learning] Concept Learning Across Domains and Modalities - Jiajun Wu (Stanford University), November 04, 2024
Abstract: I will discuss a concept-centric paradigm for building agents that can learn continually and reason flexibly across multiple domains and input modalities. The concept-centric agent utilizes a vocabulary of neuro-symbolic concepts. These concepts, including object, relation, and action concepts, are grounded on sensory inputs and actuation outputs. They are also compositional, allowing for the creation of novel concepts through their structural combination. To facilitate learning and reasoning, the concepts are typed and represented using a combination of symbolic programs and neural network representations. Leveraging such neuro-symbolic concepts, the agent can efficiently learn and recombine them to solve various tasks across different domains and data modalities, ranging from 2D images, videos, 3D scenes, temporal data, and robotic manipulation data.