LMSS @ Cornell Tech: Phillip Isola (MIT)

Learning Machines Seminar Series What: LMSS: Phillip Isola (MIT) When: Thursday, November 20, 1:30-2:45 pm Where: Bloomberg 081, Bloomberg Center, Cornell Tech (map) The series is organized by Associate Professor Yoav Artzi and sponsored by Bloomberg. Pizza will be served at 1:15 p.m. "Revisiting the Symbol Grounding Problem in the Age of LLMs" A classic question in the study of intelligence asks how symbols acquire meanings grounded in physical reality. It has been sometimes assumed that this process requires paired sensory experience, where symbols get linked to the percepts they co-occur with. I will question this assumption along two axes. First, I will present work showing that unpaired data can suffice to partially ground symbols in their referents. Second, I will show that a pure language model can be prompted to act as if it is a sensory encoder, and that this role playing is not simply a hallucination but rather can be an accurate inference of the missing sensory modality. These results, together with other recent findings, suggest that symbolic meaning may arise under weaker grounding conditions than traditionally believed. BIO Phillip Isola is the Class of 1948 Career Development associate professor in EECS at MIT. He studies computer vision, machine learning, robotics, and AI. He completed his Ph.D. in Brain & Cognitive Sciences at MIT, and has since spent time at UC Berkeley, OpenAI, and Google Research. His work has particularly impacted generative AI and self-supervised representation learning. Dr. Isola's research has been recognized by a Google Faculty Research Award, a PAMI Young Researcher Award, a Samsung AI Researcher of the Year Award, a Packard Fellowship, and a Sloan Fellowship. His teaching has been recognized by the Ruth and Joel Spira Award for Distinguished Teaching. His current research focuses on trying to scientifically understand human-like intelligence.