CAM Colloquium - Sanjay Shakkottai, Department of Electrical and Computer Engineering, University of Texas at Austin

This talk is sponsored by the Cornell Center for Data Science for Enterprise and Society as part of the Data Science Distinguished Lecture Series. Title: Towards discrete diffusion models for language and image generation We discuss discrete diffusion models that offer a unified framework for jointly modeling categorical data such as text and images. We present a new model that we have developed for language generation called the Anchored Diffusion Language Model (ADLM). ADLM is grounded in a novel two-stage framework that first predicts distributions over important tokens via an anchor network (e.g., key words or low-frequency words that anchor a sentence), and then predicts the likelihoods of missing tokens conditioned on the anchored predictions. ADLM significantly improves test perplexity on LM1B and OpenWebText, achieving up to 25.4% gains over prior DLMs, and narrows the gap with strong AR baselines. It also achieves state-of-the-art performance in zero-shot generalization across seven benchmarks and surpasses AR models in MAUVE score, which marks the first time a DLM generates better human-like text than an AR model. Beyond diffusion, anchoring boosts performance in AR models and enhances reasoning in math and logic tasks, outperforming existing chain-of-thought approaches. Project page: https://anchored-diffusion-llm.github.io/ Bio: Sanjay Shakkottai received his Ph.D. from the ECE Department at the University of Illinois at Urbana-Champaign in 2002. He is with The University of Texas at Austin, where he is a Professor in the ECE and CS Departments, and holds the Cockrell Family Chair in Engineering #15. He is also the Director of the Center for Generative AI, a campus-wide computing cluster at UT Austin. He received the NSF CAREER award in 2004 and was elected as an IEEE Fellow in 2014. He was a co-recipient of the IEEE Communications Society William R. Bennett Prize in 2021. He has served as the Editor in Chief of IEEE/ACM Transactions on Networking. His current research interests are in diffusion models and Generative AI, with applications in language models, image editing, and decision-making in wireless networks.