June 30 at 1:30pm

Abstract: We show that calibration, which is naturally encouraged by the loss minimized during pre-training of language models, leads to certain type of hallucinations. Moreover, the rate of hallucinations depends on the domain via the classic Good-Turing estimator. Interestingly, this estimate is small for domains like paper references which have been a notorious source of hallucinations. The analysis also suggests methods for mitigating hallucinations.

This is joint work with Santosh Vempala and was done while the speaker was at Microsoft Research New England.

July 1 at 1:30pm

Abstract: Reinforcement learning from human feedback (RLHF) is a go-to solution for aligning large language models (LLMs) with human preferences; it passes through learning a reward model that subsequently optimizes the LLM's policy. However, an inherent limitation of current reward models is their inability to fully represent the richness of human preferences and their dependency on the sampling distribution. In the first part we turn to an alternative pipeline for the fine-tuning of LLMs using pairwise human feedback. Our approach entails the initial learning of a preference model, which is conditioned on two inputs given a prompt, followed by the pursuit of a policy that consistently generates responses preferred over those generated by any competing policy, thus defining the Nash equilibrium of this preference model. We term this approach Nash learning from human feedback (NLHF) and give a new algorithmic solution, Nash-MD, founded on the principles of mirror descent. NLHF is compelling for preference learning and policy optimization with the potential of advancing the field of aligning LLMs with human preferences. In the second part of the talk we delve into a deeper theoretical understanding of fine-tuning approaches as RLHF with PPO and offline fine-tuning with DPO (direct preference optimization) based on the Bradley-Terry model and come up with a new class of LLM alignment algorithms with better both practical and theoretical properties. We finish with the newest work showing links between and building on top of them.

July 2 at 1:30pm

Abstract: From playing complex games at super-human level to training deep generative models to training deep neural net-based models that interact with each other and with humans, many outstanding challenges in Machine Learning lie at its interface with Game Theory. These applications motivate the study of games with infinitely many strategies and/or non-concave utilities, which are challenging in that standard notions of equilibrium are not guaranteed to exist, and when they do exist they may be infinitely supported and thus impractical to compute or implement. We develop new connections between Online Learning, Complexity Theory, and Game Theory to shed light on these equilibrium existence and computation challenges, providing game-theoretic and algorithmic foundations for this intricate setting.