Robust Estimation under Local and Global Adversarial Corruptions]{Robust Distribution Learning with Local and Global Adversarial Corruptions

Nietert, Sloan; Goldfeld, Ziv; Shafiee, Soroosh

Session: 4A-2 (Robust/Adversarial Learning), Wednesday, Jul 03, 08:30-10:00

Abstract: We consider learning in an adversarial environment, where an $\eps$-fraction of samples from a distribution $P$ are arbitrarily modified (\emph{global} corruptions) and the remaining perturbations have average magnitude bounded by $\rho$ (\emph{local} corruptions). Given access to these $n$ corrupted samples, we seek a computationally efficient estimator $\hat{P}_n$ that minimizes the Wasserstein distance $\Wone(\hat{P}_n,P)$. In fact, we attack the fine-grained task of minimizing $\Wone(\Pi_\sharp \hat{P}_n, \Pi_\sharp P)$ for all orthogonal projections $\Pi \in \R^{d \times d}$, with performance scaling with $\mathrm{rank}(\Pi) = k$. This allows us to account simultaneously for mean estimation ($k=1$), distribution estimation ($k=d$), as well as the settings interpolating between these two extremes. We characterize the population-limit optimal risk for this task and then proceed to develop an efficient finite-sample algorithm with error bounded by $\sqrt{\eps k} + \rho + d^{O(1)}\tilde{O}(n^{-1/k})$ when $P$ is sub-Gaussian. Assuming that the data distribution has isotropic covariance, our finite-sample bounds match the population-level optimum for large sample sizes. Our efficient procedure relies on a novel trace norm approximation of an ideal yet intractable 2-Wasserstein projection estimator. We apply this algorithm to robust stochastic optimization, and, in the process, uncover a new method for overcoming the curse of dimensionality in Wasserstein distributionally robust optimization.