Adji Bousso Dieng: evaluating diversity in generative AI

Written by
Allison Gasparini
Jan. 21, 2025

Over the last few years, scientists have developed generative artificial intelligence models which produce increasingly realistic, high-quality images. But while AI researchers have been enticed to set their focus to creating aesthetically pleasing images, Princeton University Assistant Professor of Computer Science Adji Bousso Dieng had the sense another, important concept had been left overlooked. 

“Evaluating an image generative model on diversity wasn’t a priority,” said Dieng, despite a diversity, which here refers to the variety of forms that the model generates, being fundamental to a model’s success.  

“I'm motivated by tackling challenging problems in the sciences,” said Dieng, who leads Vertaix, a research lab at Princeton. “Diversity, as a concept, has been mainly studied in ecology spaces, but diversity is key to sciences beyond ecology, including machine learning.”

The fundamental importance of diversity led Dieng and colleagues to develop the Vendi Score, a metric which can be applied to evaluate and promote better diversity in generative models and datasets, among other applications. 

Tackling challenging problems

On Dec. 3, Dieng gave a seminar at the Center for Statistics and Machine Learning in which she presented on how the Vendi Score works and the various ways it can be applied. 

The Vendi Score works like this: imagine a collection which contains a million items. “That collection can be anything, it can be a collection of people, can be a collection of species, can be a collection of molecules or a collection of galaxies,” said Dieng. “The Vendi Score  will take that collection and give you a number corresponding to the distinct items it contains.”

Say that this collection of a million items is a collection of shapes. Though there are a million items, each of those items is either a triangle, a square, or a circle. In this way, the vast collection contains only duplicates of three different types of item. An analysis of this set would yield a Vendi score of three. 

The Vendi Score has broad applications. Among other things, researchers have used it as a metric for measuring the memorization abilities of AI models. “The good thing is that this metric is differentiable, so we've been using it to do all sorts of things, including accelerating molecular simulations, and discovering novel materials with desirable properties.”

 A metal-organic framework (MOF), a particular type of material, Dieng and colleagues discovered using the Vendi Score within a search algorithm

Image of a metal-organic framework (MOF), a particular type of material Dieng and colleagues discovered using the Vendi Score within a search algorithm. This material is the best MOF in terms of ammonium adsorption, which is an environmentally important application. The MOF is thermally stable and energy efficient. Image courtesy of Adji Bousso Dieng.

The differing problems facing the sciences are generally tied together by their need for diversity – which is why the Vendi Score is ultimately a useful metric. One example of a major problem in the sciences are those relating to search, such as when scientists working with a vast database struggle to locate specific materials or molecules or other items of interest contained within the database. 

“In all those types of problems, you need diversity for things to work well,” said Dieng. “When you're searching a vast database, you want to make sure that you don't just focus on a part of the database, otherwise you may miss out on molecules or materials that correspond to what you're looking for.” 

Similarly, in generative modeling, an AI which generates the same thing over and over again isn’t of much use. If you’re searching for a new molecule with specific properties, for example, “you want a model that will generate a diverse set of candidate molecules and materials,” said Dieng. 

Researchers are already seeing positive results from embedding the Vendi Score into generative AI. “There are researchers at Meta who use the Vendi Score to improve the diversity of image generative models,” said Dieng. “Others have used it to determine the diversity of spacecraft trajectories, which is important in aerospace engineering.”

Dieng hopes others are encouraged to pick up the Vendi Score method and that more and more scientists start dedicating their efforts in the space of diversity. “We should be paying more attention to diversity,” said Dieng. “At my research lab Vertaix, we have been pioneering this line of work to make diversity more general in the sciences.”