There hasn’t been an artificial intelligence which has surpassed or even matched human intelligence – yet. Many researchers believe that Artificial General Intelligence (AGI) is just beyond the horizon. A central effort in the quest for AGI is the advancement of the architectures, or structural designs, of neural networks.

Princeton University Professor of Computer Science and Director and co-founder of Google AI, Princeton Elad Hazan is researching a different way of designing neural networks. His group’s novel architecture design has been experimentally shown to have the power to bolster the computation performance of large language models. 

“We hope to improve the architectures that underlie current AI to give rise to more efficient and better machines and drive the field forward,” said Hazan. “We’re excited about the potential to produce strong AI.”

Drawing inspiration from dynamical systems

The architecture Hazan and colleagues are working on is an advancement on transformers – a deep learning architecture developed by researchers at Google and that is popular for use in LLMs. OpenAI’s ubiquitous ChatGPT uses the transformer architecture. “We have an architecture that has interesting properties and is an addition to transformers,” said Hazan. “Our architecture performs better in certain situations.” 

Models such as ChatGPT and other generative language models can be seen as sequence prediction models. They generate a sequence of words by predicting each next word based on the previous word. These predictive models must be trained to give each word weight, or meaning. Transformers provide one architecture approach for model training. 

Hazan and colleagues’ approach to sequence modeling draws inspiration from dynamical systems – which are mathematical models that describe physical systems which are dynamic in time. Imagine the weather system, or an object balancing precariously on a hand. “I've been studying dynamical systems for several years now in the context of robotics and reinforcement learning,” said Hazan. “And some of these techniques we thought would be useful for sequence prediction.” 

So far, experiments have shown the group’s novel architecture is more robust and more efficient computationally, allowing for a neural network to faster and more accurately compute the next word in a sequence. There are still engineering challenges for applying this technology in scale. “Nevertheless, there is work that justifies that these are theoretically sound methods,” said Hazan.

Producing a machine that surpasses human intelligence would have wide-reaching implications across all facets of humanity. There’s the potential to advance a multitude of areas of life from healthcare to autonomous driving. “It's very hard to produce AGI, but it seems like humanity is getting closer,” said Hazan. “And I'm excited to be right in the middle of it, in the sense that we're producing the architecture that might host or give rise to it.”