When you click on any YouTube video or buy a book on Amazon, machine learning algorithms behind the scene takes the information generated from your online activity and predicts what video or book you may be susceptible to viewing or buying the next time you log on the website.
Broadly speaking, this machine learning technology is called reinforcement learning, a type of machine learning system that is taking inputs from the environment, learning from those inputs and making decisions at the same time, according to Gene X. Li, graduate of Princeton University’s class of 2019 and electrical engineering major. Basically, these algorithms iteratively use prior knowledge to make decisions, forming feedback loops. They then decide what is the best course of action given a specific situation.
“Self driving cars are one example of reinforcement learning,” said Li, 21 years old. “For example, cameras observe the road, and they must interpret whether the objects are obstacles that they should stop for, or decide when to turn to a different lane. All of this is happening in real time, so reinforcement learning algorithms must be able to process information about their surroundings and compute new decisions efficiently.”
“However, we lack theoretical understanding of reinforcement learning: Can we show that these algorithms work provably and efficiently?” Li said.
Li explored part of this issue in his independent project for the Center for Statistics and Machine Learning’s (CSML) Undergraduate Certificate Program in Statistics and Machine Learning. Li’s project, titled “Learning Linear Dynamical Systems with Sparsity Structure,” won CSML’s annual poster session competition which was held on May 14th. Li’s adviser is Yuxin Chen, an assistant professor of electrical engineering.
Li’s project, which explores the fundamental and theoretical underpinnings behind machine learning, tackles an issue that is now becoming important due to the rising prevalence of complicated decision-making algorithms in production.
Part of the reason why these algorithms are poorly understood is because they use neural networks. Neural networks learn a task by analyzing training examples, i.e. a computer vision application of neural networks is to feed the object recognition system large amounts of road driving data, Li said.
To explore whether reinforcement learning algorithms are working reliably, researchers have been setting these algorithms to work on simpler problems, Li said. Many researchers have begun by revisiting simpler, older problems, such as controlling linear dynamical systems. Linear dynamical systems are mathematical models that use linear equations to model how a system changes over time, i.e. behavior of a stock market.
Li takes a similar approach by also looking at linear dynamical systems. His project studies how quickly we can identify the unknown matrices that govern the behavior of these systems, when we make assumptions about the structure of these unknown matrices. Li applies linear regression algorithms to learn the unknown matrices in these systems and see how well these algorithms perform.
“You want to see how the simplest algorithm works on this problem,” said Li. Linear regression's performance can then be compared to how reinforcement learning algorithms work on the same systems in order to better understand where the gaps are in performance in these reinforcement learning algorithms, Li explained.
Li’s own entry into machine learning research began after several years of studying math from a competitive standpoint. Li, originally a Tennessee resident, grew up attending math contests in middle and high school.
By the time he got to college, Li said he was a little burnt out in studying math for just competitions and wanted to see how he could apply it to engineering issues. This ultimately lead to his decision in majoring in electrical engineering.
“Over my junior and senior year, I became interested in machine learning and data science and statistics,” he said, explaining his enrollment in the CSML certificate.
With his independent project garnering praise, Li wants to continue in a similar vein of research when he enrolls for the start of his doctoral program this fall at the Toyota Technological Institute at Chicago, an academic computer science institute that opened in 2003 and is closely affiliated with the University of Chicago’s computer science department. Li plans on studying theoretically-grounded research on machine learning.
“Many machine learning algorithms lack guarantees if they will work or why they work,” he said. “They could be easily exploited. We want to understand how these algorithms work and make them more robust against attackers.”
The term for this specific area of research is called "adversarial machine learning,” said Li.
After he earns his doctoral degree, Li is not sure if he will continue in academia or work in industry, but both sound intriguing.
When he’s not in the classroom, Li exercises by doing power lifting, watching NBA and NFL games, reading up on sport analytics and making and drinking coffee. Li is a coffee aficionado. One of his favorite things to do when traveling to new cities is sampling the coffee at the best shops in town.
Before he starts his doctoral program, he will be traveling this summer and moving to Chicago. While he makes these important life transitions, he said he will be missing the times he spent at Princeton and at CSML.
“I’ve enjoyed the classes I’ve taken with CSML, as they’ve introduced me to areas of academic research that I had no idea existed when I was a freshman,” he said.
“Princeton is great because of the small class sizes,” Li continued. “And I’ve been very fortunate to have interacted with and learned from some great professors.”