Can AI Forecast the Next President of the US? UC Berkeley Research

Hello, fellow human! Sophia here. Please check out the upcoming talk we are hosting soon. Hope to see you at one of our talks.

Table of Contents:

Virtual Talk on April 4th: Approaching Human-Level Forecasting with Language Models by UC Berkeley.

Video Recording: The Lecture by Avi Singh, a Research Scientist at Google DeepMind, on Approaches to Test the Quality of Synthetic Data.

Virtual Talk: Approaching Human-Level Forecasting with Language Models by UC Berkeley.

Can AI give us a heads up on when the next pandemic will happen, or maybe it “knows” who will likely be the next president of the US?

There are many impactful events that will happen in the future, affecting billions of lives. It would be nice to have an oracle who can forecast those events.

The UC Berkeley researchers thought so too. They have built a retrieval-augmented Language Model (LM) system for automatic search of relevant information, generation of forecasts, and aggregation of predictions on a given question about the future. 

In other words, the model retrieves and summarizes relevant articles, reasons about them, and predicts the probability that the event will occur. Interestingly, in some questions, AI surpassed human predictions.

In this talk, the co-author of the research work, Danny Halawi, will share with the BuzzRobot community the technical details behind this work. 

The virtual talk will happen on April 4th. Register here to attend.

The Short Summary and Video Recording of the Talk On Testing the Quality of Synthetic Data

The race for computing power has begun. In the upcoming years, we will see an order of magnitude increase in computing power for training large-scale models. The challenge it presents—there isn't enough high quality human generated data to train the next generation of models. 

That's why the importance of model generated synthetic data, which offers scalability and cost-efficiency, will increase. But let's not get too excited—ensuring the quality of synthetic data, especially for problem-solving tasks that require domain knowledge, will be crucial.

In a recent talk, Avi Singh, a research scientist from Google DeepMind, shared with the BuzzRobot community how he and his team are tackling the challenge. 

The team developed a self-training approach for language models that generates samples from the model and evaluates them. They named the method ReSTEM, inspired by Reinforced Self-Training.

The experiment results showed that ReSTEM self-training significantly improves the test performance of PaLM 2 large-language models. Initially, the tests were conducted on Gemini, but Gemini wasn’t publicly released at the time of the experiment, so they tested on PaLM 2. Overall, the trend of the results was similar on both language models. 

The generated data was tested on math and coding related benchmarks. The choice of benchmarks is not surprising –  it's easier to evaluate compared to creative writing. Assessing more 'vague' model generated data that is open to many interpretations would be the next challenge for researchers.

With ReSTEM applied to PaLM 2 models, the research team has seen impressive results in mathematical reasoning and code generation capabilities of the models. The insight from this experiment: models fine-tuned on model generated synthetic data had significantly better results compared to the models trained on human created data.

We are entering the era of large-scale synthetic data, models, and enormous computing power. It will be fun!

To learn the details of the research work, watch the full video of Avi Singh's talk on our YouTube channel.