Introducing Databricks LLM - DBRX: Architecture and Technical Details

Happy Friday, fellow humans! I'm Sophia, founder of BuzzRobot, sharing some interesting virtual talks we are hosting soon.

Introducing Databricks LLM - DBRX

On May 2nd at 10 am PT, we are hosting a virtual talk with Shashank Rajput, a research scientist from Databricks Mosaic Research. He will walk us through the new LLM (Large Language Model), DBRX, recently released by Databricks.

He'll elaborate on the choice of the model's architecture - The Mixture of Experts, explain how they made technical decisions regarding the model's components, discuss the challenges the team faced while training a model of that scale, and share how they addressed them. 

According to Databricks, the model outperforms GPT-3.5 and competes with Gemini 1.0 Pro, Google's LLM.

Use this link to learn more details about the talk and to register.

I hope to see you at our virtual event!

AI Forecaster of Future Events by UC Berkeley Researchers

I had the pleasure of hosting a talk with Danny Halawi, a researcher at UC Berkeley, who shared insights with the BuzzRobot community about AI capable of predicting future events.

You can watch the video of the lecture on our YouTube channel. 

Here's a brief summary of his talk:

This language model is based on OpenAI's GPT-4. It's designed for judgmental forecasting and is capable of answering questions like

• when the next pandemic might occur.

• whether NVIDIA's stocks will continue to go up.

• where the next geopolitical military conflict might happen and when.

So, how exactly did the researchers build the model?

Establishing a benchmark for the model

The researchers collected binary questions along with aggregated human predictions from 5 competitive forecasting platforms. Then, they set a retrieval date, up to which the language model could gather information from these platforms. 

The next step –  feeding this information to the language model and prompting it to make predictions. These predictions were then compared to human predictions using the Brier score, where a score of 0 means absolutely accurate predictions and a score of 1 –  completely inaccurate predictions.

As for the dataset, the research team collected questions before June 1st, 2023, and used them to train the language model.

They then tested the performance of the model on questions after June 1st, 2023 – after the model’s pre-training cut-off.

The structure of the system: How did they build the model?

The researchers prompted the model to generate search queries about future events, which they fed to News APIs. This helped them build up an initial set of articles for prediction analysis. 

Then, the language model rated the relevance of those articles in respect to the questions about future events. After that, the highly rated curated set of articles was provided to the model for summarization. 

The team used two GPT-4 based models.

The first model was fed with questions about future events and news summaries, along with scratchpads on how to reason about those articles. 

The second one was a fine-tuned model that received only questions and summaries but no further human input. Interestingly, the fine-tuned model achieved higher accuracy on predictions compared to the human-instructed base model.

What are the results?

The GPT-4-based language model achieved an accuracy of 71.5% in predicting future events, while human predictions scraped from forecasting platforms had an accuracy of 77%. 

To put this into perspective, 71.5% is considered a very good result, especially when compared to other AI forecasting systems that achieved only 67.9% accuracy. 

Also, OpenAI's safety guards prevented the model from providing predictions on certain events or giving specific answers, which may have affected its accuracy.

If you're interested in learning more about how the AI forecaster was developed – watch the lecture on the BuzzRobot YouTube channel. 

Also, please join our developer community on Slack.

Until next time!