Microsoft Phi-1 Small Language Model Outperforms ChatGPT Using Fewer Resources Due To Improved Data Quality

Microsoft has proven that "Quality matters", with its latest language model, Phi-1. With just 1.3 billion parameters, the "Small Language Model" has outperformed ChatGPT because it has "better quality" data.

Microsoft's Phi-1 language model, built on the Transformer architecture, has outperformed GPT-3.5, the Large Language Model (LLM) that powers the ChatGPT Generative AI chatbot. Let's see how Microsoft is improving the efficiency of Generative AI platforms, which would result in huge cost savings.

Microsoft Proves Stack Quality, Not Size, Is Essentially For Improved Performance

A research paper titled, Textbooks Are All You Need, describes that the Phi-1 model was trained for just four days on 8 NVIDIA A100s with a 'textbook quality' dataset from the internet. It used only 1 billion tokens along with synthetically generated textbooks from GPT-3.5 with 100 billion tokens.

Microsoft's approach involved improving the quality of training data, which goes against the commonly accepted notion that the bigger the data set, the better the output.

Comparative tests have demonstrated that Phi-1 achieved a remarkable accuracy score of 50.6 percent HumanEval and 55.5 percent on MBPP. While this may sound a little low, the Generative AI platform built on this model surpassed GPT-3.5's performance of 47 percent.

What's even more impressive is that Phi-1 has just 1.3 billion parameters, compared to GPT-3.5's staggering 175 billion parameters. There is another, even smaller model, called the Phi-1-Small with just 350 million parameters.

Despite being trained with the same pipeline as the larger one, Phi-1-Small still achieved 45 percent on HumanEval. Any other model that achieves greater than 50 percent accuracy on HumanEval is 100 times bigger than this with an equally large dataset size.

Is Microsoft Killing ChatGPT With Phi-1 Language Models?

Microsoft has been actively adopting and incorporating ChatGPT Generative AI within several of its products and services. In fact, OpenAI, the developer of the GPT language model, is largely funded by Microsoft, but the latter does not own the Generative AI organization.

The Phi-1's phenomenal leanness and efficiency, wouldn't have been possible without vastly improving the quality of data. It was GPT-3.5 that refined and produced the high-quality synthetic dataset fed into Phi-1.

Some researchers recently indicated that using data generated from one Generative AI or using one LLM to train others, would eventually result in incomprehensible outputs further down the line. However, that doesn't seem to be happening, at least in the short term.