Microsoft has recently introduced Phi-3, a new family of small language models (SLMs) that offer impressive performance in a compact size. The first model released, Phi-3 Mini, has 3.8 billion parameters and rivals the performance of much larger models like GPT-3.5 on various benchmarks testing language understanding, reasoning, math, and coding abilities.
The innovation behind Phi-3's performance lies in its training dataset, which uses heavily filtered web data and synthetic data generated by larger models to teach reasoning skills, rather than just crawling huge amounts of web data. This approach enables Phi-3 to learn more efficiently and effectively.
One of the key advantages of Phi-3 Mini is its small size, making it suitable for resource-constrained environments like smartphones. It can run locally on devices without requiring an internet connection, enabling fast, low-latency responses. This opens up new possibilities for businesses and developers to incorporate AI into applications where a massive cloud-based model would be impractical.
Microsoft achieved the compact yet capable Phi-3 model by using a "curriculum" inspired by how children learn from simple books. They had a larger model generate synthetic children's book data to train Phi-3, which allows it to learn complex concepts in a more efficient way.
Phi-3 Mini is now available on Microsoft's Azure platform, Hugging Face, and the Ollama framework for local deployment. Microsoft plans to release two larger Phi-3 models soon - Phi-3 Small with 7 billion parameters and Phi-3 Medium with 14 billion parameters - to provide more options across cost and quality.
The strong performance of Phi-3 Mini demonstrates the potential for small language models to punch above their weight, and Phi-3 represents a breakthrough in making highly capable AI models that can run efficiently on a wider range of devices.
