https://www.reddit.com/r/LocalLLaMA/comments/14wbmio/gpt4_details_leaked/ Here’s a summary:

GPT-4 is a language model with approximately 1.8 trillion parameters across 120 layers, 10x larger than GPT-3. It uses a Mixture of Experts (MoE) model with 16 experts, each having about 111 billion parameters. Utilizing MoE allows for more efficient use of resources during inference, needing only about 280 billion parameters and 560 TFLOPs, compared to the 1.8 trillion parameters and 3,700 TFLOPs required for a purely dense model.

The model is trained on approximately 13 trillion tokens from various sources, including internet data, books, and research papers. To reduce training costs, OpenAI employs tensor and pipeline parallelism, and a large batch size of 60 million. The estimated training cost for GPT-4 is around $63 million.

While more experts could improve model performance, OpenAI chose to use 16 experts due to the challenges of generalization and convergence. GPT-4’s inference cost is three times that of its predecessor, DaVinci, mainly due to the larger clusters needed and lower utilization rates. The model also includes a separate vision encoder with cross-attention for multimodal tasks, such as reading web pages and transcribing images and videos.

OpenAI may be using speculative decoding for GPT-4’s inference, which involves using a smaller model to predict tokens in advance and feeding them to the larger model in a single batch. This approach can help optimize inference costs and maintain a maximum latency level.