On-device large models are continuing to scale up in parameter size.

On October 13, it was learned that ModelBest officially released its new-generation on-device multimodal large model MiniCPM-V 4.7, including two versions: 35B-A3B and 1B.

Among them, 35B-A3B adopts a MoE (Mixture of Experts) architecture, with a total parameter count reaching 35 billion and only about 3 billion parameters activated per inference.

ModelBest claims this is the first open-source MoE model capable of running losslessly and smoothly on-device. In its published tests, the model requires only about 3GB of running memory to achieve a decoding speed of over 15 tokens per second on an iPad Pro.

Previously, due to limitations in phone running memory, computing power, and power consumption, deploying large models on-device faced many constraints. Especially as model parameter sizes expanded, how to balance model capability and running speed under limited hardware resources became a problem the industry needed to solve.

The application of MoE architecture provides a new approach to this challenge.

Compared with dense models that require calling all parameters for every inference, MoE models contain multiple different "experts" internally, activating only some experts per inference. This allows the model to maintain a large total parameter size while reducing the amount of computation per single pass.

But new problems followed. Although only some parameters are activated each time, the model still needs to store the complete expert weights. Phone running memory is limited and cannot hold all expert weights in memory, while frequently reading data from storage devices easily causes inference lag.

To address this issue, ModelBest's solution is LES (Lookahead Expert Scheduling) technology.

The core idea of ModelBest is to predict in advance which experts the model will need to call next. While the current computation has not yet finished, it preloads the expert weights that may be used in the next stage, allowing data reading and model computation to proceed simultaneously.

Compared with traditional on-demand loading, this technology reduces the waiting time for expert weight loading while retaining the model's original expert selection mechanism, avoiding any impact on model capability from reducing expert calls.

According to technical materials published by ModelBest, combined with weight quantization and storage layout optimization, LES can increase on-device inference speed by about 5 times.

It is worth noting that the approximately 3GB refers to the running memory usage under a specific deployment scheme and does not mean that the complete model with 35 billion parameters only requires 3GB of storage space.

In addition to the 35B-A3B version, ModelBest also launched a 1B lightweight version this time, further lowering the threshold for on-device deployment and targeting more devices with limited hardware configurations.

In terms of ecosystem adaptation, MiniCPM-V 4.7 already supports iOS, Android, and HarmonyOS systems, and is adapted to chip platforms including Rockchip, Intel, and Apple. ModelBest has also simultaneously opened its on-device adaptation code, supporting developers in deploying based on open-source inference frameworks.

As of now, the MiniCPM series of open-source models jointly launched by ModelBest and the OpenBMB open-source community has accumulated over 60 million global downloads, covering multiple fields including text, vision, and speech.

This breakthrough by ModelBest also reflects that the technological path for on-device large models is changing.

In the past, limited by phone hardware conditions, the industry focused more on compressing and optimizing small-parameter models. Today, as MoE architecture, low-bit quantization, and expert scheduling technologies gradually mature, models with tens of billions of parameters are beginning to attempt entering mobile terminals, and the capability boundaries of local AI on phones are being further broadened.

However, there is still a distance from getting models to run to truly entering ordinary consumer-grade phones. How to control the power consumption and heat generated by continuous inference, how to adapt to chip platforms at different price points, and whether larger models can bring sufficiently obvious experience improvements all require further verification.

As model and chip technologies continue to evolve, on-device large models on phones are expected to carry more complex multimodal understanding and agent tasks, further reducing dependence on the cloud. Whoever can first bring models with tens of billions of parameters from technical demonstrations to daily applications may become an important focus of the next round of on-device AI competition.