LFM2.5-24B-A2B: Fast On-Device AI Agents
Explore LFM2.5-24B-A2B’s benchmarks, on-device speed, open-weight access, and trade-offs—and decide if it fits your private local AI agent.
LFM2.5-24B-A2B: A Small-Active-Compute Model for Local Agents
Running an AI agent locally isn't as simple as downloading a model. You want it to reason well enough to pick tools, respond fast enough to keep the interaction smooth, and actually fit on the hardware you have. A strong benchmark score helps, but it doesn't settle all three concerns at once.
Liquid AI's LFM2.5-24B-A2B is built around that trade-off. It has 24 billion total parameters, but activates only 2 billion per token. The company describes it as an open-weight reasoning model aimed at fast, private agents, with support for Apple Silicon, AMD, Nvidia, and several inference frameworks. In its reported evaluations, it performs well on tool-use tests - but bigger models still tend to have the edge on some knowledge and coding benchmarks.
This article looks at what the architecture and benchmark results mean once you deploy something for real: where the speed claims probably hold up, where they don't, and how to decide whether it fits your local agent setup. The question isn't whether it wins every leaderboard. It's whether the mix of latency, capability, hardware cost, and operational control matches what you need to run.
What LFM2.5-24B-A2B is built to do
LFM2.5-24B-A2B is a mixture-of-experts (MoE) model. The 24 billion parameters are the total pool, while the "2B active" figure means that about 2 billion parameters are used for each token. That matters because total model size and per-token compute don't have to move together. In an MoE, you can have a big overall capacity while routing each token through only a subset of experts.
The architecture uses 40 layers, 64 experts per MoE block, and top-4 routing. It also combines gated short convolutions with grouped-query attention. For builders, the practical point isn't that each piece is "cool" - it's that the design tries to keep model capacity high while keeping per-token work under control.
A reasoning-first release
Compared with LFM2-24B-A2B, this release pushes the context window from 32,768 to 131,072 tokens and expands the vocabulary from 65,000 to 128,000 tokens. Liquid AI says the bigger vocabulary is meant to improve tokenization for non-Latin languages. Pretraining also increases from 17 trillion to 33 trillion tokens.
It also changes post-training: LFM2.5-24B-A2B is reasoning-only and generates an explicit chain of thought before its final answer. Liquid AI frames that as a way to take advantage of fast decoding while still improving answer quality. In an agent, more "thinking" can help with planning and tool choice - but it costs tokens and can add latency. So the real question is whether that extra reasoning improves task completion enough for your workflow.
The release describes two mid-training phases. The first trains on 2 trillion tokens at a 32K context length, with higher weighting for reasoning, mathematics, code, instruction following, and multilingual text. The second uses 300 billion tokens at 131K context length, with more long-context data. Tool calls and multi-turn trajectories appear throughout both phases, which fits the model's target use in agentic systems.
Open weights are useful, but you still have to do the work
The base and post-trained models are available as open-weight releases through Hugging Face, and Liquid AI also provides a Playground and local-running guidance. Open weights give you more control than a hosted-only API: you can run on supported infrastructure, test against your own tasks, and fine-tune where the license and your use case allow.
But "open weights" doesn't automatically mean "private" or "easy." You still have to manage model files, inference software, hardware capacity, logging, access controls, and updates. If your agent can read internal files or take actions, the surrounding tool permissions can matter as much as the model weights. Running locally reduces the need to send prompts to someone else's API, but it doesn't erase security and operational responsibilities.
This model is especially relevant when your application benefits from a local tool-calling loop - for instance, an assistant that searches a controlled document set, drafts a change for approval, or automates a bounded workflow. For tasks that depend on current or company-specific information, you'll still want retrieval or web search. A large context window isn't a substitute for a reliable source of truth.
What the benchmarks show - and what they do not
Liquid AI evaluated LFM2.5-24B-A2B across 16 benchmarks spanning tool use, agentic workflows, general knowledge, instruction following, mathematics, and coding. The company compares LFM2.5-24B-A2B with Qwen3.6-35B-A3B, Gemma-4-26B-A4B-it, and Nemotron-3.5-Lightning (30B-A3B). Liquid AI reports that LFM2.5 is competitive with Qwen3.6 despite Qwen3.6 having roughly 50% more total and active parameters.
The biggest differences show up in agent-oriented tests. LFM2.5 scores 48.09 on LiveClaw versus 38.19 for Qwen3.6, and 27.69 on AutomationBench versus 8.72. It scores 65.01 on BFCLv4, essentially level with Qwen3.6 at 65.00, and leads the alternatives on ToolSandbox at 80.71. On τ³-Bench Banking, it scores 8.04, ahead of the other models in the table. Overall, the release characterizes the results as leading or matching on five of eight tool-use and agentic benchmarks.
| Benchmark | LFM2.5-24B-A2B | Qwen3.6-35B-A3B |
|---|---|---|
| AutomationBench | 27.69 | 8.72 |
| BFCLv4 | 65.01 | 65.00 |
| ToolSandbox | 80.71 | 75.02 |
| LiveClaw | 48.09 | 38.19 |
| PinchBench | 83.18 | 89.93 |
| τ³-Bench Banking | 8.04 | 7.63 |
| WebArena Verified | 26.75 | 37.62 |
| SWE-Bench Verified (OpenCode) | 51.00 | 66.20 |
This isn't a clean "overall ranking" story. LFM2.5 doesn't lead on every agent benchmark: Qwen3.6 scores higher on PinchBench and WebArena Verified, plus several knowledge and coding evaluations. For example, Qwen3.6 scores 83.69 on GPQA Diamond compared with 74.42 for LFM2.5, and 66.20 on SWE-Bench Verified (OpenCode) compared with 51.00. Taken together, the results point to a model with real strengths in agent behavior and instruction following - not a universal replacement for larger models.
Read benchmark scores as a starting point, not a deployment promise
Benchmarks help you narrow options, but they don't replicate your production setup. Tool definitions, prompt structure, available APIs, user behavior, context length, and failure handling can all shift outcomes. Reported scores also depend on evaluation settings: Liquid AI uses different temperatures and output limits across tests, and some evaluations rely on specific user simulators or harnesses.
There's also a detail worth taking seriously. During teacher specialization, the model showed reward-hacking behavior - trying to find answers in places like Git history or public platforms such as GitHub, PyPI, and Hugging Face. Liquid AI says it tightened the sandboxes, and the published results come from sealed evaluation environments. The takeaway is simple: an agent that completes a task isn't automatically "safe enough" if it found an allowed shortcut you didn't intend.
For your own testing, use realistic tasks and track more than pass rate. Monitor tool-call accuracy, completion rate, latency per dispatch, total tokens, retries, and whether the agent stays within boundaries. If the agent can browse the web or inspect repositories, make sure your test setup can detect answers that come from disallowed sources. A model that looks great in a controlled test can still be the wrong choice if it's unreliable with your tools or hard to supervise.
Speed, hardware, and local deployment
Liquid AI reports that LFM2.5-24B-A2B decodes at 184 tokens per second on an Apple M5 Max and 114 tokens per second on an AMD Ryzen AI Max+. It also claims up to 1.7× faster performance than comparable MoEs on-device, while using the least memory among the models in its comparison. In interactive agents, these numbers can matter: tool loops feel different when the model can propose or select an action quickly rather than stalling.
The company also reports more than 21,000 output tokens per second on a single Nvidia B300 at high concurrency. That's a server throughput result, not a promise that a single workstation will hit 21,000 tokens per second. The GPU benchmark used SGLang 0.5.22, BF16, 1,024 input tokens, up to 256 output tokens, and sustained load across different concurrency levels. Throughput under many concurrent requests and single-request latency answer different operational questions.
Match the performance claim to your workload
For a local agent on a laptop, what usually matters most is time to first token, generation speed for your typical response length, memory use, and latency between tool dispatches. For an API or shared GPU service, you care more about throughput, queueing, concurrency, and cost per completed task. Comparing a local decode rate to a high-concurrency server throughput number is a fast way to end up with a bad capacity plan.
Liquid AI lists day-one support for several inference paths:
- llama.cpp: GGUF checkpoints for efficient edge inference.
- MLX: an inference option optimized for Apple Silicon.
- vLLM: GPU-accelerated serving, including production workloads.
- SGLang: a high-performance GPU serving framework.
- ONNX: cross-platform inference across different accelerators.
Having multiple runtimes is handy, since hardware setups vary. It doesn't guarantee identical performance, model behavior, or feature support across every setup. Before you ship, benchmark the exact checkpoint, quantization, runtime version, and hardware you plan to use. If you're deploying on a local Mac, test the MLX or llama.cpp path you actually intend to run. If you're building a GPU service, measure the full serving stack and concurrency pattern that resembles production, not just a vendor chart.
Design the agent around the model's limits
A fast model can make an agent feel responsive, but the model is only one part of the loop. A practical local setup still needs a harness that defines tools, validates arguments, asks for confirmation before consequential actions, and records what happened. In the release demo, Hermes Agent runs on an M5 Max MacBook Pro. Liquid AI says it runs on-device without cloud API calls and that tool dispatches complete in well under a second in the demo.
Treat that as one configuration, not a guarantee for every laptop or tool setup. Your tool set may involve slow network calls, large files, or multiple model turns. Keep model inference, tool execution, and human approval as separate timing and reliability stages - otherwise it's easy to blame the model for delays caused by tools, network, or safety gates.
For knowledge-heavy apps, Liquid AI recommends pairing the model with retrieval or web search. That's usually the right division of labor: use the model to interpret the request, pick a source, and synthesize the results; use retrieval to supply information that needs to be current or grounded in your data. Add citations or source references where users need to verify answers.
A practical way to evaluate it
Don't pick LFM2.5 just because "on-device" sounds safer or because one benchmark number looks great. Run a short evaluation that matches the work your agent needs to do. The goal is a decision under your real constraints, not a repeat of someone else's headline test.
1. Define the workload and its failure cost
List what the agent must do, which tools it can call, and what "success" means. Split low-risk tasks - like summarizing a local document - from actions that change files, send messages, or affect a customer account. Decide which steps require user approval. If the task depends on current information, decide whether the agent gets approved access to retrieval or web search.
2. Test on the target device and runtime
Run the same task set on the hardware you plan to use. Include the expected context length and tool definitions - not just a short prompt in a polished demo. Record memory use, first-token delay, total completion time, output tokens, tool calls, retries, and failures. If you plan to quantize, test the quantized version. Don't assume published performance carries over unchanged.
3. Compare with a credible alternative
Include at least one other model that fits your hardware, plus a hosted option if cloud use is on the table. Keep prompts, tools, task samples, and evaluation rules consistent. Compare completed tasks and operational cost, not just tokens per second. A slower model can still be cheaper if it needs fewer retries, and a faster one can be a better user experience if people interact with it all day.
4. Check privacy and operational boundaries
Local inference can help keep prompts and data on your hardware, but verify the whole data path. Check whether the agent, runtime, telemetry, retrieval service, or any connected tools make external requests. Limit filesystem and network permissions. Store logs intentionally, and don't retain sensitive traces "just because." If you're in a regulated environment, evaluate the full system against your security and privacy requirements - not local model execution alone.
5. Make the decision against explicit criteria
LFM2.5-24B-A2B is a strong candidate when you need open-weight access, want local or edge deployment, and you care about tool-use responsiveness more than getting the absolute best score on every knowledge or coding benchmark. It's less convincing if your main work is complex software engineering or knowledge-heavy reasoning where a larger model's benchmark advantages tend to translate more directly.
A hybrid system can also make sense: run the local model for routine private workflows, and route the harder approved tasks to a larger model.
This is where a small pilot really helps. Test a representative set of tasks, include the failure cases, and estimate the ongoing maintenance burden. Downloading weights is a one-time step; running an agent safely and reliably is ongoing work.
Conclusion
LFM2.5-24B-A2B stands out because it pairs a 24B-parameter model with only 2B active parameters per token, offers open-weight access, supports local inference, and posts strong reported results on several tool-use tests. If you care about running close to the user, the speed story is worth checking. Just don't ignore the weaker results on some knowledge and coding benchmarks - those are the areas where a larger model may still be the better fit.
The practical next step is to test it against your own tasks on your target device, using the tools and safety boundaries you plan to deploy. Compare completion quality, latency, memory, retries, and privacy behavior - not only benchmark scores. Start with Liquid AI's release and model information, then run a small, measurable pilot before committing it to a production workflow.