---
title: "Introducing Pipette: A benchmarking suite for on-device intelligence"
description: "Meet Pipette, an open-source platform for reproducible on-device AI benchmarks across models, quantization, runtimes and hardware."
canonical: "https://www.liquid.ai/blog/pipette-on-device-ai-benchmarking-by-liquid-ai"
locale: "en"
datePublished: "2026-08-24"
dateModified: "2026-10-27"
---

# Introducing Pipette: A benchmarking suite for on-device intelligence

Today, in partnership with [Artificial Analysis](http://artificialanalysis.ai/articles/mobile-phone-intelligence-inference), we release **Pipette**, an open-source platform for benchmarking foundation models on edge devices. Pipette is built around a simple premise: on-device behavior is a property of the deployed system, not the model in isolation. It turns edge deployment into an empirical systems problem, making the interactions that determine real-world behavior visible and testable.

What is being released today:

- **A public dataset of lab-verified results generated under published reproducibility protocols.** It contains five on-device performance metrics for more than 1,000 model × quantization × runtime × device × context configurations. The current data spans 30+ models, multiple quantization formats, llama.cpp builds for macOS, iOS, Windows, and Android, and context lengths from 256 to 8,192 tokens. Initial published results cover MacBook Pro with M5 Max, iPhone 17 Pro, and Galaxy S26 Ultra, with AMD Ryzen AI Max+ 395 and Radeon 8060S results coming soon. Explore the coverage details [here](https://pipette.liquid.ai/coverage).
- **Open-source** [**benchmark clients**](https://github.com/Liquid4All/pipette-clients) **for macOS, Windows, iOS, and Android.** These include native [iOS](https://apps.apple.com/us/app/pipette-by-liquid/id6772314671) and [Android](https://play.google.com/store/apps/details?id=ai.liquid.pipette) apps for executing performance benchmarks directly on target devices. Publication of community-submitted results is currently in beta.
- **An** [**interactive dashboard**](https://pipette.liquid.ai/leaderboard) **for analyzing deployment configurations.** It connects quality evaluations with measured throughput, latency, context scaling, and memory use on supported devices. More models are already in the pipeline.
- **Artificial Analysis combines Pipette’s on-device performance** data with [equally weighted quality evaluations chosen to represent real-world mobile use](http://artificialanalysis.ai/articles/mobile-phone-intelligence-inference).
- **Apache 2.0-licensed infrastructure for operating the complete benchmark pipeline.** The release includes [pipette-mgmt](https://github.com/Liquid4All/pipette-mgmt), [pipette-clients](https://github.com/Liquid4All/pipette-clients), and [pipette-scores](https://github.com/Liquid4All/pipette-scores).

![pipette-thumbnail](https://aypchzzf9pftwuto.public.blob.vercel-storage.com/pipette-thumbnail-OOiemVmgHBb5dupOs9qdgYH2jcw2Ue.png)

[Explore our leaderboard](https://pipette.liquid.ai/leaderboard)

[View clients on GitHub](https://github.com/Liquid4All/pipette-clients)

[Explore the Artificial Analysis mobile benchmark results](http://artificialanalysis.ai/hardware-inference-stack/mobile-phones)

[Request your model, runtime, processor and device](https://pipette.liquid.ai/feedback)

Model releases commonly include capability scores produced under server-class, full-precision conditions. These results provide a useful baseline, but they do not necessarily predict how a model will behave in its intended deployment environment. On-device deployment introduces different constraints, including whether the model fits in memory, how quickly it responds, and how its runtime behaves on a specific processor.

Pipette addresses these questions with versioned benchmark definitions and published measurement protocols designed for reproducibility. 

> Understanding the true performance of a device-scale model requires careful and simultaneous attention to many dimensions across capability, speed, latency, memory, runtime, and processor or device type. Most available benchmark suites do not capture this. Designing an unbiased and fair benchmarking suite for device-class models is not trivial, given the diversity of mobile devices outside data centers, runtimes, quantization profiles, device conditions, and benchmarking setups. This is why we decided to build Pipette, open-source it, and partner with Artificial Analysis, an independent expert validator to ensure the fairness and reproducibility of our methodology and results at scale.
>
> — Ramin Hasani, Liquid AI CEO and co-founder

## Why On-Device Benchmarks Need Deployment Context

Model cards typically report quality for the original full-precision weights, while on-device deployments often use quantized artifacts. Quantization can affect output quality, and the effect varies by model, format, and task. Pipette evaluates quantized artifacts on [IFBench](https://pipette.liquid.ai/docs/evaluations/ifbench-latest), [GPQA Diamond](https://pipette.liquid.ai/docs/evaluations/gpqa-diamond-latest), and [MATH-500](https://pipette.liquid.ai/docs/evaluations/math-500-latest), with FP16 or BF16 results providing a reference where available. This reveals how much of the reference artifact’s evaluated capability is retained after quantization. 

In Pipette, every measurement starts with a tested deployment configuration:

**model + quantization + runtime + device**

A benchmark then defines the metric and token shape to measure, producing a latency, throughput, or memory result for that configuration. Quality results are identified separately by model, quantization, evaluation version, thinking mode, and evaluation authority. The currently published quality scores come from llama.cpp evaluation runs on NVIDIA H100 80GB reference systems. These quality values are paired with compatible on-device performance measurements in the dashboard.

Cloud deployments can often add capacity or move a workload to larger or more specialized hardware. Small language models run on phones, laptops, and embedded devices, often customized, specialized, and quantized considerably to fit in memory. Runtime behavior, processor architecture, memory pressure, power state, and thermal headroom can therefore determine which configuration is viable.

The data shows how strongly these interactions can shape a deployment decision:

**Two 350M models can scale very differently with context.** At Q4_K_M on Galaxy S26 Ultra, [Granite-4.0-H-350M](https://huggingface.co/ibm-granite/granite-4.0-h-350m-GGUF) retains 78.4% of its decode throughput from 256 to 4,096 input tokens, while [Granite-4.0-350M](https://huggingface.co/ibm-granite/granite-4.0-350m-GGUF) retains only 33.8%. [Compare Granite context scaling](https://pipette.liquid.ai/leaderboard?device=Galaxy+S26+Ultra&models=ibm-granite%2Fgranite-4.0-350m-GGUF%7Cibm-granite%2Fgranite-4.0-h-350m-GGUF&quants=q4_k_m&scalingView=decode&scalingMode=throughput#how-performance-scales-with-context-length)

![Two 350M models can respond completely differently to longer contexts. At Q4_K_M on Galaxy S26 Ultra, Granite 4.0 H 350M retains 78.4% of its decode throughput from 256 to 4,096 input tokens, while Granite 4.0 350M retains only 33.8%. Compare Granite context scaling](https://aypchzzf9pftwuto.public.blob.vercel-storage.com/image-8FPSq7FJ69TcdL64SziNpccfNmbH5t.png)

---

**Sparse activation can deliver small-model speed without small-model memory use.** At 2,048 input tokens on Galaxy S26 Ultra, [LFM2.5-8B-A1B](https://huggingface.co/LiquidAI/LFM2.5-8B-A1B-GGUF) decodes 2.4x faster than [Qwen3.5-4B](https://huggingface.co/unsloth/Qwen3.5-4B-GGUF) and 2.6x faster than [Ministral-3B-Instruct-2512](https://huggingface.co/mistralai/Ministral-3-3B-Instruct-2512-GGUF). Despite activating only 1.5B of its 8.5B parameters per token, it still peaks at 5.29 GiB because all expert weights contribute to the model’s memory requirements. [Compare sparse and dense configurations](https://pipette.liquid.ai/leaderboard?device=Galaxy+S26+Ultra&models=LiquidAI%2FLFM2.5-8B-A1B-GGUF%7Cunsloth%2FQwen3.5-4B-GGUF%7Cmistralai%2FMinistral-3-3B-Instruct-2512-GGUF&quants=q4_k_m&input=2048#what-fits-in-memory-and-how-fast)

![Sparse activation can deliver small-model speed without small-model memory use. At 2,048 input tokens on Galaxy S26 Ultra, LFM2.5-8B-A1B decodes 2.4x faster than Qwen3.5-4B and 2.6x faster than Ministral-3B-Instruct-2512.](https://aypchzzf9pftwuto.public.blob.vercel-storage.com/pasted-image-au2KZ5kxub1FpZammBzJ7P2fAQzX9O.png)

---

**Two similarly sized iPhone models expose a direct speed-quality choice.** At Q4_K_M, [MiniCPM5-1B](https://huggingface.co/openbmb/MiniCPM5-1B-GGUF) completes the workload with 2,048 input tokens and 256 output tokens in 3.47 seconds versus 4.12 seconds for [LFM2.5-1.2B-Instruct,](https://huggingface.co/LiquidAI/LFM2.5-1.2B-Instruct-GGUF) a 15.8% reduction in elapsed time. In a quality evaluation of the same Q4_K_M artifacts, LFM scores 9.0 points higher on MATH-500. Neither configuration dominates both axes. [Open the iPhone Pareto comparison](https://pipette.liquid.ai/leaderboard?device=iPhone+17+Pro&models=openbmb%2FMiniCPM5-1B-GGUF%7CLiquidAI%2FLFM2.5-1.2B-Instruct-GGUF&quants=q4_k_m&input=2048&thinking=off&quality=accuracy_math_500&speed=e2e#which-models-are-fast-without-cutting-corners)

![speed-quality choice](https://aypchzzf9pftwuto.public.blob.vercel-storage.com/image-qy3cmlmMBNF1shrVYeqI71ELjjaklw.png)

---

**Nearly identical 8B deployment profiles can hide a task-level reversal.** At Q4_K_M and 2,048 input tokens on M5 Max, [Granite-4.1-8B](https://huggingface.co/ibm-granite/granite-4.1-8b-GGUF) and [Ministral-3-8B-Instruct-2512](https://huggingface.co/unsloth/Ministral-3-8B-Instruct-2512-GGUF/blob/b490e7c8263d22a7816832f3b9588ad98cdefb57/Ministral-3-8B-Instruct-2512-Q4_K_M.gguf) differ by only 2.4% in decode throughput and 1.2% in peak RAM. In quality evaluations of the same Q4_K_M artifacts, Granite leads IFBench by 7.3 points while Ministral leads GPQA Diamond by 14.0 points. [Compare performance and task quality](https://pipette.liquid.ai/leaderboard?device=MacBook+Pro&models=ibm-granite%2Fgranite-4.1-8b-GGUF%7Cmistralai%2FMinistral-3-8B-Instruct-2512-GGUF&quants=q4_k_m&input=2048&thinking=off&heroQuality=accuracy_ifbench%7Caccuracy_gpqa_diamond%7Caccuracy_math_500&heroPerf=decode%7Ce2e%7Cram#per-model-performance-across-the-board)

![task-level reversal - Granite-4.1-8B and Ministral-3-8B-Instruct-2512](https://aypchzzf9pftwuto.public.blob.vercel-storage.com/image-IpbCRg02RahyPjucj8L6NaKnJBVjen.png)

Pipette keeps these tradeoffs separate and inspectable rather than collapsing deployment readiness into a single ranking.

## How Pipette Works

**Pipette is designed around reproducibility and transparency**. Performance benchmarks follow Pipette’s [published measurement methodology](https://pipette.liquid.ai/docs/methodology/performance-methodology): timing and memory runs use fixed token shapes, greedy decoding, a discarded warm-up, five measured repetitions, and readiness gating. Pipette’s methodology has been reviewed and verified by [Artificial Analysis](http://artificialanalysis.ai/). Evaluations follow a [separate published protocol](https://pipette.liquid.ai/docs/methodology/evaluations) using standard datasets, completion generation through a reference runner, and deterministic, model-blind scoring. \
\
Three open-source components implement the pipeline:

- [pipette-mgmt](https://github.com/Liquid4All/pipette-mgmt): serves the versioned benchmark catalog and ingests benchmark submissions.
- [pipette-clients](https://github.com/Liquid4All/pipette-clients): runs benchmarks on target devices and generates evaluation completions.
- [pipette-scores](https://github.com/Liquid4All/pipette-scores): provides evaluation prompts and scores completions without access to their generation provenance.

A [client](https://github.com/Liquid4All/pipette-clients) fetches versioned benchmark definitions from [pipette-mgmt](https://github.com/Liquid4All/pipette-mgmt) and runs inference on the target device. The client returns performance measurements directly to [pipette-mgmt](https://github.com/Liquid4All/pipette-mgmt). Before each timed repetition, a platform-specific readiness check verifies acceptable thermal and load conditions. Pipette publishes results only for runs that pass this check. Power and cooling conditions for phones are documented in our [device conditions methodology](https://pipette.liquid.ai/docs/methodology/device-conditions). 

For evaluations, [pipette-mgmt](https://github.com/Liquid4All/pipette-mgmt) forwards the generated completions to [pipette-scores](https://github.com/Liquid4All/pipette-scores) for model-blind scoring, then stores the scores with their generation provenance. Each submission records the benchmark and token shape, model artifact and quantization, runtime version and settings, and the device hardware and operating system. Pipette uses this metadata to keep results from different conditions separate and make the basis of each comparison explicit. The [documented scope and measurement boundaries](https://pipette.liquid.ai/docs/methodology/limitations-and-future-directions) clarify how the current results should be interpreted.

[Inspect Pipette submissions](https://pipette.liquid.ai/submissions)

## Inside the Dashboard

The dashboard is organized around three complementary surfaces: a per-device [Leaderboard](https://pipette.liquid.ai/leaderboard) for deployment decisions, a [Results explorer](https://pipette.liquid.ai/results) for configuration-level tables, and a [Submissions browser](https://pipette.liquid.ai/submissions) for the underlying records. Across these surfaces, users can narrow the data by model, quantization, runtime, device, benchmark, context length, and thinking mode where applicable.

**Pareto frontiers under real constraints.** Plot IFBench, GPQA Diamond, or MATH-500 against time to first token, end-to-end latency, prefill throughput, decode throughput, or peak RAM. Lines connect quantization variants of the same model, while the constraints panel filters results by minimum quality or throughput and maximum latency or memory.

![pipette-pareto-intro](https://aypchzzf9pftwuto.public.blob.vercel-storage.com/pipette-pareto-uiC5O6rZmJzwgqcTfWRtdA73s879Y9.gif)

**Context scaling and quantization tradeoffs.** Dedicated views group models by parameter tier, trace throughput and memory across context lengths from 256 to 8,192 tokens, and map speed against peak RAM. Quantization views compare each artifact’s evaluation score with its full-precision reference where available.

![piette-context-length](https://aypchzzf9pftwuto.public.blob.vercel-storage.com/pipette-context-length-whLKZg6eESYBz4UhU9aU4KzxU4mCTJ.gif)

**Traceability down to the raw record.** The [Results page](https://pipette.liquid.ai/results) lists each published model, quantization, device, and runtime configuration. The [Submissions page](https://pipette.liquid.ai/submissions) exposes the records behind those results, including benchmark version, token shape, measured value, and standard deviation where applicable. A [coverage explorer](https://pipette.liquid.ai/coverage) reports how many submissions are represented across runtimes, devices, quantizations, benchmarks, and models.

![pipette-coverage-launch](https://aypchzzf9pftwuto.public.blob.vercel-storage.com/pipette-coverage-launch-tAk2E8oeT7bpkr7BJPtDO7BPcOyq4U.gif)

**Shareable, documented, and versioned.** Each chart on the Leaderboard generates a shareable link that preserves its filters. Our [documentation covers](https://pipette.liquid.ai/docs/introduction) the measurement methodology, benchmark definitions, evaluation protocols, and glossary. Our records retain the client and runtime versions needed to audit how each result was produced. For implementation details and setup instructions, see [pipette-mgmt](https://github.com/Liquid4All/pipette-mgmt), [pipette-clients](https://github.com/Liquid4All/pipette-clients), and [pipette-scores](https://github.com/Liquid4All/pipette-scores) on GitHub. 

## Limitations and Future Directions

Pipette is just getting started. Coverage is expanding through Liquid AI’s device lab, with community-contributed configurations planned as the submission workflow matures.

**NPUs:** NPU support depends on model-specific kernel and operator coverage. In this initial release, no NPU path supports enough of the published model set to provide consistent comparisons across model classes, so NPU results are not included. We are actively working to expand this coverage and [welcome contributions from hardware and runtime partners](https://pipette.liquid.ai/feedback).

**Android GPU coverage:** Under the current benchmark setup and model coverage, none of the stable GPU backends tested on Android consistently outperformed the selected CPU path across the full model set. The current llama.cpp Android path is therefore CPU-focused, while the iPhone path uses the Metal backend.

**Comparison across devices:** We do not currently recommend interpreting cross-device results as controlled hardware comparisons. Android and iOS runs differ in flash-attention support, thread counts, accelerator use, and execution environment. Android runs use a CPU-based CLI path, while iOS runs inside the app using Metal. Current Pipette results are most reliable for comparing configurations within the same device.

**Quality evaluations for device-class applications:** The current suite covers instruction following, science reasoning, and competition mathematics, but it is not comprehensive for agentic behavior, knowledge-intensive tasks, multimodal workloads, or other device-oriented use cases. For this release, Artificial Analysis [reports the average of common evaluations chosen to represent real-world mobile use](http://artificialanalysis.ai/hardware-inference-stack/mobile-phones). We welcome [feedback](https://pipette.liquid.ai/feedback) on which capabilities and benchmarks should be included to better represent on-device model quality.

## Get Started

[Explore the Pipette dashboard](https://pipette.liquid.ai/leaderboard) to compare models, quantizations, runtimes, and context lengths on the device you plan to target. Every filtered chart has a shareable URL, so individual deployment comparisons can be inspected and discussed directly.\
\
Run Pipette on your own hardware using the [open-source clients](https://github.com/Liquid4All/pipette-clients), the native [iOS app](https://apps.apple.com/us/app/pipette-by-liquid/id6772314671), or the [Android app](https://play.google.com/store/apps/details?id=ai.liquid.pipette). Publication of community-submitted results is currently in beta, and we encourage model providers, runtime developers, hardware teams, and application builders to help test the workflow.\
\
If a model, runtime, device, or benchmark you need is missing, [open an issue](https://github.com/Liquid4All/pipette-dashboard/issues/new) or [contact Liquid AI](https://www.liquid.ai/connect) to request coverage or share feedback. More configurations will be added as they complete the measurement and publication process.

[Read Pipette docs](https://pipette.liquid.ai/docs/introduction)

###  

### Appendix

Pipette launches with the following models and quantization formats:\
\
**LFM2.5-230M:** Q4_0, Q4_K_M, Q5_K_M, Q8_0. [Hugging Face](https://huggingface.co/LiquidAI/LFM2.5-230M-GGUF)

**LFM2.5-350M:** Q4_0, Q4_K_M, Q5_K_M, Q8_0. [Hugging Face](https://huggingface.co/LiquidAI/LFM2.5-350M-GGUF)

**LFM2.5-1.2B-Instruct:** Q4_0, Q4_K_M, Q5_K_M, Q8_0. [Hugging Face](https://huggingface.co/LiquidAI/LFM2.5-1.2B-Instruct-GGUF)

**LFM2.5-2.6B:** Q4_0, Q4_K_M, Q5_K_M, Q8_0. [Hugging Face](https://huggingface.co/LiquidAI/LFM2.5-2.6B-GGUF)

**LFM2.5-8B-A1B:** Q4_0, Q4_K_M, Q5_K_M. [Hugging Face](https://huggingface.co/LiquidAI/LFM2.5-8B-A1B-GGUF)

**Qwen3.5-0.8B:** Q4_0, Q4_K_M, Q5_K_M, Q8_0. [Hugging Face](https://huggingface.co/unsloth/Qwen3.5-0.8B-GGUF)

**Qwen3.5-2B:** Q4_0, Q4_K_M, Q5_K_M, Q8_0. [Hugging Face](https://huggingface.co/unsloth/Qwen3.5-2B-GGUF)

**Qwen3.5-4B:** Q4_0, Q4_K_M, Q5_K_M, Q8_0. [Hugging Face](https://huggingface.co/unsloth/Qwen3.5-4B-GGUF)

**Qwen3.5-9B:** Q4_K_M. [Hugging Face](https://huggingface.co/unsloth/Qwen3.5-9B-GGUF)

**Qwen3.5-27B:** IQ1_M. [Hugging Face](https://huggingface.co/mradermacher/Qwen3.5-27B-i1-GGUF)

**Qwen3.6-27B:** IQ1_M. [Hugging Face](https://huggingface.co/mradermacher/Qwen3.6-27B-i1-GGUF)

**Gemma 4 E2B Instruct:** Q4_0, Q4_K_M, Q5_K_M, Q8_0. [Hugging Face](https://huggingface.co/unsloth/gemma-4-E2B-it-GGUF)

**Gemma 4 E4B Instruct:** Q4_0, Q4_K_M, Q5_K_M, Q8_0. [Hugging Face](https://huggingface.co/unsloth/gemma-4-E4B-it-GGUF)

**Gemma 4 12B Instruct:** Q4_K_M. [Hugging Face](https://huggingface.co/unsloth/gemma-4-12b-it-GGUF)

**Granite 4.0 350M:** Q4_0, Q4_K_M, Q5_K_M, Q8_0. [Hugging Face](https://huggingface.co/ibm-granite/granite-4.0-350m-GGUF)

**Granite 4.0 H 350M:** Q4_0, Q4_K_M, Q5_K_M, Q8_0. [Hugging Face](https://huggingface.co/ibm-granite/granite-4.0-h-350m-GGUF)

**Granite 4.0 H 1B:** Q4_0, Q4_K_M, Q5_K_M, Q8_0. [Hugging Face](https://huggingface.co/ibm-granite/granite-4.0-h-1b-GGUF)

**Granite 4.0 H Micro:** Q4_0, Q4_K_M, Q5_K_M, Q8_0. [Hugging Face](https://huggingface.co/ibm-granite/granite-4.0-h-micro-GGUF)

**Granite 4.1 8B:** Q4_K_M. [Hugging Face](https://huggingface.co/ibm-granite/granite-4.1-8b-GGUF)

**Ministral 3 3B Instruct 2512:** Q4_0, Q4_K_M, Q5_K_M, Q8_0. [Q4_0 source](https://huggingface.co/unsloth/Ministral-3-3B-Instruct-2512-GGUF). [Q4_K_M, Q5_K_M, and Q8_0 source](https://huggingface.co/mistralai/Ministral-3-3B-Instruct-2512-GGUF)

**Ministral 3 8B Instruct 2512:** Q4_K_M. [Hugging Face](https://huggingface.co/mistralai/Ministral-3-8B-Instruct-2512-GGUF)

**Llama 3.2 1B Instruct:** Q4_0, Q4_K_M, Q5_K_M, Q8_0. [Hugging Face](https://huggingface.co/unsloth/Llama-3.2-1B-Instruct-GGUF)

**Llama 3.2 3B Instruct:** Q4_0, Q4_K_M, Q5_K_M, Q8_0. [Hugging Face](https://huggingface.co/unsloth/Llama-3.2-3B-Instruct-GGUF)

**Falcon-H1R-7B:** Q4_K_M. [Hugging Face](https://huggingface.co/tiiuae/Falcon-H1R-7B-GGUF)

**Olmo-3-7B-Think:** Q4_K_M. [Hugging Face](https://huggingface.co/unsloth/Olmo-3-7B-Think-GGUF)

**Ornith-1.0-9B:** Q4_K_M. [Hugging Face](https://huggingface.co/unsloth/Ornith-1.0-9B-GGUF)

**Ornith-1.5-9B:** Q4_K_M. [Hugging Face](https://huggingface.co/bartowski/Ornith-1.5-9B-GGUF)

**Nanbeige4.2-3B:** Q4_K_M. [Hugging Face](https://huggingface.co/bartowski/Nanbeige_Nanbeige4.2-3B-GGUF)

**Ling-3.0-tiny:** Q4_K_M. [Hugging Face](https://huggingface.co/bartowski/Ling-3.0-tiny-GGUF)

**MiniCPM5-1B:** Q4_K_M. [Hugging Face](https://huggingface.co/openbmb/MiniCPM5-1B-GGUF)

**NVIDIA Nemotron Nano 9B v2:** Q4_K_M. [Hugging Face](https://huggingface.co/bartowski/nvidia_NVIDIA-Nemotron-Nano-9B-v2-GGUF)

**ai9stars G9v3 3B:** Q4_K_M. [Hugging Face](https://huggingface.co/bartowski/ai9stars_G9v3-3B-GGUF)

**Bonsai-27B:** Q1_0. [Hugging Face](https://huggingface.co/prism-ml/Bonsai-27B-gguf)

**Ternary-Bonsai-27B:** Q2_g64. [Hugging Face](https://huggingface.co/prism-ml/Ternary-Bonsai-27B-gguf)
