It listens · entirely in your browser

Whisper, and who said it.
In this tab.

franken_whisper rebuilds OpenAI's Whisper and NVIDIA's Streaming Sortformer speaker diarizer in pure, memory-safe Rust: ggml parser, log-mel frontend, encoder, decoder, word timestamps, four-speaker diarization, and the fusion that stitches them into who said what. No PyTorch, no Python, no FFI, no subprocess. The same code that ships as the fw CLI compiles to WebAssembly, so the full 809-million-parameter large-v3-turbo transcribes your recordings and the 123-million-parameter Sortformer labels the speakers, entirely on your CPU, in this page. Your audio never leaves the tab. The CSP's connect-src 'self' makes that a browser-enforced promise rather than a privacy-policy one.

Try it in your browser Install the CLI

100% Rust 809M + 123M parameters, in-tab 1.3 GB of weights, SHA-256-pinned 4 speakers, turn by turn 0 bytes uploaded
The franken_whisper monster listening with a stitched ear horn, sorting voices into labeled threads

Live · in this tab

Run it right now.

This panel runs the real pipeline. Whisper large-v3-turbo (874 MB, q8_0 — transcript-gated identical to the f16 reference) transcribes, Sortformer (469 MB, f32) works out who is speaking when, and the CLI's own projection-fusion attributes every line, all in hand-written Rust kernels compiled to wasm and running on your CPU. Both downloads are SHA-256-verified against pinned manifests, cached in browser storage, and re-verified on every visit. The Sortformer package is additionally re-authenticated inside the engine against its conversion receipt, exactly as the CLI does it. There is no server behind this panel and no mocked demo.

01 · The models

Starting engine…

02 · The recording

Drop an audio file here (mp3, m4a, wav), or click to pick one. A meeting, an interview, a voice memo: anything with people talking.

Decoded in-tab by Symphonia (pure Rust), downmixed and resampled to 16 kHz mono. Honest expectation, measured on an M-series desktop: Chrome gets the threaded engine at about 1.6 times real time (a 1-minute clip takes about 100 seconds); browsers without cross-origin isolation, iPhones included, get the single-thread build at about 8 times real time. The progress bar below shows the live estimate either way.

03 · Language and speakers (optional)

Auto-detection listens to the first 30 seconds; if a recording opens with music or noise it can guess wrong (and then transcribe the whole file in the wrong language). If you know the language, pin it.

Names help twice. They feed Whisper's decoding prompt, the same mechanism as the CLI's --prompt, so names and titles come out spelled right. They also replace the anonymous SPEAKER_00…03 labels, assigned in order of first appearance. The voices themselves are matched by sound alone, so if the first person to talk is not the first name you listed, swap the order and export again.

04 · Run

The transcript

Results land here: every line timestamped and speaker-attributed, with the same SPEAKER_00…03 labels and fusion rules the CLI emits. The measured wall-clock and realtime ratio print with each run, from your hardware.

Anatomy · one code path

The same bytes as the CLI.

The wasm crate mounts the native engine's source files directly, so the browser runs the identical Rust that decodes on a workstation, behind one platform seam: a host-fed clock and a serial thread scope. What differs is named and documented; everything else is the same pipeline, stage for stage.

  1. ggml parser. The 874 MB q8_0 turbo file is scanned once for its tensor directory, then each tensor hydrates straight from browser storage through a positioned reader; the file never exists as one in-memory blob. Quantized blocks stay block-RESIDENT and dequantize per row inside the kernels — bit-identical math to a dequantize-at-load reference, at less than half the memory.
  2. Neural clean-up. FastEnhancer‑S denoises the 16 kHz signal before anything downstream hears it (see the denoiser section below), and leading silence is trimmed so Whisper never free-runs on an empty first window. Both are length-honest: timestamps stay true to the original file.
  3. Log-mel frontend. The reference mel pipeline, bit-faithful to the native engine: 25 ms windows, 10 ms hop, 128 mel bins for turbo.
  4. Encoder + decoder. 32 encoder layers, 4 decoder layers, greedy temperature-0 decode with the CLI's window retry and context-carry policies. The wasm build multiplies f16 weights with the same batched dequant-GEMV kernels the native decoder ships by default.
  5. Sortformer diarizer. NVIDIA's streaming four-speaker model (FastConformer frontend, transformer stack, per-frame speaker lanes), hand-ported to the same Rust and verified in-engine against its conversion receipt before a single frame is scored.
  6. Projection fusion. The CLI's projection-fusion-v1 attributes each transcript segment to the dominant speaker turn, fills gap segments from the turn timeline, and merges runs into speaker-attributed blocks. The wasm build calls the very function the CLI calls; the code was relocated into a shared module so the two can never drift apart.

The denoiser · clean audio in, clean words out

Noise makes Whisper hallucinate.

A transcription inherits the recording it came from, noise and all. Feed Whisper hiss, rumble, or a silent stretch and it does something worse than missing words: it invents them, sometimes in the wrong language. So every recording is cleaned automatically before the model hears a sample, in the CLI and in this playground, using a pure-Rust port of FastEnhancer‑S, a 207 K-parameter speech-enhancement network. The same port cleans voice enrollments in our sibling project franken_tts; the weights are an 0.8 MB file that loads alongside the models. It is on by default and one checkbox (or FW_DENOISE=0 in the CLI) turns it off.

How it decides what is noise

The recording is sliced into overlapping 21 ms spectrogram frames. For every time-frequency cell the network predicts a complex ratio mask: a number that scales the cell toward zero if it looks like noise and leaves it alone if it looks like speech. Two structural tricks let a model this small make that call well. Small GRUs carry memory from frame to frame, so the network builds a running picture of the steady background; hiss and rumble are steady, speech never is. And a tiny attention layer compares frequency bands within each frame; voiced speech is a comb of harmonics that move together, while broadband static has no such structure. The mask is complex, so it corrects phase as well as loudness, avoiding the underwater smearing that magnitude-only methods produce.

Why it matters for transcription

Whisper was trained mostly on reasonably clean speech, and its failure mode on noise is not graceful: confidence drops, words vanish, and on the worst stretches the decoder free-runs and invents text. The diarizer suffers too, since speaker identity lives in exactly the spectral detail that noise buries. Cleaning first attacks both problems at the source. The stage is length-preserving, so every timestamp stays truthful to your original file. It processes audio in one-minute chunks with carried context and reports each chunk as it finishes. On the measured single-thread path it costs about 0.24× audio duration (roughly 11 minutes for a 45-minute call), so turn the playground checkbox off when clean audio or turnaround time matters more.

207 K parameters

The weights are an 0.8 MB pinned file, verified by SHA-256 like every other artifact on this page. It loads in a blink next to the 874 MB transcription model.

On by default

Zero flags to remember. The playground checkbox and the CLI's FW_DENOISE=0 both turn it off when you want the raw signal, and the run metadata records which way it ran.

Same code, both surfaces

One Rust implementation serves the CLI and this page. In the CLI the stage engages whenever the pinned artifact is present in the model cache; the browser ships the artifact with the models, so it is simply always available.

Diarization · who said what

Names on every line.

Transcription without attribution is half an answer, so diarization runs as a first-class stage: Sortformer scores four speaker lanes at 80 ms resolution, turns become a timeline with per-turn confidence and overlap suspicion, and the fusion pass projects that timeline onto the transcript so consecutive same-speaker segments merge into readable blocks. The CLI exposes the same result as speaker_segments in its JSON envelope; the playground renders it directly.

Turn timeline

Start, end, speaker, confidence, and overlap suspicion for every detected turn: the raw evidence, preserved alongside the merged view.

Conservative first

A segment is labeled only when one speaker owns at least 70% of it at calibrated confidence; the fusion pass then fills the gaps from the same evidence instead of returning null.

Receipt-verified

The diarizer's weight package re-authenticates against its frozen conversion receipt (SHA-256 chain, tensor census) inside the engine, in the browser, every load.

Speed · every number measured

Honest numbers.

On native CPU, franken_whisper's engine measures 2.99× faster than whisper.cpp at matched greedy decode settings, side-by-side in one interleaved harness; the methodology and evidence hashes live in the repo's performance ledger. On Apple-silicon GPU, the Metal encoder went from 15.4 s to 1.85 s per window across one ledgered kernel campaign. The wasm build in this page started at 45 times real time; one byte-identical kernel pass brought it to 8.4, and the threaded lane (rayon over Web Workers, the same recipe the sibling projects ship) measured 1.56× real time in headless Chrome with 8 workers, transcript unchanged. Chrome gets the threaded engine; everything else gets the serial build. The page prints the exact ratio after each run, so the number you read is from your machine.

2.99× vs whisper.cpp

Native CPU, matched greedy settings, interleaved A/B pairs, transcript-equality gate, evidence bundle hashed in the ledger.

Every claim gated

One lever per pass, a kill-switch env for every optimization, byte-identical or ledgered. The repo's PERF_LEDGER.md and NEGATIVE_EVIDENCE.md are the record.

This page tells the truth

The playground prints wall-clock and realtime ratio per run. If it is slow on your hardware, it says so, in numbers.

Agent-first · scriptable to the bone

Built for robots too.

The CLI answers to agents first: NDJSON streaming events, a stable JSON envelope with raw_output introspection (which encoder route actually ran, window stats, dropped-window accounting), durable SQLite job storage, and exit codes that mean things. If you are wiring transcription into a pipeline, the contract is the product. 

fw robot run --input meeting.m4a | jq -r '
  select(.event == "run_complete") | .segments[]
  | "[\(.speaker // "?")] \(.text)"'

Install · one line

Take it home.

The native CLI is faster than this tab will ever be, runs every model tier, and speaks NDJSON. One line, no Python environment, no CUDA archaeology. 

# macOS / Linux
curl -fsSL https://raw.githubusercontent.com/Dicklesworthstone/franken_whisper/main/install.sh | bash

# or Homebrew
brew install dicklesworthstone/tap/franken-whisper

# or cargo
cargo install franken_whisper

# Homebrew and cargo installs do not bundle the model packages
fw pull all --json
fw doctor --json

Then: fw transcribe --input recording.mp3 --json