Benchmarks and tasks
A benchmark is a fixed collection of OCR tasks. A task pairs a dataset with the image input given to the model. The Scandinavian OCR v2 benchmark contains 12 tasks from 10 datasets. Danish OCR v2 contains seven tasks, including Danish Fraktur. Norwegian and Swedish benchmarks contain their respective subsets. Each benchmark page lists its exact tasks and links to examples and source data.
Cropped text line. The model receives an image of one line and returns that line’s text. Existing line images are used where available; page datasets are cropped using their annotations.
Whole document page. The model receives an entire page and returns its text in reading order. The score measures text accuracy, including omissions, additions and ordering errors. It does not separately evaluate bounding boxes, tables or layout structure.
Sample selection
Each dataset has a fixed evaluation sample shared by all models evaluated on that task. The original 11 tasks target 1,000 lines or 100 pages per dataset, using seed 20260917. Modern Danish has 977 available lines, so all are used. Danish Fraktur uses its complete set of 20 pages.
Line-only datasets use uniform sampling across the source split, except Riksarkivet, where sampling is balanced across 19 source files. Page samples are balanced across documents. Historical Danish lines are sampled from the 100 selected pages; modern Danish lines are drawn from the full collection.
The dataset catalogue lists each source split and reference type. A source labelled train is used for evaluation here; it is not an official held-out test split. The benchmark does not train or fine-tune models on these samples.
The Danish Fraktur task uses all 20 curated pages from two books, ten per book. Its initial five open models were selected by their saved Swedish Fraktur CER. Swedish inputs are cropped lines; Danish inputs are whole pages, so this is not a matched language comparison. References combine validated Wikisource transcriptions with an additional AI scan review. Danish Fraktur contributes one equally weighted task to both the Danish and Scandinavian benchmarks.
Character and word error rates
CER measures edits at the character level. WER measures edits at the word level. Both use the minimum number of substitutions, deletions and insertions needed to turn a prediction into its reference.
Counts are accumulated across samples within each dataset before division: these are corpus error rates, not averages of per-image percentages. A CER of 5% means five character edits per 100 reference characters. Lower is better; insertions can push a score above 100%.
Text normalization
Prediction and reference are Unicode NFC-normalized, and consecutive whitespace is collapsed to one space. Leading and trailing whitespace is removed. Case, punctuation, diacritics and historical spelling are preserved. Metrics are computed with jiwer.
Unsuccessful predictions
A failed or timed-out attempt is scored as an empty prediction against its full reference. It is not dropped from the score or retried. Repetition and other incorrect output remain part of the evaluation.
Aggregation and ranking
The table groups scores by dataset, with separate subcolumns for cropped lines and full pages where both are evaluated. Each subcolumn is a task, and the overall score aggregates all tasks in the selected benchmark. Task cells always contain corpus CER or WER in percent, regardless of aggregation. Click a dataset name to read its description, the arrow beneath it to sort by that task, or a score to inspect example predictions. Bold marks the lowest error rate on a task; darker red shading indicates a worse position among the models evaluated on that task. Shading is relative within each column, not a shared error-rate scale.
Mean
The overall score is the arithmetic mean of task error rates, with each task weighted equally. Task scores of 2% and 8% produce a mean of 5%, regardless of image counts. A dataset evaluated as both lines and pages contributes two tasks. Lower is better.
Borda rank
Models are ranked by error rate on each task, then these positions are averaged. A score of 1 means first on every task. Exact ties share the average of their occupied positions: two models tied for first each receive 1.5. Lower is better.
Win rate
Each task supplies a pairwise comparison between models: lower error wins, and a tie awards half a win to each. A Bradley–Terry model is fitted to these comparisons using 200 minorization–maximization iterations. For model strengths sᵢ and sⱼ, the fitted probability of i beating j is sᵢ / (sᵢ + sⱼ). The displayed score averages this probability over all other eligible models and expresses it as a percentage. Higher is better. This is a fitted win probability, not the fraction of tasks won outright; no confidence intervals are estimated.
The aggregation choices follow the Danish ASR leaderboard. Borda ties use average positions here, and a numerical floor of 10⁻¹² keeps Bradley–Terry strengths positive. Win rate is unavailable if fewer than two models qualify.
Incomplete evaluations and ties
All three overall scores require a completed, valid evaluation on every task in the selected benchmark. Only these fully evaluated models form the Borda and win-rate comparison pool. Models with partial results remain visible with their available task scores, but receive no overall score. A dash means no score is available; it never means zero error. Per-image failures inside a completed evaluation are already counted in its task error rate as described above.
Sorting uses full precision before display rounding. Equal scores sort by model name. Search only hides rows: it does not change aggregate scores, task shading, or the comparison pool. Borda and win-rate scores can change when models enter or leave the benchmark, even if their transcription results are unchanged. The comparison page shows separate mean scores for lines and pages; its page mean includes Danish Fraktur.
Performance by model size
The Pareto view plots CER or WER against estimated active parameters on a logarithmic horizontal axis. Lower and further left is better. Overall uses the mean across every task in the selected benchmark, regardless of the table’s aggregation setting. Selecting a single task includes models evaluated on that task, including specialist line recognisers.
A model is on the Pareto frontier when no other eligible model has both fewer or equal active parameters and lower or equal error, with a strict improvement in at least one. Identical points remain on the frontier together. The dashed steps mark the frontier, not interpolated model results. Search hides points without recalculating frontier membership.
Counting parameters
Counts are estimates derived from each linked, revision-pinned model configuration using Transformers 5.15.0. We construct the inference architecture on PyTorch’s meta device, tie shared weights and count unique parameters. This requires no model weights or inference. Total parameters cover that architecture; unused checkpoint tensors, such as training-only prediction layers, are not included. The counting script reproduces and checks all recorded estimates.
For active parameters we exclude embedding lookup tables, including positional and per-layer embeddings, and audio-only modules unused for image transcription. Vision encoders and image projectors are included. This follows MTEB’s non-embedding convention, extended to the image transcription path. A shared embedding/output weight is excluded under this convention even though it also participates in output projection. The estimate is not a measured FLOP count, speed or RAM requirement, and can differ from the rounded size in a model’s name.
Only models with verified estimates and completed scores for all selected tasks are plotted. Models with unknown parameter counts stay in the leaderboard table. Each source configuration is pinned to the revision recorded in the evaluation results. These estimates were reviewed on 24 October 2026.
Model openness
Six icons show what each model makes available: weights, license, training code, training data, a technical paper and model documentation. Confirmed items are highlighted; hover over an icon for its status or select it to read the evidence. Partial, closed and unknown items are labelled separately. The criteria are not combined into a score and do not affect OCR rankings. Read the definitions and model-by-model evidence →
Model settings
Results cover OCR engines, specialised recognition models and vision-language models. Each uses its adapter’s preprocessing, prompt and decoding settings. Image resolution and output-token limits can differ between models; these are comparisons of the recorded configurations.
The local sweeps used an NVIDIA RTX 3090 with 24 GB of VRAM and 32 GB of system RAM. Memory and time limits kept inference bounded. Exact settings, model revisions, batch sizes and runtime limits are retained per evaluation in the results download.
Hosted evaluations use the Mistral OCR 4.1 API and GPT-6.1 Sol through the standard OpenAI API with low reasoning. Both receive the original image dimensions. Mistral returns raw Markdown; GPT-6.1 Sol uses transcription prompts, with output limits of 8,192 tokens for cropped lines and 16,384 for pages, including reasoning tokens.
Dataset descriptions explain relevant reference and training context, including the NorHand specialist model, line cropping, and references extracted from PDFs. No uncertainty intervals or statistical-significance claims are reported for these fixed samples.
Reproducibility
The published scores combine 5 runs, with fixed samples shared within each task. Each exported result was checked against its report and sample manifest before publication.
- Evaluation code ↗Dataset loaders, model adapters and scoring implementation.
- Full evaluation reports ↗Full predictions, references, scores and settings for all evaluated models.
- Results JSON ↓Scores, sample counts, source revisions, manifest hashes, inference settings and run provenance.
- Dataset sources →Original collections, transcriptions and the subsets used here.
The comparison view includes ten real examples per task and dataset: 120 images and 2270 saved predictions. Cases are selected by a fixed hash of their identifiers, independently of model scores. These examples illustrate model behaviour; the leaderboard still scores the complete frozen samples.
The results snapshot contains aggregate scores and provenance. The separate example gallery republishes selected benchmark images, references and predictions with source attribution. Highlighting compares words after Unicode and whitespace normalization; switching it off shows the saved prediction text. Per-example CER and WER are the original report metrics. Adapter changes after a run do not change its saved scores or recorded source hashes.