We check AI against the source.
Independent measurement of language models, speech recognition and data for Norwegian public administration. Methods, data and corrections are public.
Latest from the workbench
- How much gold is speech? Kroner per correctly recognised word in Norwegian dialects
- DPR-PREREG-0001: How much gold is speech? Kroner per correctly recognised word in Norwegian dialects
- Smritir v1.0: large language models on Old Norse text restoration at diplomatic and normalised levels (Menota)
- Smritir: preregistration for measuring large language models on Old Norse text restoration at diplomatic and normalised levels (Menota)
- nortrebenk: coding agents on public code from Norwegian government agencies
- Preregistration: nortrebenk v1.0 pilot measurement (P3)
Same power, different emissions.
In 2025, electricity in NO4 had almost 70% higher carbon intensity than in NO1.
Selected work
-
Rates and deadlines in Norwegian law
A benchmark of dated numeric values in Norwegian statutes, regulations and parliamentary decisions, with a closed answer key. We class each answer as correct, outdated, wrong or uninterpretable. The plan for the second run was published before its first call.
-
AI judges as error finders
NOS-FD is a reference-based pilot set that tests whether AI judges find the errors in model answers about Norwegian law, health entitlements and public administration.
-
Production types at ENTSO-E
Two preregistered datasets on how production types are labeled on the ENTSO-E Transparency Platform. They cover multi-fuel reporting, register vintages, persistent misclassification and consistency between the products.
-
Source support in municipal AI search
Kildefeste is a protocol that measures what share of the answers from municipal AI search is supported by the source they cite. We fixed the protocol before data collection.
-
Carbon intensity per bidding zone
Open and reproducible carbon intensity for NO1 to NO5 and SE1 to SE4 in 2025, with data and code. The preprint describes the production-based method, and the v2.0.1 dataset adds consumption-based intensity hour by hour.
-
Rule-based fee calculation: 0 of 840
Three free local language models got 0 of 840 fee calculations right by correct rule use in RuleArena (airline). The single exact hit came from a numeric collision.
-
Evacuation distances for hazardous materials
Free language models state evacuation distances that are too short in text about hazardous materials, compared with the ERG handbook. We locked the hypothesis and the metric before the run.
All 36 works, with preregistrations and corrections, are listed on ORCID.
Contributions to other tools
-
EuroEval
Two contributions to EuroEval (formerly ScandEval), a benchmark for language models in European languages.
-
Emission factors in CodeCarbon
Factors for biomass, biofuel and waste, with a source for each value.
-
Provenance in SimpleAudit
The response from each target records which endpoint answered, and whether the proxy settings route the traffic through a proxy.
-
Descriptions in the ldproxy configuration
Descriptions from the model are carried over to object types, properties and code lists in the generated ldproxy configuration.
Services
- Test before you buy
- We measure speech recognition and AI tools on your own data before the contract is signed. Audio and documents never leave the building.
- Documentation for tenders
- For suppliers: an independent quality report in Norwegian to attach to your bid.
- Norwegian test sets
- Built from the source, with an answer key and a plan locked before measurement.
- Verification
- We review the numbers, method and code behind a system or a report.
Open method
Report with source list
Method
- We lock and publish the analysis plan before the run.
- Code and data get a DOI when we publish them.
- We publish negative results on the same footing as other results.
- We correct errors openly, in separate corrections.
Contact
Other projects
Walk-to-Zoom: an art experience in AR with gigapixel images, where distance controls the zoom.
Stifinner: a map for route planning on foot and by bike, with steepness, barriers and protected areas.