Case study ·
BPCE builds its AI benchmark: "A chosen cost versus costs imposed"
Original title, in French: “BPCE construit son benchmark IA : « Un coût choisi contre des coûts subis »”
- Publication
- Silicon.fr (France)
- Author
- Clément Bohic
- Published on
- 17 September 2026 · read in the page structured data · updated on 21 September 2026
Summary
At the Big Data & AI Paris 2026 trade show, BPCE presented its internal AI evaluation benchmark, developed by Mikaël Le Bars's team. The approach addresses the limitations of public benchmarks (data errors, saturation, overfitting, anglocentrism, opaque governance). The team identified the skills and tasks to be evaluated based on the group's projects, built internal datasets and added metrics specific to banking. A use case on extracting financial information shows that the pipeline configuration (OCR, embedding, chunking) matters more than the choice of model. According to Mikaël Le Bars, the approach carries a significant human cost but amounts to "a chosen cost versus costs imposed", with 70% of the code reusable. BPCE is considering extending the benchmark to agentic systems and eventually making it open source.
Summary based on the full text of the article.
Read the article on Silicon.fr →Figures cited
-
42 %
Errors in the problems of FrontierMath v1
FrontierMath benchmark, acknowledged by Epoch AI in June
« Lorsqu’il en a publié la v2 en juin, Epoch AI a reconnu que 42 % des problèmes de la v1 contenaient des erreurs. »
Source: Epoch AI
-
50 %
Share of the most popular public benchmarks nearing saturation
60 most popular benchmarks, according to the Stanford AI Index
« Dans son dernier AI Index, Stanford estime que le phénomène touche près de la moitié des 60 benchmarks les plus populaires. »
Source: Stanford AI Index
-
25 %
MMLU questions requiring Western thinking
MMLU benchmark
« Plus d'un quart des questions exigent une « pensée occidentale » et 85 % des références géographiques se focalisent sur l'Amérique du Nord ou l'Europe. »
-
85 %
MMLU geographical references centred on North America or Europe
MMLU benchmark
« Plus d'un quart des questions exigent une « pensée occidentale » et 85 % des références géographiques se focalisent sur l'Amérique du Nord ou l'Europe. »
-
— percentage points
Impact of the model on answer accuracy in financial document extraction
BPCE use case on extracting information from financial documents
« Celui-ci n'a qu'un impact de 1 à 4 points de pourcentage. »
-
— percentage points
Impact of embedding on answer accuracy
BPCE use case on extracting information from financial documents
« Contre 12 à 17 points pour l'embedding, 10 à 20 pour la phase de chunking / récupération et 14 à 21 pour l'OCR. »
-
70 %
Share of the benchmark code that is reusable
BPCE internal benchmark
« Et 70 % du code est réutilisable. »
Classification
- Country
- France
- Company size
- Large companies
- Industries
- Banking, insurance, asset management, finance departments
- Companies
- BPCE (deploys AI), Stanford (cited), Epoch AI (cited), OpenAI (cited), Apple (cited)
- Keywords
- ai benchmark, model evaluation, banking, audio transcription, document extraction, agentic systems
Entry written by Marsanne Studio from the cited article, with the help of an AI model (claude-sonnet-5). The rights to the article and its illustrations belong to Silicon.fr and its author. Entry translated from French by Claude Sonnet 5.5.