Product Data Scientist (AI Evaluation & Quality)

• You’ll join the AI Team — the group driving all AI products and technology at Finom
• We build and ship AI across the company: AI financial co-pilot, voice agent, and internal AI-powered processes
• Our belief: your AI agent is only as good as your eval loop — we can build AI as good as the evals we run on it
• Your mission: own that eval loop across every AI product we ship — pre-launch quality gates, post-launch monitoring, continuous improvement
• You’ll work directly with our AI Quality lead, Igor Kolodkin
• Close collaboration with AI engineers, Product, and domain experts across the company
• Core stack: Databricks, DeepEval, Claude Code
• Own and extend our offline eval suite across products — datasets (capability + regression), judges, metrics
• Build and maintain online quality dashboards: resolution rate, CSAT, thumbs up/down, LLM-as-judge signals, error rate, latency
• Close the production feedback loop: mine failure patterns from real traffic → turn them into regression cases → propose fixes to Product and domain experts
• Harden methodology: judge stability, non-determinism handling
• Translate numbers into decisions – weekly syncs, clear trade-offs, no dashboards for their own sake- Analytical mindset — you start from the business question, not from the tool
• Solid foundation in statistics — sampling, hypothesis testing, variance, understanding what a noisy metric is
• Python and SQL — you can build an analysis end-to-end
• 3+ years in analyst / data scientist roles, at least one in a product context
• Experience in quality analytics for ML systems — ranking, recommendations, classification, etc
• Experience building LLM agents — side projects, toy builds, personal experiments all count
• Hands-on experience evaluating LLM applications (RAG, agents, tool use, judges)

Izvor: https://www.welcometothejungle.com/en/companies/finom/jobs/product-data-scientist-ai-evaluation-quality_berlin_7ylnpiej

Nivo: mid level