DeveloperWeek 2025

Wojtek Kuberski, NannyML, Co-Founder and CTO
The talk will cover the core ML Monitoring Flow necessary to maintain and maximize the business impact of models deployed to production. We will focus on the three main steps of the Flow: Performance monitoring, Root Cause Analysis, and Issue resolution. In the performance monitoring part, we will cover the two core algorithms that allow us to estimate the predictive performance without ground truth: Confidence-Based Performance Estimation (CBPE) and Direct Loss Estimation (DLE). The Root Cause Analysis part will go over various Drift Detection algorithms, focusing especially on multivariate drift detection and linking the drop in performance to drift signals. In the issue resolution part, we will briefly cover the typical steps to fix ML failure and their applicability and limitations.