ml-auditing

The auditing process is to formally evaluate the performance of a new pipeline beyond training/testing metrics. Basically ask where can the model go wrong? Examples include:

Data and model bias
Frequency of specific types of errors (e.g. misclassified groups)
Performance on rare events

Part of the auditing process involves designing adequate metrics that probe these aspects of the model as part of the [[ml-ops]] lifecycle. Discussions with the customer/collaborator will also help identify possible reasons.

Backlinks

ml-ops

Part of the challenge in MLOps is to deliver performance beyond good training and testing metrics. A few reasons for when "*good isn't good enough* are like unfair algorithms (e.g. discrimination), uninterpretable models, and more primitively when the application data comes from a different distribution or domain. This part is considered in [[ml-auditing]].