Machine Learning System Monitoring

 Monitoring Machine learning system is a cumbersome process that involves quite a lot of skills other than constant business feedback.

Broadly there are 3 kinds of monitoring that one need to focus on:

1) Data : Data Monitoring includes drift monitoring, monitoring data variance and feature monitoring 

2) Model : Monitoring of model accuracy over a period of time to make decisions for retraining and A/B testing, It  also include model health check

3) System : Monitoring system health parameters such as average response time, Capacity utilization, Number of service requests, down time etc.

Machine Learning Model Governance Process

Like Data Governance, predictive model also has its own governance process. There are multiple teams like but not limited to Core team, extended team, decision making/Steering committee & implementation team. This Governance process typically requires following steps.

1) Inputs : The generation of a request for a new or updated version of model

2) Model Need, Design and Direction:  Technical process to validate the requirement, scope and high level implementation

3) Model Build: Creates the model and develops implementation requirements (along with legal and regulatory considerations)

4) Model Approval: Multistep approval process (technical, business, risk, legal) to affirm and ascertain the model

5) Model Implementation: Data integrity, end to end testing and detailed implementation

6) Monitoring: This process is done for post implementation monitoring and understanding the data drift.

In addition to this, there is a  model review process at a regular frequency to decision on refreshing the model.