Setup open source dashboard for machine learning model monitoring

Introduction

Machine learning models have become an integral part of many organizations. However, just creating a machine learning model is not enough, it is important to monitor the model's performance continuously.

Typically a ML monitoring dashboard is build

1. to monitor the accuracy metric like F1 score, data drift and concept drift of the model.
2. to alert when ML model decay over time and need retraining

The first hunch of most teams is to ask the frontend developers to build a new web page for dashboard in the app and develop the graphs and dashboard interface using Angular.

However, this have few drawbacks

1. Creates dependency with frontend developers for adding new plots and minor changes
2. Technical debt in creating a new set of front end code and maintaining it for decades.
3. Reinventing the wheel since dashboarding is a solved problem and need to be invented again

The better solution from a software architechture point of view is to use opens source database visualziation/dashboarding tools instead so that

1. Data scientists and product owners can create new/change plots independently and quickly
2. Lower technical debit and maintenance cost

Graphana fits this usecase very well. It have intergration with various databases to read data and create custom plots. So once the model serving and ground truth metadata to a database and Graphana can create custom model monitoring plots.

Other opens source solutions like EvidentlyAI, TorchDrift, WhyLogs, etc did not fit the usecase because they are rigid on what metric it can plot and not flexible enough to support custom use cases like NLP.

My Github Repo database-visualizer-tool setups Graphana and Postgres database locally and demonstrates how to integrate them for easy Low/No Code dashboarding for machine learning monitoring and alerting.

Note: You can embedd graphana plots as inline frame in the browser https://grafana.com/docs/grafana/latest/dashboards/share-dashboards-panels/

Local setup

Clone the Github Repo database-visualizer-tool and follow the below instructions

1. Run docker-compose to setup the necessary services specified

docker compose up -d

2. Setup Postgres & Pgadmin (http://localhost:9001)

1. Test pgadmin connection to postgres

psql -d dev -h localhost -p 5436 -U test   #-d specify database name

2. Add postgres server to pgadmin
3. Create tables and alter tables using code in db_create.sql
4. To import csv directly to table. Refer https://belowthemalt.com/2021/06/10/import-data-from-local-to-postgresql-and-pgadmin4-running-on-docker/

3. Setup Graphana (http://localhost:3010/)

1. Add data source to Graphana Use the below values to configure

Host : postgres:5432 # use name of service from docker-compose file
User: superset
Password : superset # database password from docker-compose file
SSL Mode: disable 

2. Create dashboard
3. Add panel
4. Specify the sql query to plot the chart

SELECT
  $__time(datestamp), # specify time field
  connected_users
FROM
  connected_users
WHERE
  $__timeFilter(datestamp) # specify time field

Note: The neat thing is you can specify the date columns in the SQL and Graphana will automatically use it to filter time in the dashaboard.

4. Graph added to graphana