training-notes

Here is a checklist of milestones for developing and testing model architectures: if we don't do well in one step, probably a good idea to go back one. This more or less pertains to the model part of the [[ml-ops]] lifecycle.

1. Synthetic data flows through the model (i.e. you receive an output without errors).
2. The output looks qualitatively correct; arrays in the expected shape, and there are no NaNs.
3. Real training data flows through the model (e.g. PyTorch Lightning does a batch of validation data).
4. Successfully iterate through a single training epoch.
5. The training loss goes down over several training epochs.
6. The training metric does not behave unexpectedly; for example NaN or negative for minimization.
7. Model is able to overfit a (small) training set.
8. Does well on the full training set.
9. Does well on dev/test sets.
10. Does well on application metrics and/or project goals.¹

Error analysis

One thing I haven't really done in the past, which might be helpful is to take smaller subsets of data and annotate tags of my own that might be helpful in diagnosing why a model performs poorly in any of checklist items.

Degraded model performance

Possible reasons why models perform worse in production:

• Data drift; statistics change over different time frames that could be seasonal or unpredictable (e.g. pandemics). At a low level, feature importances could change over time.
• Bad data collection, e.g. damaged sensors on cameras, etc.