Zegami

ALT From goodness-of-fit to goodness-of-data The current AI revolution is metrics-driven, as Thomas points out ([120]), but practitioners largely used system metrics to measure the goodness of the fit of the model to the data. Goodness-of-fit metrics, such as F1, Accuracy, AUC, do not tell us much about the phenomenological fidelity (representation of the phenomena) and validity (how well the data explains things related to the phenomena captured by the data) aspects of the data. Currently, there are no standardised metrics for characterising the goodness-of-data [11, 13]; research on metrics is emerging [15, 91] but not yet widely adopted in AI system building. As a result, there is an extreme reliance on goodness-of-fit metrics and post-deployment product metrics. First, these metrics give us no assurances about the quality of the data. Second, they are too late to detect and course-correct from the unforeseen effects of data cascades. Even more importantly, deployment of AI systems in h