Black-box models, such as neural networks, are renowned for their accuracy but often criticized for their lack of transparency. These models operate by assimilating sample data to identify patterns and anomalies, subsequently formulating rules for detecting and making decisions based on these patterns. In practice, they process data related to specific scenarios, utilizing the established rules to determine answers to relevant questions.
The new formula, known as SHAP (SHapley Additive exPlanations), is a game-changer in the realm of black-box models. It works by training a much simpler and interpretable model on top of the originally trained model, so that the simpler model approximates the original predictions. This allows for the predictions to be explained by the simpler model, making it easier to understand how the black-box model arrived at its decision.