Do Models Capture Individuals? Evaluating Parameterized Models for Syllogistic Reasoning
- Nicolas Riesterer, Cognitive Computation Lab, University of Freiburg, Freiburg, Germany
- Daniel Brand, Cognitive Computation Lab, University of Freiburg, Freiburg, Germany
- Marco Ragni, Cognitive Computation Lab, University of Freiburg, Freiburg, Germany
AbstractThe prevailing focus on aggregated data and the lacking group-to-individual generalizability it entails have recently been identified as a major cause for the low performance of cognitive models in the field of syllogistic reasoning research. This article attempts to add to the discussion about the performance of current syllogistic reasoning models by considering the parameterization capabilities some cognitive models offer. To this end, we propose a model evaluation setting targeted specifically toward analyzing the capabilities of a model to fine-tune its inferential mechanisms to individual human reasoning data. This allows us to (1) quantify the degree to which models are able to capture individual human reasoning behavior, (2) analyze the efficiency of the parameters used by models, and (3) examine the functional differences between the prediction capabilities of competing models on a more detailed level. We apply this method to two state-of-the-art models for syllogistic reasoning, mReasoner and the Probability Heuristics Model, analyze the obtained results and discuss their implication with respect to the general field of cognitive modeling.