Reliable Idiographic Parameters From Noisy Behavioral Data: The Case of Individual Differences in a Reinforcement Learning Task
- Yinan Xu, Psychology, University of Washington, Seattle , Washington, United States
- Andrea Stocco, Psychology, University of Washington, Seattle, Washington, United States
AbstractBehavioral data, though has been an influential index on cognitive processes, is under scrutiny for having poor reliability as a result of noise or lacking replications of reliable effects. Here, we argue that cognitive modeling can be used to enhance the test-retest reliability of the behavioral measures by recovering individual-level parameters from behavioral data. We tested this empirically with the Probabilistic Stimulus Selection (PSS) task, which is used to measure a participant’s sensitivity to positive or negative reinforcement. An analysis of 400,000 simulations from an Adaptive Control of Thought - Rational (ACT-R) model of this task showed that the poor reliability of the task is due to the instability of the end-estimates: because of the way the task works, the same participants might sometimes end up having apparently opposite scores. To recover the underlying interpretable parameters and enhance reliability, we used a Bayesian Maximum A Posteriori (MAP) procedure. We were able to obtain reliable parameters across sessions (Intraclass Correlation Coefficient ~ 0.5), and showed that this approach can further be used to provide superior measures in terms of reliability, and bring greater insights into individual differences.