Yannick Morvan @[email protected]

NEW PAPER ALERT: "A Latent Oscillator Measurement Model To Simulate Emotional-Expression Score Dynamics In Video". [Or why you should not use GIMME or GraphicalVAR]. In our new preprint, Aleksandar Tomašević, Alexander P. Christensen, and I introduce the Latent Oscillator Measurement Model (LOMM): a way to test whether an analysis can recover the structure we put into simulated data before we interpret patterns in real emotion-expression scores. AI now can give you 11 emotion scores (or more) for every frame of a video. However, that does not mean it has measured 11 psychological processes. This is, interestingly, a measurement problem inside an AI framework. An AI zero-shot classifier can generate scores for classes like “fear,” “anxiety,” “sadness,” and so on. These classes can overlap. Their relationships can reflect visible expression, measurement error, recording conditions, and the scoring pipeline itself. The number of labels is therefore not the number of underlying processes. And scores assigned to visible expressions do not, by themselves, establish what someone feels. So how do we evaluate an analysis when we do not know the underlying structure of the real data? We generated data for which that structure is known, in a Monte-Carlo Simulation Study, and then made recovery difficult in controlled ways. LOMM separates three elements: • Dynamics: how a latent process evolves, with fluctuations that can diminish, persist, or amplify. • Activity: how strongly that process contributes to the observed expression at each moment. • Measurement: how the expressed signal becomes multiple observed indicators, each with measurement error. Think of activity as a volume control on a changing signal. A process can continue evolving while its contribution to the observed indicators becomes lowers. Turning that volume up or down can produce an abrupt observed change even when the underlying trajectory remains continuous. A dimension can exist in the generating model yet be difficult to detect during a short recording. In the paper we asked two separate questions. First: can the simulation reproduce important statistical properties of real classifier zero-shot-score series? We used 100 MAFW videos scored with our R package transforEmotion and OpenAI's CLIP model. In each of four folds, 75 videos calibrated the generator and 25 evaluated it. We compared LOMM with a calibrated static logistic-normal generator that retained score structure and between-video heterogeneity but drew observations independently over time. Across 22 statistical features, median results were: •Plausibility: 97% for LOMM versus 51% for the static generator. • Coverage: 92% versus 37%. Plausibility asks whether simulated series have a sufficiently close real counterpart. Coverage asks whether real evaluation series have a sufficiently close simulated counterpart. "Close" is defined using a threshold calibrated from real-to-real comparisons. These are measures of statistical resemblance, not emotion-recognition accuracy. LOMM reproduced many features of variation, short-term change, and persistence. Differences remained, especially in entropy measures. This is evidence for a useful simulator within the tested setting, with specific properties still to improve. Second: when the dimensions are known, can our methods recover them? In the main comparison, four methods analyzed the same 21,600 simulated datasets, each with 100 observations per clip. Here we used continuous indicators, with known dimensions and known indicator assignments. The percentage recovering the correct number of dimensions was: • DynEGA: 93.9% • Static EGA: 88.4% • GraphicalVAR: 77.7% • GIMME: 17.6% Failures and timeouts counted as unsuccessful recovery. That matters particularly for GIMME: among its valid outputs, recovery was 78.3%, but only 22.4% of attempts returned valid outputs. We also adapted GIMME to a shared-grouping task that differs from its usual use. These results evaluate a particular recovery target and implementation. They do not rank the methods for every scientific question. And the shorter-record result makes that point concrete. With 25 observations per clip, static EGA led at 91.3%, followed by DynEGA at 86.3%. A method that uses temporal information does not automatically recover this grouping better under every observation schedule. Sensitivity analyses: Changing the starting states reduced recovery. The main design used common fixed initialization; replacing it with independent stationary starts for stable dimensions and bounded independent starts for amplifying dimensions made recovery harder. The original initialization was optimistic relative to that alternative. Then came an especially relevant result for AI scores: transforming continuous indicators into positive scores that sum to one sharply reduced recovery of their original grouping. Why is that consequential? When scores share a fixed total, increasing one channel’s share necessarily reduces the combined share of the others. The transformation can change the relationships that a dimensionality method uses to identify groups. Analyzing centered log-ratios did not preserve the original grouping in these tests either. Success with continuous indicators cannot simply be carried over to classifier scores constrained to sum to one. That is why the two studies have distinct claims: one evaluates resemblance to real score series; the other evaluates recovery from specified continuous-indicator models. For me, the broader contribution is the ability to ask sharper questions: Is a dimension missing, or weakly expressed? Did the process change, or did its visibility change? Did the analysis recover the generating structure, or did the observation transformation alter what was recoverable? LOMM gives us a controlled setting for investigating those possibilities. It is currently a generator with specified parameters; fitting those parameters to observed data is a next step. Before giving a psychological interpretation to a pattern, test whether your analysis can recover a known structure under the conditions that produced your measurements. Read the preprint in the link below: osf.io/preprints/psyarxiv/hf…

The AIGENIE package is now live on CRAN: cran.r-project.org/web/packa… AIGENIE: Automatic Item Generation and Validation via Network-Integrated Evaluation Automated psychological scale development and structural validation using large language models (LLMs) and network psychometric methods. Implements the AI-GENIE framework (Automatic Item Generation and Validation via Network-Integrated Evaluation) to generate candidate items, compute embedding representations, and estimate dimensional structure using Exploratory Graph Analysis (EGA). Item quality is evaluated using Unique Variable Analysis to identify redundant items and Bootstrap EGA to assess item and dimension stability. Supports both fully automated item generation and analysis of user-provided item sets, facilitating efficient, theory-informed measurement development prior to empirical data collection.

AI-GENIE Version 2.1.1 is now available on Github (some bugs were fixed). Install the package again! github.com/laralee/AIGENIE

For about ten years now, one idea has kept showing up in our Exploratory Graph Analysis work: changing the representation of a problem can be a game changer. Our new paper is probably the cleanest example of that idea. Traditional multidimensional IRT estimates each item's discrimination vector all at once (i.e., direction and magnitude together), and that's part of why these models are computationally expensive. To create an EGA-native multidimensional item parameter estimation methodology, we split the problem in two. The first half is a geometry problem that EGA already addresses: the communities, the item directions (network loadings), the relationships among dimensions. For the second half, I developed the DMD calibration method, which estimates ONLY ONE number per item: how far along its fixed direction the item reaches. The EGA dimensionality structure does not change, no rotation is necessary. No factor model, no iterative MIRT estimation. Only vector length changes. What I discovered is that every classical item parameter (discrimination, difficulty, intercept) follows analytically from that one scalar. The figures below show the process visually: from the EGA geometry, to a network loading vector, to a length change. In one dimension a threshold is a point on the dimension distribution. In two, the same distribution becomes a surface and the threshold becomes a line cutting through it. In K dimensions it's a hyperplane. The criterion [P(endorse) = .50] does not change, but the representation does. EGA identifies the geometry and DMD equips that geometry with a metric. Paper and code coming soon. #Measurement #EGA #Psychometrics