|
|
- A Two-Stage Regression Approach to Detecting Section Score Inconsistency
-
Free pre-print version: Loading...
Rate this result:
What is this?
Please help us test our new pre-print finding feature by giving the pre-print link a rating.
A 5 star rating indicates the linked pre-print has the exact same content as the published article.
Authors: Yi-Hsuan Lee, Charles Lewis
Abstract: Journal of Educational and Behavioral Statistics, Ahead of Print.
For an assessment with multiple sections measuring related constructs, test takers with higher scores on one section are expected to perform better on the related sections. When the sections involve different test designs, test takers with preknowledge of an administration may score unusually high on some sections but not on others. To address such inconsistency, regression approaches have been successfully applied to compare section scores for many years in operational settings. With a focus on outlier analysis, we propose a new two-stage regression approach to detecting score inconsistency among different sections of a test. It is designed to leverage rich historical information from large-scale assessments to help detect unusually high scores on the easier-to-cheat sections based on the scores on the harder-to-cheat sections in new administrations. This paper presents a statistical framework for the two-stage regression procedure and develops analytical results under a null model of no exposure. It also describes an analysis procedure to guide applications. An empirical example is provided to illustrate the proposed method, to evaluate the performance and robustness of the analytical results in real settings, and to compare with two other methods for the detection of inconsistent section scores.
Citation: Journal of Educational and Behavioral Statistics
PubDate: 2024-08-08T05:02:08Z
DOI: 10.3102/10769986241263974
- Generally Applicable Variance Estimation Methods for Common-Population
Linking-
Free pre-print version: Loading...
Rate this result:
What is this?
Please help us test our new pre-print finding feature by giving the pre-print link a rating.
A 5 star rating indicates the linked pre-print has the exact same content as the published article.
Authors: Paul A. Jewsbury
Abstract: Journal of Educational and Behavioral Statistics, Ahead of Print.
Educational assessments require periodic administration changes, such as transitioning from paper to digital administration. During such transitions, a linking function relating the metrics of the new and previous administration is estimated, but uncertainty in this estimation introduces variance into comparisons between administrations. Similarly, different assessments provided to overlapping populations may be linked, introducing linking or equating error. We introduce new generally applicable variance estimation methods, generalize prior methods to be more widely applicable, and confirm the validity of the methods via simulation. Our methods account for dependencies between linking and other sources of error, complex sampling, and nonlinear linking functions, while applying to a wide range of score comparisons and statistics such as means, standard deviations, percentiles, and differences.
Citation: Journal of Educational and Behavioral Statistics
PubDate: 2024-08-08T01:01:41Z
DOI: 10.3102/10769986241263976
- Estimating the Reliability of Skill Transitions in Longitudinal Diagnostic
Classification Models-
Free pre-print version: Loading...
Rate this result:
What is this?
Please help us test our new pre-print finding feature by giving the pre-print link a rating.
A 5 star rating indicates the linked pre-print has the exact same content as the published article.
Authors: Madeline A. Schellman, Matthew J. Madison
Abstract: Journal of Educational and Behavioral Statistics, Ahead of Print.
Diagnostic classification models (DCMs) have grown in popularity as stakeholders increasingly desire actionable information related to students’ skill competencies. Longitudinal DCMs offer a psychometric framework for providing estimates of students’ proficiency status transitions over time. For both cross-sectional and longitudinal DCMs, it is important that researchers estimate and report reliability so stakeholders and end-users can evaluate the trustworthiness of results. Over the past decade, researchers have developed and applied various metrics for reliability in the DCM framework. This study extends these metrics onto the longitudinal DCM context and consists of three parts: (a) the theory and development of the new longitudinal DCM reliability metrics, (b) a simulation study to examine the performance of the developed metrics and establish thresholds, and (c) an empirical data analysis to illustrate an application of the developed metrics. This paper concludes with a discussion of our recommendations for applying the developed metrics.
Citation: Journal of Educational and Behavioral Statistics
PubDate: 2024-07-31T11:29:17Z
DOI: 10.3102/10769986241256032
- Measurement and Uncertainty Preserving Parametric Modeling for Continuous
Latent Variables With Discrete Indicators and External Variables-
Free pre-print version: Loading...
Rate this result:
What is this?
Please help us test our new pre-print finding feature by giving the pre-print link a rating.
A 5 star rating indicates the linked pre-print has the exact same content as the published article.
Authors: Roy Levy, Daniel McNeish
Abstract: Journal of Educational and Behavioral Statistics, Ahead of Print.
Research in education and behavioral sciences often involves the use of latent variable models that are related to indicators, as well as related to covariates or outcomes. Such models are subject to interpretational confounding, which occurs when fitting the model with covariates or outcomes alters the results for the measurement model. This has received attention in models for continuous observable variables but to date has not been examined in the context of discrete variables. This work demonstrates that interpretational confounding can occur in models for discrete variables, and develops a multistage Bayesian estimation approach to deal with this problem. The key features of this approach are that it is (a) measurement preserving, in that it precludes the possibility of interpretational confounding, and (b) uncertainty preserving, in that the uncertainty from the earlier stage of estimating the measurement model is propagated to the second stage of estimating the relations between the latent variable(s) and any covariates or outcomes. Previous work on these methods had only considered models for continuous observed variables, and software was limited to models with a single latent variable and either covariates or outcomes. This work extends the approach and software to a more general class of solutions, including discrete variables, illustrating the procedures with analyses of real data. Functions for conducting the analyses in widely available software are provided.
Citation: Journal of Educational and Behavioral Statistics
PubDate: 2024-07-31T11:27:57Z
DOI: 10.3102/10769986241254348
- A Family of Cognitive Diagnosis Models for Continuous Bounded Responses
-
Free pre-print version: Loading...
Rate this result:
What is this?
Please help us test our new pre-print finding feature by giving the pre-print link a rating.
A 5 star rating indicates the linked pre-print has the exact same content as the published article.
Authors: Youxiang Jiang, Qingrong Tan, Wei Wen, Daxun Wang, Yan Cai, Dongbo Tu
Abstract: Journal of Educational and Behavioral Statistics, Ahead of Print.
Continuous bounded responses in psychometrics usually come from the visual analog scale (VAS). The VAS is a rating scale measurement tool that requires respondents to report their agreement with items by tracing a mark somewhere on a fixed-length continuous horizontal segment with ends that are generally labeled “0% disagreement” to “100% agreement” (or other possible labeling) using continuous data. In recent years, the VAS has gradually appeared in medical, educational, and psychological research, such as research on pain, worry, rumination, anxiety, risk perception, and even personality trait measurement. However, there are very few cognitive diagnosis models (CDMs) in cognitive diagnostic assessment that can analyze such continuous bounded data from VAS-type scale. In this study, we propose a family of CDMs for the continuous bounded data in VAS-type scale and provide model selection methods for practice. Three simulation studies were used to examine parameter recovery, the impact of model misspecification on parameter recovery, and the effectiveness of the model selection method. Moreover, real data are used as an illustration to demonstrate the application and effectiveness of the proposed models.
Citation: Journal of Educational and Behavioral Statistics
PubDate: 2024-07-26T11:59:57Z
DOI: 10.3102/10769986241255970
- Exploiting Network Information to Disentangle Spillover Effects in a Field
Experiment on Teens’ Museum Attendance-
Free pre-print version: Loading...
Rate this result:
What is this?
Please help us test our new pre-print finding feature by giving the pre-print link a rating.
A 5 star rating indicates the linked pre-print has the exact same content as the published article.
Authors: Silvia Noirjean, Marco Mariani, Alessandra Mattei, Fabrizia Mealli
Abstract: Journal of Educational and Behavioral Statistics, Ahead of Print.
A key element in the education of youths is their sensitization to historical and artistic heritage. We analyze a field experiment conducted in Florence (Italy) to assess how appropriate incentives assigned to high-school classes may induce teens to visit museums in their free time. Noncompliance and spillover effects make the impact evaluation of this clustered encouragement design challenging. We propose to blend principal stratification and causal mediation by defining subpopulations of units according to their compliance behavior and using the information on their friendship networks as mediator. We formally define principal natural direct and indirect effects and principal controlled direct and spillover effects, and use them to disentangle spillovers from other causal channels. We adopt a Bayesian approach for inference.
Citation: Journal of Educational and Behavioral Statistics
PubDate: 2024-07-26T11:54:18Z
DOI: 10.3102/10769986241254351
- How Do We Demonstrate AI Responsibility: The Devil Is in the Details
-
Free pre-print version: Loading...
Rate this result:
What is this?
Please help us test our new pre-print finding feature by giving the pre-print link a rating.
A 5 star rating indicates the linked pre-print has the exact same content as the published article.
Authors: Matthew S. Johnson
Abstract: Journal of Educational and Behavioral Statistics, Ahead of Print.
This commentary examines the Duolingo English Test Responsible AI standards and provides some thoughts on specific ways we can evaluate the use of AI for automated scoring.
Citation: Journal of Educational and Behavioral Statistics
PubDate: 2024-07-24T06:18:23Z
DOI: 10.3102/10769986241257963
- Predictive Performance of Bayesian Stacking in Multilevel Education Data
-
Free pre-print version: Loading...
Rate this result:
What is this?
Please help us test our new pre-print finding feature by giving the pre-print link a rating.
A 5 star rating indicates the linked pre-print has the exact same content as the published article.
Authors: Mingya Huang, David Kaplan
Abstract: Journal of Educational and Behavioral Statistics, Ahead of Print.
The issue of model uncertainty has been gaining interest in education and the social sciences community over the years, and the dominant methods for handling model uncertainty are based on Bayesian inference, particularly, Bayesian model averaging. However, Bayesian model averaging assumes that the true data-generating model is within the candidate model space over which averaging is taking place. Unlike Bayesian model averaging, the method of Bayesian stacking can account for model uncertainty without assuming that a true model exists. An issue with Bayesian stacking, however, is that it is an optimization technique that uses predictor-independent model weights and is, therefore, not fully Bayesian. Bayesian hierarchical stacking, proposed by Yao et al. further incorporates uncertainty by applying a hyperprior to the stacking weights. Considering the importance of multilevel models commonly applied in educational settings, this paper investigates via a simulation study and a real data example the predictive performance of original Bayesian stacking and Bayesian hierarchical stacking along with two other readily available weighting methods, pseudo-BMA and pseudo-BMA bootstrap (PBMA and PBMA+). Predictive performance is measured by the Kullback–Leibler divergence score. Although the differences in predictive performance among these four weighting methods in Bayesian stacking are small, we still find that Bayesian hierarchical stacking performs as well as conventional stacking, PBMA, and PBMA+ in settings where a true model is not assumed to exist.
Citation: Journal of Educational and Behavioral Statistics
PubDate: 2024-06-18T11:36:04Z
DOI: 10.3102/10769986241255969
- The Rank-2PL IRT Models for Forced-Choice Questionnaires: Maximum Marginal
Likelihood Estimation with an EM Algorithm-
Free pre-print version: Loading...
Rate this result:
What is this?
Please help us test our new pre-print finding feature by giving the pre-print link a rating.
A 5 star rating indicates the linked pre-print has the exact same content as the published article.
Authors: Jianbin Fu, Xuan Tan, Patrick C. Kyllonen
Abstract: Journal of Educational and Behavioral Statistics, Ahead of Print.
The rank two-parameter logistic (Rank-2PL) item response theory models refer to a set of models applying the 2PL model in a sequential ranking process that occurs in forced-choice questionnaires. The multi-unidimensional pairwise preference with 2PL model (MUPP-2PL) is a Rank-2PL model for items with two statements. Focusing on items with three statements, we develop a maximum marginal likelihood estimation with an expectation-maximization algorithm to estimate item parameters and their standard errors. A simulation study is conducted to check parameter recovery, and then the model is applied to a real dataset. Finally, the findings are summarized and discussed, and future research is suggested.
Citation: Journal of Educational and Behavioral Statistics
PubDate: 2024-06-18T01:16:51Z
DOI: 10.3102/10769986241256030
- Extending the Cluster Approach to Differential Item Functioning in
Polytomous Items-
Free pre-print version: Loading...
Rate this result:
What is this?
Please help us test our new pre-print finding feature by giving the pre-print link a rating.
A 5 star rating indicates the linked pre-print has the exact same content as the published article.
Authors: Martijn Schoenmakers, Jesper Tijmstra, Jeroen Vermunt, Maria Bolsinova
Abstract: Journal of Educational and Behavioral Statistics, Ahead of Print.
To objectively compare groups on any latent trait using tests, the absence of differential item functioning (DIF) is crucial. While the importance of DIF has been well-established in research, the question of how to identify DIF-free items is still largely open. The fact that item difficulty is not identified from observations may explain this. Recently, DIF tests utilizing the differences between item difficulties across groups, which are identified, were proposed for the Rasch and 2-parameter logistic models. The current paper aims to extend these approaches to the polytomous case using the partial credit model. Performance of the new approach is assessed using a simulation study, and practical recommendations are made.
Citation: Journal of Educational and Behavioral Statistics
PubDate: 2024-06-17T12:33:35Z
DOI: 10.3102/10769986241256033
- Using Response Times in Answer Similarity Analysis
-
Free pre-print version: Loading...
Rate this result:
What is this?
Please help us test our new pre-print finding feature by giving the pre-print link a rating.
A 5 star rating indicates the linked pre-print has the exact same content as the published article.
Authors: Kylie Gorney, James A. Wollack
Abstract: Journal of Educational and Behavioral Statistics, Ahead of Print.
Recent decades have seen a tremendous growth in the development of collusion detection methods, many of which rest on the assumption that examinees who engage in collusion will display unusually similar scores/responses. In this article, we expand the definition of answer similarity to include not only the item scores/responses but also the item response times (RTs). Using detailed simulations and an experimental data set, we show that (a) both the new and existing similarity statistics are able to control the Type I error rate in most of the studied conditions and (b) the new statistics are much more powerful, on average, than the existing statistics at detecting several types of simulated collusion.
Citation: Journal of Educational and Behavioral Statistics
PubDate: 2024-05-30T09:17:25Z
DOI: 10.3102/10769986241248770
- Approaches to Statistical Efficiency When Comparing the Embedded Adaptive
Interventions in a SMART-
Free pre-print version: Loading...
Rate this result:
What is this?
Please help us test our new pre-print finding feature by giving the pre-print link a rating.
A 5 star rating indicates the linked pre-print has the exact same content as the published article.
Authors: Timothy Lycurgus, Amy Kilbourne, Daniel Almirall
Abstract: Journal of Educational and Behavioral Statistics, Ahead of Print.
Sequential, multiple assignment randomized trials (SMARTs), which assist in the optimization of adaptive interventions, are growing in popularity in education and behavioral sciences. This is unsurprising, as adaptive interventions reflect the sequential, tailored nature of learning in a classroom or school. Nonetheless, as is true elsewhere in education research, observed effect sizes in education-based SMARTs are frequently small. As a consequence, statistical efficiency is of paramount importance in their analysis. The contributions of this manuscript are twofold. First, we provide an overview of adaptive interventions and SMART designs for researchers in education science. Second, we propose four techniques that have the potential to improve statistical efficiency in the analysis of SMARTs. We demonstrate the benefits of these techniques in SMART settings both through the analysis of a SMART designed to optimize an adaptive intervention for increasing cognitive behavioral therapy delivery in school settings and through a comprehensive simulation study. Each of the proposed techniques is easily implementable, either with over-the-counter statistical software or through R code provided in Supplemental Material.
Citation: Journal of Educational and Behavioral Statistics
PubDate: 2024-05-28T07:09:54Z
DOI: 10.3102/10769986241251419
- Artificial Intelligence and Educational Measurement: Opportunities and
Threats-
Free pre-print version: Loading...
Rate this result:
What is this?
Please help us test our new pre-print finding feature by giving the pre-print link a rating.
A 5 star rating indicates the linked pre-print has the exact same content as the published article.
Authors: Andrew D. Ho
Abstract: Journal of Educational and Behavioral Statistics, Ahead of Print.
I review opportunities and threats that widely accessible Artificial Intelligence (AI)-powered services present for educational statistics and measurement. Algorithmic and computational advances continue to improve approaches to item generation, scale maintenance, test security, test scoring, and score reporting. Predictable misuses of AI for these purposes will result in biased scores, construct underrepresentation, and differential impact over time. Recent efforts to develop standards for AI use in testing like those of Burstein are promising. I argue that similar efforts to develop AI standards for educational measurement will benefit from increased attention to the context of test use and explicit commitment to ongoing monitoring of bias and scale drift over time.
Citation: Journal of Educational and Behavioral Statistics
PubDate: 2024-05-09T12:06:17Z
DOI: 10.3102/10769986241248771
- Using Permutation Tests to Identify Statistically Sound and Nonredundant
Sequential Patterns in Educational Event Sequences-
Free pre-print version: Loading...
Rate this result:
What is this?
Please help us test our new pre-print finding feature by giving the pre-print link a rating.
A 5 star rating indicates the linked pre-print has the exact same content as the published article.
Authors: Yingbin Zhang, Luc Paquette, Nigel Bosch
Abstract: Journal of Educational and Behavioral Statistics, Ahead of Print.
Frequent sequential pattern mining is a valuable technique for capturing the relative arrangement of learning events, but current algorithms often return excessive learning event patterns, many of which may be noise or redundant. These issues exacerbate researchers’ burden when interpreting the patterns to derive actionable insights into learning processes. This study proposed permutation tests for identifying sequential patterns whose occurrences are statistically significantly greater than the chance value and different from their superpatterns. Simulations demonstrated that the test for detecting sound patterns had a low false discovery rate and high power, while the test for detecting nonredundant patterns also showed a high accuracy. Empirical data analyses found that the patterns detected in training data were generalizable to test data.
Citation: Journal of Educational and Behavioral Statistics
PubDate: 2024-05-09T12:06:17Z
DOI: 10.3102/10769986241248772
- Modeling Partial Knowledge in Multiple-Choice Cognitive Diagnostic
Assessment-
Free pre-print version: Loading...
Rate this result:
What is this?
Please help us test our new pre-print finding feature by giving the pre-print link a rating.
A 5 star rating indicates the linked pre-print has the exact same content as the published article.
Authors: Kentaro Fukushima, Nao Uchida, Kensuke Okada
Abstract: Journal of Educational and Behavioral Statistics, Ahead of Print.
Diagnostic tests are typically administered in a multiple-choice (MC) format due to their advantages of objectivity and time efficiency. The MC-deterministic input, noisy “and” gate (DINA) family of models, a representative class of cognitive diagnostic models for MC items, efficiently and parsimoniously estimates the mastery profiles of examinees. However, the existing models often overestimate the latent traits of examinees when they respond with partial knowledge, which is often observed in educational assessment. Therefore, the novel models of the MC-DINA family that can appropriately handle such responses were developed in this study. Unlike the existing models, the proposed models placed no restrictions on the Q-vector, which represents attribute specifications. Simulation and empirical studies verified that the proposed approach could resolve the overestimation problem.
Citation: Journal of Educational and Behavioral Statistics
PubDate: 2024-05-06T09:32:26Z
DOI: 10.3102/10769986241245707
- Using Regularized Methods to Validate Q-Matrix in Cognitive Diagnostic
Assessment-
Free pre-print version: Loading...
Rate this result:
What is this?
Please help us test our new pre-print finding feature by giving the pre-print link a rating.
A 5 star rating indicates the linked pre-print has the exact same content as the published article.
Authors: Daoxuan Fu, Chunying Qin, Zhaosheng Luo, Yujun Li, Xiaofeng Yu, Ziyu Ye
Abstract: Journal of Educational and Behavioral Statistics, Ahead of Print.
One of the central components of cognitive diagnostic assessment is the Q-matrix, which is an essential loading indicator matrix and is typically constructed by subject matter experts. Nonetheless, to a large extent, the construction of Q-matrix remains a subjective process and might lead to misspecifications. Many researchers have recognized the importance of estimating or validating the Q-matrix, but most of them focus on the conditions of relatively large sample sizes. This article aims to explore Q-matrix validation possibilities under small sample conditions and uses regularized methods to validate the Q-matrix based on the compensatory reparametrized unified model and generalized deterministic inputs, noisy “and” gate models. Simulation studies were conducted to evaluate the viability of the modified least absolute shrinkage and selection operator (Lasso) and modified smoothly clipped absolute deviation (SCAD) methods, comparing them with existing methods. Results show that the modified Lasso and the modified SCAD methods outperform the stepwise, Hull, and MLR-B methods in general, especially under the conditions of small sample sizes. While good recovery in all small sample size conditions is not guaranteed, the modified methods demonstrate advantages across various item quality conditions. Also, a real data set is analyzed to illustrate the application of the modified methods.
Citation: Journal of Educational and Behavioral Statistics
PubDate: 2024-04-12T07:34:31Z
DOI: 10.3102/10769986241240084
- A Novel Numerical Method for Solving Unknown Statistical Quantities in
Multivariate Regression Models-
Free pre-print version: Loading...
Rate this result:
What is this?
Please help us test our new pre-print finding feature by giving the pre-print link a rating.
A 5 star rating indicates the linked pre-print has the exact same content as the published article.
Authors: William R. Dardick, Jeffrey R. Harring
Abstract: Journal of Educational and Behavioral Statistics, Ahead of Print.
Simulation studies are the basic tools of quantitative methodologists used to obtain empirical solutions to statistical problems that may be impossible to derive through direct mathematical computations. The successful execution of many simulation studies relies on the accurate generation of correlated multivariate data that adhere to a particular model with known parameter values. In this article, we use a kernel inspired by path tracing rules to algebraically solve unknown causal effects in the context of a multivariate general linear model. The algebraic solution is the basis of the mathematical extension, which integrates a model solver. Examples are used to illustrate a range of applications, where information regarding parameter values and predictor correlations can be partially specified. Code for examples is provided.
Citation: Journal of Educational and Behavioral Statistics
PubDate: 2024-04-08T04:36:15Z
DOI: 10.3102/10769986241240083
- Using Extant Data to Improve Estimation of the Standardized Mean
Difference-
Free pre-print version: Loading...
Rate this result:
What is this?
Please help us test our new pre-print finding feature by giving the pre-print link a rating.
A 5 star rating indicates the linked pre-print has the exact same content as the published article.
Authors: Kaitlyn G. Fitzgerald, Elizabeth Tipton
Abstract: Journal of Educational and Behavioral Statistics, Ahead of Print.
This article presents methods for using extant data to improve the properties of estimators of the standardized mean difference (SMD) effect size. Because samples recruited into education research studies are often more homogeneous than the populations of policy interest, the variation in educational outcomes can be smaller in these samples than is reflective of the true variation in the population. This affects effect size estimation since the sample standard deviation is used in the denominator of the SMD. We propose leveraging extant data on sample variance estimates from multiple studies, made available via clearinghouse databases such as the What Works Clearinghouse, to standardize a mean difference. This allows effect sizes to be benchmarked across a common and broad population, thus enabling better comparability across studies and interventions. We derive the new estimators of the population variance and the corresponding SMD, which pool sample variances from multiple studies using both an analysis of variance and a meta-analytic framework. We demonstrate the properties of these estimators via analytic and simulation results and offer recommendations for when these estimators are appropriate in practice.
Citation: Journal of Educational and Behavioral Statistics
PubDate: 2024-04-08T04:36:13Z
DOI: 10.3102/10769986241238478
- Strive for Measurement, Set New Standards, and Try Not to Be Evil
-
Free pre-print version: Loading...
Rate this result:
What is this?
Please help us test our new pre-print finding feature by giving the pre-print link a rating.
A 5 star rating indicates the linked pre-print has the exact same content as the published article.
Authors: Derek C. Briggs
Abstract: Journal of Educational and Behavioral Statistics, Ahead of Print.
I consider recent attempts to establish standards, principles, and goals for artificial intelligence (AI) through the lens of educational measurement. Distinctions are made between generative AI and AI-adjacent methods and applications of AI in formative versus summative assessment contexts. While expressing optimism about its possibilities, I caution that the examples of truly generative AI in educational testing have the potential to be overexaggerated, that efforts to establish standards for AI should complement the Standards for Educational and Psychological Testing and focus attention on the issues of fairness and social responsibility, and that scientific advance and transparency in the development and application of AI in educational assessment may be incompatible with the competitive marketplace that is funding this development.
Citation: Journal of Educational and Behavioral Statistics
PubDate: 2024-04-08T04:36:13Z
DOI: 10.3102/10769986241238479
- Disentangling Person-Dependent and Item-Dependent Causal Effects:
Applications of Item Response Theory to the Estimation of Treatment Effect
Heterogeneity-
Free pre-print version: Loading...
Rate this result:
What is this?
Please help us test our new pre-print finding feature by giving the pre-print link a rating.
A 5 star rating indicates the linked pre-print has the exact same content as the published article.
Authors: Joshua B. Gilbert, Luke W. Miratrix, Mridul Joshi, Benjamin W. Domingue
Abstract: Journal of Educational and Behavioral Statistics, Ahead of Print.
Analyzing heterogeneous treatment effects (HTEs) plays a crucial role in understanding the impacts of educational interventions. A standard practice for HTE analysis is to examine interactions between treatment status and preintervention participant characteristics, such as pretest scores, to identify how different groups respond to treatment. This study demonstrates that the identical patterns of HTE on test score outcomes can emerge either from variation in treatment effects due to a preintervention participant characteristic or from correlations between treatment effects and item easiness parameters. We demonstrate analytically and through simulation that these two scenarios cannot be distinguished if analysis is based on summary scores alone. We then describe a novel approach that identifies the relevant data-generating process by leveraging item-level data. We apply our approach to a randomized trial of a reading intervention in second grade and show that any apparent HTE by pretest ability is driven by the correlation between treatment effect size and item easiness. Our results highlight the potential of employing measurement principles in causal analysis, beyond their common use in test construction.
Citation: Journal of Educational and Behavioral Statistics
PubDate: 2024-04-05T01:09:25Z
DOI: 10.3102/10769986241240085
- Power Analyses for Estimation of Complier Average Causal Effects Under
Random Encouragement Designs in Education Research: Theory and Guidance-
Free pre-print version: Loading...
Rate this result:
What is this?
Please help us test our new pre-print finding feature by giving the pre-print link a rating.
A 5 star rating indicates the linked pre-print has the exact same content as the published article.
Authors: Peter Z. Schochet
Abstract: Journal of Educational and Behavioral Statistics, Ahead of Print.
Random encouragement designs evaluate treatments that aim to increase participation in a program or activity. These randomized controlled trials (RCTs) can also assess the mediated effects of participation itself on longer term outcomes using a complier average causal effect (CACE) estimation framework. This article considers power analysis methods for such CACE analyses for a range of RCT designs, including nonclustered, clustered, and random block designs. The focus is on behavioral encouragements to promote action, such as text messaging, that are increasingly being tested in education trials. We derive asymptotic distributions of the CACE estimators using generalized estimating equations theory, which underlie the power formulas. We incorporate noncompliance from both the actual receipt of the encouragement and participation itself. An illustrative power analysis provides sample size guidance using an available Shiny R dashboard.
Citation: Journal of Educational and Behavioral Statistics
PubDate: 2024-03-12T04:50:47Z
DOI: 10.3102/10769986241233790
- Bayesian Adaptive Lasso for the Detection of Differential Item Functioning
in Graded Response Models-
Free pre-print version: Loading...
Rate this result:
What is this?
Please help us test our new pre-print finding feature by giving the pre-print link a rating.
A 5 star rating indicates the linked pre-print has the exact same content as the published article.
Authors: Na Shan, Ping-Feng Xu
Abstract: Journal of Educational and Behavioral Statistics, Ahead of Print.
The detection of differential item functioning (DIF) is important in psychological and behavioral sciences. Standard DIF detection methods perform an item-by-item test iteratively, often assuming that all items except the one under investigation are DIF-free. This article proposes a Bayesian adaptive Lasso method to detect DIF in graded response models (GRMs), where the DIF effects for all items can be identified simultaneously. The multiple-group GRMs are specified, and the possible DIF effects for each item are reparameterized using the increment components. Then, a Bayesian adaptive Lasso procedure is developed for parameter estimation, in which DIF effects can be automatically obtained. Our method is evaluated and compared with the commonly used likelihood ratio test method in a simulation study. The results show that our method can recover most model parameters well and has better control of false positive rates in almost all conditions. An application is presented using data from the National Longitudinal Study of Adolescent to Adult Health (Add Health).
Citation: Journal of Educational and Behavioral Statistics
PubDate: 2024-03-07T01:38:05Z
DOI: 10.3102/10769986241233777
- Equivalencies Between Ad Hoc Strategies and Multivariate Models for
Meta-Analysis of Dependent Effect Sizes-
Free pre-print version: Loading...
Rate this result:
What is this?
Please help us test our new pre-print finding feature by giving the pre-print link a rating.
A 5 star rating indicates the linked pre-print has the exact same content as the published article.
Authors: James E. Pustejovsky, Man Chen
Abstract: Journal of Educational and Behavioral Statistics, Ahead of Print.
Meta-analyses of educational research findings frequently involve statistically dependent effect size estimates. Meta-analysts have often addressed dependence issues using ad hoc approaches that involve modifying the data to conform to the assumptions of models for independent effect size estimates, such as by aggregating estimates to obtain one summary estimate per study, conducting separate analyses of distinct subgroups of estimates, or combinations thereof. We show that these ad hoc approaches correspond exactly to certain multivariate models for dependent effect sizes. Specifically, we describe classes of multivariate random effects models that have likelihoods equivalent to those of models for effect sizes that have been averaged by study, classified into subgroups, or both. The equivalencies also apply to robust variance estimation methods.
Citation: Journal of Educational and Behavioral Statistics
PubDate: 2024-03-05T08:45:16Z
DOI: 10.3102/10769986241232524
- Using Regularization to Identify Measurement Bias Across Multiple
Background Characteristics: A Penalized Expectation–Maximization
Algorithm-
Free pre-print version: Loading...
Rate this result:
What is this?
Please help us test our new pre-print finding feature by giving the pre-print link a rating.
A 5 star rating indicates the linked pre-print has the exact same content as the published article.
Authors: William C. M. Belzak, Daniel J. Bauer
Abstract: Journal of Educational and Behavioral Statistics, Ahead of Print.
Testing for differential item functioning (DIF) has undergone rapid statistical developments recently. Moderated nonlinear factor analysis (MNLFA) allows for simultaneous testing of DIF among multiple categorical and continuous covariates (e.g., sex, age, ethnicity, etc.), and regularization has shown promising results for identifying DIF among many covariates. However, computationally inefficient estimation methods have hampered practical use of the regularized MNFLA method. We develop a penalized expectation–maximization (EM) algorithm with soft- and firm-thresholding to more efficiently estimate regularized MNLFA parameters. Simulation and empirical results show that, compared to previous implementations of regularized MNFLA, the penalized EM algorithm is faster, more flexible, and more statistically principled. This method also yields similar recovery of DIF relative to previous implementations, suggesting that regularized DIF detection remains a preferred approach over traditional methods of identifying DIF.
Citation: Journal of Educational and Behavioral Statistics
PubDate: 2024-02-05T01:20:16Z
DOI: 10.3102/10769986231226439
- The Use of Reparametrization and Constraints on Linear Models to Parse
Qualitative and Quantitative Information for a Set of Predictors-
Free pre-print version: Loading...
Rate this result:
What is this?
Please help us test our new pre-print finding feature by giving the pre-print link a rating.
A 5 star rating indicates the linked pre-print has the exact same content as the published article.
Authors: Ernest C. Davenport, Mark L. Davison, Kyungin Park
Abstract: Journal of Educational and Behavioral Statistics, Ahead of Print.
The following study shows how reparameterizations and constraints of the general linear model can serve to parse quantitative and qualitative aspects of predictors. We demonstrate three different approaches. The study uses data from the High School Longitudinal Study of 2009 on mathematics course-taking and achievement as an example. Results show that all mathematics courses are not equally predictive of math achievement. Thus, taking into account qualitative aspects of mathematics courses is useful. The study ends with a justification of quantifying qualitative aspects of predictors relative to a criterion with extensions to other linear models.
Citation: Journal of Educational and Behavioral Statistics
PubDate: 2024-01-19T07:12:53Z
DOI: 10.3102/10769986231223769
- Improving Balance in Educational Measurement: A Legacy of E. F. Lindquist
-
Free pre-print version: Loading...
Rate this result:
What is this?
Please help us test our new pre-print finding feature by giving the pre-print link a rating.
A 5 star rating indicates the linked pre-print has the exact same content as the published article.
Authors: Daniel Koretz
Abstract: Journal of Educational and Behavioral Statistics, Ahead of Print.
A critically important balance in educational measurement between practical concerns and matters of technique has atrophied in recent decades, and as a result, some important issues in the field have not been adequately addressed. I start with the work of E. F. Lindquist, who exemplified the balance that is now wanting. Lindquist was arguably the most prolific developer of achievement tests in the history of the field and an accomplished statistician, but he nonetheless focused extensively on the practical limitations of testing and their implications for test development, test use, and inference. I describe the withering of this balance and discuss two pressing issues that have not been adequately addressed as a result: the lack of robustness of performance standards and score inflation. I conclude by discussing steps toward reestablishing the needed balance.
Citation: Journal of Educational and Behavioral Statistics
PubDate: 2024-01-08T07:07:01Z
DOI: 10.3102/10769986231218306
|
Hybrid journal (It can contain Open Access articles)
