Authors:Blakeley B. McShane, Ulf Böckenholt Abstract: Advances in Methods and Practices in Psychological Science, Ahead of Print. Meta-analysis typically involves the analysis of summary data (e.g., means, standard deviations, and sample sizes) from a set of studies via a statistical model that is a special case of a hierarchical (or multilevel) model. Unfortunately, the common summary-data approach to meta-analysis used in psychological research is often employed in settings where the complexity of the data warrants alternative approaches. In this article, we propose a thought experiment that can lead meta-analysts to move away from the common summary-data approach to meta-analysis and toward richer and more appropriate summary-data approaches when the complexity of the data warrants it. Specifically, we propose that it can be extremely fruitful for meta-analysts to act as if they possess the individual-level data from the studies and consider what model specifications they might fit even when they possess only summary data. This thought experiment is justified because (a) the analysis of the individual-level data from the studies via a hierarchical model is considered the “gold standard” for meta-analysis and (b) for a wide variety of cases common in meta-analysis, the summary-data and individual-level-data approaches are, by a principle known as statistical sufficiency, equivalent when the underlying models are appropriately specified. We illustrate the value of our thought experiment via a case study that evolves across five parts that cover a wide variety of data settings common in meta-analysis. Citation: Advances in Methods and Practices in Psychological Science PubDate: 2019-11-26T03:59:58Z DOI: 10.1177/2515245919884304

Authors:Frank A. Bosco, James G. Field, Kai R. Larsen, Yingyi Chang, Krista L. Uggerslev Abstract: Advances in Methods and Practices in Psychological Science, Ahead of Print. In this article, we provide a review of research-curation and knowledge-management efforts that may be leveraged to advance research and education in psychological science. After reviewing the approaches and content of other efforts, we focus on the metaBUS project’s platform, the most comprehensive effort to date. The metaBUS platform uses standards-based protocols in combination with human judgment to organize and make readily accessible a database of research findings, currently numbering more than 1 million. It allows users to conduct rudimentary, instant meta-analyses, and capacities for visualization and communication of meta-analytic findings have recently been added. We conclude by discussing challenges, opportunities, and recommendations for expanding the project beyond applied psychology. Citation: Advances in Methods and Practices in Psychological Science PubDate: 2019-11-26T03:54:09Z DOI: 10.1177/2515245919882693

Authors:Zoltan Dienes Abstract: Advances in Methods and Practices in Psychological Science, Ahead of Print. To get evidence for or against a theory relative to the null hypothesis, one needs to know what the theory predicts. The amount of evidence can then be quantified by a Bayes factor. Specifying the sizes of the effect one’s theory predicts may not come naturally, but I show some ways of thinking about the problem, some simple heuristics that are often useful when one has little relevant prior information. These heuristics include the room-to-move heuristic (for comparing mean differences), the ratio-of-scales heuristic (for regression slopes), the ratio-of-means heuristic (for regression slopes), the basic-effect heuristic (for analysis of variance effects), and the total-effect heuristic (for mediation analysis). Citation: Advances in Methods and Practices in Psychological Science PubDate: 2019-11-14T04:12:19Z DOI: 10.1177/2515245919876960

Authors:Sam Parsons, Anne-Wil Kruijt, Elaine Fox Abstract: Advances in Methods and Practices in Psychological Science, Ahead of Print. Psychological science relies on behavioral measures to assess cognitive processing; however, the field has not yet developed a tradition of routinely examining the reliability of these behavioral measures. Reliable measures are essential to draw robust inferences from statistical analyses, and subpar reliability has severe implications for measures’ validity and interpretation. Without examining and reporting the reliability of measurements used in an analysis, it is nearly impossible to ascertain whether results are robust or have arisen largely from measurement error. In this article, we propose that researchers adopt a standard practice of estimating and reporting the reliability of behavioral assessments of cognitive processing. We illustrate the need for this practice using an example from experimental psychopathology, the dot-probe task, although we argue that reporting reliability is relevant across fields (e.g., social cognition and cognitive psychology). We explore several implications of low measurement reliability and the detrimental impact that failure to assess measurement reliability has on interpretability and comparison of results and therefore research quality. We argue that researchers in the field of cognition need to report measurement reliability as routine practice so that more reliable assessment tools can be developed. To provide some guidance on estimating and reporting reliability, we describe the use of bootstrapped split-half estimation and intraclass correlation coefficients to estimate internal consistency and test-retest reliability, respectively. For future researchers to build upon current results, it is imperative that all researchers provide psychometric information sufficient for estimating the accuracy of inferences and informing further development of cognitive-behavioral assessments. Citation: Advances in Methods and Practices in Psychological Science PubDate: 2019-11-06T05:05:21Z DOI: 10.1177/2515245919879695

Authors:Jeffrey J. Starns, Andrea M. Cataldo, Caren M. Rotello, Jeffrey Annis, Andrew Aschenbrenner, Arndt Bröder, Gregory Cox, Amy Criss, Ryan A. Curl, Ian G. Dobbins, John Dunn, Tasnuva Enam, Nathan J. Evans, Simon Farrell, Scott H. Fraundorf, Scott D. Gronlund, Andrew Heathcote, Daniel W. Heck, Jason L. Hicks, Mark J. Huff, David Kellen, Kylie N. Key, Asli Kilic, Karl Christoph Klauer, Kyle R. Kraemer, Fábio P. Leite, Marianne E. Lloyd, Simone Malejka, Alice Mason, Ryan M. McAdoo, Ian M. McDonough, Robert B. Michael, Laura Mickes, Eda Mizrak, David P. Morgan, Shane T. Mueller, Adam Osth, Angus Reynolds, Travis M. Seale-Carlisle, Henrik Singmann, Jennifer F. Sloane, Andrew M. Smith, Gabriel Tillman, Don van Ravenzwaaij, Christoph T. Weidemann, Gary L. Wells, Corey N. White, Jack Wilson Abstract: Advances in Methods and Practices in Psychological Science, Ahead of Print. Scientific advances across a range of disciplines hinge on the ability to make inferences about unobservable theoretical entities on the basis of empirical data patterns. Accurate inferences rely on both discovering valid, replicable data patterns and accurately interpreting those patterns in terms of their implications for theoretical constructs. The replication crisis in science has led to widespread efforts to improve the reliability of research findings, but comparatively little attention has been devoted to the validity of inferences based on those findings. Using an example from cognitive psychology, we demonstrate a blinded-inference paradigm for assessing the quality of theoretical inferences from data. Our results reveal substantial variability in experts’ judgments on the very same data, hinting at a possible inference crisis. Citation: Advances in Methods and Practices in Psychological Science PubDate: 2019-09-17T05:26:59Z DOI: 10.1177/2515245919869583

Authors:Malte Elson Abstract: Advances in Methods and Practices in Psychological Science, Ahead of Print. Research synthesis is based on the assumption that when the same association between constructs is observed repeatedly in a field, the relationship is probably real, even if its exact magnitude can be debated. Yet the probability that the relationship is real is a function not only of recurring results, but also of the quality and consistency of the empirical procedures that produced those results and that any meta-analysis necessarily inherits. Standardized protocols in data collection, analysis, and interpretation are foundations of empiricism and a healthy sign of a discipline’s maturity. I propose that meta-analysis as typically applied in psychology will benefit from complementing aggregation of observed effect sizes with systematic examination of the standardization of the methodology that deterministically produced them. I describe potential units of analysis and offer two examples illustrating the benefits of such efforts. Ideally, this synergetic approach will advance theory by improving the quality of meta-analytic inferences. Citation: Advances in Methods and Practices in Psychological Science PubDate: 2019-08-22T02:39:05Z DOI: 10.1177/2515245919863296

Authors:Marc-André Goulet, Denis Cousineau First page: 199 Abstract: Advances in Methods and Practices in Psychological Science, Ahead of Print. When running statistical tests, researchers can commit a Type II error, that is, fail to reject the null hypothesis when it is false. To diminish the probability of committing a Type II error (β), statistical power must be augmented. Typically, this is done by increasing sample size, as more participants provide more power. When the estimated effect size is small, however, the sample size required to achieve sufficient statistical power can be prohibitive. To alleviate this lack of power, a common practice is to measure participants multiple times under the same condition. Here, we show how to estimate statistical power by taking into account the benefit of such replicated measures. To that end, two additional parameters are required: the correlation between the multiple measures within a given condition and the number of times the measure is replicated. An analysis of a sample of 15 studies (total of 298 participants and 38,404 measurements) suggests that in simple cognitive tasks, the correlation between multiple measures is approximately .14. Although multiple measurements increase statistical power, this effect is not linear, but reaches a plateau past 20 to 50 replications (depending on the correlation). Hence, multiple measurements do not replace the added population representativeness provided by additional participants. Citation: Advances in Methods and Practices in Psychological Science PubDate: 2019-06-10T02:58:02Z DOI: 10.1177/2515245919849434

Authors:Sara J. Weston, Stuart J. Ritchie, Julia M. Rohrer, Andrew K. Przybylski First page: 214 Abstract: Advances in Methods and Practices in Psychological Science, Ahead of Print. Secondary data analysis, or the analysis of preexisting data, provides a powerful tool for the resourceful psychological scientist. Never has this been more true than now, when technological advances enable both sharing data across labs and continents and mining large sources of preexisting data. However, secondary data analysis is easily overlooked as a key domain for developing new open-science practices or improving analytic methods for robust data analysis. In this article, we provide researchers with the knowledge necessary to incorporate secondary data analysis into their methodological toolbox. We explain that secondary data analysis can be used for either exploratory or confirmatory work, and can be either correlational or experimental, and we highlight the advantages and disadvantages of this type of research. We describe how transparency-enhancing practices can improve and alter interpretations of results from secondary data analysis and discuss approaches that can be used to improve the robustness of reported results. We close by suggesting ways in which scientific subfields and institutions could address and improve the use of secondary data analysis. Citation: Advances in Methods and Practices in Psychological Science PubDate: 2019-06-11T03:26:17Z DOI: 10.1177/2515245919848684

Authors:J. Toby Mordkoff First page: 228 Abstract: Advances in Methods and Practices in Psychological Science, Ahead of Print. Accurate estimates of population effect size are critical to empirical science, for both reporting experimental results and conducting a priori power analyses. Unfortunately, the current most-popular measure of standardized effect size, partial eta squared ([math]), is known to have positive bias. Two less-biased alternatives, partial epsilon squared ([math]) and partial omega squared ([math]), have both existed for decades, but neither is often employed. Given that researchers appear reluctant to abandon [math], this article provides a simple method for removing bias from this measure, to produce a value referred to as adjusted partial eta squared (adj [math]). Some of the many benefits of adopting this measure are briefly discussed. Citation: Advances in Methods and Practices in Psychological Science PubDate: 2019-07-30T06:37:44Z DOI: 10.1177/2515245919855053

Authors:Scott A. Cassidy, Ralitza Dimova, Benjamin Giguère, Jeffrey R. Spence, David J. Stanley First page: 233 Abstract: Advances in Methods and Practices in Psychological Science, Ahead of Print. Null-hypothesis significance testing (NHST) is commonly used in psychology; however, it is widely acknowledged that NHST is not well understood by either psychology professors or psychology students. In the current study, we investigated whether introduction-to-psychology textbooks accurately define and explain statistical significance. We examined 30 introductory-psychology textbooks, including the best-selling books from the United States and Canada, and found that 89% incorrectly defined or explained statistical significance. Incorrect definitions and explanations were most often consistent with the odds-against-chance fallacy. These results suggest that it is common for introduction-to-psychology students to be taught incorrect interpretations of statistical significance. We hope that our results will create awareness among authors of introductory-psychology books and provide the impetus for corrective action. To help with classroom instruction, we provide slides that correctly describe NHST and may be useful for introductory-psychology instructors. Citation: Advances in Methods and Practices in Psychological Science PubDate: 2019-06-27T03:12:34Z DOI: 10.1177/2515245919858072

Authors:Tom E. Hardwicke, Michael C. Frank, Simine Vazire, Steven N. Goodman First page: 240 Abstract: Advances in Methods and Practices in Psychological Science, Ahead of Print. Readers of peer-reviewed research may assume that the reported statistical analyses supporting scientific claims have been closely scrutinized and surpass a high-quality threshold. However, widespread misunderstanding and misuse of statistical concepts and methods suggests that suboptimal or erroneous statistical practice is routinely overlooked during peer review in psychology. Here, we explore whether psychology journals could ameliorate some of the field’s statistical ailments by adopting specialized statistical review: a focused technical assessment, performed by statistical experts, that addresses the analysis and presentation of quantitative information and supplements regular peer review. We discuss evidence from a recent survey of journal editors suggesting that specialized statistical review may be unusual in psychology journals and is regarded by many editors as unnecessary. We contrast these views with those in the biomedical domain, where statistical review has been considered a partial preventive measure against the improper use of statistics since the late 1970s. We suggest that the current “credibility revolution” presents an opportune occasion for psychology journals to consider adopting specialized statistical review. Citation: Advances in Methods and Practices in Psychological Science PubDate: 2019-07-03T02:51:51Z DOI: 10.1177/2515245919858428

Authors:Peter Boedeker, Nathan T. Kearns First page: 250 Abstract: Advances in Methods and Practices in Psychological Science, Ahead of Print. In psychology, researchers are often interested in the predictive classification of individuals. Various models exist for such a purpose, but which model is considered a best practice is conditional on attributes of the data. Under certain conditions, linear discriminant analysis (LDA) has been shown to perform better than other predictive methods, such as logistic regression, multinomial logistic regression, random forests, support-vector machines, and the K-nearest neighbor algorithm. The purpose of this Tutorial is to provide researchers who already have a basic level of statistical training with a general overview of LDA and an example of its implementation and interpretation. Decisions that must be made when conducting an LDA (e.g., prior specification, choice of cross-validation procedures) and methods of evaluating case classification (posterior probability, typicality probability) and overall classification (hit rate, Huberty’s I index) are discussed. LDA for prediction is described from a modern Bayesian perspective, as opposed to its original derivation. A step-by-step example of implementing and interpreting LDA results is provided. All analyses were conducted in R, and the script is provided; the data are available online. Citation: Advances in Methods and Practices in Psychological Science PubDate: 2019-07-09T02:17:38Z DOI: 10.1177/2515245919849378

Authors:Alex Bradley, Richard J. E. James First page: 264 Abstract: Advances in Methods and Practices in Psychological Science, Ahead of Print. The ubiquitous use of the Internet in daily life means that there are now large reservoirs of data that can provide fresh insights into human behavior. One of the key barriers preventing more researchers from utilizing online data is that they do not have the skills to access the data. This Tutorial addresses this gap by providing a practical guide to scraping online data using the popular statistical language R. Web scraping is the process of automatically collecting information from websites. Such information can take the form of numbers, text, images, or videos. This Tutorial shows readers how to download web pages, extract information from those pages, store the extracted information, and do so across multiple pages of a website. A website has been created to assist readers in learning how to web-scrape. This website contains a series of examples that illustrate how to scrape a single web page and how to scrape multiple web pages. The examples are accompanied by videos describing the processes involved and by exercises to help readers increase their knowledge and practice their skills. Example R scripts have been made available at the Open Science Framework. Citation: Advances in Methods and Practices in Psychological Science PubDate: 2019-07-30T06:41:04Z DOI: 10.1177/2515245919859535

Authors:Jonas W. B. Lang, Paul D. Bliese, Amy B. Adler First page: 271 Abstract: Advances in Methods and Practices in Psychological Science, Ahead of Print. Over time, groups can change in at least two important ways. First, they can display different trajectories (e.g., increases or decreases) on constructs of interest. Second, the configuration of group members’ responses within a group can change, such that the members become more or less similar to each other. Psychologists have historically been interested in understanding changes in groups over time; however, there is currently no comprehensive quantitative framework for studying and modeling group processes over time. We present a multilevel framework for such research—the multilevel group-process framework (MGPF). The MGPF builds on a statistical approach developed to capture whether individual members of a group develop a shared climate over time, but we extend the core ideas in two important ways. First, we describe how researchers can gain insights into group phenomena such as group leniency, group learning, groupthink, group extremity, group forming, group freezing, and group adjourning through modeling change in latent mean levels and consensus. Second, we present a sequence of model-testing steps that enable researchers to systematically study and contrast different group processes. We describe how the MGPF can lead to novel research questions and illustrate its use in two example data sets. Citation: Advances in Methods and Practices in Psychological Science PubDate: 2019-03-07T04:29:43Z DOI: 10.1177/2515245918823722

Authors:Lesa Hoffman First page: 288 Abstract: Advances in Methods and Practices in Psychological Science, Ahead of Print. The increasing availability of software with which to estimate multivariate multilevel models (also called multilevel structural equation models) makes it easier than ever before to leverage these powerful techniques to answer research questions at multiple levels of analysis simultaneously. However, interpretation can be tricky given that different choices for centering model predictors can lead to different versions of what appear to be the same parameters; this is especially the case when the predictors are latent variables created through model-estimated variance components. A further complication is a recent change to Mplus (Version 8.1), a popular software program for estimating multivariate multilevel models, in which the selection of Bayesian estimation instead of maximum likelihood results in different lower-level predictors when random slopes are requested. This article provides a detailed explication of how the parameters of multilevel models differ as a function of the analyst’s decisions regarding centering and the form of lower-level predictors (i.e., observed or latent), the method of estimation, and the variant of program syntax used. After explaining how different methods of centering lower-level observed predictor variables result in different higher-level effects within univariate multilevel models, this article uses simulated data to demonstrate how these same concepts apply in specifying multivariate multilevel models with latent lower-level predictor variables. Complete data, input, and output files for all of the example models have been made available online to further aid readers in accurately translating these central tenets of multivariate multilevel modeling into practice. Citation: Advances in Methods and Practices in Psychological Science PubDate: 2019-08-02T04:05:48Z DOI: 10.1177/2515245919842770

Authors:Laura M. Stapleton, Tessa L. Johnson First page: 312 Abstract: Advances in Methods and Practices in Psychological Science, Ahead of Print. When researchers model multilevel data, often a shared construct of interest is measured by individual-level observations, for example, students’ responses regarding how engaging their instructor’s teaching style is. In such cases, the construct of interest, “engaging teaching,” is shared at the cluster level across individuals, yet rarely are these shared constructs modeled as such. To address this gap, we discuss multilevel confirmatory factor analysis models that have been applied to item-level data obtained from multiple raters within given clusters, focusing particularly on measuring a characteristic at the cluster level. After discussing the parameters in each potential model, we make recommendations as to the appropriate modeling approach and the steps to be taken for model assessment given a set of data and hypothesized construct of interest. In particular, we encourage applied researchers not to use a model without constraints across the within-cluster level and the between-cluster level because such models assume that the average amount of the individual-level construct in a cluster does not differ across clusters. To illustrate this issue, we present simulation results and evaluate a series of models using empirical data from the Trends in International Mathematics and Science Study. Citation: Advances in Methods and Practices in Psychological Science PubDate: 2019-08-21T02:34:56Z DOI: 10.1177/2515245919855039