Useful Resources for Social Science Research

The list of resources below is intended for (my) personal use. It contains primers, tutorials, guides I have used/encountered and/or want to learn more about it. These are related to producing better science, open-science, methods, statistics, visualization, R & RStudio.

---

Open Science

• 8 Easy Steps to Open Science: An Annotated Reading List

• Open science manual

• What is open science? [with Vazire, Lakens, Corker, MicNuijten, and Simons by SPSP]

Better science

• Minimizing Mistakes In Psychological Science

• OSF Guides

• Manual of Best Practices in Transparent Social Science Research (BITSS)

• The BITSS Working Paper Series

• F1000 Guide

Meta-science

• Studying the science of science

• Meta-Research

Reproducibility

• Hidden moderators impacting reproducibility

Preregistration

• Preregistration template [pdf file]

• Preregistration template of Open Stats Lab and Project Tier

• Preregistration template for existing data (secondary data analysis)

• Comparison of Registered Reports

Writing

• Writing Empirical Articles: Transparency, Reproducibility, Clarity, and Memorability

preprints

• Preprints could promote confusion and distortion

---

Reproducibility & Replicability Crisis

All Sciences

• Most published research findings are false (Ioannidis 2005)

• Introduction to the replicability crisis

• The Irreproducibility Crisis of Modern Science: Causes, Consequences and the Road to Reform

• 1,500 scientists lift the lid on reproducibility

• Wiki Primer (Good!)

• Reality check on reproducibility

• Statistical literacy

• Quality Control

• Peer-review role & Science self-correction myth

• On Sample Quality

• Scientists Self-Deception, Bias, Cognitive Fallacies in Research making

• Misunderstanding P-values

• Loss of confidence project

• Is ‘Reproducibility Crisis’ Overblown?

• Is science really facing a reproducibility crisis, and do we need it to? [Fanelli]

Publication Bias (file drawer problem) & Null Findings

• Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling [ReproducibiliTea Podcast]

• Questionable Research Practices Revisited

• Coursera Video on QRP by Zand Scholten

Questionable research Practices

• Publication bias and the canonization of false facts

• Coursera Video

• Replication, Communication, and the Population Dynamics of Scientific Discovery

• Emphasis on (Rate of) Published Positive Results - per scientific field

samples

WEIRD (Western, Educated, Industrialized, Rich, and Democratic) 

• 99% of all published studies rely on WEIRD samples

• WEIRD samples and external validity

• Attracting representative samples requires thought

MTurk

• Students, CCS, MTurk similar on Economic Games
  

---

Replication Rate in Social Sciences

• Across 6 large-scale replication projects, replication rate is 90 of 190 (47%):

  • Many Labs 1

  • Many Labs 2 (summary)

  • Many Labs 3

  • SSRP

  • EERP

  • RPP

---

 

Psychology

• Estimating the Reproducibility of Psychological Science

• Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015

• Evaluating replicability of laboratory experiments in economics

• Variation in Replicability: A “Many Labs” Replication Project

• [P-B/F-D] Quantifying Support for the Null Hypothesis in Psychology

• Primer, timeline, & future

• Studies

  • In 72% of these cases, nonsignificant results were misinterpreted (Quantifying Support for the Null Hypothesis in Psychology

• Critiques & Replies

  • Questions about what counts as replication

  • Psychology is not in crisis (Feldman's Barret, 2015)

Political Science

• Summary of open-science initiatives in Pol-Sci

• Take from Editors of Journal of Experimental Political Science

• Vignette Experiment

The Reproducibility & Replicability Crisis & Politics

• Andrew Gelman's blog 1 (timeline of events + Fiske fiasco), 2 (why psychology), 3 (centered around social psychology)

• The crisis of politicization within and beyond science

• As Political Ammunition

---

Statistics

Statistical practice

• The little book of statistical practice

• Why model?

• Formal modelling - we need more

Study Design

• Declaring and Diagnosing Research Designs

degrees of freedom

• Illustrating degrees of freedom in terms of sample size and dimensionality

• Degrees of freedom tutorial

• Degrees of freedom in inferential statistics

• Twitter thread with discussions and references

Power Analysis

Multilevel design

• simr: R package for Power Analysis for Generalised Linear Mixed Models by Simulation

• Power analysis at OSF from SIPS with Soderberg, Hennes, and Lane

• Power analysis through Monte Carlo simulation

• Andrew Gelman's You need 16 times the sample size to estimate an interaction than to estimate a main effect

• Sufficient Sample Sizes for Multilevel Modeling by Cora Maas & Joop Hox

• Tom Snijder's summary of available software and guides on Multilevel modeling

• MLPowSim Guide and Software

Robust methods

• Exploratory graph analysis: A new approach for estimating the number of dimensions in psychological research

• Discussion

• Regression estimation with MM-estimator (vs. OLS) -> resistance to outliers

Linear & Generalized Linear Models

• J. Fox: Applied Regression Analysis and Generalized Linear Models with car and effect packages

Time Series

• Forecasting in R: ARIMA, Hierarchical time-series

• Course on Modern Methods in Time-series (in R, with code) by Kevin Kotzé

Causal Inference in graphs with Animated (GIF) Plots

• Controlling for and Matching on a variable, Instrumental Variables, Fixed Effects, Difference-in-Differences, Regression Discontinuity (github repo, thread) by Nick Huntington-Klein

 Survey Methodology PODCASTs

• More or Less: Behind the Stats

• Not so standard deviations

• Stats + Stories

• Casual Inference

• Everything hertz

• CNN Margins of Error

• Freakonomics

• Probable Causation

• After the Fact

• ICPSR Data Brunch

• Hidden brain

• No stupid questions

• What is wrong with UX

• Mixed Methods

• FiftyThirtyEight

• Politics and Polls

---

Psychometrics

Reliability

• R-tutorial on Reliability: alpha to omega [Revelle]

Item Response Theory [IRT]

• R-tutorial on Reliability: alpha to omega [Revelle]

Multidimensional Item Response Theory [MIRT]

• Multidimensional multiple group IRT models with skew normal latent trait distributions

Network Analysis

• R Tutorial

• ggnetwork

• Estimating psychological networks and their accuracy: A tutorial paper

  • Bootstrapping edges after regularization & Power and Robustness of Networks [both R tutorials]

• ggnetwork

---

Scientific Debates

Bayesians vs. Frequentists [in social media]

• TOST vs. BF

---

R & oPEN sCIENCE

Reproducible Research

• Reproducible Research with R and R Studio [GitHub Repo]

• Implementing Reproducible Research

• Dynamic Documents with R and Knitr

• Intros, Courses, & Examples:

  • Tools for Reproducible Research (Karl Broman, Spring, 2016)

  • Marian Schimdt

r Resources

R-Intros

• R for psychologists with piping [Nick Michalak]

R-Series: Complete COurses

Data Science Methods for Psychology - University of Oregon (HardSci)

• General Programming in R

• Data Wrangling

• Data Visualization

• Text Processing

• Web Scraping

• Machine Learning

• Network Analysis

• Package Creation

• Website Building in R

Data Science for Social Sciences

• Intro, data types, conditionals, data im/export, data/string manipulation, dataviz, anova, LM and GLM, Reproducible Reports, Web Scraping, APIs, Machine Learning, Natural Language Processing (Text Mining).

Nifty Tricks & Tips

• Copy & Paste data from Web, Text [Twitter]

• Highlight parts of plots (lines, groups, etc) with gghighlight [1,2]

Data Visualization

Intro

• Data Visualization [in R]: A practical introduction

R-packages

• ggplot2: is a system for declaratively creating graphics, based on The Grammar of Graphics.

  • ggplot extensions and gallery.

    • ggdag: An R Package for visualizing and analyzing directed acyclic graphs

    • ggnetwork

    • ggalluvial

Publication geared

• ggally: the ally of ggplot2 (display of multiple regression coefficients and its diagnostics, networks, time-series, distributions, etc.).

• ggstatsplot: Plots with Statistical Details (most common types of statistical tests (parametric, non-parametric, and robust versions of t-tets, anova, correlation, and contingency tables)

• ggpubr: ggplot2 based publication ready plots [tutorial at sthda]

• cowplot: publication-ready theme for ggplot2 (e.g., easy add panels A, B, C). Here's code for adding panels to non-ggplot2 R-base graphs.

Guides

• Hadley Wickham’s Data Visualization Guide

• Data Visualization Cheatsheet

• R-graphics Cookbook

Colors and Pallets 

• Rose’s RColorBrewer Guide

miscellaneous

• Style Guide for Figures

• Using Map Data

Per type of data

Distributions 

• ggpairs: Visualizing distributions with groupings [examples: 1]

Group differences

• ggradar

Time Series

• CGPfunctions: Slopegraphs

Strings

• Handling Strings with R

Blogdown

• Intro to blogdown via Twitter thread [Dan Quintana]

Bookdown

• Bookdown examples

---

[Scholarly]Social Media

• Twitter search operators

---

Philosophy of Science

Primer on Philosophy of Science

On Karl Popper

• rejection of empiricism

• virtue of refutation

• rejecting politics as science

• Lakatos, Kuhn and Feyerabend (twitter thread & slides)

• the craving to be right

Pot-pourri

• Logical fallacies - list thereof (youtube video -> great resource)

• Bias - list thereof with examples.

---

Everything else

Economics (Behavioral)

• On "the superiority of economists" (twitter thread)

• Gelman already explains why economists get paid more back in 2014: economists come up with ideas that rich people like.

• Evaluating replicability of laboratory experiments in economics

• Replication & Reproducibility in Economics

• The Power of Bias in Economics Research

• Representativeness of women and URM in Economics [link; link2]

• Course on Econometrics

  • SciencesPo

Writing

R Journalism - Reproducible Journalism by Washington Post Investigative Data Reporter

• Logical fallacies - list thereof (youtube video -> great resource)

• Bias - list thereof with examples.

Political Science

• A dataset with political datasets