This course covers the foundations of designing, implementing and analysing basic online survey experiments in political science.
Instructor
Benjamin E. Goldsmith is a Professor in the School of Politics and International Relations at the Australian National University. His research is in the areas of international relations, comparative foreign policy, and atrocity forecasting. He is the author of the book Imitation in International Relations: Observational Learning, Analogies, and Foreign Policy in Russia and Ukraine, as well as articles in leading academic journals including American Political Science Review, Comparative Political Studies, European Journal of International Relations, Journal of Conflict Resolution, Journal of Peace Research, Journal of Politics, Political Psychology, PNAS Nexus, Quarterly Journal of Political Science, and World Politics. He was the founding President of the Australian Society for Quantitative Political Science, and is a member of the executives of the Pacific International Politics Conference and the Asian Political Methodology meeting. He holds a Ph.D. in Political Science (Michigan 2001) and an M.A. in Russian Area Studies (Georgetown 1995).
This course covers the foundations of designing, implementing and analysing basic online survey experiments in political science. It is intended for those with no or limited experience with online survey experiments.
Survey experiments are now widely used in political science, and in other areas of social science. They are a powerful methodological tool that combines causal inference based on randomized experiments and statistical inference from a surveyed sample to a population. Online delivery allows for a range of options for the use of text and images, and delivery to respondents almost anywhere in the world. But they also have important limitations and can be difficult to design and implement well. This course will cover central conceptual and practical topics of experimental design and implementation to equip participants with fundamental knowledge across topics including treatment design, statistical power, pre-registration, ethics, and analysis. In-class examples and homework exercises are provided using the free statistical software platform “R”.
Topics covered
Survey Design:
- The inferential logic of experiments and the potential outcomes framework
- Choosing from different types of treatments (vignettes, conjoint, list experiments)
- Designing experiment to achieve useful results
- Attention and manipulation checks
- Ensuring sufficient statistical power
Implementation:
- Piloting and testing,
- Pre-registration and pre-analysis plans,
- Survey platforms (Qualtrics) and sample providers
- Soft-launch and fielding your survey.
Analysis:
- Data cleaning and checking,
- Hypothesis testing and data visualization in R,
- Confirmatory and exploratory analysis,
- Issues of external validity.
This course is presented as part of the ANU Online Summer School in Political Analysis
Image
|
Pre-Reading (Complete Before Day 1)
- Grady, Christopher. n.d. "10 Things to Know About Survey Experiments." Evidence in Governance and Politics (EGAP). https://egap.org/resource/10-things-to-know-about-survey-experiments/
- Pink, Sophia L., James Chu, James N. Druckman, David G. Rand, and Robb Willer. 2021. "Elite party cues increase vaccination intentions among Republicans," PNAS 118(32). https://www.pnas.org/doi/10.1073/pnas.2106559118
For those interested in conjoint experiments:
Clayton, Katherine, Yusaku Horiuchi, Aaron R. Kaufman, Gary King, and Mayya Komisarchik. [forthcoming] “Correcting Measurement Error Bias in Conjoint Survey Experiments,” American Journal of Political Science. https://gking.harvard.edu/conjointE
Day 1: Concepts and Principles (Monday)
Topics:
- Course overview and introductions
- What is a survey experiment?
- The potential outcomes framework of causality
- Key concepts: surveys, sampling, experiments, randomization
- Treatment effects: ATE, ATT, heterogeneous effects
- Types of survey experiments: Question wording, Vignette, Conjoint, List, Endorsement
- Survey modes: Online, telephone, in-person
- Attention checks and manipulation checks
Reading:
- Grady, "10 Things to Know About Survey Experiments" (EGAP)
- Druckman, J. N., Green, D. P., Kuklinski, J. H., and Lupia, A. 2006. "The Growth and Development of Experimental Research in Political Science," American Political Science Review 100: 627-635. https://doi.org/10.1017/S0003055406062514
Homework:
- Identify a research question you could address with a survey experiment. Write 1-2 paragraphs describing the question and why it matters, and how you would design and experiment to study it.
- Using R: Do step 9 “EXERCISE” in “01_Introduction_to_R” script (heterogeneous treatment effects)
Day 2: The Basics from Start to Finish (Tuesday)
Topics:
- From research question to experimental design
- Power calculations and sample size determination
- Pre-testing and revision
- Pre-registration: Open Science Framework (OSF)
- Ethics and IRB considerations
- Survey platforms: Qualtrics
- Sample providers: PureSpectrum, Prolific, MTurk
- Introduction to R for survey analysis
Reading:
- Gregory Huber and Matthew H. Graham. 2025. “Designing survey experiments.” In Handbook of Experimental Methodology (Elsevier). Volume 1, Issue 1, pp. 283-314. https://doi.org/10.1016/bs.hbem.2025.09.006
- Chen, Yan & Christopher Grady. "10 Things to Know About Pre-Analysis Plans." EGAP. https://egap.org/resource/10-things-to-know-about-pre-analysis-plans/
Homework:
- Draft 3-5 survey questions (including at least one experimental treatment) for your research question. Consider what your outcome variable(s) will be.
- Using R: EXERCISE 1: Power Analysis & EXERCISE 2: Data Manipulation
Day 3: Design and Pre-Analysis Plans (Wednesday)
Topics:
- Moving from theory to hypotheses
- Identifying mechanisms
- Pre-treatment questions and covariates
- Randomization strategies: Between-subjects vs. within-subjects
- Factorial designs
- Treatment design principles: Subtlety is out of place
- Control conditions: Blank control? No control?
- Outcome measurement: Why continuous variables are preferable
- Writing a pre-analysis plan (PAP)
Reading:
- Pink et al. 2021 (focus on research design and methods)
- Gaines, B. J., Kuklinski, J. H., and Quirk, P. J. 2007. "The Logic of the Survey Experiment Reexamined," Political Analysis 15: 1-20.
Homework:
- Draft a brief pre-analysis plan (1-2 pages) for your survey experiment, including hypotheses, treatment conditions, outcome measures, and planned analysis.
- Using R: EXERCISE 3: Treatment Effect Estimation & EXERCISE 4: Heterogeneous Treatment Effects
Day 4: Treatments and Manipulation Checks (Thursday)
Topics:
- Vignette experiments: Design and implementation
- Conjoint experiments: Design and analysis
- Factual manipulation checks
- Attention checks: Types and implementation
- Common problems and solutions
- External validity considerations
Reading:
- Hainmueller, J., Hangartner, D., and Yamamoto, T. 2015. "Validating Vignette and Conjoint Survey Experiments Against Real-World Behavior," Proceedings of the National Academy of Sciences. https://doi.org/10.1073/pnas.1416587112
- Mullinix, K. J., Leeper, T. J., Druckman, J. N., and Freese, J. 2015. "The Generalizability of Survey Experiments." Journal of Experimental Political Science https://doi.org/10.1017/XPS.2015.19
Homework:
- Revise your survey experiment design based on class discussion. Prepare any questions for the final session.
- Using R: EXERCISE 5: Balance Check & EXERCISE 6: Visualization
Day 5: Analysis and Inference (Friday)
Topics:
- Data cleaning and preparation
- Difference-in-means and t-tests
- Linear regression for experimental analysis
- Why linear models often outperform logit/probit for experiments
- Should you include control variables?
- Balance tests: When and why
- Confirmatory vs. exploratory analysis
- Presenting and visualizing results
- Publishing and replication
Reading:
- Arel-Bundock, Vincent, et al. 2024. "Quantitative Political Science Research is Greatly Underpowered," Journal of Politics. https://doi.org/10.1086/734279
In-Class Activity: Hands-on R analysis of Pink et al. replication data
This course will be run in one session per day running over over 5 days.
The course hours are 1.00pm - 3.30pm each day
This course is being held online via Zoom and run on Australian Eastern Daylight Time (GMT +11)
All course materials, including slides, lecture notes, and R scripts, will be provided via the course platform. Example data from Pink et al. (2021) will be used for in-class exercises.
Software Requirements:
- R (download from https://r-project.org )
- RStudio (download from https://posit.co/download/rstudio-desktop/ )
Helpful Resources:
- Introduction to R video: https://vimeo.com/711660732/b2394e5409
- Statistical Inference with R: https://moderndive.com/v2/
- Druckman, J. N. 2022. Experimental Thinking. Cambridge University Press.
- Mutz, Diana C. 2011. Population-Based Survey Experiments. Princeton University Press.
- Druckman, Green, Kuklinski, and Lupia (eds). 2011. Cambridge Handbook of Experimental Political Science. Cambridge University Press.
- Blair, Graeme, Alexander Coppock, and Macartan Humphreys. 2023. Research Design in the Social Sciences: Declaration, Diagnosis, and Redesign. Princeton University Press (with declaredesign software). Access for free here: https://press.princeton.edu/books/paperback/9780691199573/research-design-in-the-social-sciences
Additional Readings
- Imbens, Guido W., and Donald B. Rubin. 2015. Causal Inference in Statistics, Social, and Biomedical Sciences: An Introduction. Cambridge University Press.
- Fowler, Floyd J. 2014. Survey Research Methods. 5th Edition. Sage.
- Groves, Robert M., et al. 2009. Survey Methodology 2nd Edition. Wiley.
This is a non-graded professional development course. Active participation is expected, including:
- Completion of daily homework assignments
- Participation in class discussions
- Engagement with in-class R exercises
Thank you very much for your detailed feedback and for taking the time to review my materials so thoroughly. I truly appreciate the comments you added to the documents, as well as the written responses to my questions. Your guidance came at exactly the moment when I needed it most, and it has been immensely helpful for refining both components of my experimental design. It has been my great fortune to take your course this week. I learned a tremendous amount from your lectures and discussions, and your willingness to engage with each student’s project made the experience especially valuable.