A Primer on Field Experiments For Qualitative Researchers: or How I Learned to Stop Worrying and Love experiments…

4-5 November 2022

Elisa Volpi, coordinatrice CUSO

Prof. Paolo Spada (University of Southampton)

Objective: The Objective of this intensive seminar is to provide a preliminary overview of experimental methods in social science for students with no or limited background in quantitative methods. The class will focus on the situations in which experimental methods are appropriate within a more general multi-method approach with a particular focus on integrating experimental research during fieldwork. The class is tailored for PhD students that are going to conduct research in the field aimed at collecting primary data (ethnography, participants' observations, interviews, surveys, focus groups etc.) in the upcoming months, but might be of interest to any students that want to explore randomized controlled trials for the first time. Particular emphasis will be given to the relationship between in depth qualitative research and experiments, the epistemology of experiments, ethics of experiments, and the craft of implementing simple experiments that require little mathematical knowledge.

Skills that will be acquired: By the end of the class the students will be able to design and analyze their first experiment, present experimental research and prepare an ethical application to get an experiment approved.

Prerequisite: The seminar is designed to be a primer and thus does not require any statistical knowledge to participate. We will use middle school level mathematics, averages, and excel.

Day 1 agenda (Friday, November 4th) 9:30-18:30

· Introduction

· What is an experiment? Definitions and key assumptions. Types of experiments

· Laboratory 1: implementing a simple experiment

Break

· Laboratory 2: how to present an experimental design (skim the assigned paper, we will ask you to present them)

· The epistemology of experiments: are RCT positivists?

· Recognizing opportunities for field experiments while preparing/conducting fieldwork

Day 2 agenda (Saturday, November 5th) 10:00-17:00

· Experimental problems and their solutions:

o Non-compliance

o Spillover effects

o Clustering

· Laboratory 3: can I do an experiment during my fieldwork?

Break

• The ethics of field experiments
•  Devil Advocate Game: applying for ethics for an unethical experiment
• Discussion & feedback

Genève

PREPARATION BEFORE THE SEMINAR

Watch videos/read articles about experiments that challenge our ethics or created a scandal (we are going to use this in the Devil Advocate Game asking you to fill an ethics application for one of these experiments)

1) Milgram experiment https://www.youtube.com/watch?v=xOYLCy5PVgM

2) The recent case of an audit experiment in the UK that caused a commotion www.bbc.com/news/uk-politics-56196967

3) GOTV experiment scandal in Montana https://www.nytimes.com/2014/10/29/upshot/professors-research-project-stirs-political-outrage-in-montana.html

4) Oxford prison experiment https://www.youtube.com/watch?v=IRR7CwdHxUE

5) Blue eyes/brown eyes exercise (not an experiment, but interesting to discuss, and an example of usage of confederates in an inoculation treatment) https://www.youtube.com/watch?v=DwhVnPTi-jk

Skim the following 7 articles. During Laboratory 2 we will ask participants to present one of the paper to the class. There is no need to understand the statistical analysis, read intro, experimental design section and conclusions, and try to figure out the way they did the experiment in practice.

[EXPERIMENT ON DELIBERATION IN A CLASS] Spada P. & Vreeland J., (2013) "Who Moderates the Moderators? The Effect of Non-neutral Moderators in Deliberative Decision Making", Journal of Public Deliberation 9(2). doi: doi.org/10.16997/jdd.165 http://www.publicdeliberation.net/jpd/vol9/iss2/art3/

[EXPERIMENT DURING FIELDWORK USING CARS!] Fried, B.J., Lagunes, P. and Venkataramani, A., 2010. Corruption and inequality at the crossroad: A multimethod study of bribery and discrimination in Latin America. Latin American Research Review, pp.76-97. (https://papers.ssrn.com/sol3/papers.cfm?abstract_id=1301010)

[EXPERIMENT USING RADIOS] Paluck, E.L., 2009. Reducing intergroup prejudice and conflict using the media: a field experiment in Rwanda. Journal of personality and social psychology, 96(3), p.574. https://static1.squarespace.com/static/5186d08fe4b065e39b45b91e/t/51e2ba30e4b084ee7b28f499/1373813296578/Paluck+2009+JPSP+Rwanda.pdf

[AUDIT EXPERIMENTS ON PUBLIC OFFICIALS] Spada, P. and Guimarães, F.D.S., 2013. Investigating Elite Behavior through Field Experiment in Brazil: do candidates answer more to core or swing voters?. Brazilian Political Science Review, 7, pp.56-73.https://www.scielo.br/j/bpsr/a/QcPbLP344FVsnLjRwYVz3Xd/abstract/?lang=en

[EXPERIMENTS IN SCHOOLS TO PREVENT BULLYSIM] Paluck, E.L., Shepherd, H. and Aronow, P.M., 2016. Changing climates of conflict: A social network experiment in 56 schools. Proceedings of the National Academy of Sciences, 113(3), pp.566-571. https://www.pnas.org/doi/pdf/10.1073/pnas.1514483113

[EXPERIMENTS DURING ELECTIONS GOTV]: Gerber, A.S. and Green, D.P., 2000. The effects of canvassing, telephone calls, and direct mail on voter turnout: A field experiment. American political science review, 94(3), pp.653-663. https://www.jstor.org/stable/2585837#metadata_info_tab_contents

 [JPAL STYLE EXPERIMENTS IN COLLABORATION WITH A LARGE EVALUATION PROJECT] Olken, B.A., 2007. Monitoring corruption: evidence from a field experiment in Indonesia. Journal of political Economy, 115(2), pp.200-249. https://www.journals.uchicago.edu/doi/full/10.1086/517935?casa_token=We9bGfJIKu4AAAAA%3AURUB-RqsxeldgpDTUXPBekU_Vn1ePi67Y3wiUG64j26ACFoiFOoHp_Gr7dLhBWlfmfyfCrGWxbI

THE SEMINAR IS BASED ON THE FOLLOWING BOOKS AND ARTICLES (I suggest you read them after the seminar if you want to deepen your knowledge)

Gerber, A.S. and Green, D.P., 2012. Field experiments: Design, analysis, and interpretation. WW Norton.

Teele, D.L. ed., 2014. Field experiments and their critics: Essays on the uses and abuses of experimentation in the social sciences. Yale University Press.

Bonell, C., Moore, G., Warren, E. and Moore, L., 2018. Are randomised controlled trials positivist? Reviewing the social science and philosophy literature to assess positivist tendencies of trials of social interventions in public health and health services. Trials, 19(1), pp.1-12.

15