Experiments on the web¶

by Christophe, Sander, & Lore

Retreat 2020

Intro¶

• Our lab already has a history in doing online experiments (check L-POST for an example).

• So far most of those tests were developed by Christophe (or his predecessor).

• Indeed, in general experiments in the browsers are more complex than regular offline experiments: Christophe's intro to the logic of the web

• However, increasingly tools are becoming available that facilitate experiment building for the web for non-professional web-builders: I will introduce a few

• Browsers have advanced a lot in capabilities, allowing very complex, dynamic environments and tasks (like video games). (examples)

• Online recruitment platforms have become easy to use: Lore and I will introduce two popular options

Why can’t we just ask Christophe to continue to do this for us?¶

• That’s still a possibility but experimenters generally need a lot of control over things and go through iterative development
• Faster (Christophe will have a waiting list)
• Understanding how it works is important when conceiving the design and when interpreting your data

Note: we focus on actual experiments only here, if you need options for questionnaire-only studies:¶

• FormR (Eline has experience with this)
• LimeSurvey
• Qualtrics
• Experiment factory (paper)
• Google forms

Why experiments on the web?¶

• Larger samples (bigger pool), easy recruitment (potentially more diverse, more tunable to study).
• Reaching less accessible populations (cf. autism, social anxiety, visually impaired populations,...)

• Inexpensive though it is recommended to give ethical rewards
• Increased anonymity, cf. sensitive subjects (but depersonalization: cf. low wage workers)
• No face-to-face interactions -> standardization & clarity of instructions. Reducing undocumented experimenter bias (e.g. additional instructions, subtle reinforcing cues, repeating a question) and demand characteristics

• Easy (direct) replication
• Easy sharing, ‘forking’: Increasing reproducibility & transparency
• Faster data collection (many people do your study at the same time)

Advantages (contd.)¶

• Phone or tablet-based surveys (cf. experience sampling) or experiments
• Leverage the ideas/tools of an immense web development community may help you develop better/more engaging experiments (for larger samples, more data per participant)
• You can still run the exact same experiment in the lab as well!
• Learn JS/HTML/CSS/db (valuable, transferable skill outside of academia)

Disadvantages:¶

• Attention & motivation (e.g. participants might be listening to music, watching tv/youtube/social media at the same time, or be interrupted by others): fierce online competition for attention: increased probability of drop-out or low-quality data. (Partial) solutions:
  • Recording how often participants change windows/take breaks might help clean data
  • Catch questions could be useful
  • Optimize reward scheme
  • More thought into how to engage participants (actually a plus)
  • Forcing fullscreen
• Low social control & no face to face interactions
  • Checking participants' understanding of the task
• Honesty & naiveness (not necessarily worse than lab, cf psych students): excluding participants based on participation in previous studies

Disadvantages (contd.)¶

• Contextual factors are not (easily) controlled: noise, lighting, distractions
• Hardware/software (browser) heterogeneity and feature support (old versions, old hardware) (Note: speed of internet is usually not an issue: once (down)loaded, it runs locally). Avoiding variability:
  • Enforcing versions, browsers, excluding tablets/mobile (when desired)
  • Checking refresh rates (checking other browser capabilities)
  • Preloading media
  • Size on screen: matching real-life objects, rule of thumb, participants can make matching between stimuli themselves
• Data analysis: dealing with extreme response times because of lapses, distraction, hardware/software issues (outliers)
• Learn the new environment: (basics of) JS/HTML/CSS/(db), web logic

So when are web experiments (un)suitable?¶

• Despite the various problems direct comparisons often show similar quality data
• Many classic psychology findings replicate with reasonably high fidelity
• When larg(er) samples are needed (e.g. individual differences)
• Research that relies on fewer participants should still aim for the highly controlled lab environment
• In practice: will depend on your research question(s) and methods
• Obviously unsuitable when using physiological measures, fMRI/MEG/EEG (though who knows in the future)
• Even psychophysical tasks with brief exposure durations may be possible according to recent benchmarks
• Comparison papers suggest reaction times are often slower/noisier but that 1) usually doesn't compromise comparisons between conditions, 2) can be (partly) compensated by larger samples and/or trial numbers
• trade-off between technical complexity/time investment (when learning/designing/implementing) vs ease (& speed) of data collection

Benchmarking and direct comparison studies¶

• Online psychophysics: Reaction time effects in cognitive experiments
• Online versus offline: The Web as a medium for response time data collection
• Is the Web as good as the lab? Comparable performance from Web and lab in cognitive/perceptual experiments
• Presentation and response timing accuracy in Adobe Flash and HTML5/JavaScript Web experiments
• Online Timing Accuracy and Precision: A comparison of platforms, browsers, and participant's devices
• The timing mega-study: comparing a range of experiment generators, both lab-based and online

Switch to: Christophes intro to the logic of the web...¶

Existing libraries & frameworks to build/deploy your experiments online¶

Many tools exist, most are pretty recent. Some are builders (graphical point-and-click interface, wysiwyg), some also provide the coding libraries for more flexibility. Some are client-side only ('front-end'), others (usually commercial ones) provide integrated solutions (server space for deploying and storing data). Some 'natural selection' of the field still has to happen. New, more succesful tools may pop up.

This can be overwhelming, but it is intrinsic to working on new frontiers (+ competition is good in software). Don't take my choices as gospel. Choose based on your needs. For me the tool needs to be

• maximally flexible (allow me to use custom javascript code to implement unconventional structure, custom, dynamic/interactive stimuli,...),
• free of cost,
• open source,
• with some track record,
• and good documentation/examples.

Vanilla js from scratch: most flexible but time-consuming, technically challenging. Using "canvas": an html element that allows drawing anything (including complex, animated stimuli) using javascript (js). Only for experienced (or ambitious) web developers.

I'm putting aside (for now):¶

Integrated (commercial) frameworks with more limited experiment building capabilities (but they might be sufficient for your needs)

• Psytoolkit (paper): low threshold (no js knowledge needed), but very limited, bitmap-based functionality. Aimed at cognitive psychologists. Many classic experiments and questionnaires already implemented and usable through (free) web interface.
• Labvanced: commercial tool, integrated development environment, no js needed, bitmap or questionnaire-based. Includes deploying & data storage (you don't need your own server). No js library
• Gorilla: commercial builder tool (examples). No js library.

Real race is between the following three¶

These are free, open source js libraries and all include a builder. They provide a timeline/scheduler and tools for controlling the flow of an experiment and an (extensible) set of stimuli.

• lab.js (builder)
• jsPsych: (builder). Also: tools for R/Rstudio (jspsychr; jaysire).
• psychojs: builder (same as psychopy builder). Support for building online experiments is under development: (features currently available). Has the psychopy reputation/resources behind it. Will eventually probably be more advanced in stimulus presentation and recording data. Extra benefit: structure is similar to psychopy.

Frameworks¶

Agnostic to frontend experiment building (js) tools.

• Experiment factory (paper; Poldrack lab): library of reusable experiments to increase reproducibility.
• Psiturk: framework (dashboard) to manage connection to Mturk. Example experiments available.
• Pavlovia: based on git, but straight uploading from psychopy builder is possible (still under development) see intro (Peirce) Works with jspsych as well.

A brief intro to jspsych¶

• jspsych: Modular structure.

• adding js code: walkthrough (editor suggested vscode [ms, open source, many tools for web developers])

• Github to host/serve (static) pages -> show + why (iterative development)?

• sending data to server (https://psytests.be/tests/curimoon/admin/)

• debugging in console (right-click, inspect, console)

• Give explicit instructions and show warnings about closing the page, going back in the browsers, etc.

• Consider measures to avoid bots (especially for Mturk)

• Test your experiment on different browsers/screens.

• Pilot extensively before moving to larger pools

https://www.wooji-juice.com/blog/javascript-article.html

• Ethical considerations (data protection?) Informed consent/debriefing
• http://blog.thefirehoseproject.com/posts/why-is-javascript-so-hard-to-learn/

References¶

• Conducting perception research over the internet: a tutorial review
• The Experiment Factory: Standardizing Behavioral Experiments
• Law and Psychology Grows Up, Goes Online, and Replicates
• Retention of participants recruited to a one-year longitudinal study via Prolific
• An Evaluation of Amazon’s Mechanical Turk, Its Rapid Rise, and Its Effective Use
• Prolific.ac—A subject pool for online experiments
• Beyond the Turk: Alternative platforms for crowdsourcing behavioral research
• The state of web-based research: A survey and call for inclusion in curricula.
• The pitfall of experimenting on the web: How unattended selective attrition leads to surprising (yet false) research conclusions.
• Should we trust web-based studies? A comparative analysis of six preconceptions about Internet questionnaires.
• Running experiments on Amazon Mechanical Turk.
• Amazon's Mechanical Turk A New Source of Inexpensive, Yet High-Quality, Data?
• Using Mechanical Turk to Study Clinical Populations
• Evaluating Amazon's Mechanical Turk as a tool for experimental behavioral research.