Cogsci 2020 W6: Workshop on Scaling Cognitive Science
July 29th, 2020
CogSci 2020 Conference Banner

The proliferation of web-connected devices has presented significant opportunities and challenges to cognitive science — opportunities in that cognitive scientists can collect data relevant to human cognition orders of magnitude faster than before, addressing questions that were otherwise impossible to address; and challenges in that cognitive scientists require new infrastructure to collect these data and new methods to analyze them once collected.

This workshop brings together cognitive scientists who are at the forefront of these opportunities and challenges in scaling cognitive science, along with cognitive scientists who would like to be, to engage in a day of interactive exchange and development of ideas related to scaling cognitive science. Each presentation addresses a different opportunity or challenge. One set of presentations highlights opportunities. A second set highlights challenges of statistical analysis and data collection. In a break-out session, attendees address these opportunities and challenges head on.

CogSci Website >>>

Start Time (EDT) Presentation
9am Tom Griffiths on Scaling Cognitive Science
9:30am Joshua Hartshorne on Scaling Language Studies
10am Mayank Agrawal on Scaling Moral Decisions
10:30am Break
10:45am Andrea Simenstad on Scaling the Zooniverse
11:15am Jordan Suchow on Scaling Experiment Design
11:45am Todd Gureckis on Scaling Brain Data via Behavior
12:15pm Lunch
12:45pm Joshua Peterson on Scaling Choices and Categories
1:15pm Lauren Rutter on Scaling via Mobile Applications
1:45pm James Houghton on Scaling Social Cognition
2:15pm Break
2:30pm Platform Advertisements on Dallinger,, Pushkin, and Zooniverse. Click here to go to forum
3:15pm Breakout Sessions on Open Questions - Big Data with Tom Griffiths and Todd Gureckis, Big Groups with Jordan Suchow and James Houghton, Citizen Science with Andrea Simenstad, Lauren Rutter, and Josh Hartshorne

Tom Griffiths
Tom Griffiths on Scaling Cognitive Science, 9-9:30am

Thomas Griffiths is a professor of psychology and computer science at Princeton University, where he leads the Computational Cognitive Science group. Griffiths’ research focuses on developing mathematical models of higher-level cognition and understanding the formal principles that underlie our ability to solve the computational problems we face in everyday life.

Griffiths will introduce the workshop by discussing opportunities and challenges associated with scaling cognitive science. Griffiths will discuss how novel approaches to data collection and analysis — particularly “big data” — can change psychological research.

Joshua Hartshorne
Joshua Hartshorne on Scaling Language Studies, 9:30-10am

Joshua Hartshorne is an assistant professor of psychology at Boston College, where he directs the Language Learning Laboratory Hartshorne is interested in understanding what allows humans, but not current machines, to learn language — and why it is that children, despite their salient limitations of both cognition and experience, are so much more successful at language learning than adults. Hartshorne is particularly interested in applying new and emerging methods (such as computational modeling and crowdsourcing) to core problems in the language sciences.

Hartshorne will introduce the workshop by discussing opportunities and challenges associated with scaling cognitive science. Hartshorne will discuss these challenges in the context of running, a web-based research laboratory that has tested hundreds of thousands of participants in a variety of experiments on language and cognition. Hartshorne will also discuss Pushkin, a platform for running large-scale online experiments.

Mayank Agrawal
Mayank Agrawal on Scaling Moral Decisions, 10-10:30am

Mayank Agrawal is a doctoral student in psychology and neuroscience at Princeton University. Agrawal is interested in building formal computational models to understand how humans learn and make decisions.

Agrawal will discuss scaling studies of moral decision making. Standard methods of exploratory data analysis are vulnerable to noise in large datasets. To combat this problem, he proposes a methodology called Scientific Regret Minimization, which focuses on minimizing errors with respect to the data that are predictable. Agrawal applies this methodology to large datasets in the domains of moral reasoning and economic choice, and demonstrates how this approach helps build powerfully predictive (and interpretable) models and identify interesting new phenomena.

Andrea Simenstad
Andrea Simenstad on Scaling the Zooniverse, 10:45-11:15am

Andrea Simenstad is a developer with the Zooniverse project at the University of Minnesota, where she collaborates with researchers, museum professionals, volunteers, and other developers to create crowdsourced citizen-science research projects with Zooniverse. Simenstad has a B.A. in cognitive science from Carleton College.

Simenstad will discuss Zooniverse. Zooniverse is the world’s largest and most popular platform for crowdsourced citizen-science research, with more than 1,000,000 volunteers. The goal of the Zooniverse organization is to enable research that would not be possible, or practical, otherwise. Zooniverse research results in new discoveries, datasets useful to the wider research community, and many publications.

Jordan Suchow
Jordan Suchow on Scaling Experiment Design, 11:15-11:45am

Jordan Suchow is an assistant professor at Stevens Institute of Technology, where he runs the Cognition Lab, which focuses on computational models of human learning, memory, and decision-making. As part of DARPA’s Next Generation Social Science program, Suchow led a team to develop Dallinger, an online experimental platform for running behavioral and social science experiments at the scales necessary to understand emergent social phenomena.

Suchow will discuss Dallinger and associated techniques for experiment design at scale.

Todd Gureckis
Todd Gureckis on Scaling Brain Data via Behavior, 11:45am-12:15pm

Todd Gureckis is a professor at New York University, where he runs the Cognition and Computation lab. The goal of the lab’s research is to better understand the memory, learning, and decision processes which allow humans to carry out intelligent and adaptive behaviors.

Gureckis will discuss scaling brain data through behavior. One goal of computational cognitive neuroscience is to infer the relationship between latent variables in a cognitive model and particular features of brain data (e.g., fMRI or EEG). Typically, the precision of this relationship is constrained by the amount of brain data that can be collected (e.g., the number of subjects in a fMRI study). In this talk, Gureckis will leverage ideas from the statistics literature on inference with missing data and hierarchical modeling to explore how one can potentially learn more about a neuroimaging signal by collecting only additional behavioral data.

Joshua Peterson

Joshua Peterson on Scaling Choices and Categories, 12:45-1:15pm

Josh Peterson is a postdoctoral fellow at Princeton University, where he studies the relationship between the representations learned by deep neural networks and those learned by people.

Peterson will discuss scaling research on decision making and categorization. Technological advances have made it possible to observe human behavior at an unprecedented scale, and the massive datasets that result provide two exciting new tools for cognitive modeling. First, such datasets serve as a test of the generalizability of our models and theories and therefore have the potential to improve them. Second, they offer the essential fuel for training machine learning models. Peterson will present case studies within two of the oldest modeling paradigms in cognitive science: categorization and decision making.

Lauren Rutter
Lauren Rutter on Scaling via Mobile Applications, 1:15-1:45pm

Lauren Rutter is a Associate Research Scientist in the Department of Psychological and Brain Sciences at Indiana University. Rutter studies basic cognitive and affective processes that influence the etiology and maintenance of internalizing psychopathology across the lifespan.

Rutter will discuss mobile applications for scaling mental- health assessment as well as large-scale citizen-science experiments. Rutter will demonstrate that digital technologies provide benefit to a variety of clinical populations and participants from the general public to generate scientific insights that would not have been possible with traditional models.

James Houghton
James Houghton on Scaling Social Cognition, 1:45-2:15pm

James Houghton is a doctoral candidate at the MIT Sloan School of Management. He explores the mechanisms and outcomes of social contagion using a combination of data science and massive online experimentation, informed by rigorous simulation-based theory building.

Houghton will discuss the Empirica platform in the context of his research on social contagion. Social contagion research has for decades explored the effect of social network structure on the adoption of a single belief or practice. However, new theoretical models show that interaction between diffusants has a strong influence on their patterns of adoption. Houghton will present an experimental design called the ‘detective game’ for studying the diffusion of interacting beliefs in a controlled setting, explaining how an online laboratory experiment handles the confounds of pre-existing biases and meanings while cost-effectively achieving the scale necessary to draw statistical inferences.