On Wednesday, September 18th, 2019 as part of IBM Research's AI week we will be hosting the first workshop on Neuro-Symbolic Computation and Machine Common Sense (NSC-MCS).

Neuro-symbolic methods are relatively new in the DL community and at present, they are largely driven by the findings at the intersection of the fields of child-psychology, generative modelling and neuroscience suggesting symbol manipulation may be at the core of human common sense. This workshop will bring researchers from the aforementioned fields together with the highly diverse set of researchers in the wider fields of representation learning and reasoning in order to advance discussion and promote collaborations across its various subdomains, with a special focus on Neuro-symbolic AI and Neuro-Symbolic Computing and Machine Common Sense.


9:00-9:15 AMWelcome and Opening Remarks
09:15-09:45 AMInvited talk: T.S. Jayram
"Can External Memory Help in Visual and Symbolic Reasoning?"
09:45-10:15 AMInvited talk: Tomer Ullman
"Computational models of core intuitive physics (and some psychology)"
10:15-10:30 AMCoffee break
10:30-11:00 AMInvited talk: Jiajun Wu
"Neuro-Symbolic Concept Learning"
11:00-11:30 AMInvited talk: Kate Saenko
"Modular and Generalizable Deep Learning"
11:30-11:45 AMPanel (Q&A) with all the speakers
11:45-12:45 PM Lunch break
12:45-1:30 PMThreeDWorld tutorial (Jeremy Schwartz and Seth Alter)
1:30-2:15 PMThreeDWorld use cases (Ronald Alvarez, Martin Schrimpf, James Traer)
2:15-2:45 PMInvited talk: Rebecca Saxe
2:45-3:15 PMInvited talk: Josh Tenenbaum
3:15-3:30 PMCoffee break
3:30-4:00 PMInvited talk: Dan Yamins
4:00-4:30 PMInvited talk: Akash Srivastava
"Learning Simulators with Disentangled Representations"
4:30-4:45 PMPanel (Q&A) with all the speakers

Invited Speakers

Daniel Yamins

Stanford University
Daniel Yamins is a cognitive computational neuroscientist at Stanford University, where he's an assistant professor of Psychology and Computer Science, a faculty scholar at the Wu Tsai Neurosciences Institute, and an affiliate of the Stanford Artificial Intelligence Laboratory. His research group focuses on reverse engineering the algorithms of human cognition, both to learn both about how our brains work and build more effective artificial intelligence systems. He is especially interested in how brain circuits for sensory information processing and decision making arise via the optimization of high-performing cortical algorithms for key behavioral tasks. He received his AB and PhD degrees from Harvard University, was a postdoctoral research at MIT, and has been a visiting researcher at Princeton University and Los Alamos National Laboratory. He is a recipient of an NSF Career Award, the James S. McDonnell Foundation award in Understanding Human Cognition, the Sloan Research Fellowship, and is a Simons Foundation Investigator.
Rebecca Saxe is an associate investigator of the McGovern Institute and the John W. Jarve (1978) Professor in Brain and Cognitive Sciences at MIT. She obtained her Ph.D. from MIT and was a Harvard Junior Fellow before joining the MIT faculty in 2006. She was awarded tenure in 2011. Saxe was chosen In 2012 as a Young Global Leader by the World Economic Forum, and she received the 2014 Troland Award from the National Academy of Sciences. Rebecca Saxe studies human social cognition, using a combination of behavioral testing and brain imaging technologies.

Kate Saenko

MIT-IBM Watson AI Lab /
Boston Univeristy
Kate Saenko is an Associate Professor of Computer Science at Boston University and a consulting professor for the MIT-IBM Watson AI Lab. She leads the Computer Vision and Learning Group at BU, is the founder and co-director of the Artificial Intelligence Research (AIR) initiative, and member of the Image and Video Computing research group. Kate received a PhD from MIT and did her postdoctoral training at UC Berkeley and Harvard. Her research interests are in the broad area of Artificial Intelligence with a focus on dataset bias, adaptive machine learning, learning for image and language understanding, and deep learning.
Josh Tenenbaum is a professor of Computational Cognitive Science in the Department of Brain and Cognitive Sciences at MIT, and a member of the Computer Science and Artificial Intelligence Laboratory (CSAIL). He studies learning, reasoning and perception in humans and machines, with the twin goals of understanding human intelligence in computational terms and bringing computers closer to human capacities.

T.S. Jayram

IBM Research, Almaden
T.S. Jayram (Jayram Thathachar) is a research scientist in the Machine Intelligence Team at IBM Research AI. His work in deep learning focuses on cognitively inspired memory-augmented neural network architectures with a view towards visually grounded language learning and reasoning with memory. This was inspired partly by his past work on the role of memory in theoretical computer science where he has made fundamental contributions to both algorithms in data science as well as demonstrating their limitations using information theory. For these contributions, which includes starting two new areas namely, Information Complexity in theoretical computer science and Index Coding in information theory, he was awarded the IBM Research Accomplishment (Outstanding) in the Science category (along with David Woodruff) in 2012. He was also an invited speaker at the ACM PODS conference in 2010. In the past he has worked on time-space tradeoffs for branching programs; part of this work won the Machtey Award in FOCS 1994.

Akash Srivastava

MIT-IBM Watson AI Lab
Akash Srivastava is a research scientist at the MIT-IBM lab in Cambridge, MA where he works on building machines with child-like common-sense and intuitive physics using probabilistic modelling and Bayesian inference. Before this, he was a PhD student in the Informatics Forum at University of Edinburgh where he worked with Dr Charles Sutton and Dr Michael U. Gutmann on variational inference for generative models using deep learning.

Tomer Ullman

Tomer Ullman is a cognitive scientist interested in common-sense reasoning, and building computational models for explaining high-level cognitive processes and the acquisition of new knowledge by children and adults. In particular, he is focused on how children and adults come to form intuitive theories of agents and objects, and providing both a functional and algorithmic account of how these theories are learned. Tomer D. Ullman is a postdoctoral associate in the Computational Cognitive Science group at MIT, and the Lab for Developmental Studies at Harvard. He studies people's common-sense reasoning about physics and psychology, using computational models and behavioral experiments with children and adults. His research is funded by the Center for Brains, Minds and Machines.

Jiajun Wu

Jiajun Wu is a Ph.D. student in Electrical Engineering and Computer Science at Massachusetts Institute of Technology. He received his B.Eng. from Tsinghua University in 2014. His research interests lie in the intersection of computer vision, machine learning, robotics, and computational cognitive science. His research has been recognized through the IROS Best Paper Award on Cognitive Robotics and fellowships from Facebook, Nvidia, Samsung, Baidu, and Adobe, and his work has been covered by major media outlets including CNN, BBC, WIRED, and MIT Tech Review.

ThreeDWorld Tutorial

Researching Intelligence using Virtual Worlds

ThreeDWorld (TDW) is a simulation framework that uses state-of-the-art video game technology to generate photo-realistic scenes in a virtual world. TDW’s flexible, general design allows researchers to collect next-generation experimental data and generate large-scale datasets for training AI systems with complete control over data generation and full access to all associated generative parameters.


Seth Alter
Seth Alter is the back-end programmer and API developer for TDW. He is an experienced game designer and developer, specializing in python, Unity 3D development and procedural content creation. He is also the co-organizer of Boston Indies (a local monthly meetup group for independent game developers).
Jeremy Schwartz
Jeremy Schwartz is the Project Lead for TDW. He is a veteran interactive technology developer and CGI production manager, who has worked for many years in the video game industry. While at Electronic Arts and Acclaim Entertainment he ran studio operations for motion capture and audio/video post-production, and managed 3D animation and tool-development teams. Jeremy also spent several years as a technology analyst with Forrester Research, and has worked for design consultancies and interactive agencies developing both digital and hybrid physical/digital interactive experiences.

ThreeDWorld Use Cases

Use Case I

The aim of our project is to understand the underlying neural computations involved in intuitive physics, i.e., common sense about physical interactions. We present video stimuli of physical events unfolding over time to rhesus macaques and compare the brain response (with fMRI and fMRI-guided electrophysiology) to intentional interactions versus purely physical interactions. Crucially, by using TDW to create parametric animations we will achieve unprecedented control over relevant physical properties (like magnitude and direction of force), which we can subsequently decode from neural activity using computational models.


Ronald Alvarez
Ronald is an Electrical Engineering and Mathematics student at Mercer University. He works on developing computational models of planning and decision-making with the Tenenbaum lab at MIT, investigating neural correlates of intuitive physics with the Freiwald lab at Rockefeller University, and control theory with the Thitsa lab at Mercer.

Use Case II

Computer Vision researchers have limited control over various dimensions such as object size, angle, etc. in large-scale photorealistic datasets. Using TDW, we can generate extremely flexible and fully controlled datasets that allow for training models with transfer performances approaching those of photographic datasets such as ImageNet.


Martin Schrimpf
Martin completed his Bachelor and Master degrees in Software Engineering in Germany at TUM, LMU and UNA, conducting his Master’s Thesis on recurrent visual processing at Harvard with Gabriel Kreiman. He has founded the non-profit startup Integreat and worked at Salesforce Research on deep learning for NLP with Richard Socher before joining MIT’s Brain and Cognitive Sciences program for his PhD. There, he is advised by James DiCarlo and works on evaluating and building neural network models of brain processing.

Use Case III

Humans can readily interpret the sounds made when objects collide, scrape, and clatter against each other. We know little about how humans can make such inferences, nor can we replicate their success with machine hearing algorithms. We are building a perceptually inspired generative model of contact sounds, which can be interfaced with TDW to simulate sounds made by everyday objects. We can use these sounds to study human perception of natural sounds.


James Traer
James was formally trained as a physicist, and during his PhD work (Univ. of California, San Diego) he studied sound generation by ocean waves and developed algorithms to map the ocean and the Earth's crust with ambient sound. In 2013 he joined the McDermott Laboratory for Computational Audition (MIT) as a post-doctoral researcher. He has since studied how humans infer physical attributes of the world (e.g. source distance, room size, object material, force of impact) from the sounds made by everyday objects as they collide, scrape and deform.


Registration to the Neuro-Symbolic Computing and Machine Common Sense workshop is open!

In order to register please use Eventbride website.

Organizing Committee

IBM Research, MIT-IBM Watson AI Lab, Cambridge, MA, USA
IBM Research, MIT-IBM Watson AI Lab, Cambridge, MA, USA
IBM Research, MIT-IBM Watson AI Lab, Cambridge, MA, USA
IBM Research, Almaden Research Center, San Jose, CA, USA
IBM Research, Thomas J. Watson Research Center, Yorktown Heights, NY, USA
IBM Research, Almaden Research Center, San Jose, CA, USA
IBM Research, MIT-IBM Watson AI Lab, Cambridge, MA, USA
IBM Research, Almaden Research Center, San Jose, CA, USA
IBM Research, MIT-IBM Watson AI Lab, Cambridge, MA, USA


MIT Samberg Conference Center
Chang Building (E52), 50 Memorial Drive, Cambridge, MA 02142
7th floor, Salon M

Samberg center is located in Kendall square and easily accessible by public transportation. The Samberg Center is a short walk from the Kendall/MIT stop on the red line. We highly encourage using public transportation to get here.


Dan Gutfreund, dgutfre(ta)us(tod)ibm(tod)com
Christian Muise, christian(tod)muise(ta)ibm(tod)com
Akash Srivastava, akash(tod)srivastava(ta)ibm(tod)com