Francois Chollet – On the Measure Of Intelligence

And this is the machine learning street talk episode where Dr. Tim Scarfe, Yannic Kilcher and Connor Shorten covered François Chollets “On the Measure of Intelligence” paper. Chollet thinks that deep learning methods are great for pattern recognition but not the route to AGI. Generalisation comes from a high level of abstraction and reasoning capability. He advocates strongly that we need to start looking at program synthesis methods. He created the ARC dataset and Kaggle challenge to test developer-aware generalisation and created a formalism for measuring intelligence as a function of generalisation difficulty and priors.

00:00:00 MAIN SHOW FLASHY INTRO
00:09:51 SHOW STARTS
00:11:21 GENERALISATION LEVELS
00:14:21 THE G FACTOR
00:22:20 INCLUDING THE CONTEXT OF INTELLIGENCE i.e. Creators, society, evolution
00:26:51 DERMGAN PAPER – GANS TO HELP US MODEL KNOWN UNKNOWNS(?)
00:37:41 WOZNIAK COFFEE CUP vs AlphaGo and broad intelligence
00:43:11 PRIORS, CORE KNOWLEDGE (DONT MISS THIS!)
00:46:31 MULTI TASK BENCHMARKS
00:47:01 ARC CHALLENGE (DONT MISS!)
00:48:51 LEG AND HUTTER, UNIVERSAL INTELLIGENCE
00:54:21 CHOLLETS formalism of intelligence
01:02:17 SPARSE FACTOR GRAPH TO LEARN RELATIONSHIPS
01:03:31 HOW SMART IS ALPHA GO, DEVELOPER AWARE GENERALISATION
01:04:41 AUTOML ZERO
01:05:41 THE EXTENDED MIND
01:12:41 ARC CHALLENGE 2
01:20:21 Hofstadter’s string analogy problem
01:22:31 HOW WOULD WE SOLVE ARC? (DONT MISS!)
01:34:31 META LEARNING AND PROGRAM SYNTHESIS (DONT MISS!)
01:37:21 SIMPLEST SOLUTION TO ARC, CHOLLET MAKES IMPLICIT UNSPOKEN ASSUMPTIONS?
01:40:21 DNNS ARE GLORIFIED HASH TABLES SKIT
01:43:31 MORE ARC CONVERSATION, RULE FINDING, GAN solution
01:47:11 REDDIT COMMENTS
01:51:51 COMMENT RE:MLST FROM REDDIT! (FUNNY)
01:55:31 REDDIT Q continued- meta learning, consciousness, alphago
02:14:51 LOOKING AT KAGGLE SOLUTIONS
02:16:41 BACK TO REDDDIT COMMENTS
02:17:51 BUILDING A GENERATIVE MODEL
02:23:51 FINAL TAKES ON PAPER
02:32:31 TIM’S FINAL TAKES ON CHOLLET

Paper: https://arxiv.org/abs/1911.01547
Abstract:
“To make deliberate progress towards more intelligent and more human-like artificial systems, we need to be following an appropriate feedback signal: we need to be able to define and evaluate intelligence in a way that enables comparisons between two systems, as well as comparisons with humans. Over the past hundred years, there has been an abundance of attempts to define and measure intelligence, across both the fields of psychology and AI. We summarize and critically assess these definitions and evaluation approaches, while making apparent the two historical conceptions of intelligence that have implicitly guided them. We note that in practice, the contemporary AI community still gravitates towards benchmarking intelligence by comparing the skill exhibited by AIs and humans at specific tasks such as board games and video games. We argue that solely measuring skill at any given task falls short of measuring intelligence, because skill is heavily modulated by prior knowledge and experience: unlimited priors or unlimited training data allow experimenters to “buy” arbitrary levels of skills for a system, in a way that masks the system’s own generalization power. We then articulate a new formal definition of intelligence based on Algorithmic Information Theory, describing intelligence as skill-acquisition efficiency and highlighting the concepts of scope, generalization difficulty, priors, and experience. Using this definition, we propose a set of guidelines for what a general AI benchmark should look like. Finally, we present a benchmark closely following these guidelines, the Abstraction and Reasoning Corpus (ARC), built upon an explicit set of priors designed to be as close as possible to innate human priors. We argue that ARC can be used to measure a human-like form of general fluid intelligence and that it enables fair general intelligence comparisons between AI systems and humans.”

YouTube Source for this AI Video

AI video(s) you might be interested in …