Unicorn AI - Computerphile Computerphile English (auto-generated) English

Unicorn AI - ComputerphileYouTube↗ IA↗(long shot)
Education 339.5K views 7.8K likes 189 dislikes 04 Jul 2019

Computerphile 2.6M subscribers N/A YouTube↗

Click to Play Video

in the previous video we were talking

In the previous video we were talking about

about transformers this architecture that uses attention to give

transformers this architecture that uses attention to give

unprecedented ly good performance on sort of language modeling tasks and some

Unprecedented ly good performance on sort of language modeling tasks and some other tasks as well

other tasks as well but when were looking at language modeling and that was in preparation to make a video about

but when were looking at language modeling and that was in preparation to make a video about

GPG 2 which is this very giant language model that has been there was recently

GPG 2, which is this very giant language model that has been there was recently

well it was recently not released actually by open AI the way that they generated the data set for this is

Well, it was recently not released actually by open AI the way that they generated the data set for this is pretty cool

pretty cool to get enough text they went to Reddit and they pulled every website that is linked to from reddit do we have any idea of how many days lots

to get enough text they went to Reddit and They pulled every website that is linked to from reddit. Do we have any idea of how many days lots?

literally everything was everything that

Literally, everything was everything that had more than three karma

had more than three karma I think or maybe more than two karma something like

I think or maybe more than two karma something like that like

that like anything that had somebody had thought to post around it and at least two or three people who had thought was

Anything that had somebody had thought to post around it and at least two or three people who had thought was good enough to upload

good enough to upload they scraped the text from that it's pretty much just a transformer it's not the the

They scraped the text from that. It's pretty much just a transformer. It's not the the

architecture is not especially novel they haven't done any like amazing new

Architecture is not especially novel. They haven't done any like amazing new

new discovery but what they realized was

new discovery, but What they realized was?

transformers it seems like the more data

Transformers it seems like

you give them the better they do and the bigger you make them the better they do

the more data you give them the better they do and the bigger you make them the better they do and

and everything that we built up until this point is clearly not like we

Everything that we built up until this point is clearly not

haven't hit the limits of what this can

Like we haven't hit the limits of what this can do

do we they thought we think we're

We they thought we think we're probably

probably bottle necked on data and maybe network size so what

Bottle necked on data and maybe network size

happens if we'd like to turn that 211 what happens if we just give this all

So what happens if we'd like to turn that 211 what happens if we just give this all?

the data and make a really big one it makes sense to talk about the acronym right so it's a generative pre-training

The data and make a really big one. It makes sense to talk about the acronym right so it's a generative pre-training

transformer so generative same as generative adversarial network it generates outputs to generate samples

Transformer so generative same as generative adversarial network. It generates outputs to generate samples

your pre-trained is this thing I was

Your pre-trained is this thing. I was talking about all of the different things

talking about all of the different things you can use a language model for right you can do you can do translation

You can use a language model for right you can do you can do translation. You can try and resolve ambiguities

you can try and resolve ambiguities you can do summarization you can answer questions you can use the probabilities for augmenting other systems so yeah

You can do summarization. You can answer questions. You can use the probabilities for augmenting other systems

there's a bunch of different benchmarks for these different tasks that you might

So yeah, there's a bunch of different benchmarks for these different tasks

want your language model to do and this

that you might want your language model to do and

is what we talked about in the grid worlds video of having these like standardized problems with standardized metrics and standardized data sets so

This is what we talked about in the grid worlds video of having these like standardized problems with standardized metrics and standardized data sets

that if you're comparing two different methods you know that you're actually comparing apples to apples and this is like very important it gives you numbers

So that if you're comparing two different methods, you know that you're actually comparing apples to apples And this is like very important it gives you numbers on these things. It's often quite difficult

on these things it's often quite

difficult expected to like you're generating samples of text and it's like how plausible is this text how realistic

Expected to like you're generating samples of text and it's like how plausible is this text? How realistic does it look like?

does it look like how do you put a number on that it's kind of difficult so there's all of these standardized metrics and the thing that people came

How do you put a number on that it's kind of difficult. So there's all of these standardized metrics and the thing that

to realize which actually I mean I say that as though it's like some amazing

People came to realize which actually I mean I say that as though it's like some amazing discovery

discovery it's fairly obvious if you train your system in a like an unsupervised way on a large corpus of just general English text and then you

It's fairly obvious. If you train your system in a like an unsupervised way on a large corpus of just general English text and

take that and train that with the data from this benchmark or the data from

then you take that and Train that with the data from this benchmark or the data from that benchmark

that benchmark you can like fine-tune it so you start

You can like fine-tune it so you start with something which has like a decent

with something which has like a decent understanding of how English works more or less and then you say now I'm going to give you these samples for like

Understanding of how English works more or less and then you say now I'm going to give you these

question answering or I'm going to build a system using that to solve to go for

Samples for like question answering or I'm going to build a system using that to solve to go for this benchmark

this benchmark so it's pre trained you start with something that's like a general-purpose language model and then you from that a fine-tuned it to

So it's pre trained you start with something. That's like a general-purpose language model and then you from that a

whichever actual benchmark or problem

Fine-tuned it to whichever Actual benchmark or problem you're trying to solve

you're trying to solve and this can give

you better performance than to starting from nothing and training to each of the benchmarks from scratch make sense and

Can give you better performance than to starting from nothing and training to each of the benchmarks from scratch make sense

so the point of the GPT 2 paper the thing that makes it cool is they said okay if we make a really huge one what

and so The point of the GPT 2 paper the thing that makes it cool is they said okay if we make a really huge one

if we don't fine tune it at all what if

What if we? don't Fine tune it at all

we just make a giant model and then just try and run it on the benchmarks without

What if we just make a giant model and then just try and run it on the benchmarks without messing with it?

messing with it without showing it any of their specialized data for that benchmark just the raw general-purpose

Without showing it any of their specialized data for that benchmark. Just the raw

language model how does that perform and it turns out surprisingly well so this

general-purpose language model, how does that perform and it turns out surprisingly well, so this is a

is a very very large data set for text

Very very large data set for text

it's about 40 gigabytes which actually

It's about 40 gigabytes which

doesn't sound like very much but like for text text that's insane right it's

Actually doesn't sound like very much but like for text text that's insane, right? It's

somebody said that this was the size of

somebody said that this was the size of

Google's entire index of the Internet in

Google's entire index of the Internet in 98

98 so like it's yeah it's a lot of text

So like it's yeah, it's a lot of text

and they trained it on that and they

and they trained it on that and they ended up with a

ended up with a 1.5 billion parameter model but which is like a previous state of the art system was 345 million this

1.5 billion parameter model, but which is like a previous state of the art system was 345 million

is 1.5 billion so they've just made the

This is 1.5 billion

thing much much bigger and it performs really well some of their samples that they published quite captured the public

So they've just made the thing much much bigger and it performs really well some of their samples that they published quite

imagination you could say and now that we've talked a little about the problems

captured the public imagination You could say and now that we've talked a little about the problems that

that neural networks or any language

Neural networks or any language model really?

model really has with a long term

Has with a long term dependency

dependency we can now realise just how impressive these samples are because when you look at them as a you know if you look at

we can now realise just how impressive these samples are because when you look at them as a you know,

them uninitiated you're like yeah that's

If you look at them uninitiated, you're like yeah, that's pretty realistic

pretty realistic it seems to like make sense and it's cool but when you look at it knowing how language models work it's

It seems to like make sense and it's cool. But when you look at it knowing how language models work, it's like

like very impressive the the coherence

very impressive the the coherence and the

and the consistency and the long-range dependencies so we can look at this one that got everybody's attention the

Consistency and the long-range dependencies so we can look at this one that got everybody's attention the unicorns one

unicorns one right so they prompted it

with in a shocking finding scientists discovered a herd of unicorns living in

So they prompted it with in a shocking finding scientists discovered a herd of unicorns

a remote previously unexplored valley in

living in a remote previously unexplored valley in the Andes Mountains

the Andes Mountains even more surprising to the researchers was the fact that the unicorns spoke

Even more surprising to the researchers was the fact that the unicorns spoke perfect English

perfect English and from there you then say you go to your language model gbgt and you say given that we started with

And from there you then say you go to your language model gbgt, and you say given that we started with this

this what's the next word and what's the word after that and so on so it goes on

What's the next word and what's the word after that and so on?

the scientist named the population after their distinctive horn of its unicorn

So it goes on the scientist named the population after their distinctive horn of its unicorn

these four horned silver white unicorns were previously unknown to science we do

These four horned silver white unicorns were previously unknown to science

have a clue here as a human being unicorns for horned doesn't quite make

We do have a clue here as a human being unicorns for horned doesn't quite make sense

sense but nonetheless we're going okay

But nonetheless we're going okay

now after almost two centuries the mystery of what sparked this odd phenomenon is finally solved dr. Budetti

Now after almost two centuries the mystery of what sparked this odd phenomenon is finally solved. Dr

Jorge Jorge Perez Jo are G an

Budetti Jorge Jorge Perez

evolutionary biologist from the

Jo are G an evolutionary biologist from the University of La Paz

University of La Paz this is impressive because we've mentioned the Andes Mountains in our prompt and so now it's

This is impressive because we've mentioned the Andes Mountains in our prompt and so now it's saying okay

saying okay this is clearly you know in a shocking finding this is a science

This is clearly, you know in a shocking finding. This is a science press release news article

press release news article it's seen enough of those because it has every single one that was ever linked to from reddit right so it knows how these go it

It's seen enough of those because it has every single one that was ever linked to from reddit, right? So it knows how these go it knows. Okay third paragraph

knows okay third paragraph this is when

we talk about the scientist we interview the scientist right okay

This is when we talk about the scientist, we interview the scientist, right? Okay

first word of the scientist paragraph dr. obviously right because this is the

First word of the scientist paragraph, dr. Obviously, right because this is the now we're in the name of the scientist

now we're in the name of the scientist what name are we going to give it needs

What name are we going to give?

to be a name conditioning on the fact

It needs to be a name

that we have the Andes Mountains so we

conditioning on the fact that we have the Andes Mountains

need to get where we're in South America

So we need to get where we're in South America

the name probably should be Spanish or maybe Portuguese so we get we get dr. Perez here

The name probably should be Spanish or maybe Portuguese So we get we get dr. Perez here

and then evolutionary biologist makes sense because we're talking about

And then evolutionary biologist makes sense because we're talking about animals

animals from the University of La Paz

from the University of La Paz again

again this is the first sentence like when you have that first clause that introduces the scientist you always say

This is the first sentence like when you have that first clause that introduces the scientist you always say where they're from

where they're from so we say from the University of and then university names tend to be the

So we say from the University of and then university names tend to be the name of a city

name of a city what's the city where we have the Andes Mountains so we're going to Bolivia lapaz perfect and the thing

What's the city where we have the Andes Mountains, so we're going to Bolivia lapaz. Perfect

that's cool about this is it's remembered all of these things that were quite a long time ago several sentences

And the thing that's cool about this is it's remembered all of these things that were quite a long time ago several sentences ago

ago well it hasn't remembered them it's paid attention to them across that distance which is impressive but also

Well, it hasn't remembered them. It's paid attention to them across that distance, which is impressive

this is encoding a bunch of understand understanding a bunch of information about the real world right all that was given all it knows is statistical

But also this is encoding a bunch of understand understanding a bunch of information about the real world Right all that was given all it knows is statistical relationships between words, but the way that it comes out to us

relationships between words but the way

that it comes out to us is that it knows

Is that it knows?

where the Andes Mountains are what kind of names people in that area have what their cities are what the universities are all of those facts about the real

Where the Andes Mountains are what kind of names people in that area have what their cities are what the universities are all of those

world because in order to have a really good language model it turns out you have to kind of implicitly encode

Facts about the real world because in order to have a really good language model it turns out you have to kind of implicitly encode

information about the world because we

information about the world because

use language to talk about the world and knowing what's likely to come next

We use language to talk about the world and knowing what's likely to come next

requires actual real world understanding and that's something that we see in some

Requires actual real world understanding and that's something that we see in some of the other

of the other things that they got it to do you can see the real world

Things that they got it to do you can see the real world understanding coming through

understanding coming through let's keep

Let's keep going

going University of a person several companions were exploring the Andes Mountains when they found a small valley

University of a person several companions were exploring the Andes Mountains when they found a small valley with no other animals or humans peres see

with no other animals or humans peres

see we're hanging on to him yep we're

We're hanging on to him. Yep. We're referring to him again

referring to him again but now we've changed it to be just the surname

but now we've changed it to be just the surname because that's the

because that's the format that people use in news articles Peres noticed that the valley had what appeared to be a natural fountain surrounded by two peaks

format that people use in news articles Peres noticed that the valley had what appeared to be a natural fountain surrounded by two peaks of

of rock and silver snow presently others then ventured further into the valley a

Rock and silver snow presently others, then ventured further into the valley a round about here in our article

round about here in our article we should have a quote from the scientist

We should have a quote from the scientist right quote

right quote by the time we reached the top of one peak the water looked blue with some crystals on top and we're

By the time we reached the top of one peak the water looked blue with some crystals on top and we're talking about this fountain

talking about this fountain I guess it's natural fountain we're referring back to the previous int it's like everything is

I guess it's natural fountain. We're referring back to the previous int. It's like everything is

relying on in contingent on earlier parts of the text while examining there by snipped paragraph while examining

Relying on in contingent on earlier parts of the text while examining there by snipped paragraph while examining these bizarre

these bizarre creatures the scientists discovered that the creatures also spoke some fairly regular English know when I

Creatures the scientists discovered that the creatures also spoke some fairly regular English know when I read that I like, okay

read that I like okay this is now unusually good because that's the second sentence of the lead right where six paragraphs in

this is now unusually good because that's the second sentence of the lead right where six paragraphs in and

and it knows about this point I've covered the first sentence of this

It knows about this point. I've covered the first sentence of this

initial paragraph now it's time to talk

initial paragraph

about this second sentence of the lead even more surprising to the research of

now it's time to talk about this second sentence of the lead even more surprising to the research of us of the fact that they

us of the fact that they spoke English

spoke English and

and it completely ignored the speaking English part until it got to the part of the news article where that comes in

It completely ignored the speaking English part until it got to the part of the news article where that comes in

you've gone six whole paragraphs the

You've gone six whole paragraphs

idea of accurately remembering that the

unicorn speak perfect English is like

Accurately remembering that the unicorn speak perfect

that's very impressive to me and then it goes into its gets a little bit unhinged

English is like that's very impressive to me and then it goes into its gets a little bit unhinged

starts talking about it's likely that the only way of knowing for sure if

Starts talking about it's likely that the only way of knowing for sure if unicorns are indeed

unicorns are indeed the descendants of a lost alien race is through DNA that's read it really well it's not actually

The descendants of a lost alien race is through DNA. That's read it really

stuff on reddit it's stuff linked to from reddit but yeah this is this is news articles men they seem to be able

Well, it's not actually stuff on reddit. It's stuff linked to from reddit. But yeah, this is this is news articles men

to communicate in English quite well

They seem to be able to communicate in English quite well

which I believe is a sign of evolution or at least a change in social organization said the scientist that's

Which I believe is a sign of evolution or at least a change in social organization said the scientist

his evolutionary biology there right right right yeah we know here's an evolutionary biologist so so the the

That's his evolutionary biology there. Right? Right, right. Yeah, we know here's an evolutionary biologist. So so the the

coherence of this text is really

coherence of this text is

dependent on its ability to condition

really dependent on its ability to

what it's generating on things that it's

Condition what it's generating on

generated a long time ago so yeah so it

Things that it's generated a long time ago

can generate really nice news articles and it can generate all kinds of text things that it anything that is

So yeah So it can generate really nice news articles and it can generate all kinds of text things that it anything that is

sufficiently well represented in the original data set so that's GPG - it's a

Sufficiently well represented in the original data set. So that's GPG - it's a really

really unusually powerful and like versatile language model that can do all of these different natural language

Unusually powerful and like versatile language model that can do all of these different natural language processing

processing tasks without actually being trained specifically on those tasks it's

Tasks without actually being trained specifically on those tasks

really and that's that's why it's

It's really and that's that's why it's impressive

impressive it's not that it's a it's a brand new architecture or a brand new

It's not that it's a it's a brand new architecture or a brand new approach or whatever

approach or whatever it's just when you make these things really huge and give them tremendously large amounts of data

It's just when you make these things really huge and give them tremendously large amounts of data

the results are really impressive in the

The results are really impressive

original data set so it will it will write you the Lord of the Rings fan

In the original data set. So it will it will write you the Lord of the Rings fan fiction

fiction it will write you cake recipes if we're like there's all kinds of examples of different samples here's a recipe for

It will write you cake recipes if we're like, there's all kinds of examples of different samples. Here's a recipe for

some kind of peppermint chocolate cake and it's got a bunch of different

Some kind of peppermint chocolate cake and it's got a bunch of different

Similar videos:

Is Papa John's Extra Cheesy Alfredo Pizza TOO Cheesy?
TheReportOfTheWeek
1M views 38.9K likes -1 dislikes 04 Jan 202032% match

Pascal (Not Just Nickel & Dime) - Computerphile
Computerphile
43.6K views 2.1K likes 18 dislikes 14 Aug 202032% match

AI "Stop Button" Problem - Computerphile
Computerphile
1.1M views 28.6K likes 1.1K dislikes 03 Mar 201731% match

How Cheesy Is KFC's NEW Cheetos Sandwich?
TheReportOfTheWeek
1.1M views 39.5K likes -1 dislikes 02 Jul 201931% match

Classical Singer Reaction - Tengri | Heaven. Fascinating Singing Style! He's Rare and Amazing!
SingerDad Reacts
12.3K views 805 likes 10 dislikes 08 Oct 202031% match

Trying Burger King's NEW Impossible Whopper!!
TheReportOfTheWeek
1.3M views 44K likes -1 dislikes 08 Aug 201931% match

Just how Cheesy is Pizza Hut's Stuffed Cheez-It?
TheReportOfTheWeek
953.3K views 28.2K likes -1 dislikes 19 Sep 201931% match

we can't keep doing this (van life update)
Naick & Kim
34K views 1.9K likes 21 dislikes 24 Feb 202131% match

Taco Bell's NEW Bacon Club Chalupa Review!
TheReportOfTheWeek
173.3K views 11.1K likes 113 dislikes 08 Jan 202130% match

I React To My FIRST VIDEO!
TheReportOfTheWeek
1.3M views 63.5K likes -1 dislikes 26 Feb 201930% match

Is Popeyes NEW Rip'n Chicken Big Box the BEST Deal of 2021?
TheReportOfTheWeek
211.9K views 13.4K likes 147 dislikes 12 Jan 202130% match

Hot and STEAMY at a Turkish Bath | Our First Time
Eileen Aldis
72.4K views 2.5K likes 30 dislikes 10 Oct 202030% match

Wendy's NEW Bacon Double Stack Review!
TheReportOfTheWeek
114.8K views 6.5K likes 86 dislikes 25 Mar 202130% match

Classical Singer First Time Reaction- Dimash | Bastau - Kazakhstan. Charismatic & Impressive!!
SingerDad Reacts
19K views 2K likes 12 dislikes 31 Jul 202030% match

Does McDonald's McRib Still Live Up To The Hype?
TheReportOfTheWeek
826.3K views 42.3K likes -1 dislikes 04 Dec 202030% match

0 views 0 likes -1 dislikes 05 Jun 202630% match

Building VINCENT From Disney The Black Hole 3d Printing the Head (Part 2)
Ricardos Gaming
578 views 19 likes 0 dislikes 04 Jun 201930% match

We Made the WORLD's HEAVIEST Stormbreaker!
Hacksmith Industries
3.7M views 68.9K likes 1.4K dislikes 22 Nov 201830% match

How Good Is Wendy's NEW Spicy Crispy Chicken Sandwich?
TheReportOfTheWeek
222.5K views 13.1K likes 148 dislikes 15 Aug 202030% match

Classical Singer First Time Reaction. Nightwish | Ghost Love Score (Wacken). LOVE THIS BAND!
SingerDad Reacts
31.8K views 1.8K likes 43 dislikes 22 Sep 202030% match

Primeras impresiones NICHO: Montale, Mancera y Molinard 📦 YUNIQU | Smarties Reviews
Smarties Reviews
2.8K views 239 likes 3 dislikes 25 Oct 202030% match

How Difficult is Travelling Japan without Japanese? | Travel Tips
Abroad in Japan
4.1M views 141.2K likes 1.5K dislikes 27 Mar 201830% match

SEASON 4 ROCKET CRASH SITE FOUND | Fortnite Battle Royale
SXVXN
2.5M views 25.9K likes 3K dislikes 27 Jun 201830% match

why am I so embarrassed..
PewDiePie
6M views 383.2K likes 10.1K dislikes 21 Oct 201630% match

The Coke vs. Pepsi Challenge
TheReportOfTheWeek
2M views 104.5K likes -1 dislikes 11 May 201930% match

Are Red Baron's Pizza Melts a Frozen Pizza Game Changer?
TheReportOfTheWeek
147.4K views 9K likes 83 dislikes 06 Feb 202130% match

Genius inventions that make the world a better place!
AzzyLand
3.2M views 35.6K likes 870 dislikes 26 Sep 201830% match

Taco Bell's NEW Toasted Cheddar Chalupa Review!
TheReportOfTheWeek
276.1K views 17.7K likes 193 dislikes 10 Nov 202030% match

My First McRib from McDonald's
TheReportOfTheWeek
2.4M views 76.8K likes 2.5K dislikes 09 Nov 201830% match

Halloween BLOWOUT! Trying Krispy Kreme's NEW Halloween Doughnuts
TheReportOfTheWeek
108.1K views 9.5K likes 70 dislikes 31 Oct 202030% match

Settings Privacy About Contact Donate