Unit 10 10 Passive Temporal Difference Learning.mp4 knowitvideos English Japanese

Unit 10 10 Passive Temporal Difference Learning.mp4YouTube↗ IA↗(long shot)
Film & Animation 21.8K views 228 likes 8 dislikes 05 Nov 2011

knowitvideos 11.8K subscribers N/A YouTube↗

Click to Play Video

Let's start by looking at passive reinforcement learning.

受動強化学習について見てみましょう

I'm going to describe an algorithm called Temporal Difference Learning--or TD.

TD学習もしくはTDと呼ばれるアルゴリズムを書いてみます

And what that means--sounds like a fancy name,

有名人の名前のようですが

but all it really means is we're going to move from one state to the next;

ある状態から次へ動くという意味です

and we're going to look at the difference between the 2 states,

そして2つの状態の差を見て

and learn that--and then kind of back up the values, from one state to the next.

学んだあとに次々と値を元に戻します

So if we're going to follow a fixed policy, pi,

固定されたポリシーであるΠに従うと

and let's say our policy tells us to go this way, and then go this way.

ポリシーは次に進む方向を教えるでしょう

We'll eventually learn that we get a plus 1 reward there

結局＋1の報酬を手に入れると学び

and we'll start feeding back that plus 1, saying:

＋1に戻り始めるでしょう

if it was good to get a plus 1 here, it must be somewhat good to be in this state,

もしここで＋1を獲得すべきならこの状態にいるのがよいでしょう

somewhat good to be in this state--and so on, back to the start state.

この状態にいるのもよいと思います出発状態に戻ります

So, in order to run this algorithm,

このアルゴリズムを実行するため

we're going to try to build up a table of utilities for each state

それぞれの状態に関する効用や方法に沿った効用の表を作り

and along the way, we're going to keep track of the number of times that we visited each state.

それらを訪ねた回数を追ってみましょう

Now, the table of utilities, we're going to start blank--

さて効用の表の空白から始めます

we're not going to start them at zero or anything else where they're just going to be undefined.

0や定義がされていない場所からは出発しません

And the table of numbers, we're going to start at zero,

数の表では0で始めます

saying we visited each state a total of zero times.

それぞれ状態に向かうのは 0回ということです

What we're going to do is run the policy,

私たちがやるのはポリシーの実行です

have a trial that goes through the state;

その状態を経験する実験をしましょう

when it gets to a terminal state,

終着の状態にたどり着く時

we start it over again at the start and run it again;

私たちはまた出発点から実行します

and we keep track of how many times we visited each state,

それぞれの状態を訪ねる回数を追います

we update the utilities, and we get a better and better estimate for the utility.

効用を更新して効用をよりよく評価します

And this is what the inner loop of the algorithm looks like--

これがアルゴリズムの内側のループです

and let's see if we can trace it out. So we'll start at a start state,

では追ってみましょう　出発点から初めて

we'll apply the policy--and let's say the policy tells us to move in this direction.

ポリシーを適用します　するとポリシーが動く方向を教えてくれます

Then we get a reward here,

ここで報酬を手にします

つまり0です

and then we look at it with the algorithm,

そしてアルゴリズムで報酬を見ます

and the algorithm tells us if the state

アルゴリズムは状態が新規か教えてくれます

is new--yes, it is; we've never been there before--

ここは初めてですね

then set the utility of that state to the new reward, which is zero.

その状態の効用に新しい報酬を設定します　つまり0です

Okay--so now we have a zero here;

今ここに0があります

and then let's say, the next step, we move up here.

そして次の段階でここに動きます

So, again, we have a zero;

そしてまた0があります

and let's say our policy looks like a good one,

私たちのポリシーはよいものに見えます

so we get: here, we have a zero.

そしてここで手にします　0です

We get: here, we have a zero.

ここで0を手にします

We get: here--now, this state, we get a reward of 1, so that state gets a utility of 1.

そしてここで1の報酬を得るので状態は1の効用を手にします

And all along the way, we have to think about how we're backing up these values, as well.

あらゆる点においてどうやってこの値を戻すかも考える必要があります

So when we get here, we have to look at this formula to say:

ですのでここに着いた時にこの式を見る必要があります

How are we going to update the utility of the prior state?

どのように前の状態の効用を更新するでしょうか？

And the difference between this state and this state is zero.

この状態とこの状態の差は0です

so this difference, here, is going to be zero--

この差も0になるでしょう

the reward is zero, and so there's going to be no update to this state.

報酬0の状態は更新されません

But now, finally--for the first time--we're going to have an actual update.

しかし最後に初めて更新することになります

So we're going to update this state to be plus 1,

この状態を＋1に更新します

and now we're going to think about changing this state.

さてこの状態の変更を考えてみましょう

And what was its old utility?--well, it was zero.

何が古い効用ですか？　それは0です

And then there's a factor called Alpha,

αという要素があります

which is the learning rate

つまり学習率です

that tells us how much we want to move this utility towards something that's maybe a better estimate.

どのくらいこの効用をよりよい評価へ動かしたいかを教えてくれます

And the learning rate should be such that,

学習率は次のような方法です

if we are brand new, we want to move a big step;

もしまったく新しいものがあれば大きく動かしたいですよね

and if we've seen this state a lot of times, we're pretty confident of our number and we want to make a small step.

この状態を何回も見ると値に自信がついて小さなステップにしようとします

So let's say that the Alpha function is 1 over N plus 1.

さてアルファ関数が1／n＋1ですつまりnがゼロの時

Well, we'd better not make it 1 over N plus 1, when N is zero.

1／n＋1にはならないはずです

So 1 over N plus 1 would be ½;

1／n＋1は1／2になります

and then the reward in this state was zero;

そしてこの状態での報酬は0です

plus, we had a Gamma--

さらにγを加えます

and let's just say that Gamma is 1, so there's no discounting; and then

γが1ならば割引されません

we look at the difference between the utility

そして結果状態の効用との差を見ます

of the resulting state--which is 1--

つまり1です

minus the utility of this state, which was zero.

そこからこの状態の効用を引くと0です

So we get ½, 1 minus zero--which is ½.

1／2×（1－0）は1／2です

So we update this; and we change this zero to ½.

更新してこの0を1／2に変更します

Now let's say we start all over again

では最初からやり直してみましょう

and let's say our policy is right on track; and nothing unusual, stochastically, has happened.

ポリシーが計画通りに進んだ場合確率的に普通のことが起こります

So we follow the same path,

同じ道をたどっても更新しません

we don't update--because they're all zeros all along this path.

この道がずっと0だからです

We go here, here, here;

このルートを通っていきます

and now it's time for an update.

ここで更新します

So now, we've transitioned from a zero to ½--

0から1／2へと遷移しました

so how are we going to update this state?

どうやって更新しますか？

Well, the old state was zero

前の状態は0でした

and now we have a 1 over N plus 1--

今は1／n＋1になります

so let's say 1/3.

例えば1／3とします

So we're getting a little bit more confident--because we've been there twice, rather than just once.

二度も通ったので少し慣れてきましたね

The reward in this state was zero,

この状態の報酬は0でした

and then we have to look at the difference between these 2 states.

この2つの状態間の差を見てみましょう

That's where we get the name, Temporal Difference;

これがTDという名前になった部分です

and so, we have ½ minus zero--

ここが1／2－0で

and so that's 1/3 times ½-- so that's 1/6.

1／3×1／2なので答えは1／6になります

Now we update this state.

この状態を更新します

It was zero; now it becomes 1/6.

0でしたが1／6になります

And you can see how the results of the positive 1 starts to propagate

＋1という結果が逆方向に増え始める様子が分かります

backwards--but it propagates slowly.

ゆっくり増えていきます

We have to have 1 trial at a time to get that to propagate backwards.

逆方向に増える結果を得るため 1回につき実験は1つです

Now, how about the update from this state to this state?

さてこの状態の更新はどうなるでしょう？

Now, we were ½ here--so our old utility was ½;

今ここは1／2です　古い効用は1／2になります

plus Alpha--the learning rate--is 1/3.

αつまり学習率は1／3です

The reward in the old state was zero;

古い状態の報酬は0でした

plus the difference between these two, which is 1 minus ½.

この2つの状態の差を加えると 1－1／2です

So that's ½ plus 1/6 is 2/3.

そして1／2＋1／6は2／3です

And now the second time through, we've updated the utility of this state from 1/2 to 2/3.

2回目でこの状態の効用を 1／2から2／3に更新しました

And we keep on going--and you can see the results of the positive, propagating backwards.

そして続けると逆方向へ増える正の結果が分かります

And if we did more examples through here, you would see the results of the negative propagating backwards.

もしここを通して更に例を使えば反対方向へ増える負の結果が分かります

And eventually, it converges to the correct utilities for this policy.

最終的にこのポリシーに関して正しい効用へと収束します

Similar videos:

Unit 3 11b Question

10.1K views 11 likes 9 dislikes 08 Oct 2011100% match

Unit 2, Topic 29, State Spaces

16.9K views 4 likes 1 dislikes 05 Oct 2011100% match

Unit 9 14 Policy Answer 3 Question 4

7.7K views 10 likes 3 dislikes 01 Nov 2011100% match

CEOS Chatter
Fernando Garcia
1.3K views 2 likes 1 dislikes 14 Mar 2012100% match

Unit 4, 4b More Variable Elimination

8.6K views 7 likes 1 dislikes 06 Oct 2011100% match

Unit 2, Topic 10, Graph Search Answer

12.5K views 12 likes 3 dislikes 05 Oct 2011100% match

Mr. Peabody & Sherman
TheCaptionedInternet
43.6K views 74 likes 9 dislikes 03 Mar 2014100% match

Instrucciones para llorar, de Julio Cortázar
Inés Andrés
5.9K views 10 likes 0 dislikes 10 Jan 2012100% match

Unit 3 11a Answer

10.6K views 16 likes 40 dislikes 08 Oct 2011100% match

Horrible Bosses (2011)
Captionwire
13.5K views 22 likes 2 dislikes 27 May 2011100% match

Gundarr (Ep 3): Don't Have a Cow Man! (subtitulado al castellano)
daniborr
11.5K views 37 likes 7 dislikes 23 Jun 2012100% match

Unit 3 12b Question

7.6K views 2 likes 0 dislikes 08 Oct 2011100% match

Unit 3 9d Absolute vs Conditional Independence

13.5K views 4 likes 2 dislikes 08 Oct 2011100% match

Gundarr (Ep 13): Meme Goose ta! (subtitulado al castellano)
daniborr
162.2K views 414 likes 95 dislikes 04 Jul 2012100% match

Unit 2, Topic 5, Tree Search Answer

17.8K views 19 likes 3 dislikes 05 Oct 2011100% match

SD Gundam Mk I: Episodio 2 (subtitulado al castellano)
daniborr
88.4K views 251 likes 9 dislikes 14 Jul 2012100% match

Unit 2, Topic 31, Sliding Blocks Puzzle

21.1K views 15 likes 2 dislikes 05 Oct 2011100% match

Unit 3 2d Question

17.5K views 3 likes 0 dislikes 08 Oct 2011100% match

ABR terapija za cerebralna paraliza - P.J.
Vesna Gabeloska
10.3K views 13 likes 0 dislikes 05 Sep 2011100% match

Recomendación kuentalibros - pizarra digital y niños con discapacidad auditiva
anitamc20
933 views 1 likes 0 dislikes 21 Dec 2012100% match

(ENG/CHI SUB) 140216 SM True Thailand Facebook EXO
Baby Xiumin
783 views 10 likes 0 dislikes 16 Feb 2014100% match

Serebral palsi (Beyin Felci) terapi ABR
Vesna Gabeloska
8.3K views 4 likes 1 dislikes 20 Jun 2012100% match

0 views 0 likes 0 dislikes 06 Jun 2026100% match

0 views 0 likes 0 dislikes 06 Jun 2026100% match

Oh ja Ah matkal
Tuuli Koitjärv
21.5K views 30 likes 4 dislikes 01 Jan 2014100% match

Unit 6 8 Supervised vs Unsupervised Learning

20.7K views 42 likes 4 dislikes 22 Oct 2011100% match

Winnie the Witch
Pilar Moure
111K views 286 likes 57 dislikes 31 May 2013100% match

Unit 6 5b Maximum Likelihood

10.2K views 25 likes 1 dislikes 22 Oct 2011100% match

Unit 11 06.mp4 Markov Chain Answer 1.mp4

10.1K views 27 likes 1 dislikes 11 Nov 2011100% match

Gundarr (Ep 1): Crystal Whipped (subtitulado al castellano)
daniborr
16.1K views 62 likes 11 dislikes 21 Jun 2012100% match

Settings Privacy About Contact Donate