Unit 9 35 POMDP knowitvideos English Japanese

Unit 9 35 POMDPYouTube↗ IA↗(long shot)
Howto & Style 8.2K views 39 likes 0 dislikes 01 Nov 2011

knowitvideos 11.8K subscribers N/A YouTube↗

Click to Play Video

I'd like to illustrate the problem, using a very simple environment that looks, as follows:

この問題について簡単な環境を用いて説明します

Suppose you live in world like this;

このような世界に住んでいるとします

and your agent starts over here, and there are 2 possible outcomes.

エージェントはここから始まり 2つの結果があります

You can exit the maze over here-- where you get a plus 100--

ここにある迷路を出ると＋100で

or you can exit the maze over here, where you receive a minus 100.

こちらの迷路を出ると－100をもらえます

Now, in a fully observable case, and in a deterministic case,

完全観測可能かつ決定的なケースで

the optimal plan might look something like this;

最適なプランはこのようになると思います

and whether or not is goes straight over here or not, depends on the details.

ここを真っすぐ進むかどうかは詳細次第です

For example, whether the agent has momentum or not.

例えばエージェントの運動量の有無などです

But you'll find a single sequence of actions and states that might cut the corners,

しかし早く＋100に近づくためなるべく近くで角を曲がった場合の

as close as possible, to reach the plus 100 as fast as possible.

単一の行為と状態を見つけます

That's conventional planning.

これは一般的なプランニングです

Let's contrast this with the case we just learned about,

すでに学んだケースで対比してみましょう

which is the fully observable case or the stochastic case.

つまり完全観測可能ではあるが確率的なケースです

We just learned that the best thing to compute is a policy

ベストな計算方法はすべての状態に

that assigns to every possible state, an optimal action;

最適な行為を割り当てるポリシーだと学びました

and simplified speaking, this might look as follows:

単純に話すとこれは次のようなことです

Where each of these arrows corresponds to a sample control policy.

この矢印はそれぞれがサンプルの制御ポリシーに一致しています

And those are defined in part of the state space that are even far away.

遠くの状態空間も定義しています

So this wouuld be an example of a control policy where all the arrows gradually point you over here.

つまりこれはすべての矢印が徐々にここに向く制御ポリシーの一例になります

We just learned about this, using MDPs and value iteration.

これはMDPや価値反復法で学びました

The case I really want to get at is the case of partial observability--

目指すケースは部分観測可能なケースで

which we will eventually solve, using a technique called POMDP.

最終的にはこれをPOMDPで解決します

And in this case, I'm going to keep the location of the agent in the maze observable.

このケースでは観測可能な迷路にエージェントを置きます

The part I'm going to make unobservable is where, exactly, I receive plus 100

観測不可能な部分は＋100と

and where I receive minus 100.

－100を受け取る場所です

Instead, I'm going to put a sign over here

その代わりにここにサインを置き

that tells the agent where to expect plus 100, and where to expect minus 100.

エージェントに＋100や－100を予測できる場所を教えます

So the optimum policy would be to first move to the sign,

つまり最適ポリシーはまずサインまで行き

サインを読み

and then return and go to the corresponding exit,

戻って対応する出口に向かうことです

for which the agent now knows where to receive plus 100.

つまりエージェントは＋100を受け取る場所が分かっています

So, for example, if this exit over here gives us plus 100, the sign will say Left.

例えばこの出口で＋100を受け取れるならサインには左と書かれます

If this exit over here gives us plus 100, the sign will say Right.

もしこの出口が＋100になるならサインには右と書かれます

What makes this environment interesting is

この環境が興味深いのは

that if the agent knew which exit would have plus 100, it will go north, from a starting position.

エージェントがどの出口が＋100になるか分かれば出発点から北を目指すことです

It goes south exclusively to gather information.

情報を集めるためだけに南に向かいます

So the question becomes: Can we devise a method for planning

次のことを理解するプランニングの方法を考えてください

that understands that, even though we'd wish to receive the plus 100 as the best exit,

最適な出口で＋100を受け取りたい場合

there's a detour necessary to gather information.

情報を収集して迂回する必要があります

So here's a solution that doesn't work:

解決方法に効果がないのは

Obviously, the agent might be in 2 different worlds--and it doesn't know.

エージェントが2つの違う世界にいるからかもしれません

It might be in the world where there's plus 100 on the Left side

＋100は左側にある世界かもしれません

or it might be in the world with plus 100 on the Right side,

＋100は右側にある世界かもしれません

with minus 100 in the corresponding other exit.

－100は対応する他の出口にあります

What doesn't work is you can't solve the problem for both of these cases

これらのケースの問題が解けないまま解決方法をまとめても効果はありません

and then put these solutions together-- for example, by averaging.

例えば平均化してみます

The reason why this doesn't work is this agent, after averaging, would go north.

効果がないのはこのエージェントが平均化すると北に向かうからです

It would never have the idea that it is worthwhile to go south,

サインを読んで最適な出口へ向かうために

read the sign, and then return to the optimal exit.

南へ向かうことに価値があるという考えはありません

When it arrives, finally, at the intersection over here,

到着すると最後にここの交差点で

it doesn't really know what to do.

何をすべきか分からなくなります

So here is the situation that does work-- and it's related to information space or belief space.

効果がある状況にすると情報空間もしくは信念空間が関係してきます

In the information space or belief space representation you do planning,

情報空間もしくは信念空間で表現をプランニングします

not in the set of physical world states, but in what you might know about those states.

それは物理世界の状態の集合ではなくこの状態について知っていることからです

And if you're really honest, you find out that there's a multitude of belief states.

よく見るとたくさんの信念状態があることが分かります

Here's the initial one, where you just don't know where to receive 100.

これが1つ目です 100をもらえる場所は分かりません

Now, if you move around and either reach one of these exits or the sign,

動き回ってこの出口かサインのうち一方にたどり着いたら

you will suddenly know where to receive 100.

100をもらう場所が突然分かります

And that makes your belief state change--

そして信念状態を変えさせます

and that makes your belief state change.

また信念状態を変えさせます

So, for example, if you find out that 100 is Left, then your belief state will look like this--

例えば100は左にあると分かれば信念状態はこのようになります

where the ambiguity is now resolved.

あいまいさが解決しましたね

Now, how would you jump from this state space or this state space?

さてこれらの状態空間からどうやって遷移しましょうか？

The answer is: when you read the sign,

答えはこれです　サインを読んだ時に

there's a 50 percent chance that the location over here will result in a transition to the location over here--

この場所がこの場所へ移動するという結果になる確率は50％です

50 percent because there's a 50 percent chance that the plus 100 is on the Left.

左側に＋100がある確率が50％だからです

There's also a 50 percent chance that the plus 100 is on the Right,

右側に＋100がある確率も50％です

so the transition over here is stochastic;

つまりここの遷移は確率的です

and with 50 percent chance, it will result in a transition over here.

50％の確率でここへと移動する結果になります

If we now do the MDP trick in this new belief space,

この信念空間でMDPトリックを使えば

and you pour water in here, it kind of flows through here

ここに水を注いで流れると

and creates all these gradients--as we had before.

前と同じ勾配を作るでしょう

We do the same over here and all these gradients are being created point to this exit on the Left side.

ここでも同じことをします左側にある出口を目指す勾配を作ります

Then, eventually, this water will flow through here and create gradients like this;

最終的に水はこの部分を流れこのような傾斜を作ります

and then flow back through here, where it creates gradients like this.

ここで逆流してこのような勾配ができます

So the value function is plus 100 over here, plus 100 over here

つまり価値関数はここで＋100でこの部分でも＋100になります

that gradually decrease down here, down here;

ここで徐々に減っていきます

and then gradually further decrease over here--

さらにここでも徐々に減っていきます

and even further decrease over there, so we've got arrows like these.

さらに減ってこのような矢印になります

And that shows you that in this new belief space, you can find a solution.

この信念空間で解決方法が見つけられると示しました

In fact, you can use value iteration--MDP's value iteration--

実際にMDPの価値反復法を使えば

in this new space to find a solution to this really complicated partially observable planning process.

この新しい空間で複雑な部分的観測のプランニング過程の解決方法を見つけられます

And the solution--just to reiterate--

その解決方法を繰り返してください

we'll suggest: Go south first, read the sign,

提案としてはまず南に向かってサインを読みます

expose yourself to the random position to the Left or Right world

確実な自信を持って＋100を獲得できる

in which you are now able to reach the plus 100 with absolute confidence.

左か右の世界のランダムな位置に向かいます

Similar videos:

Unit 11 09.mp4 Stationary Distribution.mp4

12.4K views 34 likes 2 dislikes 11 Nov 2011100% match

Unit 2, Topic 34, Note on Implementation

11.7K views 26 likes 1 dislikes 05 Oct 2011100% match

Unit 2, Topic 28, Optimistic Heuristics

17.7K views 26 likes 3 dislikes 05 Oct 2011100% match

Gundarr (Ep 6): Beach Body Toddy (subtitulado al castellano)
daniborr
15.1K views 64 likes 8 dislikes 26 Jun 2012100% match

CEOS Chatter
Fernando Garcia
1.3K views 2 likes 1 dislikes 14 Mar 2012100% match

Mr. Peabody & Sherman
TheCaptionedInternet
43.6K views 74 likes 9 dislikes 03 Mar 2014100% match

Unit 1w, 7 Poker Answer

16.4K views 12 likes 1 dislikes 05 Oct 2011100% match

Instrucciones para llorar, de Julio Cortázar
Inés Andrés
5.9K views 10 likes 0 dislikes 10 Jan 2012100% match

Gundarr (Ep 19): Gundarr's Origin (subtitulado al castellano)
daniborr
14.3K views 53 likes 3 dislikes 15 Aug 2012100% match

Unit 10 10 Passive Temporal Difference Learning.mp4

21.8K views 228 likes 8 dislikes 05 Nov 2011100% match

Unit 3 9b Question

13.6K views 2 likes 0 dislikes 15 Oct 2011100% match

Unit 2, Topic 26, A-Star Fourth Question

28.6K views 17 likes 7 dislikes 05 Oct 2011100% match

Gundarr (Ep 25): Fish and Chumps (subtitulado al castellano)
daniborr
14.5K views 51 likes 3 dislikes 04 Oct 2012100% match

Unit 5 47 Gradient Descent Implementation

14K views 16 likes 17 dislikes 17 Oct 2011100% match

ganar habito-vince lombardi
Josito H
15.9K views 62 likes 0 dislikes 04 Mar 2012100% match

Gundarr (Ep 34): Carjackers (subtitulado al castellano)
daniborr
16.6K views 63 likes 2 dislikes 05 Dec 2012100% match

Recomendación kuentalibros - pizarra digital y niños con discapacidad auditiva
anitamc20
933 views 1 likes 0 dislikes 21 Dec 2012100% match

(ENG/CHI SUB) 140216 SM True Thailand Facebook EXO
Baby Xiumin
783 views 10 likes 0 dislikes 16 Feb 2014100% match

Anonymní laskavost se šíří internetem (A stranger's act of kindness goes viral)
Michal Skala
21.7K views 4 likes 0 dislikes 20 Dec 2013100% match

[ENG SUB] 110811Mnet Wide Super Junior Interview Part 1/3
yingxuan0806
80.6K views 567 likes 7 dislikes 18 Sep 2011100% match

Unit 1w, 1 Introduction

56.8K views 146 likes 4 dislikes 05 Oct 2011100% match

Gundarr (Ep 9): Meals on Heels (subtitulado al castellano)
daniborr
11.1K views 33 likes 5 dislikes 01 Jul 2012100% match

La double origine de l'être humain Part.1/2
Selim Aïssel
20.2K views 125 likes 16 dislikes 31 Jan 2013100% match

¿Cuál es la función de los Bancos de Inversión?
canaloie24
17.4K views 97 likes 5 dislikes 22 Nov 2011100% match

TED & TEDx explained
Kyo young Chu
1.1K views 7 likes 0 dislikes 02 Apr 2012100% match

Unit 9 06 Branching Factor Question

11.3K views 4 likes 7 dislikes 01 Nov 2011100% match

Unit 1w, 4 Terminology

75.1K views 114 likes 4 dislikes 05 Oct 201160% match

Unit 3 1 Introduction

36.6K views 95 likes 6 dislikes 08 Oct 201160% match

Unit 3 2b Question

18.4K views 1 likes 1 dislikes 08 Oct 201160% match

Unit 3 2f Question

19K views 7 likes 0 dislikes 08 Oct 201160% match

Settings Privacy About Contact Donate