OPTIMIZING EXPECTATIONS: FROM DEEP REINFORCEMENT LEARNING TO STOCHASTIC COMPUTATION GRAPHS

PDF Publication Title:

OPTIMIZING EXPECTATIONS: FROM DEEP REINFORCEMENT LEARNING TO STOCHASTIC COMPUTATION GRAPHS ( optimizing-expectations-from-deep-reinforcement-learning-to- )

Previous Page View | Next Page View | Return to Search List

Text from PDF Page: 077

(1) θ (2) θ (3) θ (4) θ x x x x y Stochastic Computation Graph Objective yf Ey [f(y)] yf Ex [f (y(x))] yf Ex,y [f (y)] f Ex [f (x, y(θ))] 5.2 preliminaries 69 Gradient Estimator ∂x∂logp(y|x)f(y) ∂θ ∂x ∂ logp(x|θ)f(y(x)) ∂θ ∂ logp(x|θ)f(y) ∂θ ∂ logp(x|θ)f(x,y(θ))+∂y∂f ∂θ ∂θ ∂y ∂ log p(x1 | θ, x0)(f1(x1) + f2(x2)) ∂θ + ∂ log p(x2 | θ, x1)f2(x2) ∂θ f2 Ex1,x2 [f1(x1) + f2(x2)] x0 x1 x2 (5) θ These simple examples illustrate several important motifs, where stochastic and de- terministic nodes are arranged in series or in parallel. For example, note that in (2) the derivative of y does not appear in the estimator, since the path from θ to f is “blocked” by x. Similarly, in (3), p(y | x) does not appear (this type of behavior is particularly useful if we only have access to a simulator of a system, but not access to the actual likelihood function). On the other hand, (4) has a direct path from θ to f, which contributes a term to the gradient estimator. (5) resembles a parameterized Markov reward process, and it illustrates that we’ll obtain score function terms of the form grad log-probability × future costs. f1 Figure 11: Simple stochastic computation graphs The examples above all have one input θ, but the formalism accommodates models with multi- ple inputs, for example a stochastic neural network with multiple layers of weights and biases, which x may influence different subsets of the stochastic and cost nodes. See Section 5.10 for nontrivial ex- amples with stochastic nodes and multiple inputs. The figure on the right shows a de- W1 b1 W2 b2 y=label h1 h2 cross- soft- entropy max loss

PDF Image | OPTIMIZING EXPECTATIONS: FROM DEEP REINFORCEMENT LEARNING TO STOCHASTIC COMPUTATION GRAPHS

optimizing-expectations-from-deep-reinforcement-learning-to--077

PDF Search Title:

OPTIMIZING EXPECTATIONS: FROM DEEP REINFORCEMENT LEARNING TO STOCHASTIC COMPUTATION GRAPHS

Original File Name Searched:

thesis-optimizing-deep-learning.pdf

DIY PDF Search: Google It | Yahoo | Bing

Cruise Ship Reviews | Luxury Resort | Jet | Yacht | and Travel Tech More Info

Cruising Review Topics and Articles More Info

Software based on Filemaker for the travel industry More Info

The Burgenstock Resort: Reviews on CruisingReview website... More Info

Resort Reviews: World Class resorts... More Info

The Riffelalp Resort: Reviews on CruisingReview website... More Info

CONTACT TEL: 608-238-6001 Email: greg@cruisingreview.com | RSS | AMP