PDF Publication Title:
Text from PDF Page: 039
3.8 experiments 31 2. Hopper. 12-dimensional state space, same reward as the swimmer, with a bonus of +1 for being in a non-terminal state. We ended the episodes when the hopper fell over, which was defined by thresholds on the torso height and angle. 3. Walker. 18-dimensional state space. For the walker, we added a penalty for strong impacts of the feet against the ground to encourage a smooth walk rather than a hopping gait. We used δ = 0.01 for all experiments. See Table 2 in the Appendix for more details on the experimental setup and parameters used. We used neural networks to repre- sent the policy, with the architecture shown in Figure 3, and further details provided in Section 3.13. To establish a standard baseline, we also included the classic cart-pole bal- ancing problem, based on the formulation from Barto, Sutton, and Anderson [BSA83], using a linear policy with six parameters that is easy to optimize with derivative-free black-box optimization methods. The following algorithms were considered in the comparison: single path TRPO; vine TRPO; cross-entropy method (CEM), a gradient-free method [SL06]; covariance matrix adap- tion (CMA), another gradient-free method [HO96]; natural gradient, the classic natural policy gradient algorithm [Kak02], which differs from single path by the use of a fixed penalty coefficient (Lagrange multiplier) instead of the KL divergence constraint; empiri- cal FIM, identical to single path, except that the FIM is estimated using the covariance ma- trix of the gradients rather than the analytic estimate; max KL, which was only tractable on the cart-pole problem, and uses the maximum KL divergence in Equation (13), rather than the average divergence, allowing us to evaluate the quality of this approximation. The parameters used in the experiments are provided in Section 3.14. For the natural gradient method, we swept through the possible values of the stepsize in factors of three, and took the best value according to the final performance. Learning curves showing the total reward averaged across five runs of each algorithm are shown in Figure 4. Single path and vine TRPO solved all of the problems, yielding the best solutions. Natural gradient performed well on the two easier problems, but was unable to generate hopping and walking gaits that made forward progress. These results provide empirical evidence that constraining the KL divergence is a more robust way to choose step sizes and make fast, consistent progress, compared to using a fixed penalty. CEM and CMA are derivative-free algorithms, hence their sample complexity scales unfavorably with the number of parameters, and they performed poorly on the larger problems. The max KL method learned somewhat more slowly than our final method, due to the more restrictive form of the constraint, but overall the result suggests that the average KL divergence constraint has a similar effect as the theorecally justifiedPDF Image | OPTIMIZING EXPECTATIONS: FROM DEEP REINFORCEMENT LEARNING TO STOCHASTIC COMPUTATION GRAPHS
PDF Search Title:
OPTIMIZING EXPECTATIONS: FROM DEEP REINFORCEMENT LEARNING TO STOCHASTIC COMPUTATION GRAPHSOriginal File Name Searched:
thesis-optimizing-deep-learning.pdfDIY PDF Search: Google It | Yahoo | Bing
Cruise Ship Reviews | Luxury Resort | Jet | Yacht | and Travel Tech More Info
Cruising Review Topics and Articles More Info
Software based on Filemaker for the travel industry More Info
The Burgenstock Resort: Reviews on CruisingReview website... More Info
Resort Reviews: World Class resorts... More Info
The Riffelalp Resort: Reviews on CruisingReview website... More Info
CONTACT TEL: 608-238-6001 Email: greg@cruisingreview.com (Standard Web Page)