%0 Journal Article
%T Humanoids Learning to Walk: A Natural CPG-Actor-Critic Architecture
%A Cai Li
%A Robert Lowe
%A Tom Ziemke
%J Frontiers in Neurorobotics
%D 2013
%I Frontiers Media
%R 10.3389/fnbot.2013.00005
%X The identification of learning mechanisms for locomotion has been the subject of much research for some time but many challenges remain. Dynamic systems theory (DST) offers a novel approach to humanoid learning through environmental interaction. Reinforcement learning (RL) has offered a promising method to adaptively link the dynamic system to the environment it interacts with via a reward-based value system. In this paper, we propose a model that integrates the above perspectives and applies it to the case of a humanoid (NAO) robot learning to walk the ability of which emerges from its value-based interaction with the environment. In the model, a simplified central pattern generator (CPG) architecture inspired by neuroscientific research and DST is integrated with an actor-critic approach to RL (cpg-actor-critic). In the cpg-actor-critic architecture, least-square-temporal-difference based learning converges to the optimal solution quickly by using natural gradient learning and balancing exploration and exploitation. Futhermore, rather than using a traditional (designer-specified) reward it uses a dynamic value function as a stability indicator that adapts to the environment. The results obtained are analyzed using a novel DST-based embodied cognition approach. Learning to walk, from this perspective, is a process of integrating levels of sensorimotor activity and value.
%K reinforcement learning
%K humanoid walking
%K central pattern generators
%K actor-critic
%K dynamical systems theory
%K embodied cognition
%K value system
%U http://www.frontiersin.org/Journal/10.3389/fnbot.2013.00005/abstract