The main focus of this research is the domains of reinforcement learning (RL) and neural network in behavior learning of goalie humanoid robot in a three-dimensional soccer simulation environment. RL is a branch of machine learning to choose action in an unknown environment to maximize the cumulative reward. One of the most important goals of development in robotics and artificial intelligence is the winning of a team of autonomous humanoid robots against human teams in a soccer game. Among all the agent’s behavior, the goalie’s behavior in a soccer match is an important problem. RL in an environment with continuous states and actions provides a proper method for learning the agent’s behavior at any time. According to advances that have been made in this field, the goalie’s humanoid robot has been able to shut out much more ground shots by designed controllers and RL. Achieving better performance requires implementing a method to control the agent’s behavior to perform a proper response in a more complex environment and with various shots, including aerial shots. Therefore, control of agent behavior in complex environments will be necessary. But traditional RL algorithms are inefficient in the environment with two following attributes: 1) high-dimensional state spaces, such as pixels of camera images, 2) high-dimensional continuous action spaces. This research tackles the goalie problem using RL algorithm where two asynchronous RL learners are utilized to achieve better performance. performance on this problem is the number of shots shut out by the goalie in the goalie challenge. Recently, powerful RL methods such as Deep RL Method and RL with Actor-Critic architecture based on Policy Gradients method, have been proposed to solve robot control problems over a wide range of action spaces. Using these two methods and deep neural networks with more robust network architecture, a new hybrid method is proposed that can solve continuous control problems. In this research, first, the problem of goalie’s humanoid soccer robot is modeled using two reinforcement learners. To determine the state of the environment, a method is proposed to predict the trajectory of the ball. Then the skill description language is used to design skills such as dive to cover more area by the goalie and the action space is specified and then, by combining two reinforcement learners, doing behavior control of goalie humanoid robot. Finally, it has been shown that the RL agent in shutting out the ground and aerial shots, is more efficient than the methods implemented by top teams. Keywords Humanoid Soccer Robot, Three Dimensional Soccer Simulation, Goalie Challenge, Deep Reinforcement Learning, Policy Gradient, Deep Neural Networks, Behavior Learning, Continuous Control