Problems in robotics domain are usually characterized by high-dimensional continuous state and action spaces. Partially observable and noisy states are accessible, rather than the actual state. Controlling autonomous robots in this domain is consequently challenging. The idea of interacting with the environment to autonomously find an optimal behavior is the essence of reinforcement learning. RoboCup Soccer is a commonly used testbed for reinforcement learning methods. Continuous multi-dimensional state space, noisy actions and perceptions and high uncertainty of this environment make reinforcement learning appropriate to use in this domain. Many tasks have so far been learned in this domain with these methods. Keepaway and Half Field Offense are two instances that incorporate suitable tasks for reinforcement learning. Such tasks have mostly been learned in 2D soccer. Because of an additional dimension and physical constraints, learning is much more difficult in 3D soccer. Applying 2D environment algorithms in 3D space faces new challenges. Extending Keepaway from 2D soccer to 3D soccer is an example of such efforts done so far. Reinforcement learning problems typically feature discrete or continuous action spaces. Parameterizing each discrete action with continuous parameters makes it possible to fine-tune actions in different situations. Learning in such parameterized action spaces is complicated by the necessity of dealing with continuous parameters. However, it provides the most fine-grained control over the agent’s behavior. One method of learning in this domain is to define separate policies for discrete actions and the continuous parameters of each action and then alternate learning these policies from interaction with the environment. In this study, a single-agent task of Half Field Offense is learned in a parameterized action space in the domain of 3D soccer simulation. The agent must learn to maintain the possession of the ball while it makes its way towards the goal and finally score on the goal at an appropriate time. The performance of the agent is evaluated by the number of goals scored. One of the reasons that makes this study important is that this task has never been implemented in the 3D environment before. Furthermore, making use of a parameterized action space and learning two separate policies for discrete actions and continuous parameters, entails using value-based methods along with policy search methods in an environment of such great complexity. Final results demonstrate that despite a large state space and noisy intricate actions, the agent succeeds in learning these two policies. The agent has been successful in maintaining an uptrend in the number of goals scored in different test scenarios. Keywords: Machine Learning, Reinforcement Learning, Parameterized Actions, Robotics, 3D Simulation, RoboCup, Half Field Offense