Cooperative control of multi-agent systems (MAS) has been an active area in the past decade. Cooperative agents in MAS must achieve a common goal under communication constraints. The goal is to design a distributed control law such that each agent can exhibit a specified behavior, such as consensus, formation, rendezvous, flocking, synchronization, coordination, and covering, using only local information from its neighbors. In several applications, agents are modeled as rigid bodies, where the state of each agent is on nonlinear space that is called Lie group. In this study, in the first step, the geometric properties of the Lie group SO(3) has been exploited in a global and coordinate-free approach to avoid singularity and ambiguities associated with the attitude representation of Euler angles and quaternion. Without embedding the Lie group SO(3) in Euclidean spaces, a new approach for analyzing the attitude synchronization problem for multiple rigid bodies is developed. Using the same procedure as that for the Euclidean case, the consensus law is proposed based on the gradient of a smooth cost function which depends on the relative attitude of neighboring agents. It is proved that for every connected and undirected graph, convergence to synchronization can be achieved using the proposed consensus law and a sufficiently small step size if the initial attitudes of agents are adequately close. The sets of initial attitudes that ensure convergence to synchronization are obtained for three types of graph topologies, i.e. tree, complete, and ring. It is shown that the initial set is more restrictive for ring graph topology with N 5 agents, owing to the existence of other critical points for the cost function induced by graph topology. Moreover, the exceptional case in which the ring graph has four agents has been discussed. However, characterizing the set of initial conditions depends on the critical points of the cost function which depends on communication graph topology. Hence, using the appropriate Lyapunov function, sufficient condition has been derived that guarantees synchronization occurs for any connected and undirected graph. Using the obtained results and the idea of auxiliary variable on the Lie group SO(3), the consensus law for any initial attitude of rigid bodies in a connected graph is proposed. Then, the consensus-based formation control laws for MAS which is described by the kinematic equation, with or without a leader, are proposed. Applying the viscosity solution for the Hamilton Jacobi problem on the rotational group, in the reverse approach, the consensus laws designed on the Lie algebra so(3) are transferred to the equivalent consensus laws on the Lie group SO(3). The advantage of these consensus laws lies in the fact that convergence to the synchronization doesn't depend on the graph topology. In the next step, considering bounded external disturbance in the dynamic equation of rigid bodies and using the robust sliding mode control, the formation problem on Lie group SE(3) is discussed in virtual structure and consensus-based approach. To deal with external disturbance in a network of rigid body an adaptive intrinsic PID sliding mode control is proposed to the formation control of MAS in a virtual structure. The advantage of this control method is that it is robust with respect to external disturbances and parametric uncertainties. Since in real applications, the performance of virtual structure-based method was greatly deteriorated by failure in central unit, or existence of communication constraint, the new error vector based on the gradient of modified error function on the Lie group SE(3) is applied to design sliding mode law for consensus-based leader-follower formation control. Finally, in the presence of uncertainty in the inertial matrix and existence of known and unknown external disturbance the robust consensus laws based on the adaptive control are proposed. Numerical simulation results have also illustrated the effectiveness of the proposed control schemes.