Better utilization of the existing ac transmission lines, without risking the instability, is a major challenge in power system operation and control. In many cases, the instability occurs because of poor damping of power oscillations in long distance or weak transmission lines. One approach to reduce the risk of instability, thus increase the power transfer capacity, is Power Oscillation Damping improvement. Nowadays, series FACTS controllers such as Static Synchronous Series Compensator (C), demonstrate superior and competitive features to improve power oscillation damping. Control issues of the C have been addressed by a number of researchers. Traditional linear Single-Input Single-Output (SISO) controllers design approaches such as eigenvalue/eigenvector analysis, and root locus are only useful when the system states remain in the vicinity of a specific operating point. In addition, in most SISO approaches the dynamic interactions among control variables and various control loops are neglected. A serious issue is the dc-link voltage dynamic, which is assumed to be constant in most SISO design. This simplification can lead to serious problems in a practical implementation. In this thesis, multivariable and nonlinear controllers are proposed to improve the power oscillation damping in a SMIB power system using C. Considerable interactions among the C control variables and power system variables are taken into consideration, and the dc-link voltage dynamic is not ignored. Accordingly, a nonlinear multivariable model for the SMIB power system equipped with an C is developed. A multivariable controller in frequency domain is designed. The proposed multivariable controller minimizes the interactions among system variables, and is less sensitive to the variation of the operating point. In nonlinear system theory, obtaining the Input-Affine model for the system, if possible, is a critical step in the design and implementation of many nonlinear and robust control approaches. In this dissertation, such a nonlinear and multivariable input-affine model for a power system installed with an C is developed for the first time. Next, based on the developed input-affine model, and differential geometric approach, a Multi-Input Multi-Output (MIMO) feedback linearized nonlinear controller is designed. The proposed controller is employed to improve power oscillation damping in addition to adjusting the dc-link voltage of the C. Simulation results, carried out in a PSCAD-V4.1 software, show the viability of the developed system model and the proposed controllers. Simulation results also justify