Optical Communication systems have higher capacity than other communication systems, so they are widely deployed in internet infrastructure. Due to extension of online services, demand for more capacity is growing fast. As a result, there is a need to increase the capacity of the optical communication systems. Exploiting coherent detection is one of the proposed solutions to fulfil capacity demand. In-phase and Quadrature components of optical signal are achievable by beating between received signal and local oscillator beam. These two components are enough to use 2D modulation. Furthermore, coherent detection is sensitive to the polarization of the optical signal. This feature can be used to deploy polarization division multiplexing, so that 4D modulation would be available at these systems and capacity could have four-fold increase. There is another advantage for exploiting coherent detection. Since complete information of optical signal is reachable by coherent detection, compensation of signal is possible through digital signal processing methods. Loss, dispersion and nonlinear effects degenerate optical signal in optical fibers. Loss and dispersion have linear response, thus they could be compensated by low computational cost. Nonlinear effects are dependent to amplitude of signal, so their compensation is more complex. According to Shannon’s theory, capacity would increase by soaring signal power. Although increasing power would rise capacity, more power makes more degeneration through nonlinear effects. As a result, nonlinear effects should be compensated for rising capacity. Fundamental of coherent optical fiber communication systems and algorithms for these systems are demonstrated in this thesis. Equations for optical signal propagation in optical fiber are defined and nonlinear effects are categorized. Digital Back Propagation (DBP) and Nonlinear Fourier Division Multiplexing (NFDM) are two methods for nonlinear compensation, those are described in this thesis. These methods are implemented in DSP unit. DBP is inversed kind of optical fiber numerical simulation, otherwise NFDM is kind of analytical method and uses mathematical model for lossless fiber by inverse scattering. There are some simulations to qualify and evaluate the linear and nonlinear compensation algorithms. In addition, a new method is introduced. This method is a combination of optical phase conjugation and NFDM. Keywords: Coherent Optical Fiber Communication, Digital Signal Processing, Nonlinear Effects Compensation, Digital Back Propagation, Nonlinear Frequency Division Multiplexing