In this dissertation, we study the problem of waveform design for improvement of detection performance in radar systems. To this end, for single-input single-output (SISO) systems, we consider the effect of the signal-dependent interference at receive side (i.e., clutter) and the fact that Doppler shift of targets are often unknown at the transmit side. The raised design problems (for various situation) are non-convex and in some cases, belong to justify; LINE-HEIGHT: normal; MARGIN: 0cm 0cm 0pt; mso-layout-grid-align: none" tackle the design problems and extend the proposed algorithms to the case of constrained design (peak-to- average-power-ratio (PAR) or similarity constraint). In case of multiple-input multiple-output (MIMO) systems, we derive the exact theoretical performance expressions of the optimal detector in Gaussian interference. Due to the complicated forms of the expressions, we use the most common information-theoretic criteria including Bhattacharyya distance, KL-divergence, J-divergence, and mutual information as design metrics. We cast the design problems under a unified optimization framework and devise two algorithms to deal with these problems. We extend the proposed methods to the case of PAR-constrained design. We also propose a novel method for (constrained) transmit code design in non-orthogonalMIMO radars.