Abstract Multiple-antenna communication system is an important research topic in the past decades. It increases the data rate or diversity in reception, without occupy- ing additional frequency or time resource. On the other hand, amplify-and-forward (AF) relaying attracts a lot of attention lately, as it is suitable in cases where the source cannot directly communicate with the destination, but is possible via a relay in the middle. The AF relay simply amplifies the received signal without decoding, thus its operation is favorable in implementation. The combination of multiple- input multiple-output (MIMO) communication and AF relaying technique is cur- rently under consideration for several future wireless communication standards. With the source, relay and destination all equipped with multiple antennas, a natural question is how to allocate the limited power resource to make the commu- nication as efficient as possible. This problem is addressed by linear transceiver design in this thesis. Transceiver designs for point-to-point MIMO or multi-user MIMO systems have been widely addressed previously. However, for AF MIMO xi relaying system, due to the relaying operation, transceiver design becomes more challenging. In this thesis, we start with a fundamental three nodes source-relay-destination MIMO system. The forwarding matrix at relay and equalizer at destination are jointly designed, under the realistic scenario that channel estimates in both hop contains Gaussian error. Two robust design algorithms are proposed to minimize the mean-square-error (MSE) of the output signal at the destination. The first one is an iterative algorithm with its convergence proved analytically. The other is an approximated closed-form solution with much lower complexity than the iterative algorithm. Next, we consider the AF MIMO orthogonal frequency division multiplexing (OFDM) system over frequency selective fading channels. Again, the forwarding matrix at relay and equalizer at destination are jointly designed by minimizing the total MSE of the output signal at the destination, under channel estimation errors. However, since OFDM is a multicarrier modulation, transceiver design in such sys- tem involves power allocation in both spatial and frequency domains, and thus is more complicated than the first system. In the proposed solution, the second-order moments of channel estimation errors in the two hops are first deduced in the fre- quency domain. Then, the optimal designs for both correlated and uncorrelated channel estimation errors are investigated. The relationship between the proposed solutions with existing algorithms is also disclosed. xii Finally, we consider the AF MIMO relaying system with multiple users. It cor- responds to the case where one base station communicates with multiple terminals via one relay station. In this system, the source precoder, relay forwarding matrix and destination equalizer are jointly designed by minimum MSE criterion. Both uplink and downlink cases are considered. It is found that the uplink and downlink transceiver designs share some common features and can be solved by a general iterative algorithm. On the other hand, another proposed algorithm for fully loaded or overloaded uplink system is shown to include several existing results as special cases. Table of Contents Page Declaration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . i Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iii List of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viii Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . x 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Background . . . . . . . . . . . . . . . . . . . . 1.1.1 Cooperative Communication . . . . . . 1.1.2 Multiple-Input Multiple-Output Systems 1.1.3 AF MIMO Relay Systems . . . . . . . . Research Motivation and Problems to be Tackled Organization and Contributions of the Thesis . . Commonly Used Notations . . . . . . . . . . . Robust Transceiver Design for AF MIMO Relay Systems Introduction . . . . . . . . . . . . . . . . . . . . . . . System Model . . . . . . . . . . . . . . . . . . . . . . Problem Formulation . . . . . . . . . . . . . . . . . . . The Proposed Iterative Algorithm . . . . . . . . . . . . 2.4.1 Updating G given F . . . . . . . . . . . . . . . 2.4.2 Updating F given G . . . . . . . . . . . . . . . 2.4.3 Summary and convergence analysis . . . . . . . The Proposed Closed-Form Solution . . . . . . . . . . Extension to Weighted MSE Criterion . . . . . . . . . . Simulation Results and Discussions . . . . . . . . . . . 2.7.1 Simulation Setup . . . . . . . . . . . . . . . . . 2.7.2 Convergence Performance of Iterative Algorithm 2.7.3 Effect of Estimation Error σe . . . . . . . . . . . . . . . 2.7.4 Effect of Correlation Coefficients, α and β . . . . . . . . 2.7.5 BER Performance . . . . . . . . . . . . . . . . . . . . . 2.8 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.9 Proof of q(γi+1 ) is monotonically decreasing and upper bound on γi+1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.10 Proof of MSEU (F) ≥ MSE(F) . . . . . . . . . . . . . . . . . . ̃ 2.11 Derivation of optimal F . . . . . . . . . . . . . . . . . . . . . . Robust Transceiver Design for AF MIMO OFDM Relay Systems . . 41 Introduction . . . . . . . . . . . . . . . . . . System Model . . . . . . . . . . . . . . . . . Channel Estimation Error Modeling . . . . . . Transceiver Design Problem Formulation . . . Proposed Closed-Form Solution . . . . . . . . 3.5.1 Uncorrelated Channel Estimation Error 3.5.2 Correlated Channel Estimation Error . 3.6 Simulation Results and Discussions . . . . . . 3.7 Conclusions . . . . . . . . . . . . . . . . . . 3.8 Proof of (3.7) . . . . . . . . . . . . . . . . . . 3.9 Proof of (3.17) . . . . . . . . . . . . . . . . . 3.10 Proof of Property 1 . . . . . . . . . . . . . . . 3.11 Proof of Property 2 . . . . . . . . . . . . . . . 3.12 Proof of Property 3 . . . . . . . . . . . . . . . LMMSE Transceiver Design for AF MIMO Relaying Cellular Net- works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . Downlink Transceiver Design . . . . . . . . . . . . . . . . . . . 4.2.1 System model and problem formulation . . . . . . . . . . 4.2.2 Proposed iterative algorithm . . . . . . . . . . . . . . . . 4.2.3 Summary and Initialization . . . . . . . . . . . . . . . . Uplink Transceiver Design . . . . . . . . . . . . . . . . . . . . . 4.3.1 System model and analogy with downlink design . . . . . 4.3.2 Uplink transceiver design for fully loaded or overloaded systems . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3.3 Special cases . . . . . . . . . . . . . . . . . . . . . . . . Simulation Results and Discussions . . . . . . . . . . . . . . . . Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98 Conclusions and Future Research . . . . . . . . . . . . . . . . . . . . 99 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 Future Research Directions . . . . . . . . . . . . . . . . . . . . . 100 List of References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102