LAUSR

For Long Term Evolution Advanced (LTE-A) network, although there exist many studies that focus on improving the performance with relays, security issues are often neglected. Due to the broadcast nature of wireless channels, relay nodes in LTE-A network may act maliciously, affect communication, reduce quality, and cause delays. Recently, physical (PHY) layer security has attracted researchers to provide secure communication and data privacy. In this study, we propose using unsupervised learning approach at the destination node to detect malicious relay attacks in cooperative LTE-A network based on received source signal in the PHY layer. Outlier detection algorithms such as one class support vector machine (OCSVM), local outlier factor (LOF) and isolation forest ( $i$ Forest) are applied to detect various malicious relay behaviors such as garbling, regenerative, and false data injection type attacks. As input to these algorithms, feature vectors are constructed by using amplitude, phase, and relative phase information of modulated baseband symbols. The performance of the outlier detectors are evaluated with respect to precision, accuracy, and under the area curve (AUC) measures under changing signal-to-noise ratio (SNR) levels, different modulation types, allocated number of resource blocks (RBs), and varying data size. The results demonstrate the effectiveness of our proposed outlier detection approach for detecting malicious relays in the LTE-A network. Accuracy and precision of the algorithms are observed to be above 90% for 10 dB and larger SNR levels for the relay attack scenarios considered here. AUC values for all algorithms for all SNR levels is also above 0.9 for the attack detection cases, and the performance of the LOF algorithm with 0.95 and above AUC values is superior to other algorithms. The results verify the contribution of this study, which is the demonstration of the effectiveness of one class outlier detection approaches for detecting malicious relays in the LTE-A network.