The rapid proliferation of Internet of Things (IoT) devices has created complex network environments increasingly vulnerable to sophisticated cyber attacks. Detecting anomalous communication patterns in such heterogeneous networks requires mathematical models capable of capturing the multi-faceted nature of IoT traffic. This paper develops a multi-layer graph-theoretic framework for detecting anomalous communication patterns in IoT networks. The proposed model represents network traffic as a multi-layer graph where each layer corresponds to a different communication modality including TCP, UDP, ICMP, HTTP, and MQTT. Unlike prior works that assume stationary Poisson processes, we propose a dynamic negative binomial model with an overdispersion parameter to capture burstiness and a time-varying function to model diurnal patterns. The framework integrates three complementary mathematical approaches: spectral analysis using random matrix theory for global structural anomalies, local neighborhood analysis using graph signal processing for node-level behavioral deviations, and inter-layer correlation analysis using tensor decomposition for coordinated multi-vector attacks. To ensure practical robustness under non-stationary conditions, we introduce permutation-based threshold calibration that controls false positive rates even when theoretical assumptions are violated. Comprehensive sensitivity analysis is provided for all hyperparameters including integration weights, time window length, and tensor rank. Fair comparative evaluation is conducted against six state-of-the-art graph-based methods including Graph Convolutional Networks (GCN), Dynamic Graph Neural Networks (DyGNN), Multi-layer Graph Convolutional Networks (M-GCN), GraphSAGE, Graph Attention Networks (GAT), and Ensemble Graph Convolutional Networks (E-GCN). Numerical experiments on real IoT traffic datasets from CICIDS2017 and Bot-IoT demonstrate that the proposed framework achieves a detection rate of 89.2\% with a false positive rate of 3.8\%, outperforming the leading baseline M-GCN by 1.7\% in detection rate. The computational complexity scales linearly with network size, enabling near real-time deployment in large-scale IoT environments.