As latency sensitive applications, such as online video chatting and virtual reality become popular, end-to-end latency prediction is becoming an important problem. Traditional methods for latency prediction are static, and are unsuitable to predict time-varying round-trip times between the servers and edge devices. Though distance-feature decomposition is able to predict time-varying round-trip times with time sampled information, it fails to utilize network similarity for better prediction. When the time correlation is weak between different sample matrices, it performs no better than static prediction algorithms. But in most cases, as long as network structure remains the same, network similarity still exists even if round-trip times change greatly over time. In this paper we show that similar patterns of round-trip time sequences exist both across time and among different pairs of devices. By exploiting both time correlation and network similarity, we can achieve much lower prediction error even if time correlation of 3D sampled data is weak.