Nowadays, many social networks can record time-to-event data for a pair of nodes. However, despite the large body of literature on social network analysis, statistical methods for analyzing time-to-event data on social networks are still lacking. On the one hand, the pairs having time-to-event data are usually sparse on the whole network. On the other hand, the pairs with recorded data also have very small sample sizes. To allow efficient pair-specific inference, here we propose a semi-parametric frailty model with the network structure to borrow strengths. We adopt the Variational Expectation-Maximization (EM) method for fast computation and apply the proposed method to a corporate email dataset.