Modern neuroimaging technologies, combined with state-of-the-art data processing pipelines, have made it possible to collect longitudinal observations of an individual's brain connectome at different ages. It is of substantial scientific interest to study how brain connectivity varies over time in relation to human cognitive traits. In brain connectomics, the structural brain network for an individual corresponds to the strength of connection between each pair of brain regions. We propose a symmetric bilinear logistic regression to learn a set of small outcome-relevant subgraphs from subjects' longitudinal structural brain networks as well as estimating the time effects of the subgraphs. The clique structure of the extracted signal subgraphs can be related to some neurological circuits and hence gives interpretable results. Time effect of each signal subgraph reflects how the predictive effect on the outcome varies with age, which may guide the optimal age for taking a brain scan to diagnose a neurological disorder. We apply the method to longitudinal brain connectome and cognitive capacity data.