Linear mixed-effects models are widely used in analyzing repeated measurement and longitudinal data, where the inferences of both fixed-effects and variance components are of importance. Although statistical inference of the fixed effects is well studied, inference of the variance component is rarely explored, which often requires strong distributional assumptions on the random effects and errors. In this paper, we develop empirical likelihood-based methods for the inference of variance components in the presence of fixed effects. A non-parametric version of the Wilks' theorem for the proposed statistics is derived. Simulation studies demonstrate that the proposed methods exhibit better coverage than the commonly used likelihood ratio method with Gaussian assumption and the results with unknown fixed effects are as good as those with known fixed effects. The new tests are illustrated in the analysis of a wearable device dataset.