The incomplete observations of the continuous time, discrete state space time series, and a potential large parameter sets makes sampling for continuous time Markov chains a challenging problem in the Bayesian framework. Generalized linear models (GLM) have been applied to design structured prior distributions with known properties of the process with applications to phylogenetics. Novel sampling algorithms has been constructed via a combination of auxiliary variable construction and an Adaptive Hamiltonian Monte Carlo (AHMC) algorithm. In our paper, we proposed an even more efficient algorithm which includes three components: the auxiliary variable construction techniques, the Local Bouncy Particle Sampler (LBPS) for sampling the bivariate features and the Hamiltonian Monte Carlo (HMC) for sampling the univariate features using the GLM parameterization. It has been demonstrated that our proposed algorithm has better scalability in terms of the increase in the dimension of the parameter space compared with the state-of-art AHMC samplers for Continuous Time Markov Chains (CTMCs).