Heritability is a common measure that quantifies the proportion of phenotypic variation explained by genetic factors. When only single nucleotide polymorphisms (SNPs) are available in genome-wide analysis, the estimate of heritability is called SNP heritability. Existing methods are mainly based upon the linear model structure; however, genetic architecture is more complicated in reality. Here we propose to consider the correlation between SNPs and the non-parametric relationship between variants and phenotype into the analysis based on the generalized additive model (GAM). Before fitting GAM, the analysis procedure begins with clustering SNPs and then dimensionality reduction is performed, for some SNPs may together contribute to the phenotype and this process can preserve the information. The results indicate that the proposed method performs better than current methods. In addition, the cluster-level heritability is estimated by our GAM-based method. This method can be applied to other types of DNA variation data, and can be extended to analyze different categorical data.