X-chromosome is often excluded from whole-genome association studies due to a number of complexities. Some are apparent, e.g. sex-specific allele frequencies, sex-gene interaction effects, and the choice of (additive or other) genetic models, while others are subtler, e.g. random, skewed or no X-inactivation, and the choice of risk allele. In this work, we aim to consider all these complexities jointly and propose a regression-based association test. We provide theoretical justifications for its robustness in the presence various aforementioned model uncertainties, as well as for its improved power under certain alternatives as compared with existing approaches. For completeness, we also revisit the autosomes and show that the proposed framework leads to a robust and sometimes much more powerful test than the standard method. Finally, we provide supporting evidence from simulation and application studies.