In modern predictive modeling process, budget constraints become a very important consideration due to the high cost of collecting data. This motivates us to develop new and efficient high-dimensional cost constrained predictive modeling methods. In this talk, to address this challenge, we first study a new non-convex high-dimensional cost-constrained linear regression problem. The non-convex budget constraint makes this problem NP-hard. In order to estimate the regression coefficient vector of the cost-constrained regression model, we propose a new discrete extension of recent first-order continuous optimization methods. We further show some extensions of our proposed method. Both theoretical and numerical studies are used to demonstrate the effectiveness of the proposed method.