Crop yield forecasting plays an important role in the planning and management of fields. Precision agriculture had a market cap of $1.2B in North America in 2016 and is expected to grow to $2.2B by 2022. Forecasting yield becomes especially challenging when we have a short time series. In this paper, we propose a model for forecasting yield based on datasets which have a few points in time but have a large number of points in space (10 m spatial resolution in 800 x 800 m domain). Different type of models exists to forecast yield at the state level and county level, but not a lot of models estimate yield at a farm level. In this paper, we develop a two-stage model to forecast yields at a farm level. In the first stage, we use clustering algorithms to form clusters based on commonly available geophysical variables. In the second stage, we apply a spatially varying auto-regressive model and obtain yield forecasts. We compare the forecasting performance of our model with the traditional time series model and a few machine learning algorithms. The results show that for short time series with high spatial resolution data, our proposed model outperforms other models.