It is long known that climatological summary statistics based on daily temperature minima and maxima suffer from bias due to the observation time. We develop a novel approach to addressing this problem, by imputing hourly temperatures at the location of the measurements, which can then be used for statistical summary purposes. We propose a spatio-temporal Gaussian process model to combine information from the observed minima and maxima, and from nearby weather stations that record hourly temperatures. To efficiently impute the hourly temperatures from our model, we develop a novel MCMC algorithm. We validate our imputation model using hourly temperature data from four meteorological stations in Iowa, one of which has its data concealed and replaced with daily minima and maxima, and show that the imputed temperatures closely match the hidden temperatures. We also demonstrate that our model can exploit information contained in the data to infer the time of daily measurements.