In the presence of outliers, estimates of regression models for censored data can be heavily biased. This issue has previously been addressed by robust estimation methods for censored data. However, in practice, covariate effects often vary over time and, to our knowledge, no robust survival models that take into account time-varying effects exist. In this paper, we propose to robustly estimate the parameters of a piecewise constant hazard model by using the Brier score as loss function. By including a large number of cut-points the resulting time-varying covariate effects can be made quite flexible. However, this might lead to overfitting issues. To avoid such potential overfitting issues, a penalty which can shrink the time-varying effects towards constant effects is applied. The combination of the robust estimation procedure and penalization of the time-varying effects is then tested and exemplified using simulated and real-world data.