Between-group comparison based on the restricted mean survival time (RMST) is getting attention as an alternative to the conventional logrank/hazard ratio test/estimation approach for time-to-event outcomes in randomized controlled trials (RCT). The validity of the commonly used nonparametric inference procedure for the RMST-based treatment contrast measures (such as difference or ratio of RMST) has been justified by large sample theories. However, we sometimes encounter small sample size cases in practice, where the large sample approximation may not work well. Generally, the permutation approach can be useful to handle these situations. However, several numerical issues arise in implementing permutation-based inference for between-group comparison of RMST. We discuss these numerical issues and consider three kinds of permutation methods for the inference of difference in RMST between two groups in RCT settings. We conduct extensive numerical studies and show the performance of these methods, compared to the commonly used asymptotic approach.