Asymptotic stability of the critical Fisher-KPP front using pointwise estimates