Precise identification of Fetal movements is necessary for continuous prenatal monitoring; however, the signals obtained from wearable accelerometers are always affected by artifacts induced by the maternal movements and environmental noises. This research proposes a hierarchical signal refinement paradigm that can iteratively refine inertial sensor signal data from wearables in order to detect fetal movements in realistic monitoring environments. The hierarchical signal refinement paradigm uses a well-defined process to eliminate noise, extract compact features, and extract discriminative features to be able to conduct multiclass classification. Performance evaluation of this approach has been conducted using a publicly available fetal movements benchmark dataset containing synchronized tri-axial accelerometer data corresponding to fetal movements, mother's respiration, and laughing. The experimental results showed 97.94% of accuracy, 97.00% of precision, 96.00% of recall, 97.19% of F1-score, and 98.21% of area under the receiver operating characteristic curve. In particular, the ablation experiment validated the contributions of each signal refinement step, while statistical significance and computational cost analysis further confirmed the effectiveness, reliability, and efficiency of the proposed framework. Compared with recent fetal movement recognition methods, HSRF provides an interpretable and effective signal refinement strategy suitable for continuous wearable prenatal monitoring and intelligent maternal healthcare applications.