Or HaChaim on Numbers 25:4/אור החיים על במדבר כ״ה:ד׳

1 וישוב חרון אף השם, so that the fierce anger of G'd may turn away, etc. G'd's anger had to be turned away even after justice had been done because whereas the courts in this world can only convict people who have been warned by accredited witnesses and who have committed their sin in the presence of these witnesses, others, equally guilty, had not been dealt with. If G'd were allowed to remain angry He would give permission to the angel of death to kill all those sinners who had not been warned or whose deed had not been witnessed by their peers. If G'd's anger could be turned away, these people would remain alive at this time.