Or HaChaim on Numbers 17:10/אור החיים על במדבר י״ז:י׳

1 הרמו מתון העדה הזאת, "get up from amongst this congregation, etc." G'd did not speak about Moses and Aaron but to all those who had not joined the ones who had accused Moses and Aaron. Proof of the fact that my interpretation is correct is the word לאמר, which always means that others are to be informed. The word הזאת, "this," further indicates that only the people who had actively accused Moses of wrongdoing were slated to be destroyed. This also explains the need for G'd to have said הרומו altogether. After all, G'd has no problem in killing the wicked in the very presence of the righteous as He had done when He killed the firstborn in Egypt and their family members did not have to leave their houses first. If you accept our interpretation, however, the instruction הרמו makes sense as even the people not guilty of accusing Moses now were under sentence of death already and unless they demonstrated that they were not involved in this sin the angel of death would execute them now. (I refer you to my commentary on Numbers 16,21).