Or HaChaim on Numbers 22:19/אור החיים על במדבר כ״ב:י״ט

1 ועתה שבו נא בזה גם אתם, "And now, you stay here also, etc." Bileam pleaded as he was afraid that these dignitaries would do what the Midianite delegates had done when they heard that Bileam was waiting to get G'd's instructions. The Midianite delegates at the time had gone home immediately without staying overnight as they anticipated Bileam's refusal (compare Bamidbar Rabbah 20,8). Now that the delegation was large and consisted of highly placed people, Bileam was afraid that they would not be willing to wait.