Sforno on Numbers 22:12/ספורנו על במדבר כ״ב:י״ב

1לא תלך עמהם. Even if you were not to curse the people they referred to you are not at liberty to go with them. The reason is that I do not want you to set your evil eye on these people even without uttering any words. (compare the expression נתן עיניו בו in B’rachot 58 In that instance the victim was reduced to a heap of bones.)