Rashi on Genesis 21:10/רש"י על בראשית כ״א:י׳

1עם בני עם יצחק WITH MY SON, WITH ISAAC —For since this is my son, even though he were not as good as Isaac really is, or if he were as good a man as Isaac really is, even though he were not my son, this one (Ishmael) is not deserving of inheriting with him: how much less is he deserving of inheriting with my son, with Isaac — with one who possesses both these qualities (of being my son and of being the good man Isaac) (Genesis Rabbah 53:11).