Logically, people should have the right of free movement. In practice, people in not-miserable places don't want to get overrun by people fleeing from places that are uninhabitable due to war, starvation, or increasingly in modern times, environmental foreclosure. I suspect the latter will win over the former.
It also won't make much difference. People won't stay and die. As climate change makes more of the world uninhabitable, people will flee. If they aren't welcomed, they'll simply overrun the barriers. Even a castle is only good for 10:1 odds. The moment 11:1 people hate you, then you're fucked.
We can either set up a rational means of handling this issue now, or get stuck with irrational ones later.