At what point do people stop being immigrants?

If you go back far enough you will find that all our ancestors emigrated from somewhere else (unless you think the first humans appeared in London or where ever you live). At what point do people cease to be immigrants, when the bigots can't detect a difference? When they think they are English? after all nobody goes on about bloody Normans or Saxons.