A study proposes that the population that gave rise to modern humans may have been reduced to roughly 1,300 reproducing individuals