M. Hofreiter / K. Fiusterweier

Researchers say some Neanderthals may have had pale skin and red hair similar to that of some modern humans. Explaining the genetic similarities, however, can lead to a tangled tale.

One of the most titillating tales in the study of human origins — focusing on whether Neanderthals interbred with modern humans — has just gotten more tangled.

Over the past couple of years, studies of Neanderthal DNA samples painstakingly extracted from ancient bones have suggested that contemporary non-Africans can trace up to 4 percent of their genetic code to our long-extinct Neanderthal cousins. The genomes of modern-day Africans, in contrast, have virtually nothing in common with the Neanderthals. Researchers assumed that the genetic contribution for the non-Africans was passed down through cross-species sex during the time that Neanderthals and Homo sapiens lived in close proximity in Europe, tens of thousands of years ago.

However, there's another possibility: Maybe that common genetic code was passed down from the common ancestor of Homo sapiens and Homo neanderthalensis, hundreds of thousands of years ago in Africa. Today, researchers at the University of Cambridge reported in the Proceedings of the National Academy of Sciences that such a scenario provides a better fit for the genetic data. They say there's no need to assume that anatomically modern humans did the Neanderthal nasty, a process known more scientifically as hybridization. 

"Our work shows clearly that the patterns currently seen in the Neanderthal genome are not exceptional, and are in line with our expectations of what we would see without hybridization," the lead researcher, Andrea Manica, said in a Cambridge news release. "So, if any hybridization happened — it's difficult to conclusively prove it never happened — then it would have been minimal and much less than what people are claiming now."

Modeling population dynamics
Manica and his colleagues set up a computer model for the last half-million years of population dynamics, with the assumption that there were two migrations from Africa. The first migration led to the settlement of Europe by the ancestors of the Neanderthals, hundreds of thousands of years ago. Then, around 300,000 to 350,000 years ago, the route from north Africa to Europe was cut off somehow. The European and north African populations showed gradual genetic divergence, but still retained a bit of common heritage from their mutual ancestors.

When the second migration from Africa took place, around 60,000 to 70,000 years ago, the north Africans who dispersed to Europe and Asia would carry that extra bit of genetic similarity with them. But the Africans who lived farther south and stayed behind on the continent wouldn't have as much genetic kinship with the Europeans.

The researchers found that their model did a fine job of accounting for the existing data without Neanderthal sex.

So what do the researchers behind the earlier DNA studies say? That's where it gets really interesting: One study, published online in April in Molecular Biology and Evolution, contends that ancient population dynamics alone can't account for the genetic patterns seen in the DNA from Neanderthals and modern humans. Another study, posted on the arXiv preprint server and due for publication in PLOS Genetics, takes a closer look at a genetic pattern known as linkage disequilibrium — and concludes that modern humans and Neanderthals interbred somewhere between 37,000 and 86,000 years ago. Nature's Ewen Callaway delves into the details surrounding those claims and counterclaims.

Either-or proposition?
So, to recap: Some scientists say the population dynamics that were in effect hundreds of thousands of years ago can explain genetic similarities between populations, even if those populations never interbred. Others say the evidence is getting stronger that modern humans and Neanderthals really did mate when they met up in Europe, tens of thousands of years ago.

University of Washington geneticist Joshua Akey says both sides just might be right.

"To me, I don't think it's a case of either-or," he told me. "I think that both things can be going on."

Akey and other researchers recently published a study in the journal Cell suggesting that a mysterious "Neanderthal sibling species" made a genetic contribution to the DNA of modern-day Africans. He said that interpreting whether genetic similarities come from a common ancestor (a process known as archaic population structure) or from more recent cross-species sex (a process that Akey calls introgression) is a tricky but essential task for those who study human origins.

"Ultimately, it's important that we come to a consensus as to one process or the other, but I find them both to be interesting interpretations," Akey said. "Introgression is a sexier mechanism, but even if it turns out to be a case of archaic population structure, that still tells us something about our past that we didn't know before."

So did modern humans do it with Neanderthals or not? And to what extent? Today, maybe it's a tangled tale, but that won't necessarily be the case forever. Akey said he was optimistic that researchers will be able to "tease apart" the different influences from the two processes within the next year or so.

Update for 2:10 a.m. ET Aug. 14: In a follow-up email, Akey clarified his view on the "did they do it" question, leaning toward the affirmative:

"Although I do think that both ancestral population structure and introgression are not mutually exclusive events, the recent papers from David Reich and Monty Slatkin show pretty compelling evidence that introgression of Neanderthal lineages into anatomically modern humans occurred. Thus, the real debate moving forward will be about the relative contributions of these two processes."

More about ancient hominid sex:

• Neanderthal-human sex rarely produced kids
• Ancestors had sex with mysterious human cousins
• How sex with Neanderthals made us stronger

Cambridge's Anders Eriksson is Manica's co-author on the paper published by the Proceedings of the National Academy of Sciences, titled "Effect of Ancient Population Structure on the Degree of Polymorphism Shared Between Modern Human Populations and Ancient Hominins."

Authors of the study in Molecular Biology and Evolution are Melinda Yang, Anna-Sapfo Malaspinas, Eric Durand and Montgomery Slatkin.

Authors of the preprint destined for PLOS Genetics are Sriram Sankararaman, Nick Patterson, Heng Li, Svante Pääbo and David Reich.

Alan Boyle is NBCNews.com's science editor. Connect with the Cosmic Log community by "liking" the log's Facebook page, following @b0yle on Twitter and adding the Cosmic Log page to your Google+ presence. To keep up with Cosmic Log as well as NBCNews.com's other stories about science and space, sign up for the Tech & Science newsletter, delivered to your email in-box every weekday. You can also check out "The Case for Pluto," my book about the dwarf planet and the search for new worlds.