GENEWATCH
 
THE BROKEN CLOCK: ACCURACY AND UTILITY OF DTC TESTS
By CRG staff - interview with James Evans
 


James Evans, MD, PhD, directs the Clinical Cancer Genetics Services at the University of North Carolina. He delivered testimony at the July congressional hearings focusing on a Government Accountability Office (GAO) investigation which criticized DTC genetic tests as "misleading and of little or no practical use to consumers."

 

How did you get involved in the GAO report hearings?

They called me a year or two ago. They told me that they were concerned about some of these direct-to-consumer testing companies' offerings, and I have some concerns as well. I don't think that the sky is falling because of the existence of these things, but I have some concerns. My biggest concerns have to do with the false claims that are made by these companies and the fact that we don't really know how to interpret this kind of information.

So the GAO was looking to design a way to  investigate some of these concerns, and I think that the strategy they ultimately took was a good way of investigating and illustrating that we are not ready to interpret this information with any degree of reliability. Then there is a second question: Even if we were able to interpret these results reliably, would it tell us anything of any real significance?

They investigated that first question and eloquently and elegantly demonstrated that we are not ready to interpret much of this data. There's just no way of reconciling claims that it's usable information with the fact that reputable companies conduct analyses on the same DNA and come up with radically different interpretations. There's no way to reconcile that information with claims that it's ready for prime time.

What the GAO did not investigate—and really could not investigate, I think—was the other issue. The question that remains is: Even if there was consistency, and we learned to interpret it, would it provide any utility to patients? And the answer to that, I think, is largely a resounding no.

Industry critics have claimed that the GAO's methods were not scientific. Do you share any of those concerns?

It's an interesting criticism, because they actually did an elegant experiment; it was entirely scientific. They had, really, the ultimate control. They took the same sample to different companies and simply presented the results: for the same exact sample, one company says the individual has an increased risk of prostate cancer, one says he has an average risk, and one says he has a low risk. One of them is right, but it's a little like the broken clock which is right twice a day.

We have no idea, as that experiment readily demonstrated, how to interpret some of this genetic information. So the idea that the report was not scientific is, I think, a rather silly accusation. They did an experiment, and the results speak for themselves.

In your testimony you took issue with the marketing claims made by DTC testing companies. Are there any specific claims that you see commonly made which you find especially egregious?

Yes. The three big players in this field, the top strata of these companies, are doing a fine job of telling you reliably which nucleotide you have at a given position, but all of these companies make implicit and explicit claims that the information will improve your health. All you have to do is look at their home page on their websites, look at their advertising, and they all make some claim along the lines of "Understanding your genes will be a roadmap to better health" or "Take control of your future with genetic analysis." They are all making explicit or implicit claims that knowing your genetic information will improve your health; and, frankly, there is no evidence that this is the case.

They spout platitudes that, for example, people will be motivated to lose weight or live a healthier lifestyle. Firstly, there is little evidence that this is the case; secondly, even if this is the case, if somehow genetic information has some magical properties that make it particularly motivating, then we have a bigger problem: it is arithmetically guaranteed that for everyone who has increased risk of a condition, there are an equal number with decreased risk. If this information is actually so motivating, we run the significant risk of altering people's behavior for the worse. And frankly, the magnitudes of the risk shift that they are giving people are practically meaningless. Finding out that you are at a twofold or a fifty percent risk of heart disease over the general population is essentially meaningless since these are common diseases that we remain at significant absolute risk for whether or not we are at some relatively decreased genetic risk.

So you are concerned not only about the possibility of false reassurance, but even reassurance that isn't necessarily false.

Exactly, it may not even be false! In other words, I might tell you that you're at a 50% risk of heart disease over the general population, but that relative risk is rather meaningless. You are still at a high risk for heart disease simply because it is a very common malady. Millions of people out there who are at relatively "low risk" for heart disease end up dying of it!

Because of environmental factors?

Yes, and because it's simply a common disease. But what you're getting at is exactly right: genetics is only one small part of our risk for most of these diseases. Therefore, even if we understood completely the genetic risk for diabetes and heart disease and cancer, we still would be left with a huge amount of uncertainty because the causation of these maladies is multifactorial.

My other gripe is that in the results they send to the consumer, some of these companies mix pure entertainment—like "Do you have thick earwax?"—with a tiny subset of information that is very medically meaningful. A small percentage of the information they give, like BRCA1 and 2 (relating to risk of breast cancer), and LRK 2 status (relating to risk of Parkinson's disease), are very predictive, and in the right circumstances have important medical implications. And yet they're being dumped into this big pot with all kinds of tests that are purely of entertainment value and some tests that are misrepresented as being medically useful when they are not.

I don't have a problem with the public gaining access to information about their own genes. I'm not so paternalistic as to say you can't have the information in your genome. What I do feel strongly about is that people shouldn't be lied to about the significance of that information, and that people should be able to be assured that the claims that are made are accurate and that their privacy will be protected.

Do you think that the whole concept of the way these tests are marketed clings to the old concept that your genome can tell you everything about yourself?

I think what's happened is that there's an understandable impatience to apply all of this wonderful, cool genetic technology to medical care. There's a seductive appeal to thinking that because we understand some things about the genome, we now understand a lot about its role in health and disease. The difficult and the sober reality, however, is that we don't have a very good grasp of precisely how to relate your genetic information to your health. That's going to be the work of many years. What we need to do before we just start willy-nilly selling this idea to people is to find evidence of what's real and what's not, what works and what doesn't. All I ask  is that we have data that back up the things that we introduce into the realm of patient care.

And these companies want to have it both ways. They implicitly and explicitly make claims about the health value of this information, and yet on every page of their results, they say "this isn't medical advice." Pick one or the other: it's either medically important or it isn't. And I would say it is not, demonstrably, as the GAO report really pointed out.

Since there are plenty of traits tested where there is not a lot of utility, do you think there are traits which are useful for consumers to know about?

You bet. And I think that there may well be reasons that a patient might want to pursue various tests in what we consider a non-traditional environment. For example, DNA Direct for many years has offered BRCA testing. I've never had any problem with what they do. They don't misrepresent what they offer, and importantly, they have genetic counselors available to talk to customers about the meaning of their results. I haven't looked at their website in a while, but for many years they offered tests that arguably were medically valid tests, with real medically important results—and they didn't conflate entertainment with medical information.

I think there are ways of doing this that are reasonable and responsible, but I don't think companies like 23andMe are doing so.

Going back to different companies' radically different interpretations of the same DNA sample in the GAO report: do you know of a reason that the interpretations would be so different?

I think there are several reasons. Now, the reason that the companies will give you is only part of the truth. They will tell you the problem is that one company did nine variants and another company did fourteen variants. The reason that explanation isn't the whole story, and the reason that settling on everyone testing the same twelve variants isn't a valid response, is the following: We still don't understand how to aggregate these independent risk factors into a net risk score. Genes interact with each other in ways that we are only dimly beginning to glimpse, and genes interact with the environment. It is entirely possible that you might have variant A, B and C and have an increased risk for a condition, but a person who has variant A, B, C and D is actually at decreased risk because of the interaction of variant D with variant B.

Secondly, genes interact with the environment. It could well be that an individual who has variant A, B and C, who should be at increased risk, is actually at average risk in the right environment—because those genes and our physiology interact with our environment.

So the reason that they come up with different results is that we don't know how to come up with a single net risk estimator from these variants. We just simply don't know enough at this point.

One of the things that was left hanging after the testimony was an idea that the industry would love to have you believe: "If we could all just get together and agree on standards, it would be fine." No, what would happen is you would all agree on standards and we would see the same risk prediction from all the companies …but that doesn't mean it's correct!

How much of this do you think could be solved just by having a genetic counselor involved throughout the process?

That's part of the solution, that the individual who avails themselves of these tests would have a first responder who could give them information. That probably isn't all of it, though. As we learned from the GAO report, people can get told all kinds of different things, especially when there are conflicts of interest that are swaying people to give certain kinds of advice.

If 23andMe would just have a line that people could call, that would go a long way toward alleviating some of my concerns. But it doesn't relieve all of them, because I still think the claims that are being made in their advertising are simply wrong. And that seems to me something that doesn't necessarily require further regulation; it requires the FTC to enforce truth in advertising.

Can I say one other thing that I didn't get a chance to say during my congressional testimony? One of the things you will hear from these companies at first sounds quite convincing. Cholesterol and blood pressure confer only subtle relative risks for heart disease; so they'll say, "High cholesterol only confers a 1.4 relative risk on somebody for a heart attack and this is similar to the degree of risk conferred by genetic variants." And that is true. But what they fail to discuss is that your doctor doesn't check your cholesterol because they are primarily seeking predictive information. Your doctor checks your cholesterol because they can change your cholesterol. They aren't doing it just so they can say, "Oh, you're at increased risk for a heart attack. Have a good day." They're checking it because they can do something about it. And that puts it in an entirely different category than these direct-to-consumer genetic tests.               

 
 
Search: GeneWatch
 
 
 
The use of forensic DNA databases by law enforcement around the globe is expanding at a rate that should be of great concern to civil libertarians.
 
View Project
 
 
The purpose of the Genetic Bill of Rights is to introduce a global dialogue on the fundamental values that have been put at risk by new applications of genetics.
 
View Project
 
 
Tools
PAGE TOOLS
 
 
 
 
ON THE WEB