Brain-based devices: How well do they work?

There are more than 10,000 patent filings for brain-based devices that claim to help people “develop muscle memory faster,” “lose weight,” “monitor and act on…sleep,” and “treat depression.” Many of the websites featuring these devices cite “science” as backing up their claims. However, a recent review by science journalist Diana Kwon concluded that the large majority of these claims are not scientifically valid. As a consumer, how can you separate hype from science when deciding to use a brain-based device?

Even when there is science, you can’t assume that a device will work for you

Many people choose to ignore scientific findings, even when there is published evidence supporting a view. While this is understandable, it makes little sense to completely ignore scientific findings when you are evaluating new technologies.

For the scientific community to believe that a device is helpful, they usually consider the following basic factors:

True positive findings: There must be a statistically significant difference between the device and a placebo or sham treatment.
Replication: There are many different experiments by different groups that show a device has worked.
Control: The device should be compared to a placebo or sham treatment to show that it had a real effect.
Blindness: People conducting the experiment should not know what they are administering, and participants should not know what they are receiving. When both researchers and participants are blind to the intervention, this is called a double-blind study. Also, the intervention should be randomized (people should receive the placebo or control interventions at random). When trials are double-blind and randomized, bias is reduced.
Peer-reviewed journal: The findings should be published in a peer-reviewed journal, and not just an open online platform.

Challenges facing neuroscience research

While scientific studies do provide one line of evidence that supports whether a device will work or not, in neuroscience research the criteria above are fraught with challenges.

False-positive findings. Many neuroscience studies are not stringent enough, and as a result, you cannot believe findings at face value. For instance, Stanford epidemiologist John Ioannidis explained that most neuroscience studies produce false alarms because they are designed poorly. The findings are blown out of proportion because the sample sizes are too small or biased. For example, one headline stated, “Brain implant ‘predicts’ epileptic seizures” but only 15 people were tested. Another study of a cognitive training program had a large sample size, but all the participants were already using the program, making this quite biased.

Can you be blind to a device? Unless researchers make a placebo device that looks and feels identical, they will not be blind to what they are administering. And unless participants cannot distinguish between an actual device and a placebo, they know what they are getting. When either researchers and participants are not blind to the intervention, this can bias study results.

Randomized, double-blind, placebo-controlled trials give you comparisons of an average effectiveness of a device in a group. As a unique individual, you can’t be certain a device will work for you because it has worked for others.

Is replication possible? Close to 50% of medical studies cannot be replicated, even once. This is especially true of neuroscience research. Also, biology changes over time, so even if you do replicate a finding on the same sample, it is essentially a new finding.

Do studies use the right control groups? Across different studies, control groups should also be comparable. Age, gender, geography, diet, and temperament can all vary, and when they do, the results are less reliable. Also, simply being in a study may make people behave differently from how they behave in their everyday lives, so you can’t assume that research findings will translate to real life.

Peer review is flawed. When a study is peer-reviewed, it means that qualified people who study a similar topic have double-checked the study for quality and accuracy. In 2006, British physician Richard Smith explained that peer review is an inherently flawed and subjective process. While peer review does ensure oversight by respected experts, peer reviewers are often doing similar research, and they may be biased if new findings oppose their own research. Also, peer reviews are dominated by men, due to gender biases that are often subtle or unconscious.

How do you assess the value of new neurotechnologies?

So, what can you do when the absence of studies gives you no information, or the validity of studies is highly questionable?

Use criteria of scientists to evaluate studies.
Look at how many subjects were studied. For studies using changes in brain blood flow, 20 to 30 subjects is typical. However, in 2016 neuroscientist Julien Dubois explained that at least 100 people should be studied. There’s no ideal number for brain device studies, but the higher the number, the better.
See if the biases mentioned above apply. For example, are all the studies done by one group of researchers only? Alternatively, are they being done at too many sites for the methods to be consistent across all sites?
Work with your physician to see how you can safely try a technology that you believe in, after evaluating the above factors.
Evaluate whether the financial cost is worth the benefit to you over time.