Tag Archives: research

Why Is So Much Published Research So Bad?

I read lots of trauma-related articles every week. And as I browse through them, I often find studies that leave me wondering how they ever got published. And this is not a new phenomenon. Look at any journal a year ago. Five years ago. Twenty years ago. And even older. The research landscape is littered with their carcasses.

And on a related note, sit down with any serious clinical question in your field you want to answer. Do a deep dive with one of the major search engines and try to get an answer. Or better yet, let the professionals from the Cochrane Library or other organization do it for you. Invariably, you will find hints and pieces of the answer you seek. But never the completely usable solution you desire. 

Why is it so hard? Even with tens of thousands of articles being published every year?

Because there is no overarching plan! Individuals are forced to produce research as a condition of their employment. Or to assure career advancement. Or to get into medical school, or a “good” residency. And in the US, Level I trauma centers are required to publish at least 20 papers every three years to maintain their status. So there is tremendous pressure across all disciplines to publish something.

Unfortunately, that something is usually work that is easily conceived and quickly executed. A registry review, or some other type of retrospective study. They are easy to get approval for, take little time to complete and analyze, and have the potential to get published quickly.

But what this “publish or perish” mentality promotes is a random jumble of answers that we didn’t really need and can’t learn a thing from. There is no planning. There is no consideration of what questions we really need to answer. Just a random bunch of thoughts that are easy to get published but never get cited by anyone else.

Bottom line: How do we fix this? Not easily. Give every work a “quality score.” Instead of focusing on the quantity of publications, the “authorities” (tenure committees and the journal editors themselves) need to focus in on their quality. Extra credit should be given to multicenter trial involvement, prospective studies, and other higher quality projects. These will increase the quality score. The actual number of publications should not matter as much as how much high quality work is in progress. Judge the individual or center on their total quality score, not the absolute number of papers they produce. Sure, the sheer number of studies published will decline, but the quality will increase exponentially!

Why Is So Much Published Research So Bad?

Yesterday, my colleague the Skeptical Scalpel wrote about an interesting (?) paper published in Emergency Medicine Australasia. It was a small study that concluded that ED wait times decreased as the number of people presenting to be seen decreased. Where’s the mystery in that? Overstating the obvious?

But if you look through almost any journal today, you will find studies that leave you wondering how they ever got published. And this is not a new phenomenon. Look at any journal a year ago. Five years ago. Twenty years ago. And even older. The research landscape is littered with their carcasses. 

And on a related note, sit down with any serious clinical question in your field you want to answer. Do a deep dive with one of the major search engines and try to get an answer. Or better yet, let the professionals from the Cochrane Library or other organization do it for you. Invariably, you will find hints and pieces of the answer you seek. But never the completely usable solution you desire. 

Why is it so hard? With tens of thousands of articles being published every year?

Because there is no plan! Individuals are forced to produce research as a condition of their employment. Or to assure career advancement. Or to get into medical school, or a “good” residency. And in the US, Level I trauma centers are required to publish at least 20 papers every three years to maintain their status. So there is tremendous pressure across all disciplines to publish something

Unfortunately, that something is usually work that is easily conceived and quickly executed. A registry review, or some other type of retrospective study. They are easy to get approval for, take little time to complete and analyze, and have the potential to get published quickly.

But what this “publish or perish” mentality promotes is a random jumble of answers that we didn’t really need. There is no planning. There is no consideration of what questions we really need to answer. Just a random bunch of easy to get published thoughts that never get cited by anyone else. 

Bottom line: How do we fix this? Not easily. Instead of focusing on the quantity of publications, the “authorities” need to focus in on their quality. Extra credit should be given to multicenter trial involvement, prospective studies, and other higher quality projects. The actual number of publications should not matter as much as how much high quality work is in progress. Sure, the sheer number of studies published will decline, but the quality will increase exponentially!

Crowdfunding: The Future Of Research Funding?

Many readers are familiar with the concept of “crowdsourcing”, or tapping into a pool of people connected via the internet to obtain something of value. This something might be information, services (think Uber), or content (99designs). And with the advent of websites like KickStarter, it is now possible to crowdsource money.

As anyone who has an academic focus can attest, there is tremendous pressure to pursue (hopefully) meaningful research. In many cases, this is an integral part of keeping one’s job. But research is expensive. Even the simplest retrospective study requires some kind of statistical analysis, and statisticians don’t work for free. And in more sophisticated research labs, there are huge personnel, equipment, as well as other infrastructure costs.

Traditionally, researchers have pursued grant dollars from single sources like the federal government, local agencies, corporations, and charitable organizations. But this is very competitive, and it’s usually an all or none proposition. Only one of many applications gets all the cash, and the rest get none.

But now, crowdsourcing has moved beyond the technology and design type projects seen on KickStarter to what is now called crowdfunding. There are a number of sites that solicit small donations from individuals, pooling them together into large amounts. The largest campaign on KickStarter was able to amass over $20 million to create a new version of the Pebble watch. A small campaign to get $10 to develop a potato salad recipe ended up collecting over $55 thousand.

Bottom line: The concept of crowdfunding has now made the jump to funding research. There are a number of sites that are structured similarly to KickStarter that allow researchers to solicit donations from the public. Some are relatively rudimentary, and some are naive in their approach to soliciting funds. In order to engage the public to contribute sums of money, large or small, research teams will need to explain their ideas simply and describe some practical or potential application. And it won’t hurt to offer some type of schwag for donors at various financial levels.

But the downside? The bean counters at funding agencies and university may come to expect you to get some (or all) of your funds from crowdfunding so they can save their dollars for other stuff!

A few interesting crowdfunding sites:

How To Tell If Research Is Crap

I recently read a very interesting article on research, and found it to be very pertinent to the state of academic research today. It was published on Manager Mint, a site that considers itself to be “the most valuable business resource.” (?) But the message is very applicable to trauma professionals, medical professionals, and probably anyone else who engages in research pursuits. The link to the full article is listed at the end of this post.

1. Research is not good because it is true, but because it is interesting.

Interesting research doesn’t just restate what is already known. It creates or explores new territory. Don’t just read and believe existing dogma.

Critique it.

Question it. Then devise a way to see if it’s really true.

2. Good research is innovative.

Some of the best ideas come from combining ideas from various disciplines.

Some of the best research ideas are derived from applying concepts from totally unrelated fields to your own.

That’s why I read so many journals, blogs, and newsfeeds from many different fields. And even if you are not doing the research, a broad background can help you sort out and gain perspective as you read the works of others.

3. Good research is useful.

Yes, basic bench level research can potentially be helpful in understanding all the nuances of a particular biochemical or disease process.But a lot of the time, it just demonstrates relatively unimportant chemical or biological reactions. And only a very small number actually contribute to the big picture. For most of us working at a macro level, research that could actually change our practice or policies is really what we need.

4. The best research should be empirically derived.

It shouldn’t rely on complicated statistical models. If it does, it means that the effect being measured is very subtle, and potentially not clinically significant. There is a big difference between statistical and clinical relevance.

Reference: If You Can’t Answer “Yes” To These 5 Questions, Your Research Is Rubbish. Garrett Stone. Click here to view on Manager Mint.