Tag Archives: research

How To Tell If Research Is Crap

I recently read a very interesting article on research, and found it to be very pertinent to the state of academic research today. It was published on Manager Mint, a site that considers itself to be “the most valuable business resource.” (?) But the message is very applicable to trauma professionals, medical professionals, and probably anyone else who engages in research pursuits. The link to the full article is listed at the end of this post. 

1. Research is not good because it is true, but because it is interesting.

Interesting research doesn’t just restate what is already known. It creates or explores new territory. Don’t just read and believe existing dogma.

Critique it.

Question it. Then devise a way to see if it’s really true.

2. Good research is innovative.

Some of the best ideas come from combining ideas from various disciplines.

Some of the best research ideas are derived from applying concepts from totally unrelated fields to your own.

That’s why I read so many journals, blogs, and newsfeeds from many different fields. And even if you are not doing the research, a broad background can help you sort out and gain perspective as you read the works of others.

3. Good research is useful.

Yes, basic bench level research can potentially be helpful in understanding all the nuances of a particular biochemical or disease process.But a lot of the time, it just demonstrates relatively unimportant chemical or biological reactions. And only a very small number actually contribute to the big picture. For most of us working at a macro level, research that could actually change our practice or policies is really what we need.

4. The best research should be empirically derived.

It shouldn’t rely on complicated statistical models. If it does, it means that the effect being measured is very subtle, and potentially not clinically significant. There is a big difference between statistical and clinical relevance.

Reference: If You Can’t Answer “Yes” To These 5 Questions, Your Research Is Rubbish. Garrett Stone. Click here to view on Manager Mint.

First, Read The Paper. THEN THINK ABOUT IT!

This is a perfect example of why you cannot just simply read an abstract. And in this case, you can’t just read the paper, either. You’ve got to critically think about it and see if the conclusions are reasonable. And if they are not, then you need to go back and try to figure out why it isn’t.

A study was recently published regarding bleeding after nonoperative management of splenic injury. The authors have been performing an early followup CT within 48 hours of admission for more than 12 years(!). They wrote this paper comparing their recent experience with a time interval before they implemented the practice.

Here are the factoids. Pay attention closely:

  • 773 adult patients were retrospectively studied from 1995 to 2012
  • Of 157 studied from 1995 to 1999, 83 (53%) were stable and treated nonoperatively. Ten failed, and all the rest underwent repeat CT after 7 days.
  • After a “sentinel delayed splenic rupture event”, the protocol was revised, and a repeat CT was performed in all patients at 48 hours. Pseudoaneurysm or extravasation initially or after repeat scan prompted a trip to interventional radiology.
  • Of 616 studied from 2000-2012, after the protocol change, 475 (77%) were stable and treated nonoperatively. Three failed, and it is unclear whether this happened before or after the repeat CT at 48 hours.
  • 22 high risk lesions were found after the first scan, and 29 were found after the repeat. 20% of these were seen in Grade 1 and 2 injuries. All were sent for angiography.
  • There were 4 complications of angiography (8%), with one requiring splenectomy.
  • Length of stay decreased from 8 days to 6.

So it sounds like we should be doing repeat CT in all of our nonoperatively managed spleens, right? The failure rate decreased from 12% to less than 1%. Time in the hospital decreased significantly as well.

Wrong! Here are the problems/questions:

  • Why were so many of their patients considered “unstable” and taken straight to OR (47% and 23%)?
  • CT sensitivity for detecting high risk lesions in the 1990s was nothing like it is today.
  • The accepted success rate for nonop management is about 95%, give or take. The 99.4% in this study suggests that some patients ended up going to OR who didn’t really need to, making this number look artificially high.
  • The authors did not separate pseudoaneurysm from extravasation on CT. And they found them in Grade 1 and 2 injuries, which essentially never fail
  • 472 people got an extra CT scan
  • 4 people (8%) had complications from angiography, which is higher than the oft-cited 2-3%. And one lost his spleen because of it.
  • Is a 6 day hospital stay reasonable or necessary?

Bottom line: This paper illustrates two things:

  1. If you look at your data without the context of what others have done, you can’t tell if it’s an outlier or not; and
  2. It’s interesting what reflexively reacting to a single adverse event can make us do.

The entire protocol is based on one bad experience at this hospital in 1999. Since then, a substantial number of people have been subjected to additional radiation and the possibility of harm in the interventional suite. How can so many other trauma centers use only a single CT scan and have excellent results?

At Regions Hospital, we see in excess of 100 spleen injuries per year. A small percentage are truly unstable and go immediately to OR. About 97% of the remaining stable patients are successfully managed nonoperatively, and only one or two return annually with delayed bleeding. It is seldom immediately life-threatening, especially if the patient has been informed about clinical signs and symptoms they should be looking for. And our average length of stay is 2-3 days depending on grade.

Never read just the abstract. Take the rest of the manuscript with a grain of salt. And think!

Reference: Delayed hemorrhagic complications in the nonoperative management of blunt splenic trauma: early screening leads to a decrease in failure rate. J Trauma 76(6):1349-1353, 2014.

“Medicine Is The Science Of Temporary Truths”

Does anyone know how to write a scientific paper anymore??! My impression is that a majority of articles published in medical journals these days are seriously flawed. Yes, sometimes it’s just not possible to design or execute a study the way it really should be due to scarcity of the data or ethical issues.

But a lot of stuff I read is just not very good. Poor design. Answers to things that no one really cares about. Use of the wrong statistical analysis. And even if the basic ideas and analysis are sound, so many are just not written well.

I believe that it all comes down to poor mentoring. Designing studies and writing scientific papers is an acquired skill that requires a lot of practice. And it’s tough to learn from reading lots of other people’s papers (because they’re not very good). An experienced mentor is invaluable and can accelerate the learning curve.

My own mentors, Charles Lucas and Anna Ledgerwood, taught me by repetition. And lots of it. They told me to read a bunch of papers, then try to emulate them using my own data. I remember turning my first draft in to Dr. Lucas and getting it back a few days letter. The entire thing was covered with scribbling in red pen. Almost none of my original text remained. So I revised it and gave it back. He returned it with a fewer red marks. After many iterations, we finally had a publishable paper!

The most recent Journal of Trauma includes a very nice article on how to construct a good Discussion section in your paper. There aren’t a lot of good articles on the actual technique of medical writing (go figure). But this one is definitely worth reading and will help researchers at any level!

Reference: The anatomy of an article: The discussion section: “How does the article I read today change what I will recommend to my patients tomorrow?” J Trauma 74(6):1599-1602, 2013.

How I Keep Up With The Trauma Literature

Medicine advances quickly. How can one keep up with new developments in their field? Well, there’s an app for that! You can also do it without a handheld device, but it’s not quite as convenient. Let me show you how I do it.

Most regularly updated online content is syndicated for RSS (Rich Site Summary). You know, those cute little  icons on web pages. There are loads of RSS readers available, both online and for handheld devices, that allow you to organize and browse content from RSS feeds. I use Google Reader on desktops or laptops, and the River of News app on my iPad.

I visited the home page of each of the journals I was interested in “following” and snagged the RSS web address from each. I then added it to my Google Reader collection of feeds. Now, at some point every day I can quickly browse through new abstracts in each journal. For most journals, abstracts are released once a month. A few publish early release abstracts more regularly.

When you click an abstract of interest, a web page opens that shows the full abstract. If it looks interesting, I save it using Instapaper, a clipping service that helps you organize web pages of interest. I just click a link at the top of the browser and the page is instantly saved for later reference.

As you probably already know, much of what is published is junk. Class III or worse data. Concluding that “additional research is necessary.” So why didn’t you do the study right in the first place? Therefore, it is vitally important that you read the whole article for the abstracts that you have saved. For this, you need access to a biomedical library, either online or brick and mortar. If you’re not lucky enough to have this, you’ll need to rely on others who do. I just run through my list of saved pages on Instapaper and pull up the full article via an online library connection through work.

Here’s a screen shot of my River of News app. I currently monitor 18 journals that publish trauma articles to a greater or lesser degree. I’m also a firm believer that some of the best ideas come from cross-pollinating your own field with ideas from others. To that end, I also scan another 22 feeds in technology, publishing, cooking, aviation and many others. I’m interested in hearing how others do this as well. Please feel free to comment below.

Related links:

Helicopter Transport and Civilian Trauma

Military helicopter experience led to widespread adoption in the US for civilian trauma beginning in the 1970s. This has had the significant side effect of extending the reach of trauma centers to a significant percentage of the US population. But because of safety considerations and concerns about appropriate use, the overall benefit continues to be questioned.

Most existing studies have been small, single institution projects. Researchers at the University of Rochester designed a very large study using the National Trauma Databank. They identified over 250,000 patients transported from the injury scene, 16% of whom were transported by ‘copter, the remainder by ground. 

Patients transported by air were more severely injured and were more likely to have a severe head injury or abnormal vital signs. They also had longer hospital and ICU stays, and were more likely to require a ventilator or emergency surgery.

Despite the fact that response and scene times were longer for helicopter transports, air transport was a predictor of survival when injury severity was taken into consideration. This type of study can’t tell why survival is better, but possibilities include distance traveled and a higher level of care provided by air EMS personnel. Aeromedical EMS personnel are more likely to trained to perform advanced techniques such as intubation, crich, and transfusion, and generally have more experience with trauma patients.

Use of this scarce resource for trauma patient transport remains expensive, and as recent accident statistics imply, somewhat dangerous. Trauma centers and systems need to develop evidence-based guidelines that use helicopters intelligently for benefit of the patient, not the aeromedical service owners.

Reference: Helicopters and the civilian trauma system: national utilization patterns demonstrate improved outcomes after trauma injury. J Trauma 69(5):1030-6, 2010.