Me vs. MM: Stats 101
How many times have you seen an abstract summarizing the results from a study or clinical trial in multiple myeloma that reads something like this:
The median progression-free survival (PFS) for 24 patients taking drug X was 18 months, and the median overall survival (OS) was 73 months.
So what useful conclusion can you make from this?
In a nutshell, not too much.
First, you need to understand what the "median" is. In statistical terms, if you have a set of values arranged in numerical order, the median is the middle value. There are some minor nuances depending on whether you have an even or odd number of values, but basically this means half the values are above the median and half are below.
Using the sample abstract above, this would mean half the patients had PFS less than 18 months, and half had PFS greater than 18 months. Similarly, half had OS less than 73 months, and half had OS greater than 73 months.
What's missing in the abstract, however, is how the values spread out above and below the median. To clarify what I mean, here are three examples, all with a median progression-free survival of 18 months, yet very different ranges and distributions:
- 24 patients, all with PFS of 18 months
- 10 patients with PFS of 1 month, 14 with PFS of 18 months
- 14 with PFS of 18 months, 10 with PFS of 120 months
In the first case, all patients were progression free for 18 months, then relapsed. In the second case, slightly less than half relapsed after 1 month, and nobody was progression free past 18 months. In the third case, all patients were progression free for 18 months and 10 patients went progression free for 120 months (i.e., 10 years).
Granted, these are probably not realistic scenarios, but they do illustrate the point.
The two additional pieces of information I would like to see are the range of the values and a measure of the distribution of the values over the range.
The range is easy; it is simply the minimum and maximum value.
The distribution is a bit trickier, and there are a few different methods for expressing it. One method I personally find useful is something called the median absolute deviation.
I won't get into the mathematics of how you determine the median absolute deviation (search for it on Wikipedia if you really want to know), but it basically means that half the values in a set of values will be within the median absolute deviation from the median, and half the values will be greater than the median absolute deviation from the median. Everybody got that?
Perhaps an example would help.
Using the example above with a median PFS of 18 months, if the median absolute deviation was 4 months, then half the patients would have a PFS within 4 months of the 18 months, or somewhere between 14 and 22 months. The other half of the patients would be more than 4 months from the median, so either less than 14 months or greater than 22 months.
The smaller the absolute deviation, the less spread out the values tend to be, and conversely, the greater the value, the more spread out they tend to be.
In conclusion, I would much prefer to read an abstract similar to the following:
The median progression-free survival (PFS) for 24 patients taking drug X was 18 months, with a minimum PFS of 3.6 months, a maximum PFS of 42 months, and a median absolute deviation of 4 months. The median overall survival (OS) was 73 months, with a minimum OS of 17 months, a maximum OS of 131 months, and a median absolute deviation of 23 months.
I don't know about everyone else, but for me, this sure seems to provide significantly more useful information.
Peace, and live for a cure.
Kevin Jones is a multiple myeloma patient and columnist at The Myeloma Beacon.
If you are interested in writing a regular column for The Myeloma Beacon, please contact the Beacon team at
.
Well said, Kevin!! And I totally agree!
Thanks for the interesting comments on statistics, and how they could be more meaningfully expressed in the abstracts (summaries) presented with a scientific paper, Kevin. When it is possible to see the graphics from a paper, as when the Beacon publishes the 'slides' from a presentation, usually they have graphs where 'clusters' of patients can be seen under the bell curve. With the numbers of patients in a survey usually being quite low, it's easier to see the results when they are presented visually. It would be nice to have the 'absolute deviation' of the stats calculated too, since then you can see if most of the patients respond in a similar way, or if the results are really differing from person to person!
Do you think that the way the stats are presented in a sort of mathematical shorthand, is just to catch the attention of the other scientists in the field?? They are trying to achieve the best results for OS, PFS, etc. for purposes of getting funding for the research perhaps?
Kevin,
I can see you have some kind of science, engineering or math background like myself and I have been making the same observations on the statistics published in these studies and trials, especially progression free and survival statistics. Maybe deeper in the published papers the more meaningful statistics and analysis you cite can be found but in the abstract summaries often posted on the internet and other media they are often not found. I find it can be difficult explaining what these statics really mean to friends and family and often I am not sure myself because of the lack of completeness of the posted data. Sometimes I wonder if these abstract summaries do more harm than good. All the more reason I feel it is important to talk everything over with a doctor who can hopefully put meaningful context to these trials, studies, and numbers for the individual patient.
Amen: I would love to see this information included in every clinical trial so we can get a handle on the distribution not the relatively meaningless central tendency median.
Before any newly diagnosed MM patient looks at the literature or frets over stats from their oncologist they should read the short essay (only 3 pages) by Steven Jay Gould entitled
"The Median Isn't the Message".
The pointer to this article is
http://cancerguide.org/median_not_msg.html
I would suggest to the Beacon staff that they devote one day to Gould. It is a masterpiece.
Statistically speaking, right on. That has been my pet peeve over and over again.
Hi Kevin,
I think you may want the confidence intervals. They give an indication of how broad the distribution of the numbers are and are more likely to be provided in clinical trials. The range of values you seek are generally available in the tables within the written study and are often provided during powerpoint presentations.
survivormm,
Thanks, I figured there may be a few people out there with similar sentiments.
nancy,
I've sometimes been able to find more details in the full papers. Unfortunately, the papers are not always available to us laypersons, or only the abstracts are presented/quoted, or I just don't have the time or desire to go off and read the entire paper - that's why I'd like to see a bit more in the abstracts.
Eric,
Yep, I have degrees in engineering and mathematics. I'm not sure if the abstracts do more harm than good, but there's no doubt anyone without some understanding of statistics could easily read too much into them.
Gary,
Read and enjoyed the referenced article. I think one of the key takeaways is that by the very nature of the median, the left distribution is bounded from zero to the median, while the right side is unbounded, so the distribution will almost certainly be right skewed (i.e., some people could live a very long time).
Eileen,
Obviously, one of my pet peeves to (which is why I wrote about it). It would be nice I there were some way to influence this and get the abstract changed.
I wonder whether there is a simple reason why medians are used:
Expediency: Wait until one-half of any patients in any trial have progressed or died, and you have your median value.
That way, you don't have to wait for results from all the patients. For this reason, I wonder whether doing a full-blown analysis is even possible in most of these studies.
Hi, Kevin:
Thanks. It's a pretty good 101. Regression is another tool of statistics that can easily mislead its readers. When a dependent or independent variable is revealed through regression analysis, it only implies a relationship between the variable and the outcome that says the two tend to occur at the same time. It does not say that one is the cause of another.
By the way, can anyone explain what a hazard ratio is and what the mysterious (p < 0.001) means? Thanks.
Ben
Hi Ben,
A p value tells us how likely an event is likely to occur. A p value of less than 0.05 is considered significant as it is highly unlikely that the results seem are due to random occurence, such that p < 0.001) would be exceedingly rare as a chance occurence.
So, if we are looking at a treatment vs placebo arm in a study we know that the drug works and it is not a random occurence for the outcome. The same would be true if we were comparing side effects between the arms.
Hazard ratios are more difficult to explain:
"A significant effect would be that the hazard of experiencing the event for patients in group 1 is different from the hazard of experiencing the event in reference category. You can tell more by looking at the hazard ratios, they tell you by how much these groups are different: e.g. an hazard ratio of .2 means that group 1 has a 80% smaller hazard than the reference category. An hazard ratio of 1.4 means that group 1 has a 40% higher hazard than the reference category. The mechanics of interpreting hazard ratios is the same as the mechanics of interpreting odds ratios."
Here is a link:
http://www.medicine.ox.ac.uk/bandolier/painres/download/whatis/what_are_haz_ratios.pdf
Dan D.,
You make a good point about only needing to wait for half the patients to progress or die. If this is the case, that could explain why ranges and distributions are not always available, or at the very least need to be continually updated. Another reason for using the median, instead of say the mean/average is that the median is not affected by outliers like the mean would be (i.e., outliers will tend to skew the mean). Basically, the median provides a nice way of saying half the patients have PFS/OS less than the median, and half have PFS/OS greater.
suzierose, Ben S.,
When I wrote this article, I debated how much detail to go into. I tried to keep it fairly basic and describe a level of detail that wouldn't be too difficult to understand. There's no doubt other data such as p value, quartiles, etc. might be useful, but I figured only the geeks in the crowd would read past the first paragraph. Thanks for the reference to hazard ratios, this was something I was not familiar with either (or maybe I was 30 years ago and have forgotten it).
Dear MM Group:
Off the subject. I live in Aurora Colorado, near the sight of the mass murder shooting, a suburb of Metro Denver. I ask that you all pray for the victims families and my community.
We MM patients try to study the stats of life expectancy, good responses to our treatments, quality of life. Today is a reminder that everyone never knows what tomarrow will bring in their life - with MM or healthy. Life is precious - don't forget to enjoy what we have of it, for however long it lasts...
Hi Nancy D, My heart goes out to all of the victims and their families from the tragic shooting in Colorado. None of us can be really sure of what each day may bring. That sort of thing has happened here too. In a perfect world we would not have such terrible weapons available to people, except in very controlled situations, such as police or military, and in a democratic society too. At least that is my spin on it.
Nancy S: I totally agree. Some states have banned the sale of assult type weapons and limit ammo clip size...most of us in the US would like it to be national. I agree with the right to bear arms, have hunting guns, etc. But assult weapons have no purpose outside of military or police SWAT team usage. They should not be allowed to be sold or owned by private individuals.
In Colorado, we had the Columbine High thing now this...it is just so difficult for everyone.
Remember to live for today.
Kevin: Excellent thoughts on the median values. I also see the use of this value as meaningless and confusing which is somewhat akin to the use of the 'chance of rain' percentage in predicting the weather forecast. Both can in essence be worthless. My tried and true method of determining if it is raining is to stick my head outside. I will, however, look at the radar maps to see if there is any green stuff (my verbiage) moving my direction with the yellow or red being the really bad stuff. So could the same be said for tracking multiple myeloma? If I feel good today or rather feeling good at this moment then I go forth and prosper as to the best of my abilities. If I feel rotten then I don't. Looking at the percentage chance of rain for today might in fact keep me from doing something today that I would otherwise attempt. Not unlike worrying about the various tracking numbers we have all learned about all too well. I realize that in determining the effectiveness of a drug there needs to be a statistical valuation but for those of us living with this condition, does it really matter? Viewing the chance of rain calculation may actually stop you from living while you are still breathing. Is this the same as obsessing over our numbers or worrying about the median life expectancy? I personally don't care how long I am to live while knowing full well that my life may in fact be cut short but I do plan to achieve as much as possible with the remaining years while also giving consideration to quality of life. Each day to me means new opportunities where I simply wake up, stick my head outside and decide what am I to do today which at times can mean nothing at all during those bad times but can also be very fulfilling on those days (or hours) of strength.
Jim
Nancy D, Nancy S,
Events like those of this past Friday truly do remind us of how uncertain life may be. As MM patients our lives may be shortened, but that's all the more reason to celebrate each day.
jim byrd,
What a great analogy with the weather. I generally do stick my head out on a daily basis to check the MM weather - one day at a time. However, my personality is such that I also tend to watch the forecast too, which is probably why I ended up in engineering. I don't necessarily let the forecast govern my decisions, but I do have this need to see the full picture. Hope you have many more good days than bad.
Hi Kevin,
I was happy that Ben raised the p value (probability) question as it is one of the key variables to determining how valid a study is, especially in terms of comparator arms. Even so, confidence intervals are more important than the p value, often times due to the small size of the majority of oncology trials.
I have often wanted to write about the stats in studies but I found that I would get bogged down in detailed minutae. So I think you did an awesome job with a very difficult topic and kept it focused without going into details that are overwhelmingly difficult for individuals. Even the verbage used in stats like "null hypothesis" is like HUH? You did not get trapped in so this was a great column.
Here is another article you might find useful, it you ever wanted to address p values and confidence intervals. : )
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2689604/
P.S. another good one is clinical significance vs statistical significance.