Journal of Education and Ethics in Dentistry

DESCRIPTIVE, CROSS SECTIONAL STUDY
Year
: 2013  |  Volume : 3  |  Issue : 2  |  Page : 88--92

Clinical versus statistical significance in the Iranian postgraduate periodontal theses


S Vahabi1, R Noormohammadi2, S Rahnama3,  
1 Department of Periodontics, Dental School, Shahid Beheshti University of Medical Sciences, Tehran, Iran
2 Department of Oral Medicine, Dental School, Shahid Beheshti University of Medical Sciences, Tehran, Iran
3 Dentist, Qazvin, Iran

Correspondence Address:
R Noormohammadi
Department of Oral Diseases, Dental School, Shahid Beheshti University of Medical Sciences, Tehran
Iran

Abstract

Context: There is so much reliance on using statistical significance testing in clinical trials that sometimes leads to ignore clinical importance and statistical significance may be assumed as substantively important. Aims: According to different concepts and lack of specific clinical criteria in this field, the purpose of this study was to evaluate clinical versus statistical significance in the postgraduate periodontal theses from the first number until January of 2011 in Iran. Materials and Methods: All of the experimental periodontal theses in all six postgraduate dental faculties in Iran were evaluated and every direct and indirect evidences of clinical significance were double checked in titles, methods and materials, results, conclusions and suggestions by two trained dental interns. About one-third of the theses were triple checked by the trained director at the end of the study. Results: About 66.66 percent of accessible experimental theses had statistically significant results and 24 percent had some evidence of clinical significance. Conclusions: The results of this study suggest that clinically significant changes related to periodontal therapy should be established and threshold values of each clinical parameters should be defined at the beginning of the study and then statistical testing can be used to validate that findings did not occur by chance.



How to cite this article:
Vahabi S, Noormohammadi R, Rahnama S. Clinical versus statistical significance in the Iranian postgraduate periodontal theses.J Educ Ethics Dent 2013;3:88-92


How to cite this URL:
Vahabi S, Noormohammadi R, Rahnama S. Clinical versus statistical significance in the Iranian postgraduate periodontal theses. J Educ Ethics Dent [serial online] 2013 [cited 2024 Mar 28 ];3:88-92
Available from: https://www.jeed.in/text.asp?2013/3/2/88/136053


Full Text

 Introduction



One of the important functions of clinical investigations is to provide results that helps clinician and cares for patients. If a study demonstrates a difference in treatment effects between two groups, it is valuable to know if this difference is clinically significant or not. [1] Annually, there are hundreds of periodontal studies including clinical trials that take lots of researchers and students' energy, time and costs. These experimental studies rely on statistical testing to evaluate the clinical relevance of new therapies or comparing them with usual approaches. Unfortunately, reliance on statistical testing to evaluate the clinical relevance is problematic at the best manner. [2] Numerous opinions of the relationship between statistical significance and clinical significance or relevance in the interpretation of dental research are much debated but little guidance is to be found in the literature. To appraise both the distinction and relationship between the two concepts, it would be valuable to have a clear definition, or at least a description of both. This is relatively straight forward for statistical significance but not for clinical significance since there is no accepted universal definition for the clinical significance. [3]

Statistical significance is a mathematical concept based on hypothesis testing. Published literature provides some evidences showing conclusions drawn by the author are not well explained by chance. [3] There are some problems associated with interpreting statistical significance. One treatment may provide a better statistically significant results than another one with regards to a specific outcome variable (e.g., clinical attachment gain), but this finding may be small with no clinical importance. [2],[4]

These shortcomings do not belong to periodontics or even medical science. According to the findings of Williamson and Goldschmidt in assessing 4233 study reports in 30 famous medical journals, only 28 articles had sufficient study design, data collection and appropriate statistical method for their analysis and interpretation. [5] Avram et al. evaluated 200 articles in two anaesthetic journals and found that only 15 percents of analyzed articles were with no major deficiencies. [6] These problems can mislead the way of the studies and produce misunderstanding in conclusions clarity. Thus, there is a compelling need to develop an interpretable, practical approach to characterize research data, thereby facilitating selection of an appropriate therapy for a particular patient.

Clinical significance in periodontology is much more ambiguous concept for which there are no agreed bases. One suggested general description of clinical significance is the importance of difference in clinical outcome between test and control patients. Other factors, including cost-benefit ratio, are also relevant and may influence decision on setting the relationship between the magnitude of the clinical outcome and clinical significance. In attempt to clarify the issue, authors have suggested that data become clinically significant when they lead to change in clinical behavior or improve the quality of life for the patient. [3] Now, there are many various definitions about clinical significance in the literature, [7],[8],[9] depending on specific field of periodontal parameters, effect size, measurement used to evaluate a therapy and the clinical importance of the results. [2] For example Hujoel et al. have suggested a working definition for clinically significance as ''statistically significant difference in a clinically important outcome identified in a definitive or phase III clinical trial''. Lefort mentioned that the term ''clinical significance'' reflected ''the extent of change, whether the change makes a real difference to subject lives, how long the effects last, consumer acceptability, cost-effectiveness and ease of implementation.'' [2] None of these definition seems to be perfectly fit in all study situations, because of different nature of expected results (e.g., mean values versus percentage of the sites). Clinicians, patients, researchers, industry representatives and third-party payers may interpret defining clinical significance differently, as they may place emphasis on different outcomes. [2]

At the first step to compensate for some shortcomings of the previously described approach, before conducting a study and in the designing stage of an experimental study, investigators can select a difference between therapies (e.g., 2 mm clinical probing attachment gain) whose detection will be clinically important. Next, power analysis, sample size detection and number needed to treat (NNT) should be performed in this way. This information would allow researchers to be reasonably confident that the study will have an adequate sample size to detect preselect, clinically relevant, statistically significant difference between treatments, if it really exists. [2],[10],[11]

With regard to the importances of these shortcomings which can inhibit the studies from properly interpreting results, in the first step, the purpose of this study was evaluation of statistically then clinically significance of postgraduate theses in all periodontics departments of medical universities before January 2011 in Iran.

 Materials and Methods



In this descriptive, cross sectional study, the postgraduate periodontal theses from the first number until January 2011 after allowing from the library of the periodontics departments or dental faculties were hold in trust. There were six faculty of dentistry (Shahid Beheshti, Tehran, Shiraz, Isfahan, Mashhad and Tabriz Medical universities) at the time of study who used to train periodontal postgraduate students as residents in Iran for many years. Two dental students in the final terms (interns), who had passed biostatistics units and dental Methodological workshops of their course inspected each of theses twice. Their procedure had been controlled before beginning of the study and during it. A pilot study was done on the theses in one of the faculties at the beginning of the study to assess any theoretic and practical limitations. All of these were accurately inspected from statistical point of view and type of study. Case report, case series, epidemiologic and cross-sectional studies from type I or II and those with suspected or obscure design were excluded and only experimental studies (type III) including clinical trials and animal studies entered to the study.

Primary outcome variables including clinical attachment levels, bone height and density; and secondary outcome variables such as probing depths, mobility patterns, bleeding on probing and biochemical and microbiological assessments were precisely explored.

All parts of each thesis including introductions, background information, literature reviews, main and specific objectives, hypothesis, Methods and materials, such as variables, scales, operating definitions, techniques, ethical considerations, internal and external validities, consent forms, questionnaires, findings, discussion, limitations, suggestions and conclusions were read and any direct or indirect attention of author about clinical and statistical significance was precisely recorded with no attention to their authors, consultants and faculties. Clinical significance was determined by direct and indirect measures as defined by Greenstein. [2]

Trained observers assay advantages and disadvantages of possible methods and criteria (absolute criteria, ratio, frequency distribution, cut point, percentage improvements, percentage of patients, disease progression inhibition, numbers needed to treat, radiographic evaluation and indirect measures) to evaluate periodontal status of theses that could help define clinical significance.

For example they assay:

Absolute criteria such as mean values of pocket depth reduction and clinical attachment gain in this way (1 to 2 probing depth reduction or 0.5 to 1.5 mm clinical attachment gain could be used as criteria)Percentage of sites or (patients) with initially deep probing depth (> 5 mm) that were reduced to 5 mm or less. Percentage of inflamed sites (for example, bleeding on probing) in which clinical inflammation was eliminated.Percentage of bone fills in osseous defects.Indirect measures: Cost-benefit ratio/length of therapy/ risk associated with new therapies [2]

Any suspected case was evaluated twice and then revised by supervisor who was training methodological researches in medical university for years. It was a critical stage because there are too many definitions and controversy about clinical relevance. At the end of the study, about one-third of these was analyzed for the third time and controlled by the experienced periodontist as a supervisor of the study.

 Results



Ninety two numbers of total 150 post graduate theses were accessible [Table 1]. Reliability was confirmed because of the same results attained by any observers at the final triple check of the study. Among the accessible theses and following excluding other types of studies such as epidemiologic, case reports, case series and those with suspected obscure design, 75 numbers of them with experimental (type III) design were entered the study. Finally 50 theses (66.66 percent) had gained statistical significant finding (P-value < 0.05). There were some evidences of clinical signification in only 18 (24 percent) numbers of theses [Figure 1].{Table 1}{Figure 1}

Statistical significance was between (16%-2.70%) and clinical significance varied between (10.6%-0.0%) among all accessible theses in different dental universities

 Discussion



Results obtained from this study indicated that only 24 percent of the experimental (type III) postgraduate these in periodontics departments had one or more evidence of clinical significance. This finding shows that two groups, being different statistically, may not necessarily indicate a clinically significance. We did not encounter with similar periodontal studies purely in this field in Iran, maybe because of difficulties in defining of this concept, but we found one study which had been done by similar concept in 2005 by Martin Addy et al. In their conclusion, in the fields of periodontitis susceptibility, activity and etiology, there are a few recognized clinically significant risk factors which appear to exacerbate the disease. In those which had paid some attentions to the philosophy of "clinical significance", there were a few markers regarding clinical relevance. In a survey of the clinical significance of preventive and therapeutic regimens, a large number of randomized controlled clinical trials have been conducted to determine the value of a very diverse range of them. An inconsiderable proportion of these studies are used to provide assertion support for preventive or therapeutic products available to the general public. The support of the claim is usually derived from finding a statistically significant difference in benefit of the test product over the control. At an effort to define some clinical meaning to the statistically significant finding, percentage differences are often calculated.

Briefly, the following may be suggested to assess the clinical significance of prevention or therapeutic regimen in periodontics in a study:

Benchmark efficacy - achieving an outcome similar to or better than other established regimensPositive efficacy - achieving an effect similar to/greater than the most effective regimens to dateDisease efficacy - achieving an effect on an etiological factor which reduces a disease;Proportional efficacy - achieving a previously agreed proportional reduction in parameter compared with control. The decision concerning clinical significance may be modulated by secondary factors, specifically cost-benefit ratio, adverse effects, patient convenience and risk. [3]

There are many different things that can affect how clinically relevant a study is to the patients. The most obvious factors are the validity of overall study and the patient population being studied. Given these two condition, to determine if a result is clinically significant, we must look at a (1) how were the results presented to us and (2) if clinically relevant outcomes were used or not. [1]

Sample size determination is one of the most important steps in designing experimental study. The larger sample size wouldn't always lead to better exporting of results, however may help to find the difference if it exists. If the study sample size is not determined sufficiently before performing the study, a large clinically meaningful result may be ignored because of statistically insignificant findings. [2],[3],[12] Statistical significant results which have been achieved by chance, could be clinically insignificant, however, possibility of a clinically important result especially in a prospective low power study may not be excluded if there is no statistically significant results. [13] In these cases, post hoc power analysis can be conducted to indicate the small likelihood of detecting a statistically significant difference that is clinically relevant because of the insufficient sample sizes used. The level of the statistical significance (alpha level) of 0.05, which was initially recommended by Fisher and often is used in clinical studies, is ''absurdly academic'' and should be flexible based on the evidence. For example if a large clinical effect is being diagnosed that has a P-value of 0.06 or 0.07 (not statistical significance if the alpha level was set at 0.05) it is not wise to inattention a potentially clinical significance. The reason for this failure probably could be attributed to inadequate planning for number of sample size was need to detect a statistical significance differences of a clinical relevant difference with a most likely. [2]

In real word, there are always severe limitations on the study size that would be obtained due to financial costs, man power, logistical issues, the length of time taken to recruit adequate subjects and, often, availability of subjects with relatively uncommon conditions. Expanding to a multicenter study in order to increase the study size can result additional problems such as variability and standardization of examiners and sometimes poorer data quality. [3] Overall, it is very important to design the size of a study in advance with an explicit calculation relating to power; the probability of detecting as statistically significant a difference of specified size that judged both plausible and clinically importance, or expected confidence interval width. The larger the study, the higher the power, and the more accurate the effect size can be estimated. [3]

The term "significant" should be referred to an important finding. There may be a range of values that reflect clinically relevant finding depending on clinical scenario and those different treatment methods may provide various benefits. [2] For instance, the clinical significance would be further enhanced if the new agent also had fewer side-effects than the gold standard and or was less expensive. [3] Identifying specific and sharp criteria related to clinical periodontal parameter could enhance interpretation of results and select of most suitable treatments for various clinical problems. [14] Finally, it is time for investigators to address the clinical importance of their data to help clinicians choosing effective therapies.

It seems that interpretation of results from experimental studies assessing the efficacy of periodontal therapy should focus primarily on the clinical relevance of the data and then statistical evaluation should be used to validate that these findings did not occur by chance.

Statistical significant findings should not be considered the only reason for comparing of two therapeutic methods. [2]

There are some analytical qualitative methods as well as quantitative strategies to help confirm the extent of clinical significance, however there is neither universally accepted guideline nor consensus to determine, for example, how much root coverage can be assumed "clinically significant" during a routine periodontal plastic surgery. [13]

Strict adherence to mean values of each group may provide limited information about how treatment response varies among individuals and sites. Almost all of theses we encountered in our study had used mean values evaluation because of quantitative nature of clinical parameter, however mean value is very sensitive to outliers. [15] In these cases converting of the statistics for example taking logarithms or reporting additional summary statistics (such as standard deviation, median, mode, minimum, maximum values and frequency distributions) by researchers could be helpful for clinicians. Otherwise, we recommend precisely revising of the data by the alert research men to overcome this error by awarely eliminating of the outliers if it is possible.

We emphasized "clinical significance "as defined by Greenstein: "a change that may alter how a clinician will treat a patient, and this value judgment varies depending on a situation". [2]

There are several approaches to evaluating clinical significance: Statistical and value based. An important question concerning clinical significance is "how small an improvement is clinically meaningful?" There is no precise answer, because the answer needs to be altered and tailored for different situation and goals of the therapy. [16]

Limitations of this study include difficult access to the theses. Actually, access to about one-third of the full theses was impossible as a result of poorly cooperation of directors of one library within the framework of the strict law. They could inhibit everyone's easily access to their full postgraduate theses because they were worried about copying them, and of course they were right. We observed marked difference between theses in last ten years and theses had been written before from statistical and methodological point of view. This difference was so noticeably in a few years ago, so that we could observe many systematically organized studies.

It seems to be related to modifying methodological concepts about biostatistics all over the world since last decade.

Eventually, recommendations to be used to identify clinically significant changes related to periodontal therapy should be established at consensus workshops by experts.

 Conclusion



Defining clinical significance will depend on interpreting publications and clinician's personal experiences according to advantages and disadvantages of each new procedure, to arrive at a conclusion that a result is clinically significant, and the finding must be clinically meaningful and statistically significant. The statistical and clinical significance of the estimated treatment effect, the safety profile and the possibility of delivering the intervention are all used to determine whether an investigational treatment should be adopted into practice. Clinical or even basic studies usually focus on statistically significant relations but it is necessary to remember the underlying hypothesis. In other words, we have to think about what kind of results which is supposed to be clinically important at the designing step and before starting of the study. We strongly recommend that clinically significant findings in each case of clinical studies should be defined at the designing stage and threshold values of each clinical parameters such as clinical attachment level changes should be determined before beginning the study.

 Acknowledgments



The authors wish to thank many of directors of central libraries of the dental faculties and some of the dental students (Samaneh Hashemi, Sarah Dehghan khalili and Ehsan Shafiee). This study had no support and the authors do not have any financial interests, either directly or indirectly.

References

1Bhandari M, Joensson A. Part πC: Understanding treatment effects. Clinical Research for Surgeons. Thieme electronic book library. Thieme Publisher; 333, Seventh Avenue, New York, NY 10001, USA, 2009. p. 139-44.
2Greenstein G. Clinical versus statistical significance as they relate to the efficacy of periodontal therapy. J Am Dent Assoc 2003;134:583-91.
3Addy M, Newcome RG. Statistical versus clinical significance in periodontal research and practice. Periodontol 2000 2005;39:132-44.
4Ferraris VA, Ferraris SP. Assessing the medical literature: Let the buyer beware. Ann Thorac Surg 2003;76:4-11.
5Williamson JW, Goldschmidt PG, Colton T. The quality of medical literature: An analysis of validation assessments. In: Bailar JC, Mosteller F, editors. Medical uses of statistics, 1 st ed. Waltham, MA: NEJM Books; 1986. p. 370-91.
6Avram MJ, Shanks CA, Dykes MH, Ronai AK, Stiers WM. Statistical methods in anaesthesia articles: An evaluation of two American journals during two six-month Periods. J Anesth Annul 1985;64:604-11.
7Lindgren BR, Wiellinski CL, Finkelstein SM, Warwick WJ. Contrasting clinical and statistical significance within the research setting. Pediatr Pulmonol 1993;16:336-40.
8Lefort BM. The statistical versus clinical significance debate. Image J Nurs Sch 1998;25:57-62.
9Hujoel PP, Armitage GC, Garcia RI. A perspective on clinical significance. J Periodontol 2000;71:1515-8.
10Stoner JA, Payne JB. Interpretation of treatment effects in periodontal research: A note on the number needed to treat. J Can Dent Assoc 2008;75:435-7.
11Greenstein G, Nunn ME. A method to enhance determining the clinical relevance of periodontal research data: Number needed to treat. J Periodontol 2004;75:620-4.
12Feinstein AR. Clinical biostatistics, II: Quantitative significance and statistical indexes for a contrast of two groups. Clin Pharmacol Ther 1980;27:567-78.
13Rosenberg EI, Bass PF 3 rd , Davidson RA. Arriving at correct conclusions: The importance of association, causality, and clinical significance. South Med J 2012;105:161-6.
14Greenstein G, Lamster I. Efficacy of periodontal therapy: Statistical versus clinical significance. J Periodontol 2000;71:657-62.
15Rethman MP, Nunn ME. Clinical versus statistical significance. J Periodontol 1999;70:700-2.
16Page RC, Armitage GC, DeRouen TA. Design and conduct of clinical trials of products designed for the prevention, diagnosis, and therapy of periodontitis. Chicago: American Academy of Periodontology; 1995. p. 1-54.