Clinical Prediction Rules: a little primer

Clinical Prediction

 How do we understand if a patient is likely to:

 

a)      have the condition/dysfunction we think they have?
b)      respond in a meaningful and positive way to a chosen intervention?
c)       get better within a particular time-period?

 

Clinical Prediction Rules (CPR)

Algorithmic tools developed to assist clinical decision-making by interpreting data from original studies into probabilistic statistics.

 

• Diagnostic CPR (DCPR)

 

• Interventional CPR (ICPR)

 

• Prognostic CPR (PCPR)

 

CPR Quality

 

Grading Scale  IV (poor) to  I (best) (McGinn et al, JAMA,2000;284:79-84)

 

IV     Derivation only  

 III      Validation in narrow population                                 

II       Validation in broad population                                  

I         Impact Analysis

 

 

Derivation

• Data derived from primary study (ideally RCTs, but usually prospective cohort studies).
• Need to clearly define target conditions, reference standards, and potential predictor variables.
• Need dichotomous outcomes (e.g. “condition present or absent” / “treatment successful of unsuccessful” / “condition persistent or not-persistent”).
• Study needs to report PROPORTIONS.

 

Validation

• Separate study with either a narrow of broad population (ideally RCT, ideally block-randomisation).
• Separate study subjects and therapists to primary study.
• To confirm that predictor variable effects are not due to chance.

 

Impact analysis

• To determine meaningful impact on clinical practice (ideally RCT)
• Multi-centre
• Is rule implemented?
• Does it maintain improved care?

 

Formal quality assessment

DCPR: no validated formal assessment tool

ICPR: 18-item tool.  Beneciuk et al, Phys Ther,2009; 89:10-11

PCPR: 18-item tool. Kuijpers et al, Pain, 2005;109:429-430

 

Statistics of interest

The whole world can be represented by a “2 x 2” contingency table!

	Ref Standard P/Outcome P	Ref Standard N/Outcome N
Test P / Control Group	a                 TP	b                 FP	a+b  TP+FP
Test N / Rx Group	c                 FN	d                 TN	c+d  FN+TN
	a+c       TP + FN	b+d         TN+FP

 

 

 

Diagnosis / Intervention

Intervention only

SENSITIVITY (“TP rate”) = a/(a+c)    SnNOut SPECIFICITY (“TN rate”)= d/(b+d)     SpPIn LIKELIHOOD RATIO +  =  sensitivity/(1-specificity) LIKELIHOOD RATIO –  =  (1-sensitivity)/specificity       Probability shifts:   LR+ 1 – 2: small, unimportant 2-5: small but possibly important 5-10: moderate >10: large, possibly conclusive   LR- 0.5 – 1: small, unimportant 0.2 – 0.5: small but possibly important 0.1 – 0.2: moderate >0.1: large, possibly conclusive

CONTROL EVENT RATE (CER) number of Control Group people with +ve outcome divided by total number of Control Group people. In other words: i.e.: a/(a+b)   EXPERIMENTAL EVENT RATE (EER) = same as above for Rx Group c/(c+d)   RELATIVE RISK, or RISK RATIO (RR): RR = EER/CER (a RR of 1 means there is no difference between groups; >1 means increased rate of outcome in Rx group, and <1 means less chance of outcome)   ABSOULTE RATE REDUCTION (ARR): ARR = CER – EER   RELATIVE RISK/RATE REDUCTION (or increase!) (RRR): RRR = (CER-EER)/CER   NUMBER NEEDED TO TREAT (NNT): NNT = 1/ARR   Ratios: EXPERIMENTAL EVENT ODDS (EEO): c/d   CONTROL EVENT ODDS (CEO): a/b   ODDS RATIO (OR): EEO/CEO   (The greater above 1, the better)     EFFECT SIZE = (mean score of group 1) – (mean score of group 2) SD (of either group, or even pooled data)

 

Other tests:

Test / Statistic	Purpose (outcome statistic)
T-test	Difference between 2 groups (p-value)
Chi-Squared	Frequency of observations (p-value)
Receiver Operating Characteristic (ROC) curve	Identifies score which maximises TPs and minimises FNs (Youden’s J)
Logistic Regression	Identifies predictor cut-off points, and predictor clusters (Beta-values (Expβ))
Recursive Partitioning	Repeated sub-group analysis to identify best-fit patients (index of diversity)
Confidence Intervals	Describe precision (variance) (%)

 

Examples of Lower Quadrant CPRs

 

Diagnosis (Medical Screening – DCPR)

 

Target Condition:  Deep vein thrombosis (lower limb).

 

Test:  Wells’ Score (Wells et al, J Intern Med, 1998;243:15-23)

 

Quality: Level  I (impact analysis: 10 relevant acceptable quality associated studies)

 

Test details (predictor variables)

 

1. Activecancer                                                                                                                1
2. Paralysis, paresis, or recent plaster immobilisation of the lower extremity            1
3. Recently bedridden for >3 days and/or major surgery with 4 weeks                         1
4. Localised tenderness along the distribution of the deep venous system                                1
5. Thigh and calf swollen                                                                                                                    1
6. Calf swelling 3cm > asymptomatic side (measured 10cm below tibial tuberosity)                1
7. Pitting oedema; symptomatic leg only                                                                                    1
8. Dilated superficial veins (non-varicose) in symptomatic leg only                                 1
9. Alternative diagnosis as or more likely than DVT                                                                             -2

 

Test scoring

≤ 0 points            Low Risk                               6% probability of DVT

1 or 2 points       Moderate Risk                  28% probability of DVT

≥ 3                          High risk                               73% probability of DVT

 

Reference standard(s): Plethysmography and venography

 

Study parameters

Inclusion:

Signs and symptoms for < 60 days

 

Exclusion:

Previous DVT of PE

Renal insufficiency

PE suspected

Pregnancy

Anticoagulation treatment for >48 hours

Below-knee amputation

Strong alternative diagnosis

 

 

Bottom Line

Best quality CPR (Level I): Recommended for clinical use within confines of study parameters.

 

 

 

 

 

 

 

Diagnosis (Orthopaedic Diagnosis – DCPR)

 

Target condition: Lumbar/buttock/leg pain arising from the sacroiliac joint

 

Test: 6-item predictor cluster based on physical examination responses (Laslett et al, Man Ther, 2005;10:207-218)

 

Quality: Level IV (derivation only (poor quality), no validation, no impact analysis); no regression analysis, no recursive partitioning.

 

Test details (predictor variables)

 

1. Positive SIJ compression test
2. Positive SIJ distraction test
3. Positive femoral shear test
4. Positive sacral provocation
5. Positive right Gaenslen’s test
6. Positive left Gaenslen’s test

 

Test scoring

3 or more predictor variable present =LR+ 4.3 (95%CI 2.3 – 8.6)

 

Reference standard(s): fluoroscopy-guided provocative SIJ injection

 

Study parameters

Mean age 42 (+/- 12.3)

Mean symptom duration (months) 31.8 (=/- 38.8)

 

Inclusion:

Buttock pain +/- lumbar/leg pain

Had imaging

Unsuccessful previous therapeutic interventions

 

Excludes:

Mid-line or symmetrical pain above L5

Nerve root compression signs

Referred for non-SIJ injection

Too frail for manual therapy

Pain free on day of assessment

Bony obstruction to injection

 

Bottom Line:

Not validated –  study findings could be due to chance. Small probability shift power of LR+. Not recommended for clinical use.
Interventional (ICPR) 

 

Target condition: Acute low back pain, manipulation

 

Test: 5-item predictor cluster (Flynn et al, Spine, 2002;27:2835-2843)

 

Quality: Level II (broad validation, no impact analysis. 4 associated high quality validation studies)

 

Test details (Predictor variables)

1. No pain below knee
2. Onset ≤ 16 days ago
3. Lumbar hypomobility
4. Medial hip rotation > 35deg (either hip)
5. Fear Avoidance Belief Questionnaire (Work subscale) <19

 

 

Test Scoring

4 or more predictor variables present = LR+ 24.4 (95% CI 4.6 – 139.4)

 

Reference Standard(s) (i.e. definition of success) 

50% or more improvement on modified Oswestry Disability Index

 

Study parameters

Mean age 37.6 (+/- 10.6)

 

Inclusion:

Lumbosacral physiotherapy diagnosis

Pain +/- numbness lumbar/buttock/lower extremity

Modified ODI score ≥ 30%

 

Bottom Line

Due to validation and very large probability shift power of LR+, this CPR is recommended for clinical

use within confines of study parameters.

 

Prognostic (PCPR)

 

Target condition: LBP, recovery

 

Test: 3-item predictor cluster (Hancock et al, Eur J Pain, 2009;13:51-55)

 

Quality: Level III (narrow validation, no impact analysis)

 

Test details (Predictor variables)

1. Baseline pain ≤ 7/10
2. Duration of symptoms  ≤5 days
3. Number of previous episodes ≤1

 

Test Scoring

All 3 predictor variables present = 60% chance recovery at 3 weeks, 95% chance recovery at 12

weeks.

 

Reference Standard(s) (i.e. definition of success)

7 days of  pain scored at 0-1 /10

 

Study parameters

Mean age 40.7 (+/- 15.6)

 

Inclusion:

LBP (between 12^th rib and buttock crease) +/- leg pain

Seen GP

< 6 weeks duration

Moderate pain and disability (SF-36)

 

Exclusion:

No pain-free period pre-current episode of at least 1 month

Known/suspected serious pathology

Nerve root compression

Taking NSAIDs

Receiving spinal manipulative therapy

Surgery within preceding 6 months

Contraindications to analgesics or manipulation.

 

 

Bottom Line

Due to validation, careful application to clinical practice is recommended, strongly within confines of study parameters.

 

 

 

 

Ref: Glynn PE, Weisbach PC 2011 Clinical Prediction Rules: A Physical Therapy Reference Manual. JBL Publishing

How to turn “Stats” into something useful: Diagnosis and Interventions

How to turn “Stats” into something useful 1: Diagnosis

Understanding diagnostic utility

If you have data for diagnostic utility studies, you can use a 2×2 contingency table to calculate the following information (this should have been reported anyway in the study, but often isn’t):

 

		Gold standard
Clinical test		+	–
	+	aTP	bFP
	–	cFN	dTN

 

 

 

Sensitivity (“TP rate”) = a/(a+c)

Specificity (“TN rate”)= d/(b+d)

Likelihood ratio + = sensitivity/(1-specificity)

Likelihood ratio – = (1-sensitivity)/specificity

You can then use something like a nomogram to calculat post-test probability. You will need to have an estimate of pre-test probability. Ideally, this will be the known prevalence of the condition

nomogram

Or, get yourself on app on your ‘phone like MedCalc3000 https://itunes.apple.com/us/app/medcalc-3000-ebm-stats/id358054337?mt=8

How to turn “Stats” into something useful 2: Interventions  

If a trial or systematic review is reporting DICHOTOMOUS outcomes, we can bring the “research” findings a little bit closer to clinical decision making… “Do you know HOW MANY subjects responded, and HOW they responded? e.g.  how many people in the TREATMENT group got better/worse, and the same for the control/placebo group?”

NO: then you can’t clinically apply findings. Doh.   YES: then go and do some evidence based decision making! Yipee.

 

 

 

Wow, how do we do that? Like this: 1)   Use a 2×2 table (again)

	Outcome
	+ve	-ve
Control/Placebo group	a		b
Rx group	c		d

 

 

 

2)   And some simple formulae…

CONTROL EVENT RATE (CER) number of Control Group people with +ve outcome divided by total number of Control Group people. In other words: i.e.: a/(a+b)

EXPERIMENTAL EVENT RATE (EER) = same as above for Rx Group c/(c+d) Now that we know the CER and EER, we can do loads of other useful things…

RELATIVE RISK, or RISK RATIO (RR): RR = EER/CER (a RR of 1 means there is no difference between groups; >1 means increased rate of outcome in Rx group, and <1 means less chance of outcome)

ABSOULTE RATE REDUCTION (ARR): ARR = CER – EER

RELATIVE RISK/RATE REDUCTION (or increase!) (RRR): RRR = (CER-EER)/CER

NUMBER NEEDED TO TREAT (NNT): NNT = 1/ARR Some other stats more USEFUL than “p-values”…

 

1)   THINK LIKE A BOOKIE..!

“What are the odds of getting this person better with this treatment?”

EXPERIMENTAL EVENT ODDS (EEO): c/d CONTROL EVENT ODDS (CEO): a/b ODDS RATIO (OR): EEO/CEO

The greater above 1, the better.

2)   EFFECT SIZE.

This is a standardised, scale-free measure of the relative size of the effect of an intervention. It is particularly useful for quantifying effects measured on unfamiliar or arbitrary scales and for comparing the relative sizes of effects from different studies.

EFFECT SIZE = (mean score of group 1) – (mean score of group 2) / SD (of either group, or even pooled data)

 

EXAMPLE: A study into effects of manual therapy on neck pain measured a Rx group (n=23) and a Control group (n=21), and considered a “cut-off” point for improved ROM as being an increase in at least 20deg rotation (so results can be dichotomised). Mean Rx score = 28deg (SD 4) Mean Control score = 16deg (SD 4) Results were:

 

		Outcome
		+ve	-ve			Total
Control/Placebo group		a9			b12	a+b21
Rx group		c18			d5	c+d23
	Total	a+c       27		b+d17		a+b+c+d44

CER a/(a+b): 0.43 or 43%

EER c/(c+d): 0.78 or 78%

RR = EER/CER: 1.83

ARR = CER – EER: -0.35 or 35%

RRR = (CER-EER)/CER: -0.83 or 83% (a minus figure, so this would be RR Increase)

NNT = 1/ARR: 2.86, so say 3.

EEO: c/d = 3.6 CEO: a/b = 0.75

OR: EOR/CEO = 4.8

EFFECT SIZE =  28– 16   = 3 4

 

The clinical  story then…

“So, if untreated, my patient would have a 43% chance of getting better anyway.  But if treated, his chance of improvement would be 78%.  He is 1.83 times more likely to improve if I treat him. The absolute benefit of being treated would be 35%.  The treatment would increase the chance of improvement by 83%. I would need to treat 3 people (in the period of time relevant to the study) to achieve 1 positive outcome. The odds of him getting better with no treatment are 0.75, whereas if I treat him, the odds are much better, at 3.6. the odds are 3.6:0.75 (i.e. 4.8) of him improving with treatment. “

However… From a clinical reasoning perspective, we still need to understand what “43%, 78%, 35%, etc etc” MEANS… and that’s where the real fun starts:-)

Here’s a little video for y’all: http://www.youtube.com/watch?v=tsk788hW2Ms