The Process
Sindhu Johnson, MD, PhD, a clinical researcher at University Health Network in Toronto, who is also helping lead the creation of the new system, described a rigorous, multilayered process of whittling down a list of criteria, literature review and organizing the criteria into seven clinical domains and three immunological domains.
The panel of experts—nine from North America and eight from Europe—chose to use an entry criterion that all cases have to meet even to be considered under the new system. The titer of anti-nuclear antibodies (ANA) must be at least 1–80. The panel also chose not to have any criteria, such as lupus nephritis, that would automatically confer an SLE classification.
The panelists submitted 167 cases, ranging from more likely to be SLE to less likely. The experts scored 20 of the cases using the criteria and ranked them.
Then they met for two days, reviewing the inter-rater reliability of their scoring and refining the definitions of the criteria. The experts entered the multicriteria decision analysis (MCDA) step, in which they each were presented with two criteria under two domains for a pair of cases. They had to choose anonymously which case was more likely to be SLE.
In one example, the panel experts were asked, “Which patient has the higher probability of being classified as SLE?” The choices were a case with malar rash (in the cutaneous domain) and anti-dsDNA (in the highly specific antibodies domain) or a case with oral ulcers and anti-Smith antibodies.
“If there was clear consensus, we knew which patient had the higher rated criteria,” Dr. Johnson said. “However, if there was a lack of consensus, then a discussion ensued.” The experts revoted until a consensus was reached.
Computer software, called 1000Minds, used these consensus decisions to calculate weights for each of the criteria, then assigned a total score to the other 147 cases and ranked them. The experts then voted on whether each should be classified as SLE.
For criteria that didn’t match expert opinion, the pair-style rankings were done again until the experts reached consensus. The software was used to recalculate the weights and rerank the cases.
The panelists were in complete agreement on the cases with the highest rankings—they were all SLE—until the scores dipped below 84, when some disagreement emerged, Dr. Johnson said. “For this reason, a threshold score of 84 was identified,” she said, meaning that any score above 83 is considered to be SLE and scores of 83 or lower are not.
