Expert curation of our indication catalog for disease-modifying treatments

Latest indication catalog

We have completed a first-draft of our indication catalog. Recently, we updated the catalog with some fresh disease cross-references. The latest catalog is now online (webpage, tsv, repository).

The catalog includes 1,388 high-confidence indications from four resources — MEDI-HPS [1], ehrlink [2], LabeledIn [3, 4], and PREDICT [5] — and 1,114 low-confidence indications from MEDI-LPS [1]. We are primarily concerned about the high-confidence associations which connect 108 diseases from DO Slim and 744 small molecules from Drugbank Slim.

Problematic indications

The source databases have a loose definition of indication — symptomatic treatments are often considered indications. For example, a narcolepsy drug approved for MS-induced fatigue is indicated for MS in our catalog. However, we are primarily interested in disease-modifying therapies. Since our method trains itself from this catalog, our predictions will recapitulate the types of indications included.

Seeking an expert

We are seeking an expert physician to manually review our 1,388 high-confidence indications and identify the disease-modifying subset. Since the classification is unlikely to be straightforward, we are looking for someone who can conceptualize the task from a high-throughput systems pharmacology perspective. We are happy to offer project authorship for a job well done.

Updated datasets

We modified the code and formats for our merged indication datasets (website, downloads). The underlying indications have not changed from above.

Pilot on 50 indications

We created a simpler tsv file as a curation template. As a start, we are going to have two UCSF physicians each classify a random subset of 50 indications. These indications will be a pilot to see if the task is well-defined or needs revision.

The curators will independently do a first pass. Then they will come to a consensus on conflicting classifications. We'll report back with our experience on the pilot.

Curation pilot results

@sergiobaranzini recruited two UCSF physicians — Ari Green (AJG) and Christine Hessler (CSH) — for the curation task. We asked the curators to independently classify 50 random indications as disease modifying or symptomatic. See the raw results for AJG and CSH.

Combined results

While we did not specify that "not an indication" was an option, both curators identified these instances. While our indication dataset derives from high precision datasets, non-indications will be present. Thus, going forward we will include "not an indication" as a classification.

I cleaned the curators free text into 3 classifications: DM for disease modifying, SYM for symptomatic, and NOT for not an indication. This cleaning step required some inference on my part and thus could have introduced a minor bias. The combined results show 66% agreement, with the following breakdown by curator:

The pilot suggests that the compiled indications are ~58% disease modifying, ~37% symptomatic, and ~5% non-indications.

Definitions

We did not provide the curators with clear definitions of disease modifying and symptomatic. Our plan is to use the pilot experience to help define the categories. Preferably, these definitions should be crafted by the physicians.

Informally, CSH defined disease modifying as:

any agent that changes the course of the illness or complications of the illness (not necessarily curing the disease, but preventing "flares" or complications). I'd ask myself, "is it poor form not to prescribe this medication to my patient with disease y and could I be sued for it?"

And symptomatic as:

agents that are used purely for patient's comfort and don't alter the course of the disease at all.

Cocaine and cavities

One bizarre indication was cocaine and dental carries, which was contributed by MEDI-HPS [1]. Since MEDI doesn't report its sources for individual indications and the authors did not release this data upon request, tracking down the provenance of this indication is difficult. However, MEDI does specify that this indication existed in SIDER 2 [2], which text mined drug labels. Presumably, this label for Cocaine Hydrochloride Topical Solution (brand name C-Topical) was the source for the cocaine–cavities indication. The label states the solution "is indicated for introduction of local (topical) anesthesia of accessible mucous membranes of the oral, laryngeal and nasal cavities." This suggests a non-indication or at most a symptomatic indication between cocaine and dental caries. However, for a SIDER 2 indication to be included in MEDI-HPS, at least one other source must report the indication. AJG informally suggested that the "reason for the hit is that cocaine accelerates dental caries and might therefore be considered disease modifying (but in a bad way)."

Next steps

The pilot experience proved the importance of expert curation of our compiled indication catalog. Before proceeding with the remaining indications, we should formally define the three classifications (DM, SYM, NOT). Clear definitions may increase the agreement between curators, but a final resolution stage for conflicts will be necessary.

Definitions

In a meeting yesterday, we (Ari Green, Christine Hessler, @dhimmel, @sergiobaranzini) discussed the pilot experience and definitions.

AJG pointed out that the term "disease modifying" is primarily used for rheumatology and multiple sclerosis. With this caveat in mind, we set out to identify a general definition that could apply broadly to complex disease.

When considering possible definitions, @sergiobaranzini and I stressed the following quality of a "disease modifying" indication:

If we predicted this indication, would the disease be considered an appropriate and precise application of the drug.

We agreed on the following definitions:

• disease modifying (DM) — a drug that therapeutically changes the underlying or downstream biology of the disease
• symptomatic (SYM) — a drug that treats a significant symptom of the disease
• non-indication (NOT) — a drug that neither therapeutically changes the underlying or downstream biology nor treats a significant symptom of the disease

We also agreed on the following guidelines:

• reasonable evidence of efficacy is required to be classified as disease modifying or symptomatic
• if no classification accurately describes an indication, the most appropriate (although imperfect) classification should be chosen

Next steps

AJG and CSH found many of their disagreements on the pilot indications resolved once a common definition was reached. With these definitions, we will now move onto the full set of indications, which AJG and CSH have agreed to curate.

Results from first two curators

The curations from AJG and CSH (@chrissyhessler) are in. Both curators went through and classified each of the 1,388 compound–disease pairs. The breakdown of their classifications are as follows:

Compared to the pilot, the curators classified a higher percentage of pairs as non-indications, while classifying a lower percentage as disease-modifying. Similar to the pilot, the curators agreed 68.0% percent of the time. The Cohen's kappa coefficient [1, 2] between AJG and CSH was 49.9% (notebook), indicating moderate agreement.

Looking at only the 944 agreements, there were 447 disease-modifying, 351 symptomatic, and 146 non-indications. For the remaining 444 disagreements, we plan to have a third curator break the tie.

Recruitment of a third curator

We have recruited a third curator, Pouya Khankhanian, to break ties. Pouya is a resident physician in neurology at Penn. He received his in MD at UCSF.

When learning of the task and before seeing this discussion and the three classifications, Pouya asked a few questions about what qualifies as an indication. While some of these may be cleared up by the definitions we decided on, I thought it would be helpful to post his questions here along my opinions.

Suppose a drug was indicated for treatment of seizures 10 years ago, but now we have a new drug that is more efficacious and has fewer side effects, so the old drug is no longer "indicated" in clinical practice. However, I suppose for the purposes of your study you would still want to call this "indicated"?

I would not disqualify this indication because it's no longer optimal. It still treats the disease and will therefore be helpful in training and validating our model.

Or, suppose a drug is clinically used for a disease, but is not actually indicated. Classic example is that almost all of our MS drugs are "not indicated" for treatment of progressive MS (meaning no trial has ever shown efficacy), but as you know most of our progressive MS patients get treated.

Off-label usages are acceptable as long as there's reasonable evidence of efficacy from a clinical perspective.

Or suppose a drug is like fifth line, and is only indicated if someone is medically refractory (i.e. they have failed the first four lines of drugs). Would you consider this "indicated"?

I don't think being far down the line should be a disqualifying factor for the reasons above.

Initial results from the third curator

I was initially recruited to break the 444 disagreements between the other curators. After an initial pilot review of the first 80 or so disagreements, I noted some ambiguity in definition of the three classes that appeared to be giving rise to some of the disagreements. I discussed these with Daniel and we reached a more precise amended set of definitions.

Definitions:

• disease modifying (DM) — a drug that therapeutically changes the underlying or downstream biology of the disease
• symptomatic (SYM) — a drug that treats a significant symptom of the disease
• non-indication (NOT) — a drug that neither therapeutically changes the underlying or downstream biology nor treats a significant symptom of the disease

Guidelines:

• reasonable evidence of efficacy is required to be classified as disease modifying or symptomatic. This includes off-label use.
• if no classification accurately describes an indication, the most appropriate (although imperfect) classification should be chosen

Amendments: (created 1/27/16, not seen by AJG and CSH)

• Amendment 1: if a drug was previously indicated, but is no longer used due to side effects, or because there are better drugs, it is still considered DM
• Amendment 2: it doesn't matter whether it is first line or fifth line, it's still considered DM

Assumptions: (by PK)

• Assumption 1: DM trumps SYM. If a drug is clearly both disease modifying and also treats symptoms, then I will call it disease modifying. This is because most disease modifying drugs also treat symptoms.
• Assumption 2: SYM trumps NOT. If a drug is clearly symptomatic treatment, but can actually exacerbate the downstream biology of disease, then I chose SYM. I made this choice because this was the choice I saw most often made by AJG and CSH

With the revised definitions above, I reviewed the 444 disagreements as well as the 944 agreements (and suggested a change on 124 of these). I was not blinded to the other curators' decisions. I was able to see both of their decisions and also any comments they had left regarding their reasoning. In general, my strategy was to look three sources for each drug (unless I clearly already knew that a drug was DM or SYM): uptodate.com, drugbank.ca (link provided by Daniel in the spreadsheet), and a basic google search (which also served as a proxy for a pubmed search). When I noted that one of the two curators was calling an indication which I was not aware of (either DM or SYM), I would do a much more detailed search including a more detailed google search and a direct pubmed search.

Below is the breakdown of classifications

The most notable difference was that I called DM more often than the other curators. There are at least two reasons for this. First, I was making use of amendment 1 and amendment 2 to make calls for DM, whereas the other curators were not using these amendments (in fact, when the other curators called NOT, they left comments such as "no longer recommended due to side effects", "not used anymore", or "rarely used"). Second, when I found a disagreement between two curators, I was more likely to agree with the curator who called DM. Specifically, of the 444 disagreements, there were 298 where one curator chose DM; of these 298 instances, I chose DM 204 times. I think this is because I did a more detailed search when I knew that one other curator thought that there was a DM indication. Of note, of the 146 times that the other curators were in disagreement between SYM and NOT, I chose SYM 76 times, I chose NOT 52 times, and I chose DM 18 times, likely for the same reasons described above.

The excel spreadsheet includes a detailed discussion of every decision that I made. I will not re-iterate here the instances where I used one of the amendments above to change a call or to resolve a disagreement. I will also not detail instances where I changed a call because I thought another curator made a human error (for example, not classifying two proton-pump inhibitors in the same way for the same disease). I will also not re-iterate cases where I felt that one of the two other curators knew about an indication (either DM or SYM) and I was able to confirm evidence of this indication.

I would like to enter into the discussion the cases where there was a tough decision to be made, and would like to welcome an open discussion to come to a consensus. In general, when there was a tough decision, I did look at the other curators' calls to see what the consensus would be. Below is a summary of this discussion (with greater detail given in the spreadsheet), organized by disease and drug class.

• hypertension, general — Hypertension (the disease entity) is a heterogenous group of diseases. The most likely subtype of hypertension was likely Essential Hypertension (ET). The disease of Hypertension (including ET) progresses to have complications such as strokes and heart attacks. Hypertension (i.e. high blood pressure) is also a symptom (of many diseases, including diseases that are not called "hypertension"). Within the disease of ET, hypertension is not just a symptom but also a marker of disease progression, i.e. controlling blood pressure (treating this symptom) will slow the advancement of the disease hypertension and prevent downstream biology (proven by evidence, guideline 1). Therefore, by assumption 1, a total of 29 drugs were called DM rather than SYM.

• hypertension, diuretics —all called DM due to amendment 1 and amendment 2

• hypertension, drugs used to treat ocular hypertension or pulmonary hypertension — "hypertension" as defined in Daniel's link as "chronic elevated blood pressure in the arteries" It is therefore not the same as "ocular hypertension" or "pulmonary hypertension" (I think the spirit of the definition is systemic arteries, not pulmonary arteries. Also, pulmonary hypertension is quite a different disease with different pharmacology).Therefore, I chose to put NOT for all of these.

• type 2 diabetes, drugs that lower blood sugar —Diabetes is similar to hypertension. The disease is a tendency to have high blood sugar (hyperglycemia). Hyperglycemia is a symptom (of both diabetes and other diseases). Within the disease of diabetes, hyperglycemia is both a symptom and a marker of disease progression. Therefore anything that lowers hyperglycemia will be DM.

• type 1 and type 2 diabetes, ACE inhibitors and ARBs — The downstream biology of DM2 includes proteinuria and eventual renal failure. ACE inhibitors and ARBs prevent this downstream biology in DM2 patients. Therefore they are DM.

• epilepsy, anti-epileptic drugs —I think that for consistency, all anti-epileptics should be either DM or SYM, as there is only very limited evidence that any of these drugs are different from each other. My thoughts would be to label them all as DM. Here is why: epilepsy syndrome (disease) is defined as a propensity to have seizures (symptom). However, the natural downstream biology of the disease is that each seizure that you actually have makes you more likely to have worse epilepsy in the future (i.e. seizures beget more seizures). One mechanism is that when you have a lot of seizures, you develop mesial temporal sclerosis, and mesial temporal sclerosis is a risk factor for further seizures. Therefore, I would argue that any drug which treats the symptom of seizure is actually affecting downstream biology, and is therefore disease modifying. And by assumption 1, DM trumps SYM.

• osteoarthritis, NSAIDs and steroids — I put everything as SYM. From MedScape: "To date, no disease-modifying or structure-modifying intervention has been proved effective in osteoarthritis." CSH agreed with this interpretation, while it was clear that AJG was conflicted. To play devil's advocate, you could potentially say that the biology of osteoarthritis (OA) that it starts with inflammation, and the "down-stream" biology is the pain (the primary symptom as well), and therefore NSAIDs prevent "down-stream" biology. However, if we want to make that decision, I think we should change all the NSAIDs and steroids to DM.

• cancers, pain medications — I think pain is a symptom of cancer and therefore I put all of these as SYM. CSH agreed, while AJG was conflicted and sometimes called NOT.

• hematologic cancers, steroids —steroids actually "treat" hematologic cancers, even though these days there are much better meds and steroids are not considered "treatment", in the past they were the first line. By amendment 1, I put all of these as DM.

• non-hematologic cancers, steroids — steroids treat the nausea symptoms associated with cancers. While many cancers can potentially cause nausea, most nausea in cancer patients is due to side effect of chemo. However, I still put SYM for these because they can treat nausea and nausea is potentially a side effect of any cancer. CSH agreed with me on most of these, AJG was conflicted.

• cancers, hydroxyurea and other chemotherapies — AJG called this DM for all cancers. CSH called it DM only for the cancers for which it is indicated. The truth is, any chemotherapy has theoretical benefit against any cancer (any quickly-reproducing cell type). One possibility would be to label all chemotherapies as DM for all cancers. I thought it would be better to be selective and only label DM for chemo that is used (or has been used) in a particular cancer. That way, the results of the drug-repurposing search would yield different results for different cancers (rather than giving the exact same result for all cancers because the input was exactly the same for all cancers).Thus, I labeled some DM and some as NOT.

• cancers, bisphosphonates — I don't think of bone loss as a "side effect" of cancers (at least not any of the cancers listed). Some people are malnourished and/or have drug-induced bone loss, or may have bone metastases, but I don't think this captures the essence of cancer. I chose to put NOT for all of these (CSH agreed, AJG generally chose SYM).

• coronary artery disease, drug to treat hypertension or diabetes — I treated this as pure coronary artery disease (CAD) in the absence of other causes. I did not interpret this as "CAD as a consequence of hypertension (HTN)" or "CAD as a consequence of diabetes (DM2)". It is true that many of these medications would help prevent CAD if CAD is considered as the "downstream biology" of HTN or DM2. However, the medications to not treat any biology downstream of CAD in the absence of HTN or DM2. I therefore labeled these as NOT.

• coronary artery disease, diuretics and other drugs used for congestive heart failure (CHF) — I consider these to be DM. Consider CHF as a common dowstream biology of coronary artery disease (CAD), specifically let's consider HFrEF. The biology of HFrEF is that the heart has poor cardiac output, thus there is fluid retention, thus there is further strain on the heart, creating a vicious cycle. Thus, diuretics should help avoid the vicious cycle and slow the downstream progression of disease. While no trial may have ever showed mortality benefit, I think there is reasonable evidence that this would be true.

• migraine, general — While AJG called everything SYM, CSH called one drug (amitryptilene) DM and everything else SYM, citing that this medications "may decrease frequency of migraines". I agreed with CSH's interpretation and actually chose to include many other medications as DM based on the same reasoning. Migraine disease is a propensity to get migraine headaches. Therefore, anything that decreased migraine frequency was considered by me to be DM (decreases the downstream biology that leads to headache). Anything that treated the pain of the headache I considered SYM.

• autoimmune diseases, steroids and NSAIDs — I labeled the steroids as DM and the NSAIDs as SYM. It is true that steroids are rarely if ever actually used for chronic disease (though often used to treat the symptoms of flares), mostly because of their terrible side-effect profile long-term. However, they do actually affect disease biology and do not specifically treat any specific symptom (for example, steroids do not cure "weakness", but do change the biology of the multiple sclerosis flare to help the patient recover from "weakness"). It was a close call for me for NSAIDs (as they do have anti-inflammatory properties, especially useful in the auto-immune arthritidies), but I went with consensus and chose SYM. The other curators tended not to call steroids DM (probably because they were thinking about side-effect profile), and they were inconsistent on their calls on NSAIDs.

• asthma, steroids and beta-agonists and anticholinergics — I put DM for all of these. I felt steroids are DM (since they are given to prevent attacks), long-acting beta-agonists are also DM in my opinion since they prevent attacks (in conjunction with steroids, despite the small increased risk of asthma-related death in people who don't use steroids). I put short acting beta-agonists as DM because they too can prevent downstream biology (since they are sometimes used, for example before exercise, to prevent downstream biology from happening).

• allergic rhinitis, steroids and anti-histamines and decongestants — CSH admittedly had a problem with this, she even noted "im having a hard time with allergic rhinitis. Maybe all of these meds are SYM.". AJG was conflicted as well. I chose to mark all of the steroids and anti-histamines as DM because they alter the immune response (the allergy). I chose to mark the decongestants (i.e. pseudoephedrine) as SYM because they treat a symptom (congestion) but not the underyling biology (the immune reaction)

• chronic obstructive pulmonary disease (COPD), general — I put steroids and beta-agonists as DM for similar reasons to asthma, though admittedly there is less evidence for this. I put all the antibiotics as SYM (they don't eradicate infection, they don't delay progression, they treat attacks) and did not differentiate between the antibiotics

• glaucoma, general — I agreed with CSH that almost every drug is DM, AJG was conflicted. I think it's more DM, based on similar discussion as hypertension.

• alcohol dependence, general — First, I included symptoms of alcohol withdrawal along with alcohol dependence, presumably because withdrawal symptoms are probably felt at some point in any person with alcohol dependence. Thus, I chose to mark chlordiazepoxide and zofran (used to treat withdrawal) as SYM (CSH agreed with both, AJG agreed with one of these). Next, there was the question of drugs designed to curb drinking (Citalopram, Disulfiram, Naltrexone, Acamprosate); I chose to mark these as DM because they are different from the drugs above in that they treat the urge to drink (modifying the disease) rather than the symptoms of not drinking (AJG agreed with all 5, CSH agreed with 1)

• psychiatric diseases other than alcohol/drug dependence, general — I agreed with both CSH and AJG that in general, the medications used are all SYM rather than DM (other than alcohol and nicotine dependence as described).

• alzheimer's disease, general — donepezil was marked as DM (agreed with AJG) because it is supposed to slow disease, not treat any specific symptom. Other cholinesterase inhibitors were changed to DM to match donepezil. I chose to group all antipsychotics as SYM in order to be consistent (AJG and CSH agreed most of the time)

• anemia — there is no good way to do this. Anemia is not a single disease, it is a very heterogeneous set of diseases (with very little overlap between sub-types in terms of incidence or pathophysiology). Though the most common cause of anemia is likely iron deficiency anemia, iron deficiency anemia accounts for probably a minority of all anemias. Specific types of anemia will of course respond to specific drugs (autoimmune anemia to steroids, folate deficiency responds to folate, etc...). I was faced with two choices: (1) choose DM for anything which could treat any type of anemia or (2) choose DM for anything which could treat most types of anemia. Choice (2) means nothing will link to anemia (no DM or SYM) and anemia will essentially be removed from analysis. Choice (1) means we are choosing a variety of drugs to treat a variety of illnesses. I chose choice (2) because I think it's best to ignore anemia, given that it is such a heterogeneous set of diseases. AJG and CSH were rather inconsistent in their answers, it is clear that they too had difficulty with anemia. In summary, everything was marked as NOT.

Overly broad and thus uninformative diseases

@pouyakhankhanian, fantastic curation!

From your comments, it appears that some of our diseases are too general from a pharmacological perspective. For example, you mention anemia and hypertension as particularly troublesome. To recap how we arrived at our 137 diseases, I selected the subset of Disease Ontology terms that have been analyzed using GWAS or were a 'body system' cancer. When diseases were redundant, a single disease was chosen (for example, coronary artery disease was chosen over myocardial infarction). In retrospect, input from physicians would have been prudent during this stage.

When we created our indication catalog (which we're curating here), I propagated indications from specific to more general terms. For example, an indication for non-small cell lung carcinoma (DOID:3908) would be considered an indication for lung cancer (DOID:1324).

In the cases of hypertension and anemia, @pouyakhankhanian found this practice problematic. Specifically, he considered pulmonary hypertension (DOID:6432) to be "quite a different disease with different pharmacology" than the definition of hypertension (DOID:10763). However the Disease Ontology defines pulmonary hypertension as a subtype of hypertension. Ocular hypertension (DOID:9282) is not a subtype — instead it's part of the glaucoma lineage. He also mentioned anemia as having heterogeneous subtypes.

While my original guidance would have been to chose DM or SYM if the drug is DM or SYM for any subtype, I see the point that some diseases may be too broad and therefore uninformative for our prediction approach.

Re: "While my original guidance would have been to chose DM or SYM if the drug is DM or SYM for any subtype, I see the point that some diseases may be too broad and therefore uninformative for our prediction approach."

I actually think either approach is reasonable. The key would be to maintain consistency throughout the curation process. After making a final decision on how to proceed, we can go back and ensure that we are being consistent.

I also think we may want to consider the relative frequency of subtypes of disease. For example, "lung cancer" has probably three very common subtypes which each account for 25-30% of the total entity of "lung cancer". Similarly, 90-95% of "Hypertension" is accounted for by "essential hypertension", even if you include "pulmonary hypertension". in contrast, most of the subtypes of "anemia" included in this curation each account for probably less than 1-2% of all "anemia".

Results from all three curators

To recap our curation effort thus far, we first had AJG and CSH (@chrissyhessler) independently classify the 1388 indications. Then a third curator, PK (@pouyakhankhanian), classified each indication with access to the picks and notes from the first two curators.

PK provided detailed documentation of his methodology, with a focus on instances of disagreement. I now report of the results from all three curators (notebook, dataset).

PK's kappa coefficient was 51.5% with AJG and 65.1% with CSH. PK classified many indications as disease-modifying that the other curators considered symptomatic. Overall, there were 34 threeway disagreements and 124 instances where PK disagreed with the consensus of the first two curators.

Reaching a consensus

The next step is to agree upon a consensus classification for each indication. These indications will go into our network and will be used to train our model for predicting drug repurposing.

We would like the first two curators to review PK's methodology and voice their opinions. In particular, do AJG and CSH agree with PK's reasoning that led him to reverse 124 instances where they agreed? Given this feedback, we will determine how to proceed.

Very thorough analysis by Pouya. I agree with the slight change in the definitions of categories. One clarification: is a drug still considered disease modifying if there are significant and dangerous side effects? amendment 1 seems to indicate that the answer is yes.

For the most part, I agree with how you classified the following. I have made some comments for the particularly challenging disease categories:

• hypertension, general
• hypertension, diuretics
• hypertension, ocular and pulmonary htn
• diabetes, drugs that lower blood sugar
• diabetes, ACEi and ARBs
• epilepsy: I think I agree with categorizing all AEDs as DM instead of SYM. I thought the pathophysiology behind MTS was not clearcut and treating someone with AEDs does not necessarily prevent MTS, although I think it may prevent morbidity and mortality.
• OA, NSAIDs and steroids
• cancers, pain meds
• hematologic cancers, steroids
• non-heme cancers, steroids
• cancers, hydroxyura and other chemotx
• cancers, bisphos
• CAD, drugs for htn or dm
• CAD, diuretics
• migraine: I think prophylactic medications should be DM; abortives may be better classified as SYM
• asthma, steroids, beta gonists, antichol
• allergic rhinitis: this categorization makes sense to me.
• COPD, general
• etoh dependence
• psych disease
• AD
• anemia: I agree. It is like trying to categorize "leukocytosis" or some other lab abnormality, without getting at the etiology.

I am still stuck on one disease entity:

• Autoimmune diseases, steroids: do steroids actually change the long-term disease of autoimmune diseases? Do they reduce morbidity and mortality?

• Re: Autoimmune diseases and steroids. I do believe that steroids are DM in a variety of auto-immune diseases. The easiest examples are Lupus and RA where the occasional difficult-to-control patient is given low-dose maintenance steroids to prevent disease flares (i.e. reduce morbidity). Tougher examples include MS but there is some evidence that it may help decrease disease activity and relapse rate [1], which puts it is a similar category to say Copaxone which we would probably mark as DM (even though copaxone also does not delay progression of disease). Given that, I felt it was reasonable to assume the same for other auto-immune diseases, although I must admit I'm probably under-qualified to comment on the subtleties of this (perhaps we can curb-side a rheumatologist). I fully agree that there is a high side-effect burden and chronic steroids are probably not clinically indicated for treatment, but perhaps they should still DM for the purposes of this study. I think of it like this, if Daniel's analysis could suggest a drug because it acted on some of the same molecular targets as do steroids, but that drug had zero side effects, then would that drug be of interest in treating auto-immune diseases? If yes, then I think we should classify steroids as DM.

• Re: epilepsy. Good point that AEDs prevent morbidity and mortality, it's clearly better than my hand-waving and highly disputed MTS argument. Additionally, I think we should consider the same argument as above, if Daniel's network were to find a drug that acts on the same molecular targets as our AEDs, then would we consider that drug when treating epilepsy? I would think yes.

• Re: migraine. I wholly agree with what you wrote. I think I just didn't state it as clearly as you did.

Time is short to finalize our indication catalog by consensus.

@pouyakhankhanian, are any changes needed to your original classifications to reach a consensus with @chrissyhessler regarding epilepsy and migraine indications?

@pouyakhankhanian, how many indications are subject to what we decide for autoimmune diseases and steroids? I agree this is a tough call. In practice, steroids are not referred to as disease modifying for multiple sclerosis. However, I'm not convinced there is a logic to this omission that could be consistently applied to other diseases. Thoughts?

We do want our catalog to be broadly applicable to projects beyond our specific study. In other words, we want the catalog to be generally useful to train and test computational approaches without too many disputable calls.

One clarification: is a drug still considered disease modifying if there are significant and dangerous side effects?

My take here is yes as long as the drug has been indicated in a disease-modifying capacity in some context. The context may be the time before the dangerous side effects were fully appreciated or the time before better tolerated therapies came to market.

• Re: migraines and epilepsy. No changes need to be made to the spreadsheet, the proposed classification presented by CSH matches what I chose.

• Re: steroids and auto-immune diseases. There are about 70 steroid-autoimmune connections that would need to be subject to this decision. I think there are about 20 which are Rheumatoid Arthritis and Lupus, which I think are safely DM. That leaves another 50 which would have to be re-evaluated based on the decision. For the case of multiple sclerosis, I still tend to favor calling steroids DM for a few reasons. First, let's consider an easy (but rare) example. Suppose a patient comes in with painful trigeminal neuralgia due to an active demyelinating lesion. One would give this patient steroids. The steroids would decrease active inflammation and demyelination, thereby decreasing the duration of time of the symptom of pain. The steroid does not actually treat the pain directly (like a "pain-killer"), but it treats the biology behind the pain. While it is true that the steroid does not slow the progression from RRMS to SPMS, nor does it prevent future attacks (hence it is not called "DM" in the clinic), it does affect the biology of the current attack. Next, let's consdier a more complex (but more common) example. Suppose a patient comes in with leg weakness making her unable to walk, due to an active demyelinating lesion. One would give this patient steroids. The steroids would decrease active inflammation and demyelination, thereby decreasing the duration of time of the symptom of weakness. The steroid does not directly treat weakness (like a drug like Ampyra might do). Again, while it is true that the steroid does not slow the progression from RRMS to SPMS, nor does it prevent future attacks (hence it is not called "DM" in the clinic), it does affect the biology of the current attack. Finally, let's consider this article. In that article, they give monthly steroids and in order to prevent future attacks. They find that the number of future attacks is decreased (though it is likely not a large enough effect to justify the use of chronic steroids long-term given all the side effects that go along with chronic steroid use). Decreasing the number of attacks is exactly what defines nearly all of the drugs typically which are "in practice... referred to as disease modifying" (most of these drugs to not prevent progression from RRMS to SPMS). For the three reasons above, I would still tend to favor calling steroids DM in MS. If we choose otherwise, then I think we should use MS as an example to set up a precise definition of what qualifies as disease modifying in auto-immune diseases, and then re-evaluate the other 50 steroid-autoimmune indications based on that definition.

Announcing the consensus curation

In the interest of time, we are finalizing the consensus curation now. We have chosen the PK curation as the consensus. Discussion is still welcome and will be helpful for future incarnations of our catalog.

Resolving steroids for autoimmune disease

Both original curators were given a change to respond to the PK curation and methodology. In CSH's response and offline discussion with AJG, questions were raised regarding calling steroids DM for autoimmune diseases. Further discussion with PK, both above and offline, helped clarify the issue and convinced @sergiobaranzini and I that the DM classification was appropriate.

According to PK, steroids are not considered DM in the clinic because their poor risk–benefit ratios generally preclude longterm use. Clinicians interpret "disease modifying" to mean a therapy for changing the longterm disease course and therefore do not consider steroids, which are usually given for only a short period of time, disease modifying. However, our definition of DM does not require longterm modification. Nevertheless, PK points to some evidence [1, 2] that steroids do modify the longterm disease course when administered over a prolonged period.

While a clinician's decision to prescribe a steroid for an autoimmune disease is motivated by reducing symptoms, PK believes the steroid reduces symptoms by modifying the underlying disease biology. In his opinion, steroids lead to a short-term suppression of the underlying biology — in the case of autoimmune disease, the overactive immune response — leading to a short-term improvement in symptoms. One litmus test is that while a steroid may be prescribed to treat a specific symptom of a multiple sclerosis relapse, the steroid would not treat the symptom in the absence of MS.

In conclusion, we are conformable with the decision that steroids modify autoimmune disease rather than treat their symptoms according to our definition. However, it's important to clarify that our indication catalog is designed primarily from a perspective of pathophysiology rather than clinical best practice.

• I think this is a valuable data source to maintain. I understand the need to freeze it at the moment for your analysis. Going forward, I'd love to see a couple additional data sources. One would be uptodate, which would add to the total number of indications (including those where side effects outweight potential benefit) and would give you precise disease-indications for drugs (no need for curation by experts). When thinking about a second data source which may help add to your total number of indications, I might suggest a more clinically relevant source (like medscape). I think your current data-sources are heavy on government approval and, per our discussion earlier, for often politico-economic reasons, drugs may be very commonly used but not had any pharma funding for official approval, and these drugs may not all be caught when surveying pharmacy or doctor's "indication notes" as those may lack sensitivity (due to under-reporting of key "disease" designations).
• I would keep an eye out for the steroids in auto-immune diseases Steroid represent a relatively large fraction of the drugs, and auto-immune diseases a reasonable fraction of the total diseases (approximately 5%). I would interpret any results that your algorithm suggests in light of this. For example, I expect this will drive your algorithm into picking things that "look" like steroids (in terms of molecular structure, and known targets of possible action). As you know, steroids are molecularly quite similar to each other, and are often associated with the same limited number of key molecular targets. The other immunosuppressive agents (i.e. all the other drugs on your typical clinical list of choices) represent a variety of shapes (molecular structure) and known targets, and may provider richer (but more subtle, and probably lower powered to get a trustworthy result) information, and hopefully provide a more nuanced drug suggestion rather than picking things that "look" like steroids (e.g. suggesting a drug that nobody would ever have considered) .
• Consider assigning mechanisms to drugs. If you note that something richer is to be gained by "decreasing the gain" in large drug classes (i.e. the class "steroids" includes about 10 drugs in the list), consider using drug classes as an attribute. This will also aid any person who will have to curate the disease-drug connections.
• The few remaining discrepancies, if using my calls as final calls. After the discussion of the major discrepancies (where multiple discrepant drug-disease connections hinged on a single discussion), there are still minor discrepancies. There appear to be 55 other discrepancies to eventually be evaluated, totaling less than 5 percent of the total number of connections. Given the small number of total calls (less than 5% of total calls) and the large amount of discussion that would be required to solve each one, it makes sense to go forward with a data freeze for your downstream analysis. But I think it would be great to have you at least aware of these, and we can decide what to do on future versions.
  Of the 55 cases of discrepant calls, there are include 31 cases where my curation changed a previous agreement between the prior 2 curators, and those 24 cases where my curation resulted in a three way tie.
  — 7 are explained directly by ammendment 1
  — 14 where I made a call which was explained by a prior call regarding the same disease being connected to another drug within the same drug class
  — 11 can be encapsulated in a discussion regarding hormone therapy in breast cancer. This is a more complex discussion than that of steroids, because the hormone therapies include "partial agonists".
  — 3 which treated symptom of chemo rather than symptom of disease (therefore changed to NOT)
  — 3 regarding SSRIs in parkinson's
  — 17 discrepancies would each require a long discussion (similar to our discussions of diseases earlier). These are briefly denoted below.