Brainstorming future directions for Hetionet

Last Friday, in an applied imagination meeting for the @caseygreene lab, we brainstormed future directions for Hetionet. The timing worked out well as we had just released our initial report [1, 2].

The following modifications to Hetionet were brought up:

• Adding biologic drugs, e.g. antibodies, since Hetionet v1.0 only includes approved small molecules. In previous private communications, @sergiobaranzini has also strongly supported adding biologics as well as a more comprehensive set of small molecules.
• Adding metabolites since the efficacy of many small molecules results from metabolites.
• Updating Hetionet continuously in an automated fashion. Versioning could enable updates, using a scheme similar to API versioning. For example, we could use versioned URLs for our Neo4j Browser, such as v1.het.io.
• Adding microbiome information, such as microbial communities and drug-microbe interactions.
• Create the network by snorkeling the literature. Hetionet v1.0 could be used as the "ground truth" to extract labeling functions. We alluded to data programming techniques in the report, writing "Going forward, advances in automated mining of the scientific literature could enable extraction of precise relationship types at omics scale [3]."
• Adding SNPs, potentially using ExAC [4] for allele frequencies.
• Adding a discovery date property to relationships. Year of earliest publication could provide a proxy for some relationship types with source publication metadata. This could allow us to see if we can predict future knowledge from past knowledge and help address knowledge biases.
• Incorporating universal edge weights representing confidence in a relationship. In the past, @larsjuhljensen has also strongly advocated against our binarization of uncertain relationships.
• Supporting the addition of user-supplied relationships
• Adopting more ontologies. Perform propagation during query-time. See Hetontology which aims to host all openly-licensed OBOFoundry ontologies in a public Neo4j instance.

The following modifications to Project Rephetio were brought up:

• Quantifying the significance of a predication. Showing enrichment over null rather than probability of treatment to provide more context.
• Algorithms that account for relationship weight/confidence when measuring path prevalence for a given metapath between two nodes.
• Focusing on drugs and diseases as perturbations to biological systems/pathways.
• Better capturing tissue specific effects. Accounting for which tissues/anatomies drug efficacy occurs in for a specific compound and disease. How do drugs work in a specific tissue?
• Queries that leverage metapatterns, rather than just metapaths. Metapatterns encompass queries that search for more than just a simple path. Metapatterns could help create tissue-specific aware features and drill down on the mechanisms of drug efficacy. Currently, we identify when a drug will work, but not necessarily why a drug will work (mechanisms).

The following new applications of Hetionet were discussed:

• Predicting drug-drug interactions.
• Allowing users to provide their own gene sets. Enabling some sort of gene set enrichment queries. For example, what drugs interact with a given gene set?
• When multiple drugs are available for a disease, what are the differences and which drug should be used for a given patient?
• Clustering nodes, such as extracting gene modules.

Thanks to everyone who participated: @caseygreene, Jaclyn Taroni, Jie Tan 💻, @gregway, Brett Beaulieu-Jones, René Zelaya, Matt Huyck 💻, Dongbo Hu, Amy Campbell, Kathy Chen, Timothy Chang, Linda Zhou, and Stephen Woloszynek (💻 denotes virtual participation). There're clearly more ideas than time, which is why these discussions are important — so we pursue only the best!

Building on Adopting more ontologies. Perform propagation during query-time, here is text I had sitting in a draft:

Currently, we don't include ontological edges. For example, there is no myelination→is_a→axon ensheathment relationship despite its presence in the Gene Ontology. We omitted this information because we didn't have a query framework that accommodated its complexity. However, Neo4j supports variable-length pattern matching, which will make realtime transitive closure feasible.

@dhimmel I would really like to find a way to collaborate more directly. We have two PhD students and a postdoc all working on extensions to the hetionet project - thanks entirely to your efforts to be open here on ThinkLab. They are interested in: DeepDive/SNorkel, extending to rare genetic diseases, expanding drug coverage (particularly in relation to the Calibr collection we now will have access to), and generally improving the framework itself.

Apart from their interests, one of the things I would personally like to see is a better exposure of the network you have assembled to the community beyond bioinformaticians. The purposes are both to support hypothesis generation for them and to tap into their collective expertise to crowdsource the curation and extension of the network. One way that might be achieved would be to load it into an incarnation of an application I am building in partnership with Richard Bruskiewich and his team at Star Informatics called knowledge.bio. The first version of knowledge.bio is online now, described in a BioarXiv article and available on in the kb1 bitbucket repo. The next version is due to be released at the end of the year, is a complete rewrite (including a move to neo4j), makes extensive use of wikidata, supports limited social features, and is available in the kb2 repo.

Everything above is at the 'strong starting point' phase. If you or other members of your group are interested in joining forces to take them to fruition, let us know!

We are also working with the Monarch consortium on a large knowledge integration project known as TransMed. This will be leveraging and extending the SciGraph system they built for Monarch. SciGraph exposes OWL ontologies and associated knowledge bases for query in the form of graph databases like neo4j.

@dhimmel thats an excellent list of potential ideas. I'd fully agree with expanding to include (reliable) data types as much as possible. The idea of automated continuous updating is very cool, but seems like a ton of workhours to maintain long term. Drug-drug interactions can easily be tested against a "gold standard" database. I'm guessing it's relatively cheap (in work hours) for you to re-write the code to create drug-drug matches?

The idea of automated continuous updating is very cool, but seems like a ton of workhours to maintain long term.

@pouyakhankhanian, if we move to more automated techniques on more standardized inputs, I thinks it's possible to entirely automate updates. One area of interest in my current lab, the Greene Lab, is automation [1]. As a simple example, see dhimmel/thinklytics which automatically backs up and analyzes Thinklab content on a daily basis. Obviously, building Hetionet is much more complex, which means automation will be more difficult but also more valuable.

Drug-drug interactions can easily be tested against a "gold standard" database. I'm guessing it's relatively cheap (in work hours) for you to re-write the code to create drug-drug matches?

Yeah that wouldn't be too bad. I'd probably rework some of the current analysis pipeline to be more reproducible and automatable. Currently, it's difficult for others to re-execute and extend our computations — see for example, this issue by @tongli.

@dhimmel I would really like to find a way to collaborate more directly. We have two PhD students and a postdoc all working on extensions to the hetionet project - thanks entirely to your efforts to be open here on ThinkLab.

@b_good definitely. First, let me say that it's been super encouraging to see your team extend Hetionet / Rephetio. I'm not entirely sure what extensions you're working on, but the GitHub Issues opened have helped me see the sticking points of my old workflows.

From your comment, I think we're all thinking along the same lines. All biomedical knowledge should be mined from the literature into a hetnet. The hetnet should be stored in Neo4j. Nodes should be from ontologies and standardized vocabularies that can be automatically maintained. Curation should be fully outsourced or, if necessary, crowdsourced. Layers on top of the Neo4j database will likely be needed to provide biologists with user-friendly interfaces for domain-specific functionality.

I think that we're at a point where we can achieve the above objectives. One outstanding issue that I'd love input on is how to deal with context-specific relationships. For example, protein interactions that only exist in a certain tissue [1]. I don't think my conception of hetnets or Neo4j's graph model are well-suited for this task.

But to the point, let's not duplicate efforts. I'm hoping that everyone who shares our vision can join forces into a collaborative open source development community. There's lot's of overlap with the NCATS Biomedical Translator Program, which funded investigators from 11 Universities. @b_good are you affiliated with any of these projects and are they something we could also leverage — i.e. will their development be open source and are they pursuing related or complementary approaches to what we've outlined?

So any suggestions on how best to structure collaboration? I personally think open source GitHub repos that operate on a pull request model are ideal. The difficultly will be coordination, recruitment, and providing an incentive structure that rewards collaboration.

Mining knowledge from the literature

In the early days of Project Rephetio, @b_good suggested mining text to populate our network. We didn't end up doing much text mining to construct Hetionet v1.0. We did indirectly use relationships extracted from the literature, though MEDLINE term co-occurrence and resources that curated literature.

In general, we were more excited about "edges from systematic technologies that are not subject to knowledge biases" and invested our time in integrating resources such as LINCS L1000 and STARGEO. Our hope was to leverage systematic high-throughput technologies to make novel and insightful drug repurposing predictions. However, as we discuss in our report [1]:

Ideally, different data types would provide orthogonal information. However, our model for whether a compound treats a disease focused on 11 metapaths — a small portion of the hundreds of metapaths available. While parsimony aids interpretation, our model did not draw on the weakly-predictive high-throughput data types — which are intriguing for their novelty, scalability, and cost-effectiveness — as much as we had hypothesized. Instead our model selected types of information traditionally considered in pharmacology.

I think increasing our number of positives and fixing edge dropout contamination — for example by training on novel clinical trail indications — could help incorporate weakly predictive metapaths into our models. However, the findings of Project Rephetio suggest that we should redirect focus to incorporating established knowledge embedded in the literature.

@b_good, I saw that knowledge.bio [2] uses the Semantic MEDLINE Database (SemMedDB) [3]. How good is SemMedDB? Unfortunately, SemMedDB appears to have troubling terms & conditions despite being a US government work and primarily factual. I know @andrewsu is in communication with the National Library of Medicine to fix these legal impediments to reuse. In my next post, I'll discuss Snorkel — the question being whether implementing Snorkel will provide sufficient advantages over existing databases, such as SemMedDB.

Snorkeling the literature

Snorkel describes itself as a "a lightweight platform for developing information extraction systems using data programming" (HazyResearch/snorkel on GitHub) [1]. There's a short video describing data programming, although the terminology is a bit foreign to me. Therefore, @caseygreene and I videochatted with the lead developer, Alex Ratner, of the Christopher Ré Lab at Stanford.

Here were the takeaways. We could use snorkel to fill in missing relationships in Hetionet. Each of the 24 relationship types would be a distinct classification problem, where we instruct snorkel to learn how to extract that relationship type from the literature. We would start with a single labeling function, which would return true for relationships in Hetionet and false otherwise (distant supervision). Snorkel would fit a generative model that outputs the existing relationships and more.

This approach is intriguing for several reasons:

• We could fill in missing relationships where Hetionet doesn't currently contain all of the knowledge from the literature.
• Each extracted relationship could refer back to the specific phrases supporting it throughout the literature.
• The resulting hetnets would be free of licensing restrictions and therefore could be released fully as CC0.
• The approach can be largely automated and deployed as new literature enters the corpus.
• The approach scales whereas existing human curation approaches do not.
• The approach generalizes meaning each relationship type doesn't require an entirely new processing pipeline.

Alex mentioned the following three steps that take the most time:

1. data processing and munging — preparing the input data. Hopefully, the burden here could be minimized since Hetionet v1.0 is already standardized and integrated. Furthermore, Alex mentioned that they've indexed certain portions of PubMed, so we wouldn't have to spend lot's of time configuring a literature corpus in the beginning.
2. test set curation — expert curation of specific phrases from the literature corpus as negative or positives to evaluate performance. Potentially, we could withhold relationships and use them for testing, but the relationships are downstream from the actual classification which occurs at the level of actual text. It sounded like uneven prior probabilities of relationship based on node degree, which we've experienced, could be an issue with testing on withheld relationships.
3. error analysis — investigating where errors are being made and adding labeling functions to rectify the errors (direct supervision). This step could be amenable to crowdsourcing by community review.

We discussed doing a pilot project based on a single relationship type in Hetionet v1.0 — perhaps Compound–treats–Disease since we've already done substantial curation to create PharmacotherapyDB.

Regarding the initial preprocessing (e.g. splitting, NER) of PubMed, I wonder if the PubAnnotation project might provide some useful structures and data ?

Hi @dhimmel this is really exciting! Thoughts (some others also in the issues discussion of your repo):

• Is compound functionally too different from chemical? We have tags for chemical and disease in Snorkel format already, for PubMed abstracts via PubTator. If compound is different, we also have / are working on tools for building custom entity taggers for this, but it's just another step that would have to be tackled

• I definitely agree with @b_good's general sentiment of starting with simpler relation types, although don't have a lot of intuition for which are more tractable. Easy things to check off when selecting extraction schema:

  • Make sure that the relation type is well defined / unambiguous enough that you can cleanly annotate some example yourself
  • Test some simple baseline methods (e.g. a regex or two) to make sure it's not too simple / general (if it is, maybe the regex is good enough, and no need for Snorkel; or maybe this is good cue to jump to the more specific, complex relation type you actually care about!)

• @b_good that resource looks really cool, thanks for sharing!

We're trying to wrap up a bunch of dev stuff that's been scattered about over last month or so, hoping to push something to dev and master in next 1-2 weeks, stay tuned!

Thanks,
Alex

I think it would be a great experimental setup to begin your extraction work using the relationships in your current hetnet as your gold standard (standards.. one per relation type). First that means you have a gold standard already which is a huge part of the work in building up an NLP system (and can likely publish good results in NLP-related arenas). Second, if it works, you have an automated way to grow the hetnet on a daily basis and potentially to scrub out the licensed content. Third you set yourself up for a great experiment - can a drug-treats-disease automatic relation finder outperform your results from the hetnet + R inference model? How can those methods be harmonized?

This will be a great project...

@b_good, your comment nails the promise of snorkeling hetnets. I was envisioning using Snorkel for network construction rather than relationship prediction. In other words, I'd want each literature extracted relationship to derive from specific phrases that directly attest to the relationships' existence. However you bring up an interesting point — will Snorkel begin to infer relationships and become a substitute for hetnet edge prediction?

The Compound–treats–Disease Pilot

you might consider starting with a relation type (or 2) that are less semantically complex and more plentiful in the research literature than 'treats'.

Is whether a compound treats a disease really so semantically complex? For example, here is an example abstract sentence which implies a CtD relationship from an abstract [1]:

Acamprosate proved to be a safe and effective aid in treating alcohol-dependent patients and in maintaining the abstinence of patients during 2 years.

I'm hoping we can steer Snorkel away from making inferences say based on the genes involved in alcohol dependence and instead focus on actual statements of clinical efficacy. PharmacotherapyDB has 755 disease-modifying treatments — @ajratner will this be sufficient to support an effective labeling function? If not, we could expand to all clinical trial indications, which would give us ~30,000 gold standard relationships.

@b_good if we struggle at snorkeling treats relationships, we could always pilot another relationship type that was more semantically straightforward and plentiful. Gene–interacts–Gene relationships would be one possibility.

Entity tagging and mapping

Is compound functionally too different from chemical? We have tags for chemical and disease in Snorkel format already, for PubMed abstracts via PubTator.

@alexratner, compound and chemical are the same thing. Looks like PubTator tags
genes, diseases, species, chemicals, and mutations [2, 3]. Of these, Hetionet contains genes, diseases, and chemicals.

• PubTator uses Entrez Gene for genes, as do we. Genes are tagged using GNormPlus [4].

• PubTator uses DNorm to tag diseases [5], which uses MEDIC. MEDIC is the disease vocabulary used by the Comparative Toxicogenomics Database (CTD) [6], which consists of OMIM and MeSH terms.

• Compounds in PubTator are tagged with tmChem [7], which uses MeSH and ChEBI, so we'll have to find a way to map MeSH to Drugbank.

Future Snorkel discussion

I'd like to make sure this brainstorming discussion doesn't explode with Snorkel implementation details. So for any Snorkel comments that will generate lot's of discussion, let's either create new discussions on Thinklab or issues on greenelab/snorkeling. We can also make a new Thinklab project for snorkeling.

@dhimmel Sorry for the delayed response here. I 100% agree with your statement "I was envisioning using Snorkel for network construction rather than relationship prediction. In other words, I'd want each literature extracted relationship to derive from specific phrases that directly attest to the relationships' existence."

In some sense this is an engineering decision: we could train any predictive model, but it seems best in our experience to train one to extract the concrete explicit mentions of relations, and then to do further reasoning / prediction / analysis over these.

In terms of the complexity of the treats relationship is probably higher than you'd expect... thank you natural language... but definitely very doable (actually, we have a chemical-induced disease extraction application using CTD distant supervision which beats state-of-the-art on a BioCreatives task, will clean up as tutorial soon...)

However as to which labeling functions / weak supervision will be good enough... no great easy answers, let's see what happens!

Great to hear that our entity types are aligned.

Also, we just released v0.5.0, so hopefully this helps!