Predicting Bcr-Abl targets using a hybrid information extraction system
Chronic myeloid leukemia (CML) is a disorder of uncontrolled leukocyte proliferation caused by a chromosome translocation resulting in Bcr-Abl, a constitutively active tyrosine kinase [1, 2]. Over a thousand Americans die from CML each year due to resistance to existing therapies like imatinib [3, 4, 5]. Difficulties in selectively inhibiting Bcr-Abl have shifted drug development towards targeting downstream Bcr-Abl interactors, many of which are still unknown . Identifying new Bcr-Abl targets would spur drug development and lead to novel CML treatments.
Informatics techniques utilizing knowledge networks built from the biomedical literature have successfully identified novel p53 kinases and prostate cancer drugs [7, 8]. By exhaustively processing millions of documents, informatics techniques discover insights previously missed by individual scientists. However, existing computer programs cannot extract a significant portion of important information due to limitations like being able to only process individual sentences . Improvements to processing programs would increase the breadth and depth of extractable information [10, 11]. Crowdsourcing, an approach using groups of untrained experts to perform complex tasks, has been applied to predict protein and mRNA structure and mine biomedical text [12, 13, 14]. Machine learning (ML) is an analytical technique that allows programs, or classifiers, to infer patterns from provided data .
I propose to identify cytoplasmic Bcr-Abl targets using an informatics technique combining crowdsourcing and automated text mining (Figure 1). I will develop an information extraction workflow using crowdsourcing and integrate machine learning methods for automated pattern recognition. This will allow a higher number of documents to be processed than what is possible with crowdsourcing alone at a quality higher than what is currently possible with automated programs. Applying the hybrid method to the literature surrounding CML and Bcr-Abl will answer the following two questions: 1) What are the limitations of existing automated information extraction programs? 2) What are previously unidentified targets of Bcr-Abl? Researchers in the natural language processing and drug discovery domains have long sought answers to these questions.
Aim 1: Validate non-expert information extraction ability
The first step to developing a hybrid information extraction technique is to demonstrate that non-experts can read and interpret scientific publications. This step is necessary to ensure that the assertions extracted by the system can be trusted, a common problem in interpreting output of purely automated methods. In preliminary experiments we have determined that the pooled judgments of six non-experts is almost as accurate as domain experts at identifying biological concepts in text (Figure 2a). On a more complex task of determining chemical-induced diseases, we found that using as few as four non-experts already gave accuracies of 65% (Figure 2b), demonstrating that non-experts can interpret publications with little training. I will expand the difficulty and variety of tasks, and continue to evaluate worker performance by comparing performance to gold standards produced by expert biocurators.
Figure 2. Non-experts can interpret biomedical text.
a) Non-expert performance on a disease annotation task steadily improves as the number of workers increases, eventually reaching that of a domain expert. b) On a more complex chemical-disease relation identification task, non-experts found 56% of relations picked by experts, but 75% of the predictions were correct, leading to a overall performance of 65%.
Aim 2: Integrate machine learning with a crowdsourcing workflow
Although preliminary studies have shown that ML techniques can be used to resolve judgment disagreements better than majority voting (Figure 3a), how the approaches can be used to complement one another is still mostly unknown, as the two approaches have historically been used independently . To address this question, I will perform experiments attempting to fuse the two techniques together in novel ways in order to utilize the strengths of each technique.
For this aim, I will identify areas where predictive classifiers would provide an advantage over currently used methods. Our preliminary experiments show that task allotment and judgment aggregation would benefit from enhanced modelling. I will apply standard classifiers trained on crowd gathered data and compare whether performance or scalability increased. As a control, I will perform the same tasks with no classifiers. I expect that classifier application will enhance performance and increase the number of documents which can be processed, but that these improvements will be partially dependent upon the consistency of the human judgments.
Aim 3: Identify downstream protein targets of Bcr-Abl
I will answer the following question: which cytoplasmic proteins in the literature are likely downstream targets of Bcr-Abl? To address this question, I will apply the hybrid information extraction system to documents related to Bcr-Abl to produce a knowledge network centered around the protein.
I will then overlay known Bcr-Abl targets and known non-targets with the extracted information graph, and use concept similarities and term connectivities to rank potential proteins based on their resemblance to known Bcr-Abl targets. For validation, I will ask a group of blinded clinicians to review the predictions based on the original supporting literature documents and judge for prediction validity. Real and fake targets will be introduced as control items to verify expert consistency. Since a similar network analysis was used to identify prostate cancer drugs (Figure 3b), I expect that the method will find at least one novel Bcr-Abl target, and that other proteins like extracellular proteins will rank near the bottom of the predictions .
Pitfalls and Alternatives
One potential complication is that processing the large set of CML documents using paid online crowdsourcing platforms will be prohibitively expensive. If this were to occur, we would instead re-adapt our crowdsourcing model to instead harness volunteers, large communities of which already exist . In addition, the concept similarity method for finding new Bcr-Abl targets may not work for sparsely connected networks, in which case we would use an alternative concept category knowledge-based method instead .
Significance and Innovation
This proposal represents one of the first studies aiming to solve biological problems using a representation of the literature generated through crowdsourcing. The technique is novel in that it can be adapted to other domain specific problems, and can therefore be reused on different parts of the literature. Exposure of the general public to scientific literature through crowdsourcing also improves scientific literacy and outreach.
Everyone's path to bioinformatics is different, and mine is no exception. During my early schooling I spent much of my time exploring the physical sciences broadly, trying to find the area where I felt the greatest interest. Although I found physics to be powerful and chemistry to be precise, I discovered that biology fascinated me the most because it has a direct and lasting influence on the people around me. Thanks to two excellent high school teachers who instilled in me an appreciation for both computer science and biology, I proceeded down the path which would eventually lead me to computational biology.
My journey towards bioinformatics began in earnest during high school. I was a member of our school's Programming Enrichment Group, an extracurricular activity focused on problem solving and critical thinking through algorithmic design. We participated in many programming competitions, and I along with other members of our club competed for a chance to represent Canada at the International Olympiad in Informatics at the Canadian Computing Competition. I relished in the logical rigor of informatics, and sought to apply it to other domains.
As an undergraduate I worked in Virginia Cornish's lab at Columbia University on a directed evolution project seeking to evolve protein binders. The project was motivated by a desire to develop protein therapeutics for targets difficult to target specifically with small molecules, such as Bcr-Abl. Under the guidance of my graduate mentor I helped to characterize variants generated by the in vivo protein recombination system. This was my first experience working in a laboratory, and I found science to be a demanding but ultimately fulfilling endeavour. I returned to the Cornish lab the following summer to continue my work, but unfortunately had to leave prior to the project's completion due to a year spent studying abroad at Oxford. The most valuable experience from my time in the lab, other than learning to balance research with my other academic commitments, was the exposure to experimental techniques requiring large scale data interpretation and analysis.
Upon starting my Ph.D. at Scripps after graduation, I quickly sought out Andrew Su's lab so that I could fulfill my interest in computation and biology. In particular the application of computational approaches to generate hypotheses from the existing literature fascinated me because I saw new opportunities to solve old problems like finding Bcr-Abl targets. In order to generate novel hypotheses, it is first necessary to create a computational representation of the literature. I realized that existing methods for generating these representations suffered either from a lack of quality or scalability, and sought to solve both problems by merging human judgments with automated machine learning. While working on this project I have had the opportunity to hone my communication skills through conference presentations, and managed to win a presentation prize.
Following completion of my Ph.D. I hope to broaden my research background by working as a postdoctoral researcher in a laboratory which is both able to make computationally guided hypotheses and able to verify them biologically. My eventual goal is to establish a research group of my own which is able to utilize and verify the predictions of rapidly advancing computational tools in order to produce treatments and results for patients directly. Support from HHMI at this early stage in my career would provide me with the freedom to guide and develop my own research path without having to worry about financial burdens. Thank you for considering my application.
I had similar thoughts (written before reading Dr. Good's comment):
I am not a biologist. I have no idea what Bcr-Abl is. I am guessing it is a protein because of "constitutively active tyrosine kinase" but I don't fully grasp what is going on with the chromosome translocation. Without the translocation, are humans devoid of Bcr-Abl? Also searching for "Bcr-Abl" hasn't been helpful because "BCR-ABL" is a gene name. Is all CML caused by Bcr-Abl? Is all Bcr-Abl bad?
I have a hard time understanding the "Knowledge Network" from Figure 1. Drawing node borders in addition to edge arrows may help. My current understanding of your network at this point in reading is that you are using text mining + crowdsourcing to construct a metabolite network.
The paragraph up till the end of this sentence has primed me: you've led me to believe that NLP programs need to span multiple sentences. The rest of the paragraph is a complete non-sequitur. I finish the paragraph confused, having just read a list of definition with no cohesive connection.
My comment on this phrase:
I feel that this is a precondition for funding you. If in general it is unknown whether non-experts can read science publications, I don't see the point of exploring such a specific application of crowdsourcing.
Now I agree that you should evaluate whether non-experts succeed on your specific problem, but you should have examples of where non-experts read science successfully. Here is one example that non-experts understood drug labels .
Update: your next sentences answer the point, so you should assert that non-experts can from the start.
I think you should be specific here in the sentence about what type of biological concept. Say that you have shown that groups of non-experts can identify biological concepts such as diseases, genes, etc with high accuracy. If you just say "biological concepts" a reader might imagine something more complex and get riled up by this statement.
You need stronger evidence than the one reference regarding prostate cancer to get here. As it stands, I would not be convinced. Further, since that work did not use crowdsourcing, why do you need to ? If you just did exactly what they did, but enhanced their workflow through crowdsourcing, could you identify better drugs than they did? WHy do you think so? How similar is that to what you are planning to do here?
This is an awkward tautology. If everyone's path is different then by saying your is no exception you mean that yours is also different? What if your path wasn't the exception then it would be the same? Impossible. I suggest you drop this first sentence and just begin the story of your unique journey to comp biol.