Statistical Tests for Associations between Two Directed Acyclic Graphs

Abstract

Biological data, and particularly annotation data, are increasingly being represented in directed acyclic graphs (DAGs). However, while relevant biological information is implicit in the links between multiple domains, annotations from these different domains are usually represented in distinct, unconnected DAGs, making links between the domains represented difficult to determine. We develop a novel family of general statistical tests for the discovery of strong associations between two directed acyclic graphs. Our method takes the topology of the input graphs and the specificity and relevance of associations between nodes into consideration. We apply our method to the extraction of associations between biomedical ontologies in an extensive use-case. Through a manual and an automatic evaluation, we show that our tests discover biologically relevant relations. The suite of statistical tests we develop for this purpose is implemented and freely available for download.

About the Authors:

Robert Hoehndorf

* E-mail: [email protected]

Affiliations Research Group Ontologies in Medicine, Institute for Medical Informatics, Statistics and Epidemiology, University of Leipzig, Leipzig, Germany, Department of Computer Science, University of Leipzig, Leipzig, Germany, Department of Evolutionary Genetics, Max Planck Institute for Evolutionary Anthropology, Leipzig, Germany, European Bioinformatics Institute, Hinxton, Cambridge, United Kingdom

Axel-Cyrille Ngonga Ngomo

Affiliation: Department of Computer Science, University of Leipzig, Leipzig, Germany

Michael Dannemann

Affiliation: Department of Evolutionary Genetics, Max Planck Institute for Evolutionary Anthropology, Leipzig, Germany

Janet Kelso

Affiliation: Department of Evolutionary Genetics, Max Planck Institute for Evolutionary Anthropology, Leipzig, Germany

Introduction

An increasing number of discoveries, particularly in biomedicine, are facilitated by statistical analyses of data annotated to biomedical ontologies [1]. Biomedical ontologies are generally represented as DAGs, and specific domains are usually represented in distinct, separate DAGs [2]–[4].

Statistical tests that utilize a single graph can only consider the given domain. However, entities from different domain are linked via biomedical relations [5]. These relations can be vital for the discovery of novel biomedical knowledge. We have designed a family of novel statistical tests to identify strong associations between nodes from two directed acyclic graphs. The tests combine measures of relevance and specificity.

We evaluated our statistical method through an extensive use-case in which we applied our tests to the detection of strong semantic associations between the Gene Ontology [3] and the Celltype Ontology [6] based on co-occurrence in scientific literature. In this use-case, we annotated the ontologies with occurrence and co-occurrence count data of the ontologies category labels in full text scientific articles. The strongest associations identified through our tests are biologically relevant relations.

An implementation of the six novel statistical tests to identify associations between directed acyclic graphs is available as free software from our project webpage at http://bioonto.de/pmwiki.php/Main/ExtractingBiologicalRelations.

State of the art

Our approach to the computation of the strength of the association between two graphs relies on approaches for capturing the semantic similarity between categories in ontologies and for propagating these similarities within DAGs. In the following, we give a brief overview of methods for computing the similarity of categories (a more complete overview can be found in [7]). Most of the existing semantic similarity approaches assume that ontologies contain categories that are annotated with terms . Based on this assumption, the computation of the semantic similarity of two categories and can be carried out by using the structure of the ontology to which and belong (edge-based approaches), the nodes and their properties (e.g., similarity between and ) (node-based approaches) or by combining structural knowledge and annotations (hybrid approaches).

The most common edge-based approach consist of using a function of the number of edges between and as semantic similarity measure [8], [9]. Other approaches combine the previous approach with the lenght of the path from the most specific common ancestor of and and the root node [10], [11]. Edge-based approaches rely on the nodes being elements of the same graph. Thus, they cannot be utilized when trying to compute the similarity of two nodes from distinct DAGs.

The second category of approaches, the node-based approaches, use the properties of the nodes themselves to compute their similarity. One of the central concept for using annotations to compute similarity is that of information content, which is the negative log-likehood of a term where is the probability of occurrence of the terms in in a certain corpus. Based on this value, several similarity metrics have been developed including the information content of the most informative common ancestor used in [12], [13] or of the disjoint common ancestors [14].

In recent years, hybrid similarity measures that combine node- and edge-based approaches have been developed. Most of these approaches utilize the information content. For example [15] utilize a combination of edge weights based on node depth and node link density and of the difference of information content of the nodes linked by that edge. Other approaches such as that described in [16] compute edge weights by using a scheme that takes the type of the edge into consideration. The semantic similarity between two terms is set to a function of the maximum of the product of best path between the terms. Again, these approaches can only compute the similarity of terms from the same DAG.

The aim of our approach is to provide a means for the computation of the association between nodes from 2 DAGs, which are, in general, distinct. We do not make similar assumptions about the annotation of edges and nodes as other approaches to semantic similarity. Instead, we go beyong current semantic similarity measures by providing a measure of statistical significance in a distribution of arbitrary node and edge annotations. When applying out method to semantic similarity between ontologies, we can compute initial semantic similarity values for categories which do not belong to the same ontologies.

Methods

Statistics on graphs

Preliminaries of directed acyclic graphs.

Our tests take as input two directed acyclic graphs, and that are disjoint (). From these two graphs, a graph with is constructed. We denote an edge as an ordered pair of vertices. If an edge connects and , , we call the child of and the parent of . If there is a path from to , we call a predecessor of and a successor of .

In addition to the two graphs, two functions and are given as input such that and . From these two functions, a graph decoration for is constructed based on the assumption that the two input functions are transitive over the DAG: the decoration of a vertex is the union of and the values of for all successors of . Similarly, the decoration of an edge for is the union of and the values of for all edges between the successors of and .

The third component of the input is a score function . We assume that the value of the score function between the vertices and depends only on the graph decorations of and of as well as the decoration of the edge .

The score function is not symmetric, i.e., it is not necessary that . It is intended to measure the association strength between two vertices from the input graphs. Our method identifies whether the score between two vertices is significantly high. A graphical overview of our test method is shown in Figure 1.

[Figure omitted. See PDF.]

Figure 1. Schematic representation of our method.

https://doi.org/10.1371/journal.pone.0010996.g001

Determining the Random Distribution.

The score between two vertices and is influenced by the topology of the input DAGs: a vertex that is more general has a larger decoration set due to our basic assumption about transitivity of input graph decorations. Similarily, the cardinality of the decoration set of the edges between nodes from the two input DAGs is larger when the edges connect more general vertices. Therefore, it is insufficient to test for a high score between vertices to consider the score between two vertices as significantly high. A random distribution of the scores of each pair of vertices and provides a means for determining the significance of the score between and . This random distribution depends on the functions and , the score function and the topology of the input graphs. Hence, we cannot assume any statistical distribution of scores ab initio. Instead, we simulate the random distribution of the scores between each vertex pair through multiple random permutations: the -values that are given as input for our method are randomly swapped with the -values of vertices in the input DAG from which they originate. There are two options for permutating the -values for edges: either they are, mutatis mutandis, permutated similarily to the -values of the vertices, or they are permutated depending on the permutation of -values; in the latter case, when the -values of and are swapped, so are the values of and for any vertex .

Because our test is intended to identify associations between vertices, we do not assume that the values of and are independent. We therefore prefer to use the second option, i.e., that the permutation of the values depends on the permutation of the -values.

Based on these permutations, we first rebuild the graph decorations and . Then, we calculate and record the values of the score function for all pairs of vertices and . In addition, for each vertex , such that is a direct successor of , we calculate and record the score difference . Further, for each vertex with the direct predecessor , we calculate and record the difference .

Hence, the results of this step are threefold. First, we approximate the random score distribution for each pair of vertices through multiple random permutations. Second, each triple of vertices , and gives rise to a random distribution of score differences between and . Third, each triple , and yields a random distribution of score differences between and .

Ontologies as graphs

While the tests we develop can be applied to any DAG that satisfies the conditions specified above, their primary application is to test the significance of an association between categories from two ontologies. An ontology is the specification of a conceptualization of a domain [17], [18]. Many biological ontologies are represented as directed acyclic graphs (DAGs) and are available in the OBO flatfile format [2]. In these DAGs, nodes represent categories and edges represent relations between these categories. A category, also called kind, class or universal, is an entity that is general in reality. Examples are dog, apoptosis or red. Categories may have instances, of which some may not be further instantiated. These are called individuals. We call the set of all categories in an ontology .

Categories may be related to other categories. The most important relation between two categories and is the relation, . The relation can be defined by using the instantiation relation: when , then all instances of are instances of [18]. This definition implies that the relation is reflexive, transitive and antisymmetric.

A set of categories with the relation among them form a taxonomy. These taxonomies are often the backbone of the OBO ontologies' DAG structure. We call the set of all successors of a category the sub-categories and its predecessors the super-categories . The direct successors of in the taxonomy are called children (), while the direct predecessors are called parents.

In the OBO flatfile format, ontologies are assigned a namespace. Category identifiers are prefixed with the namespace of the ontology to which they belong. Identifiers are therefore unique within the OBO ontologies. In addition to a unique identifier, categories are assigned a name and a set of synonyms. Neither the name nor the set of synonyms must be unique.

Results

Statistics on graphs

To identify strong associations, we designed a family of tests for the score of each edge between the two input DAGs that considers a fragment of the path in the DAG. The tests are designed to measure the significance of the score between vertices and based on three criteria: (1) the score for the association should be higher than expected; (2) for each child of , should be higher than expected; and (3) for each parent of , should be lower than expected.

The first criterion of our tests identifies hypothetical associations between nodes from two graphs. The second and third criteria are used to verify whether the pair is the best selection, or whether a more specific or more general association is preferable. For this purpose, the second and third criteria test for novelty of the association (compared to the child and parent nodes).

Within this section, let and be fixed vertices from the DAGs and , respectively. Furthermore, let be the number of permutations that were used to determine the random distributions. The first test we designed, , depends on the vertices and , the DAG structure and the number of permutations . It tests for the following properties:

* the score between and is high,

* the difference between and for every child of is high,

* the difference between and for every parent of is low.

“Being high” and “being low” are captured using the values of the cumulative distribution functions (CDFs) obtained by the permutations performed in the previous step: one function for each pair of categories and , one function for each triple of categories , and where is a child of , and one for each triple , and where is a parent of . We combine the -values of the score differences to children in a single value using their geometric mean. A similar combination of the score differences' -values to the parent categories of is carried out: here, the combined value is the geometric mean of , where is the -value in the corresponding CDF.

Formally, let and be fixed vertices from the directed acyclic graphs and , respectively, and let

* be the number of permutations,

* be the score between and in the permutation,

* , , be the cumulative distribution function (CDF) of .

* , , be the CDF of the difference between the vertex and its child vertex,

* ,

* , , be the CDF of the score difference between the vertex and its parent vertex,

* ,

* , for all and , be the CDF of the variances of the distribution , and and for the distributions and , respectively.

For each child of , we calculate the difference in scores . Then, we compute the geometric mean of all values . Similarly, we calculate for each parent of , and the geometric mean of all values . Then we define as our first test(1)

All other tests are extensions of the first test. The second test, , uses the minimum function instead of the geometric mean to combine the -values in the CDFs of the score differences to parents and children.

The first two tests and do not consider the variances of the distributions of scores, differences in scores to children and differences in scores to parents. Therefore, we extend these tests by weighting all three components of the tests with the variances of their corresponding distributions. In these tests, high variance lowers the impact of the result, while lower variance strengthens it.

We define three new distributions for the variances and choose the -value in the respective CDF as a weight in our tests. We compute the scores for each pair of category times, resulting in one distribution of scores for each pair of categories. Each of these distributions has a variance. The score variance distribution is the finite distribution (containing elements) of the variances of each of these distributions. We define the variance distribution for score difference to parent and child analogously.

The tests and use only the variance distribution of scores, while and use all three variance distributions. These tests are one-sided, i.e., they are not symmetric. We define two-sided, symmetric tests for all vertices and as(2)Table 1 lists the combination of properties for all tests. The precise formulation of all six tests can be found in the supplement S1.

[Figure omitted. See PDF.]

Table 1. Elements of the test score of .

https://doi.org/10.1371/journal.pone.0010996.t001

Application to biomedical ontologies

Occurrence and co-occurrence count data as graph decoration.

To verify whether the tests we designed yield reasonable results, we applied our method to the detection of significant co-occurrences between ontological categories in natural language texts, as a precursor to the detection of relations between ontological categories. For this purpose, we make the following assumptions:

1. A term occurs in a portion of text if it is an exact substring of this portion of text.

2. Terms can designate ontological categories; the terms that designate the same category are henceforth called the category's synset. Every occurrence of an element of the category 's synset is called an occurrence of . Every co-occurrence of an element of the category 's synset with an element of the category 's synset is called a co-occurrence of and .

3. If is a sub-category of , then every co-occurrence of with is a co-occurrence of with . Additionally, every occurrence of counts as an occurrence of .

To test our method, we used the Gene Ontology (GO) [3] and the Celltype Ontology (CL) [6] as input DAGs. The GO is an ontology specifically designed to describe gene products. It contains three separate ontologies: the biological process, molecular function and cellular component ontologies. Gene products can be tagged with ontology categories to describe and classify them. The CL is an ontology for types of cells. It classifies cells based on criteria such as structure or function.

Based on the input requirements of our test, we constructed synsets from the synonyms attached to each category in the input ontologies, and counted the occurrences and co-occurrences of the categories based on two contexts: single sentences and sentences in documents. The second context refers to whole documents, but co-occurrence is based on single sentences. Therefore, when two terms co-occur in two or more sentences within one document, their co-occurrence is only counted once. The functions that assign the occurrence and co-occurrence count values to a synset of a category for each context are called and , respectively.

We used exact string matching to identify terms in text. Our evaluation was conducted using a 2.2 GB text corpus containing 60143 fulltext articles from Open Access journals listed in Pubmed Central. The aim of our method is to test for significant co-occurrences between categories.

Text Processing.

First, we counted the number of occurrences and co-occurrences of the terms contained in synsets of categories from the input ontologies. Table 2 shows examples for the synsets of categories. We counted the total number of sentences and documents in which at least one element of a synset was found by using exact matching. For each pair of categories, we counted the total number of co-occurrences of elements of their respective synsets in sentences. Furthermore, we counted the number of documents in which they co-occured within at least one sentence. We used exact matching and abstained from using any more sophisticated methods for recognizing the ontologies' categories in text [19], [20] to evaluate our method. Exact matching provides a large dataset for the evaluation of our method. For practical applications such as relationship extraction, more advanced methods should be chosen.

[Figure omitted. See PDF.]

Table 2. Example synsets taken from the GO and the CL.

https://doi.org/10.1371/journal.pone.0010996.t002

The text processing yielded, for each category , both its frequency and the total number of documents in which occurred, . Furthermore, for each pair of categories and , we obtained both the total number of co-occurrences in sentences and the total number of documents containing these co-occurrences .

Count data over ontologies.

The first component in our method implements the assumption that the input graph decorations are transitive over the DAG structure. In the case of ontologies, this implements the assumption that occurrence and co-occurrence between categories is transitive over the relation between categories.

We assumed that when two categories and stand in the relation, , then every occurrence of is also an occurrence of . This means that the synset-closure of a category can be constructed as follows:(3)(4)For count data, the decoration value of a vertex in the DAG is equal to the sum of the input value pair and and the corresponding input values for 's successors. Therefore, for all categories , we define and to represent the sum of the values and over all of 's sub-categories . Furthermore, for all categories and , we compute the cumulated - and -values dubbed and :(5)(6)Again, for count data, co-occurrence values between nodes and can be summed up over the successors of and to yield the decoration of the edge between and .

A score for occurrences and co-occurrences.

For all categories and , we defined the following score function:(7)The first component of the score function implements the natural logarithm of the Pointwise Mutual Information (PMI) [21] score achieved by the categories with respect to their co-occurrence within sentences. PMI has been successfully used in several text mining tools (see, e.g., [22]). To avoid divisions by 0, the denominators of all members of the score function were incremented. The second component measures a similar value using documents as context. The aim of the score function is to ensure that categories that co-occur relatively often are assigned a high score. The range of the score function is between and .

Discussion

Evaluation

We applied the tests to the biological process (BP) branch of the GO and the CL. To recognize the categories in text, we used the identifier of the category, the name and all exact synonyms of the category. On average, every category had 2.1 synonyms. Using exact matching, we identified 3,751 out of BP's 14,542 (26%) categories in our text corpus. We found 491 of 754 (65%) categories from the CL. Categories from the BP co-occurred 70,967 times with CL categories.

Using our method, we identified a total number of 202,627 co-occurrences between categories. After applying our tests, 157,894 co-occurrences produced test values distinct from . The remainder obtained a test value of due to numerical restrictions. They were subsequently excluded, because they were indistinguishable from the absence of co-occurrence. We illustrate the quantiles obtained for different -values in our six tests, , in Table 3. The distribution of scores for and are shown in Figure 2. The remaining plots are included in the supplement S1.

[Figure omitted. See PDF.]

Figure 2. Distribution of test results.

The plot on the left shows the distribution of the test results for . On the right, the same is shown for . It can be seen that a test using the minimum function () is more restrictive than a test using the geometric mean (). Furthermore, weighting the tests with the CDFs of the variances () produces stronger results than the basic test (). The test results of the GO-CL dataset for each test are displayed below the distributions.

https://doi.org/10.1371/journal.pone.0010996.g002

[Figure omitted. See PDF.]

Table 3. -quantiles for different -values for all tests.

https://doi.org/10.1371/journal.pone.0010996.t003

We found that the tests using the minimum instead of the geometric mean of -values of score differences to parent and child categories are generally more restrictive, i.e., they include fewer co-occurrences for a given cutoff. Similarly, tests including the variance for scores are generally more restrictive than tests that are not weighted by the variance of score distributions. In this sense, the tests and are the most restrictive.

Table 4 shows example associations, and Table 5 shows the kind of relationship between categories that our tests identified for the top-scoring results with respect to the test . The has-participant relation is defined in the OBO Relationship Ontology (RO) [5] as a relation that holds between two categories, where every instance of one category participate in some instance of the other. We define the Participates-in relation as a relation between two categories: Participates-in , where participates-in is the primitive participation relation between individuals as defined in the RO. We extend the definition of located-in in the RO to a relation Located-in between processes and objects, which holds when all participants of a process are located-in a structure during the entire duration of the process.

[Figure omitted. See PDF.]

Table 4. Association examples.

https://doi.org/10.1371/journal.pone.0010996.t004

[Figure omitted. See PDF.]

Table 5. Manually identified ontological relations in the top-scoring association results with respect to .

https://doi.org/10.1371/journal.pone.0010996.t005

In our sample, associations do not fall under one of the three relations that we investigated. We discovered several kinds of unclassified relations. First, mismatches in granularity lead to strong associations for unrelated categories. For example, xanthine transport and erythrocyte are closely related according to . Erythrocytes are involved in the transport of xanthine. However, the GO category xanthine transport refers to the inter- and intracellular level of granularity, while erythrocytes transport nutrients between organs. Second, some categories are indirectly related via another category. For example, osteoclasts and lymph node development are related via the protein RANK. Third, when cells have closely related functions, we sometimes identify too specific or too generic cell types as in the case of the association between basophil degranulation and mast cell. Finally, out of associations in our sample seem erroneous.

We were not able to compute precision or recall for our method due to the absence of a gold standard. However, we compared our method with the GO-CL crossproducts available from the OBO Foundry. The dataset contains manually verified relations between categories from the GO and the CL that have been extracted using pattern matching on category names [23]. As this method is based on the compositional nature of terms in the GO, it exclusively identifies relations in which one category name (usually a type of cell) is a substring of another category name (usually a GO category).

The GO-CL crossproduct contains 396 relations between GO and CL categories. From these 396, we identified 73 that co-occurred in our text corpus. Table 6 shows the percentage of significant co-occurrences within these 73 relations for different cutoffs in our six tests. Figure 2 shows the distribution of the 73 pairs with respect to and .

[Figure omitted. See PDF.]

Table 6. Evaluation of our approach with respect to the GO-CL dataset [23].

https://doi.org/10.1371/journal.pone.0010996.t006

As our method relies exclusively on the distribution of terms and not on their syntactic structure, it permits the recognition of associations between categories that cannot be recognized using syntactic patterns. An example of such an association is myoepithelial cell (cells located in the mammary gland) and milk ejection.

Important potential applications for our tests arise from the fact that annotations of a large set of biomedical ontologies satisfy the conditions for our tests. Annotations satisfy the True Path Rule [3]: if two categories and stand in the is-a or part-of relation, then any annotation of is also an annotation of . Therefore, if gene annotations are used as graph decorations for the two input graphs of our method, the conditions for applying our tests are satisfied. For detecting associations between annotations, an appropriate score function must be chosen based on the hypothesis that is to be tested.

Another potential application of our tests lies in the field of relation extraction. The evaluation of our tests with the GO and CL reveals that we are able to detect biologically relevant associations between these ontologies. of the best associations retrieved by have biological meaning, as shown in Table 5. Although our approach is unable to detect the types of the biological relations, the associations provide a good starting point for an elaborate approach to the extraction of biological relations.

Our method is designed for the detection of associations between two DAGs. However, it can be generalized to test for associations between graphs. The result of the tests would then be significant -ary associations between nodes from graphs.

Conclusions

We developed a family of novel statistical tests for associations between two directed acyclic graphs. The tests account for the graphs' topologies and test for relevance and specificity of associations. The tests are suitable for the detection of associations between categories from two biomedical ontologies, in particular those which comply with the OBO criteria [24].

In an extensive use-case, we applied our tests to the discovery of associations between categories from the Gene Ontology and the Celltype Ontology that were decorated with the number of occurrences and co-occurrences of the categories' labels in a large corpus of full-text articles. Our results show that a large proportion of the associations discovered by our tests are biologically relevant relations.

The family of tests is implemented in a Java library, which is available as free software from our project webpage at http://bioonto.de/pmwiki.php/Main/ExtractingBiologicalRelations.

Supporting Information

[Figure omitted. See PDF.]

Supplement S1.

Statistical tests for associations between two directed acyclic graphs and their application to biomedical ontologies.

https://doi.org/10.1371/journal.pone.0010996.s001

(0.14 MB PDF)

Acknowledgments

We would like to thank Leonardo Bubach, Hernán Burbano and Heinrich Herre for helpful discussions and valuable comments, and Christine Green for her help in preparing the manuscript.

Author Contributions

Conceived and designed the experiments: RH ACNN MD JK. Performed the experiments: RH ACNN MD. Analyzed the data: RH ACNN MD. Contributed reagents/materials/analysis tools: RH ACNN MD. Wrote the paper: RH ACNN MD JK.

Citation: Hoehndorf R, Ngonga Ngomo A-C, Dannemann M, Kelso J (2010) Statistical Tests for Associations between Two Directed Acyclic Graphs. PLoS ONE5(6): e10996. https://doi.org/10.1371/journal.pone.0010996

References

1. Smith B, Ashburner M, Rosse C, Bard J, Bug W, et al. (2007) The OBO Foundry: coordinated evolution of ontologies to support biomedical data integration. Nat Biotech 25: 1251–1255.B. SmithM. AshburnerC. RosseJ. BardW. Bug2007The OBO Foundry: coordinated evolution of ontologies to support biomedical data integration.Nat Biotech2512511255

2. Golbreich C, Horrocks I (2007) The OBO to OWL mapping, GO to OWL 1.1! In: Golbreich C, Kalyanpur A, Parsia B, editors. C. GolbreichI. Horrocks2007The OBO to OWL mapping, GO to OWL 1.1!C. GolbreichA. KalyanpurB. ParsiaProceedings of the OWLED 2007 Workshop on OWL: Experiences and Directions, Innsbruck, Austria, Jun 6–7. Aachen, Germany: CEUR-WS.org, volume 258 of CEUR Workshop Proceedings. Proceedings of the OWLED 2007 Workshop on OWL: Experiences and Directions, Innsbruck, Austria, Jun 6–7. Aachen, Germany: CEUR-WS.org, volume 258 of CEUR Workshop Proceedings.

3. Ashburner M, Ball CA, Blake JA, Botstein D, Butler H, et al. (2000) Gene ontology: tool for the unification of biology. the gene ontology consortium. Nat Genet 25: 25–29.M. AshburnerCA BallJA BlakeD. BotsteinH. Butler2000Gene ontology: tool for the unification of biology. the gene ontology consortium.Nat Genet252529

4. Beissbarth T, Speed TP (2004) Gostat: find statistically overrepresented gene ontologies within a group of genes. Bioinformatics 20: 1464–1465.T. BeissbarthTP Speed2004Gostat: find statistically overrepresented gene ontologies within a group of genes.Bioinformatics2014641465

5. Smith B, Ceusters W, Klagges B, Köhler J, Kumar A, et al. (2005) Relations in biomedical ontologies. Genome Biol 6: B. SmithW. CeustersB. KlaggesJ. KöhlerA. Kumar2005Relations in biomedical ontologies.Genome Biol6

6. Bard J, Rhee SY, Ashburner M (2005) An ontology for cell types. Genome Biology 6: R21.J. BardSY RheeM. Ashburner2005An ontology for cell types.Genome Biology6R21

7. Pesquita C, Faria D, Falco AO, Lord P, Couto FM (2009) Semantic similarity in biomedical ontologies. PLoS Comput Biol 5: e1000443.C. PesquitaD. FariaAO FalcoP. LordFM Couto2009Semantic similarity in biomedical ontologies.PLoS Comput Biol5e1000443

8. Wu Z, Palmer MS (1994) Verb semantics and lexical selection. In: Pustejovsky J, editor. pp. 133–138.Z. WuMS Palmer1994Verb semantics and lexical selection.J. Pustejovsky133138Proceedings of the 32th Annual Meeting on Association for Computational Linguistics (ACL '94), June 27–30, 1994, New Mexico State University, Las Cruces, New Mexico, USA. Morgan-Kaufman Publishers, San Francisco, CA, USA. Proceedings of the 32th Annual Meeting on Association for Computational Linguistics (ACL '94), June 27–30, 1994, New Mexico State University, Las Cruces, New Mexico, USA. Morgan-Kaufman Publishers, San Francisco, CA, USA.

9. Wu H, Su Z, Mao F, Olman V, Xu Y (2005) Prediction of functional modules based on comparative genome analysis and gene ontology application. Nucleic Acids Res 33: 2822–2837.H. WuZ. SuF. MaoV. OlmanY. Xu2005Prediction of functional modules based on comparative genome analysis and gene ontology application.Nucleic Acids Res3328222837

10. Wu X, Zhu L, Guo J, Zhang DY, Lin K (2006) Prediction of yeast protein-protein interaction network: insights from the Gene Ontology and annotations. Nucl Acids Res 34: 2137–2150.X. WuL. ZhuJ. GuoDY ZhangK. Lin2006Prediction of yeast protein-protein interaction network: insights from the Gene Ontology and annotations.Nucl Acids Res3421372150

11. del Pozo A, Pazos F, Valencia A (2008) Defining functional distances over gene ontology. BMC bioinformatics 9: 50+.A. del PozoF. PazosA. Valencia2008Defining functional distances over gene ontology.BMC bioinformatics950+

12. Resnik P (1995) Using information content to evaluate semantic similarity in a taxonomy. Proceedings of the 14th International Joint Conference on Artificial Intelligence (IJCAI-95) San Mateo, CA: Morgan Kaufmann. 448–453.P. Resnik1995Using information content to evaluate semantic similarity in a taxonomy.Proceedings of the 14th International Joint Conference on Artificial Intelligence (IJCAI-95)San Mateo, CAMorgan Kaufmann448–453URL citeseer.ist.psu.edu/resnik95using.html. URL citeseer.ist.psu.edu/resnik95using.html.

13. Lin D (1998) An information-theoretic definition of similarity. D. Lin1998An information-theoretic definition of similarity.In: Proceedings of the Fifteenth International Conference on Machine learning (ICML-98). Madison, Wisconsin.

14. Couto FM, Silva MJ, Coutinho PM (2005) Semantic similarity over the gene ontology: family correlation and selecting disjunctive ancestors. CIKM '05: Proceedings of the 14th ACM international conference on Information and knowledge management. New York, NY, USA: ACM. pp. 343–344.FM CoutoMJ SilvaPM Coutinho2005Semantic similarity over the gene ontology: family correlation and selecting disjunctive ancestors.CIKM '05: Proceedings of the 14th ACM international conference on Information and knowledge managementNew York, NY, USAACM343344

15. Othman RM, Deris S, Illias RM (2008) A genetic similarity algorithm for searching the gene ontology terms and annotating anonymous protein sequences. J of Biomedical Informatics 41: 65–81.RM OthmanS. DerisRM Illias2008A genetic similarity algorithm for searching the gene ontology terms and annotating anonymous protein sequences.J of Biomedical Informatics416581

16. Wang JZ, Du Z, Payattakool R, Yu PS, Chen CF (2007) A new method to measure the semantic similarity of go terms. Bioinformatics 23: 1274–1281.JZ WangZ. DuR. PayattakoolPS YuCF Chen2007A new method to measure the semantic similarity of go terms.Bioinformatics2312741281

17. Gruber TR (1993) A translation approach to portable ontology specifications. Knowl Acquis 5: 199–220.TR Gruber1993A translation approach to portable ontology specifications.Knowl Acquis5199220

18. Herre H, Heller B, Burek P, Hoehndorf R, Loebe F, et al. (2006) General Formal Ontology (GFO) – A foundational ontology integrating objects and processes [Version 1.0]. H. HerreB. HellerP. BurekR. HoehndorfF. Loebe2006General Formal Ontology (GFO) – A foundational ontology integrating objects and processes [Version 1.0].Onto-Med Report 8, Research Group Ontologies in Medicine, Institute of Medical Informatics, Statistics and Epidemiology, University of Leipzig, Leipzig, Germany. Onto-Med Report 8, Research Group Ontologies in Medicine, Institute of Medical Informatics, Statistics and Epidemiology, University of Leipzig, Leipzig, Germany.

19. Doms A, Schroeder M (2005) GoPubMed: exploring PubMed with the Gene Ontology. Nucleic Acids Res 33: 783–786.A. DomsM. Schroeder2005GoPubMed: exploring PubMed with the Gene Ontology.Nucleic Acids Res33783786

20. Gaudan S, Yepes AJ, Lee V, Rebholz-Schuhmann D (2008) Combining evidence, specificity, and proximity towards the normalization of gene ontology terms in text. EURASIP Journal on Bioinformatics and Systems Biology 2008: 9.S. GaudanAJ YepesV. LeeD. Rebholz-Schuhmann2008Combining evidence, specificity, and proximity towards the normalization of gene ontology terms in text.EURASIP Journal on Bioinformatics and Systems Biology20089

21. Manning CD, Schütze H (1999) Foundations of Statistical Natural Language Processing. Cambridge, Massachusetts: The MIT Press. CD ManningH. Schütze1999Foundations of Statistical Natural Language ProcessingCambridge, MassachusettsThe MIT Press

22. Pantel P, Lin D (2002) Discovering word senses from text. Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining. ACM Special Interest Group on Knowledge Discovery in Data. New York, NY, USA: ACM Press. pp. 613–619.P. PantelD. Lin2002Discovering word senses from text.Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining. ACM Special Interest Group on Knowledge Discovery in DataNew York, NY, USAACM Press613619ISBN:1-58113-567-X. ISBN:1-58113-567-X.

23. Bada M, Hunter L (2007) Enrichment of obo ontologies. Journal of Biomedical Informatics 40: 300–315.M. BadaL. Hunter2007Enrichment of obo ontologies.Journal of Biomedical Informatics40300315

24. Smith B, Ashburner M, Rosse C, Bard J, Bug W, et al. (2007) The obo foundry: coordinated evolution of ontologies to support biomedical data integration. Nat Biotech 25: 1251–1255.B. SmithM. AshburnerC. RosseJ. BardW. Bug2007The obo foundry: coordinated evolution of ontologies to support biomedical data integration.Nat Biotech2512511255