In Morville & Louis Rosenfeld book, I came across to Precision and Recall ratio concepts. In the past in order to write a paper which was about the comparison of two search engines (Google and Bing) by searching three different words, I was familiarized with these two concepts since I chose these two as my criteria to do the comparison.
Precision is one of the most used criterion that have been used in evaluating the effectiveness of search engines. According to Spink and Zimmer (2008), “Precision measures the ability of an information retrieval (IR) system to produce only relevant results. Precision is the ratio between the number of relevant documents retrieved by the system and the total number of documents retrieved.” In my case, what happened with precision was that most of the search results were relevant to the words that were utilized (apple, amazon, and IBM). As you see these are common words and the probability of having relevant results are very high, and since the denominator for most of them were the same, no major differences could be seen.
Recall, on the other hand, is another traditional criterion in evaluating the performance of search engines. Accroding to Spink and Zimmer 2008, “recall”, measures the ability of an IR system to find the complete set of relevant results from within a collection of documents. Recall refers to the ratio of the number of relevant documents retrieved by the system to the total number of relevant documents for the given query.” What I came to conclusion was that the relative recall of Google was significantly different from relative recall of Bing. In my case and other similar studies that has been done due to small number of queries, the score and importance of precision is more than recall in search engine effectiveness evaluation. However, in some cases, with more queries used, recall can be important too (Can, Nuray and Sevdik, 2003). It is worth to mention that precision and recall are not mathematically dependent on each other, but as a rule of thumb, the higher the precision of a results set, the lower the recall, and vice versa (Spink & Zimmer, 2008).
Can, F., Nuray, R., & Sevdik, A. B. (May 01, 2004). Automatic performance evaluation of Web search engines. Information Processing & Management, 40, 3, 495-514.
Spink, A., & Zimmer, M. (2008). Web search: Multidisciplinary perspectives. Berlin: Springer.