Google Scholar as a single source for systematic reviews has not been justified
Alison Weightman, Cardiff University
13 February 2013
This paper forms part of an important debate. Google Scholar (GS) is an increasingly powerful search tool that should rightly be considered by systematic review searchers. Research studies comparing GS to other established search tools are valuable.
However we feel the paper fails to address some really important issues, and believe that the conclusion that GS could be a single search source for systematic reviews has not been justified.
Being able to find a paper once you know about it is not the same as finding the paper in the first instance. Problems identified with this publication, on which we would welcome the authors' and journal editors' responses, are:
1. Google Scholar is constantly updated and it isn't possible to know whether the references included in it at the search date for this paper would have been there at the search date of the candidate systematic reviews. GS trawls many sites and may have picked up these references as a result of their inclusion in the reviews.
2. The lack of advanced search functions is acknowledged but there seems to be an implicit assumption in the paper that review authors would have chosen the right terms to pick up relevant publications, including citations without an abstract. Further, the very low precision rates measured by the authors of this paper from example searches (circa 0.1% precision) severely limit the current value of GS as a single search source for systematic reviews. It would ask a lot of systematic reviewers to trawl/sift 36,000 results in order to find 36 relevant papers.
3. The authors did not require the title to link to an abstract or full text in Google Scholar (GS), to be regarded as a study within GS; a link to a citation was considered sufficient. Each title identified via a citation would need to be searched for in other database(s) to find an abstract, or obtained in full text to assess relevance. This would entail a substantial additional workload for systematic review authors which is not addressed in the paper.
4. The paper has several sections that should have been edited for grammatically correct English. Errors are not surprising given that the authors are not writing in their native language, but it may call the editorial process into question.
Alison Weightman, Fiona Morgan, Mala Mann and Bernadette Coles
University Library Service, Cardiff University, UK
Competing interests
A competing interest exists when your professional judgment about a paper could possibly be influenced by considerations other than the paper's validity or importance. Detail possible competing interests here... No competing interests
Interesting but...short shrift to expert searching
Michelle Fiander, A personal comment
13 February 2013
That Google Scholar is a one stop source for searching because a selection of studies included in a selection of systematic reviews can be found in GS displays, I believe, faulty logic--perhaps an association fallacy. Whether or not I am correct regarding the logic, I am bit more concerned that the authors give very short shrift to the importance of the search interface (technology aspect) and knowledge of indexing (understanding gained by experience and training) which are both integral to searching for systematic reviews. While GS and other Web applications are absolutely fantastic (no argument from me on this point), neither the search interface, nor the scope and depth of GS is yet, in my opinion, sufficiently well defined or organized to support comprehensive searching for systematic reviews.
Competing interests
I am an Information Scientist, Trials Search Co-ordinator, for the Cochrane Collaboration Effective Practice and Organisation of Care Review Group and a member of the Cochrane Collaboration Steering Group.
Of course GS will find what you already know exists
Tuulevi Ovaska, University of Eastern Finland
13 February 2013
Dear colleagues,
your article is very interesting but also rather confusing.
It would naturally be very efficient and truly economic if only one simple and freely available search engine could be used to search and find all studies needed for a systematic review.
You write that "GS was searched for all these studies one by one to assess the percentage of studies which could have been identified by searching only GS. ---All the 738 original studies included in the gold standard database were retrieved in GS (100%)."
Well, of course, you were able to retrieve them all as you already knew what they were and searched them one by one.
But would you have found them if you did not know them already? Would searching GS for the topic (instead of the studies already known) end in the result of the same studies? These are the important questions, imho.
Google Scholar as a single source for systematic reviews has not been justified
13 February 2013
This paper forms part of an important debate. Google Scholar (GS) is an increasingly powerful search tool that should rightly be considered by systematic review searchers. Research studies comparing GS to other established search tools are valuable.
However we feel the paper fails to address some really important issues, and believe that the conclusion that GS could be a single search source for systematic reviews has not been justified.
Being able to find a paper once you know about it is not the same as finding the paper in the first instance. Problems identified with this publication, on which we would welcome the authors' and journal editors' responses, are:
1. Google Scholar is constantly updated and it isn't possible to know whether the references included in it at the search date for this paper would have been there at the search date of the candidate systematic reviews. GS trawls many sites and may have picked up these references as a result of their inclusion in the reviews.
2. The lack of advanced search functions is acknowledged but there seems to be an implicit assumption in the paper that review authors would have chosen the right terms to pick up relevant publications, including citations without an abstract. Further, the very low precision rates measured by the authors of this paper from example searches (circa 0.1% precision) severely limit the current value of GS as a single search source for systematic reviews. It would ask a lot of systematic reviewers to trawl/sift 36,000 results in order to find 36 relevant papers.
3. The authors did not require the title to link to an abstract or full text in Google Scholar (GS), to be regarded as a study within GS; a link to a citation was considered sufficient. Each title identified via a citation would need to be searched for in other database(s) to find an abstract, or obtained in full text to assess relevance. This would entail a substantial additional workload for systematic review authors which is not addressed in the paper.
4. The paper has several sections that should have been edited for grammatically correct English. Errors are not surprising given that the authors are not writing in their native language, but it may call the editorial process into question.
Alison Weightman, Fiona Morgan, Mala Mann and Bernadette Coles
University Library Service, Cardiff University, UK
Competing interests
A competing interest exists when your professional judgment about a paper could possibly be influenced by considerations other than the paper's validity or importance. Detail possible competing interests here... No competing interests
Interesting but...short shrift to expert searching
13 February 2013
That Google Scholar is a one stop source for searching because a selection of studies included in a selection of systematic reviews can be found in GS displays, I believe, faulty logic--perhaps an association fallacy. Whether or not I am correct regarding the logic, I am bit more concerned that the authors give very short shrift to the importance of the search interface (technology aspect) and knowledge of indexing (understanding gained by experience and training) which are both integral to searching for systematic reviews. While GS and other Web applications are absolutely fantastic (no argument from me on this point), neither the search interface, nor the scope and depth of GS is yet, in my opinion, sufficiently well defined or organized to support comprehensive searching for systematic reviews.
Competing interests
I am an Information Scientist, Trials Search Co-ordinator, for the Cochrane Collaboration Effective Practice and Organisation of Care Review Group and a member of the Cochrane Collaboration Steering Group.
Of course GS will find what you already know exists
13 February 2013
Dear colleagues,
your article is very interesting but also rather confusing.
It would naturally be very efficient and truly economic if only one simple and freely available search engine could be used to search and find all studies needed for a systematic review.
You write that "GS was searched for all these studies one by one to assess the percentage of studies which could have been identified by searching only GS. ---All the 738 original studies included in the gold standard database were retrieved in GS (100%)."
Well, of course, you were able to retrieve them all as you already knew what they were and searched them one by one.
But would you have found them if you did not know them already? Would searching GS for the topic (instead of the studies already known) end in the result of the same studies? These are the important questions, imho.
Competing interests
None declared