Open Knowledge Creation: Improving the Peer Review and Adoption Process

Introduction  

The publication of research is a highly regarded form of capital in academia.  It is the goal of research to advance knowledge and understanding for the greater good of society.  The process used by traditional journals to review and publish papers has been in place for many years, predating the Internet.  At the ICIS 2004 meeting Watson suggested a number of areas (Gray et. al. 2006) where the IS discipline should lead with changes in the management of its journals. His suggestions included: 

1. Improved reviewing of papers by accrediting reviewers in order to improve the quality of their work
2. Creating a market for journal articles in which editors would bid for articles accepted by the accredited reviewers
3. Moving our journals to the next level of Internet sophistication

Among the advantages Watson cited (Gray et. al. 2006) were higher quality reviews, fairness to authors by shifting the balance of power between journals and authors, shorter times to decision and publication, better use of reviewers (our field’s scarcest resource), taking advantage of the Internet's capabilities, and showing leadership by demonstrating to the scientific community the improvements that are possible through the use of the Internet.  During the ICIC 2004 meeting a survey was administered concerning Watson's presentation.  The survey findings (Gray et. al. 2006) can be seen in Table 1. 
 

Table 1: Survey Responses from Richard T. Watson's Presentation at the 2004 ICIS in Washington D.C. 

The conclusions drawn from the survey (Gray et. al. 2006) include the following. 

• Universal dissatisfaction with reviewing
• We have only begun to adapt to the Internet
• Lack of structure hinders productivity
• IS should lead the way

Watson's response to the panel discussion concerning the management of our journals in the future addressed quality control, the move to markets, and the Internet.  It has been   recognized (Gray et. al. 2006, Straub 2008) that one of the problems with quality control has to do with the lack of consistency of reviewing.  Since there is no single perspective that reviewers follow, the outcomes of the reviews are varied.  Symptoms of this problem can be seen (Gray et. al. 2006) in the low acceptance rates and assessments that too many of the articles submitted are unsuitable.  The fact is many of the authors of the submitted papers are also reviewers of other scholar's submissions.  The irony (Gray et. al. 2006) in this situation is that “authors whose work is not good enough for publication in a particular journal are sometimes also asked to judge the quality of articles submitted to the same or a similar quality journal.” As Watson stated: reviewers are scarce and given the reviewer variability the prospects of successfully getting a paper accepted “is a crap shoot.” 

Watson believes the move to markets (Gray et al. 2006) may “do a better job with the allocating of scarce resources than bureaucracies” with regard to reviewers.  In a market situation “authors would be free to withdraw a paper after reading the reviews or reject an offer to publish in a journal.” The power to make decisions should be more with the authors as opposed to the reviewers and editors since they are the creators of the knowledge.  A market approach can increase reviewer availability and allow the authors to have more control over their paper.  The Internet offers practical ways to implement markets as well as other innovations that previously were not feasible. 

The prospect of a journal becoming electronic and fully utilizing the capabilities of the Internet are possible today.  Although electronic journals are now possible on the Internet, the IS discipline has not played a leadership role with changing traditional practices to take advantage of it.  Watson (Gray et. al. 2006) contends that the traditional posture of IS has been: “Too much of IS scholarship is studying what others have invented, or codifying the experience of others when we could be more influential and respected if we were creators rather than reporters.” He believes that the IS discipline should take a leadership role and act as a change agent for moving the academic community into the Information Age. 

Although a comprehensive solution to these issues is not presently in use, the survey results suggest that some changes are desirable.  This paper offers a framework that can be used to address a number of the concerns Watson expressed (Gray et. al.2006) although we are not suggesting his exact recommendations for a remedy.  This paper will primarily focus on peer review and adoption.  Peer review has been identified by Watson (Gray et. al. 2006) and Straub (2008) as a key area that needs improvement.  The Internet with Google's Knol offers a set of tools designed specifically for authoring, reviewing and publishing.  This paper takes a systemic view (i.e., W. Edwards Deming) of the peer review and adoption process by providing a framework that can be used to facilitate its redesign. 

Much has been written about open concepts versus those that are closed or tightly controlled.  Open concepts have such labels as “commons-based peer production” by Yochai Benkler (2002, 2006), “wisdom of crowds” by James Surowiecki (2004) and “open innovation” by Henry Chesbrough (2006).  These open concepts suggest a different approach to peer review from the current practice used by many journals. 

Straub (2008) notes that the traditional journal review process has two risks: (a) where the reviewers’  view of a paper is to accept while in retrospect the IS community's view would have been to reject and (b) where the reviewers’ view is to reject and the IS community's view is to accept.  Both errors are undesirable since they conflict with the objective of journals: “to publish good, and occasionally great, papers.” The later risk could be eliminated by accepting all papers but this is not realistic in the present system of publishing since (a) there is a limit to the publication space in physical journals and (b) the quality of the journals would be diluted by the addition of poorer papers.  The former risk could be reduced by accepting significantly fewer papers.  Watson (Gray et. al. 2006) has also discussed problems with the review process pointing to the reviewer variability.  One purpose of this paper is to address these risks and offer an alternative solution. 

Alternative Approaches to Peer Review 

A peer review approach used in the hard sciences is ArXiv (pronounced “archive”).  ArXiv uses electronic copies of papers to make information known.  Its existence was one of the contributing factors leading to the revolution in scientific publishing, known as the open access movement, with the prospect of the eventual disappearance of traditional scientific journals.  Professional mathematicians and scientists regularly upload their papers to ArXiv for worldwide access and sometimes for reviews before they are published in peer reviewed journals.  Authors submit their papers (i.e., manuscripts) to the archive and moderators for each area review the submissions and may re-categorize any that are deemed off-topic.  The moderators are volunteers who have been approved by the discipline-level advisory committee and by the ArXiv staff.  Although comments on the manuscripts are possible there is no assurance that helpful comments will follow.  In essence the ArXiv archive houses many manuscripts, some of which are subsequently submitted to peer-reviewed journals. 

Another case of experimenting with alternative methods in the communication of research can be seen in the hard sciences journal Nature.  Nature's (2006) ongoing debate about peer reviewing is captured in twenty articles authored by leading scientists, publishers and other stakeholders.  Nature conducted a trial of open peer review where invited authors of newly submitted manuscripts to Nature that survived the initial editorial assessment hosted their manuscripts on Nature’s website for public comment.  The manuscripts were also subjected to traditional peer review.  All comments were required to be signed.  After the traditional peer review was completed all comments from both reviews were compiled.  Afterwards an author survey was conducted with the following results. 

All authors who participated in the trial were sent a survey.  Sixty-four people were contacted and 27 responded (a 42% response rate).

• Twenty respondents thought it was an interesting experiment.
• Fourteen of the 27 who responded to the survey received comments.
• Four of the 14 who received comments described them as 'not useful', 6 as 'somewhat useful', and 4 as 'very useful'.
• Although most respondents received no additional comments by taking part in the trial (such as email messages or phone calls), those who did (five people) found them either 'useful' (four) or 'very useful' (one).
• Some authors expressed concern about possible scooping and others were disappointed that they didn't receive more comments.
• Of the 27 respondents, 11 expressed a preference for open peer review.

Although the results were not overwhelming there were indications that there is interest in a different approach to peer review.  Some combination of a modified peer review process coupled with proper incentives and embraced by the field could usher in an improved process for peer review and adoption over time.  The framework proposed in this paper is referred to as open knowledge creation (OKC).  The OKC framework is aimed at improving the peer review and adoption process. 

Implementing the Open Knowledge Creation Framework 

Open Knowledge Creation uses Google's Knol (i.e., an online web publishing tool) and Google Groups as the primary tools for authors, editors and journals.  A general discussion of a similar approach to OKC (Hardaway 2005) can be found in ”Sharing Research in the 21^st Century:  Borrowing a Page from Open Source Software.” Google Groups are used to segregate papers based on their topic.  The Groups would be created in advance by the journal with each Group representing a research theme or category.  Members of the IS community would join those groups in which they are interested after a notification of the created groups was sent out over the IS listserv.  Organizing the IS community into groups prevents everyone in the community from being inundated with emails about every posted paper.  Only members of a Group would receive notification about papers posted in their group.  When a paper is ready for review the author(s) would post an abstract and link to their Knol in the appropriate Group requesting reviews and comments.  In essence, these Groups which represent “birds of a feather”, contain the IS communities subject matter expertise.  This has the potential to offer better expertise and more available reviewers when papers are posted for review.  The creation of the Groups will have to be completed by the journal and once the IS community joins those Groups that reflect their interests then the OKC process can begin.  In addition, editors would be assigned to the various Groups so that they could have access to the papers posted when the papers are ready for consideration by the journal.  The four stage framework of OKC appears in Figure 1 and includes the creation, review/revision, evaluation/acceptance and publication stages.

Figure 1: Open Knowledge Creation Framework

Creation Stage 

OKC begins with an author creating a Knol (i.e., Google's name for a paper about a specific topic).  The author has the choice of creating the paper through any browser using the on-line editing features or authoring the paper on their PC and uploading it as a Knol.  Once the paper is stored as a Knol, any future changes are reflected by new versions of the paper while prior versions are maintained as well. 

Review/Revise Stage 

When an author is ready to start receiving feedback he/she would use a feature in Google Knols to invite a review by posting an abstract and link to the Knol in the appropriate Google Group based on the papers topic.  The abstract included in the posting would allow IS researchers to determine if the research falls within their specific area of expertise.  The address to the Knol would only be known to the subscribers of the Group so membership can be controlled to prevent unwanted participation. 

Before a Knol can be reviewed the author must choose from among three collaboration models. 

• Open collaboration model - all registered Google users are allowed to edit the Knol
• Moderated collaboration model - all registered Google users may suggest edits to the Knol, but these edits need approval from an author before the original manuscript will incorporate the suggestions
• Closed collaboration model - only owners/authors may edit the Knol

We would recommend using the closed collaboration model since OKC is a new process and it would be better to make smaller changes at first.  It would also remove the dilemma of how to manage additions by others. 

Listed below are the various choices for feedback to a Knol. 

• Ratings – are used by the reviewer to provide a rating based on a five star Likert assessment of the Knol.
• Comments – are used by the reviewer to address specific points in the manuscript or to raise specific questions.
• Edits - if a Knol allows moderated collaboration, the reviewer can click Edit at the top of the Knol to offer changes to the text of the Knol.
• Invited Reviews – authors can solicit a formal review from anyone.

Reviews will vary as people have differing attitudes and beliefs.  For example, some may be threatened by unorthodox papers (Straub 2008) and inclined to post negative responses.  We believe this will be moderated to some extent by the fact that all reviews and comments have the reviewers' identity shown. This affects their image among the IS community if their remarks are not constructive.  We believe the non-blind aspect of the review process will help encourage constructive collaboration.  Regardless of reviews, editors can always make adoption decisions based on their own evaluation as suggested by Straub (2008) if desired.  Another concern is that those with differing views might be reluctant to offer a counter view because they fear that the author(s) might harbor a grievance against them.  This could certainly happen but we feel over time a more open process will result in increased constructive, helpful reviews and be received in a more positive manner than the blind reviews that sometimes are not professional in their critique.   

Once reviewers begin to post reviews or comments the authors would monitor the remarks and make revisions based on their judgment of the critiques.  The mechanisms for offering feedback are part of Google's Knol system.  A description of the feedback options are reprinted in the Appendix.  As seen in the Appendix, Google offers a number of features to accommodate reviewing. 

At some point the author(s) are ready for an editor to consider their paper.  This procedure will have to be decided by the editors of the journal.  Editors of Groups can monitor and initiate contact with the author(s) if they are interested or they can require authors to notify them.  They may set some guidelines before being notified such as the minimum number of reviews or some other criteria before reviewing the article themselves.  The editor may want to solicit a review in addition to those from the Group.  This will have to be a decision set by each journal.  Different journals will have different thoughts on how they want to implement OKC. 

Encouraging Reviewers 

Obtaining reviewers’  participation will be needed in the OKC process.  Most people have a difficult time adjusting to major changes. Adjusting to a new process concerning the most valuable academic capital would be very challenging. This adjustment will require active and continued support by the journal adopting OKC.   

One approach to encourage participation from the IS community can be found by understanding how open source development participation has occurred.  An open source project always begins with one or more people beginning the project and carrying the burden of the coding.  If the project is of interest to others, then they would begin to develop code and offer it to the leaders of the project for consideration while other may just suggest desirable features.  If the contributions are good they are adopted.  As programmers make more contributions that are of quality, two things typically result.  One, they establish a reputation within the community of being good and second, they commonly are asked to assist with the governance of the project.   

To apply this concept to OKC we would begin with the journal selecting editors to oversee Google Groups for which they have expertise.  The journal could then communicate to the IS community (i.e., IS listserv) that additional editors will be needed but they must first exhibit evidence of their competence and subject matter expertise.  The journal could announce that additional editors will be identified by monitoring reviews and selecting those who have demonstrated expertise with the subjects they reviewed.  Following this process could lead to an increased number of editors monitoring the various Google Groups and take better advantage of the expertise that is within the IS community.  Having an editorial role is a form of capital in academia and should encourage participation.  By expanding the editorial base and using demonstrated expertise as a basis for selection gives everyone in the community a chance to show their capabilities while solving the problem of the additional personnel needed to monitor the volume of papers that are typically submitted to a top journal.  This also represents a natural process of grouping willing participants with a particular expertise in Groups with others of the same interest.  One problem with large numbers of people is knowing who has what expertise and the strength of their expertise.  As with open source projects everyone knows who the best coders are and who makes the most contributions to various projects.  This could follow that same principal and be open for the community to witness the evidence as it is with open source development.  Although reviews are critical, a journal may decide to make adoption decisions based on an individual editor’s judgments as suggest by Straub(2008).  This is a decision that will have to be decided by each journal. 

Evaluation/Acceptance Stage 

Watson (Gray et. al. 2006) suggests a paper marketplace whereby authors would post their papers and IS journals would shop for paper of interest.  In the OKC case we are not recommending an open marketplace.  It would be too different and require cooperation among several journals which may be hard to achieve.  The idea of a marketplace may be able to be implemented at some point in the future but is not advised at the outset.  We feel that one journal should use OKC and get it fully implemented before expanding to a multi-journal OKC implementation.  In either case it would involve editors monitoring the Google Groups and selecting papers.  Once a paper is posted the authors would receive feedback from the IS community and make refinements to their paper.  The authors in this context control their paper as to the modifications made and any subsequent arrangements made with the journal.  Next, editors of each Google Group review the Knols to identify papers that they believe have potential after the papers have achieved a milestone set by the journal.  Under the OKC framework, in addition to the traditional all rights reserved license, three other licensing options are available. 

• All Rights Reserved--The holder of a copyright reserves the sole right to publish a work, and is entitled to take legal action against infringement.  In this case the author may have to choose which journal to select unless journals cooperated by allowing different versions of the paper to be published by other journals.  By having additional reviews from the IS community the editors would have more information to consider when making accept/reject decisions.    Ultimately, journals will decide how and to what extent they want to leverage the OKC framework
• Creative Commons Attribution 3.0 License--free to copy, distribute and transmit the work and to adapt the work
• Creative Commons Attribution Noncommercial 3.0 License--free to copy, distribute and transmit the work and to adapt the work for noncommercial purposes
• Creative Commons Attribution-Share Alike 3.0 License--If you alter, transform, or build upon this work, you may distribute the resulting work only under the same, similar or a compatible license.

. 

If a paper is desired by a journal, then the author would be notified and the journal and author would reach an agreement as to additional work that needs to be done.  The licensing would have to be negotiated if the Knol was to be made available for public access.    

Publication Stage 

The publication options in OKC include the publication of papers (1) in the traditional manner (on paper or electronic copy of a paper) and (2) an Internet version (Knol) that includes multimedia content.  We are also suggesting that journals promote the Internet version since it may encourage reviewer participation by the IS community.  The Internet version offers not only additional types of content (i.e., multimedia) to be incorporated into the paper but also shortens the time to acceptance and publication.  The additional types of content allow papers to become much more robust since audio, video and animation are powerful illustration and communication media forms. 

Further possibilities include the use of the collections feature in Google Knol.  This feature allows for the grouping of articles such as with a journal issue or as a collection of papers with a common theme.  Knols have the flexibility to be used to create an electronic journal.  With the ability to group Knols, research can be easily organized and housed.  An electronic journal could be created using this feature along with the other features in Knols and removing much of the burden for individual journals wishing to become Internet based. 

Knols would allow the public as well as the IS community to use the power of Google's search engine to find papers of interest.  After all, Google's mission statement reads: “to organize the world's information and make it universally accessible and useful. 

Benefits 

Primary benefits of OKC include shortening the time from posting a paper to adoption and providing the potential for more reviews and comments from a larger number of the IS community.  It does not preclude the editors from soliciting additional reviews, it only augments or replaces the need at the option of the editors.  Even with these potential benefits, change for IS community will be a challenge and require patience and persistence.  Before the benefits of OKC can be realized changes will need to be made by  the leadership of our journals.   Second, there is a need for a more detailed framework to define scholarship so faculty can receive credit for the different types of scholarly work. 

Leadership to Change 

One change we recommend the leadership of the top journals to make is to redefine the process by which they will receive submissions for consideration. By redefining the submission process to direct papers to be posted using the OKC design will encourage the participation of the IS community to use the OKC process.  If faculty see an opportunity to gain recognition through reviewing and believe the OKC process would make it easier for them to publish it would provide an incentive to change.  This change will take time and patience so, unlike the test trial that the journal Nature conducted which was a one off, it is our suggestion to view OKC as a goal with adoption taking place over time as discussed by Everett Rogers (2003) in his Diffusion of Innovations book. 

Levels of Scholarly Contribution 

The Review process will need incentives aside from the potential to become a Group editor to encourage an active participation and to advance research in a more efficient and thorough fashion.  Today's practice provides credit for authorship and co-authorship.  Some institutions even break authorship into percentages of contribution when evaluating faculty.  We propose using an expanded reward structure to assign faculty credit for the different types of contributions.  Table 1 contains example categories for organizing various types of contributions in rank order of merit. 

Table 2: Levels of Scholarly Contribution

By using a reward structure, the community may be more inclined to participate in reviewing as long as IS leaders and institutions encourage its use. 

The OKC Framework 

The OKC framework is a design using Google's Knol and Groups adapted for use by traditional IS journals.  There are a number of change aspects that will require incentives.    Below is a sampling of some change aspects and their associated incentives. 

Table 3: Change Aspects

An Example Implementation 

There are several preliminary steps that must be taken before any journal would change its practices.  This could be considered analogous to developing a new business plan.  All parties with the responsibility for governance of the journal would have to agree to the new design.  Journals will have to decide if they want to appoint editors and associate editors to make acceptance decisions or use some other strategy.  This would be followed by announcements of the changes to the market place (i.e., IS Community and Institutions) with a go live date and procedures to be used for the new design.  Afterwards, authors would begin to follow the new procedure. 

The partial implementation described here is only the initial phase in taking full advantage of OKC.  We are implementing the process in the review of this paper on the Knol. It is intended to provide the IS community some initial experience with a basic form of OKC.  The process outlined below does not use the Google Groups and is not fully implemented as it would be in a production environment since it is a trial run using only part of the design to gain some insights.  The implementation will be described in the steps below. 

Step 1 

The paper is posted as a Knol.  The paper will use closed collaboration and with all rights reserved.  The editor will be assigned full access to the paper so that all postings of comments, reviews and modifications will be observable. 

Step 2 

In the initial phase of the transition the editor of the journal then sends a message and link to the paper on the IS listserv explaining that this is a new process being explored for the benefit of the IS community and to please participate if you are in favor of a change like this (or something similar) to the review and adoption process.  The reviewing time window should be stated (e.g., perhaps two or three weeks).  The reviewing can begin at this point. 

Step 3 

As reviews and comments are published the author(s) and editor will monitor the feedback with the authors making modifications as appropriate.  The Knol would remain posted until the authors have received feedback for a specified period.  Revisions would be made as determined by the authors or with suggestions from the editor.  Authors would recognize those paper reviewers who provided the most helpful feedback in the appropriate place in the paper.  This acknowledgement serves as a reviewer citation so that credit could be given for the effort as illustrated in Table 2.    In this trial instance the author(s) would include a summary of the feedback provided and describe its effect.  After all changes are made within the time window alloted, the authors will notify the editor that the review/revision phase is completed. 

Step 4 

The editor(s) would then decide if they wanted to have further revisions made although it would be more efficient if the editors did that during the Step 3.  The journal with the editors would then decide if they wanted to publish the paper in a new e-journal offering under the same brand (i.e., in essence a fully electronic version of the same journal) or publish in the traditional print method.  Google's Knol can easily accommodate setting up an electronic form of the journal. 

Concluding Remarks 

There are many challenges with changing practices that have been in place for so many years.  It is typical with anything new that you would have a continuum of faculty ranging from those that are early adopters to ones more comfortable with the traditional process as discussed by Rogers (2003). 

Although change is slow and sometimes painful, it is necessary if progress is to be made.  There are always unforeseen obstacles that have to be dealt with and a human tendency to resist change.  Leadership by those in influence is important and only with a collective effort can a new process for conducting research be achieved.  The benefits can be significant with the added ease by which faculty can connect with others with similar interests.  OKC is one example of an innovation we believe could prove to be a feasible next step to advancing how research is reviewed and adopted.

1. Chesbroug H., Vanhaverbeke, W., West, J. (editors)(2006), Open Innovation. Researching a new paradigm. ISBN 978-0-19-9290-72-7.
2. Gray P., Lyytinen K.J., Saunders C.S., Willcocks L.P., Watson R.T., and Zwass V., 2006. “How Shall We Manage Our Journals in the Future? A Discussion of Richard T. Watson’s Proposals at ICIS 2004 ,” Communications of the Association for Information Systems, Vol. 18, Article 14.
   
3. Hardaway, D., “Sharing research in the 21st century: borrowing a page from open source software,”  Communications of the ACM 48(8): 125-128 (2005).
4. Nature's peer review debate: http://www.nature.com/nature/peerreview/debate/index.html
   
5. Nature's peer review trial: http://www.nature.com/nature/peerreview/debate/nature05535.html, 2006.
6. Rogers, E., Diffusion of Innovations, Fifth Ed., Free Press, 2003.
   
7. Straub, Detmar W., 2008. "Editor's Comments: Thirty Years of Service to the IS Profession: Time for Renewal at MISQ," MIS Quarterly, (32: 1).
8. Straub, Detmar W., 2008. "Editor's Comments: Type II Reviewing Errors and the Search for Exciting Papers," MIS Quarterly, (32: 2) pp.v-vii.
   
9. Straub, Detmar W., 2008. "Editor's Comments: Why Do Top Journals Reject Good Papers," MIS Quarterly, (32: 3) pp.iii-vii.
10. Straub, Detmar W. and Ang, Soon., 2008. "Editor's Comments: Readability and the Relevance Versus Rigor Debate," MIS Quarterly, (32: 4) pp.iii-xiii.
    
11. Surowiecki, James (2004). The Wisdom of Crowds: Why the Many Are Smarter Than the Few and How Collective Wisdom Shapes Business, Economies, Societies and Nations Little, Brown ISBN 0-316-86173-1.
12. Yochai, B., Coase’s Penguin, or, Linux and The Nature of the Firm The Yale Law Journal, Vol. 112: 369, 2002.
13. Yochai Benkler, The Wealth of Networks: How Social Production Transforms Markets and Freedom, Yale University Press, New Haven, CT, 2006.

Appendix 

Writing Reviews 

Writing an effective knowledge article often requires soliciting reviews from people you consider experts in your field. Knol makes this very easy to do. 

Solicited reviews 

You can solicit reviews from anyone by using the author toolbox on the right hand side of your knol. Simply click the "invite" link next to the reviewers count (if you just started, this should say "0 reviewers") and you will be able to send a request for a review via email. You do not have to publish your knol for this, as the email contains a special URL that will allow your recipient to see and review your knol even though it has not been published yet. Once your reviewer has published the review, a link to it will appear alongside your knol. 

Unsolicited reviews 

Anyone can write reviews of any published knols. These reviews will also be shown as a link alongside the original knol. 
 
If you do not want to receive unsolicited reviews, you can turn them off, as described here. 

The Review Scorecard 

When you write a review, we ask a series of questions, which can be answered "yes", "no", or somewhere in between. If the question is not applicable to the knol, simply answer "no opinion". 
 
The Review Scorecard is an online opinion poll. It is not a scientific opinion poll, and it is certainly not a scientific peer review. It is simply a means by which readers can communicate to an author and to an author's readership whether they found the article useful, why or why not, and how it might be improved. The specific questions are:

• Do you trust this person as an expert on this knol's topic?
• Look at the author's profile page. Look over the author's collection of work, and tell us what you think -- and why.
• Do you believe this content is original and/or correctly attributed?
• Notice the Similar Content area. Look over the author's references and source material.
• Did you find this knol useful?
• Or -- just as important -- would people that you know find it useful?
• Is this knol a complete and thorough treatment of its topic?
• Does the knol cover its subject in enough depth?  A good knol should be valuable as an introduction or reference on the topic. It should not be merely a "teaser" that points elsewhere.
• Is this knol a pleasure to read, making good use of layout, style and media?
• For a sense of what's possible with knols, see this, this, this or this.
• Does this document cite adequate and reliable references to support its claims?
• See our page on Adding References.
• Is this knol as good or better than the top ten search results on its topic?
• For reviewers -- this is an easy question to answer. Just cut-and-paste the knol's title into the search box, search, and see how the knol compares! 
   
  For knol authors -- if you aren't adding something of value to what is already there, consider improving your knol, either yourself or as part of a team of authors.
• Does this knol promote a product or service?
• A "yes" or "no" does not necessarily imply anything about whether the knol is "good" or "bad". Knol's content policy allows commercial content. However, authors of such content should be especially careful to assure that their articles provide unique and valuable information, follows our best practices, and steers well clear of anything readers might deemed as "spam".

 
An article which achieves favorable reviews from many reputable readers should be approaching the ideal the Knol project is aiming for: 

      A knol is an authoritative, comprehensive article about a specific topic.

 
To the extent that a page receives negative reviews from reputable readers -- it can be deemed to be missing that mark. 

Publishing Reviews 

When you create a review, it is not automatically published. When you are done writing your review, you should publish it. 

Managing Reviews 

Seeing reviews you've written:

Visit your Profile page and click on the Reviews tab.  

Seeing review written about your knols:

Visit your Profile page and click on the Reviews tab-- then click on the button "Reviews about this author's knols".