CDIGS and Dev.Globus Roadmap Review Guidelines
Open review is vital to both CDIGS project plans and dev.Globus project plans. The dev.Globus guidelines provide very simple guidelines for open publication and review of project plans (long-term plans, short-term plans, and release plans). The following are expanded guidelines for use by CDIGS team members when developing CDIGS project plans and dev.Globus project plans.
The general philosophy of these guidelines is that team members must have flexibility in how they get their plans reviewed by the community, but at the same time, they must document how they've done it and what the results were. The review processes should be as open as possible, and the results should most definitely be open. Also, the CDIGS project is establishing a minimum bar for the amount of community review that must be performed on plans to which CDIGS resources are to be committed, and that bar is higher than the standard dev.globus bar.
High-level Review Requirements
- All CDIGS project plans should be reviewed by relevant community members. "Relevant" means: members of the community who are expected to find the products of the plan useful in their own work.
- The steps taken to review the plans with the community must be documented (who, what, where, when, how) and easily available to CDIGS management and ideally to the public.
- The results of the community review must be documented and easily available to CDIGS management and ideally to the public.
These are the only hard and fast "rules" that must be followed. The rest of these guidelines are subjective and will most likely vary from project area to project area.
CDIGS Plan Review Guidelines
The following guidelines apply to plans to which CDIGS team members will be committed. In order to justify our continued CDIGS funding, we need to demonstrate that our plans are designed to have a high impact on the NSF user community and to be useful to these users. It is not enough to publish our plans openly and wait for others to comment on the plans. We need to obtain explicit and positive feedback on the plans from relevant users, and in as many cases as possible, obtain a sense (ideally explicit) that the products of the work will be used by specific projects or users. The guidelines below are intended to provide ideas for how to obtain this feedback.
Note that these are also general guidelines. The details of how to satisfy the guidelines are left to the project team members to design. It is expected that different types of plans will require different types of review (number of people consulted, projects consulted, detail of the review, commitments obtained, etc.).
- Applicability - We need community review to be performed for all CDIGS project plans. That is, each of the plans to accomplish a high-level goal, and each dev.globus project roadmap item that CDIGS is contributing to. For high-level CDIGS goals, the team member responsible for the goal is responsible for the review as well, pulling in others to help as appropriate. For dev.globus project roadmap items, the CDIGS lead for that dev.globus project is responsible for the review, pulling in others to help as appropriate.
- Community - The people who are consulted in the review should be appropriate for the goals of the plan. If the goal of the plan is to add a new feature to a component, for example, then the plan should be reviewed by users who we think will find that feature useful. If the goal of the plan is to refactor a component's implementation to make future feature additions easier, but the current work won't change the features that users see, then the plan should be reviewed by long-term users of the component who are likely to have future feature requests and other developers of the component (if there are any).
- Magnitude - As a general rule of thumb, plans that require a great deal of effort should be reviewed more thoroughly than plans that require little effort. Having said that, the "Community" guideline above probably takes precedence over this one. Still, if a plan requires a lot of work, and it doesn't look like many users will be affected enough to be good reviewers, then one could reasonably ask whether the plan is really worthwhile or not. (Maybe it is, but the rationale should at least be documented.)
- Commitments - Whenever possible (and we should be aggressive in this respect), we should obtain commitments from affected users that they will both benefit from the work AND will use the results of the work in their own work. We can not bind anyone to these commitments, of course, but we can at least document that we obtained the commitment and so had good reason to think that the work would be worthwhile.
- Methods - Planning teams are free to use any of a wide range of methods for their community reviews. Methods include but are not limited to: interproject meetings (e.g., a caBIG meeting or a TeraGrid meeting that we participate in), invited meetings/workshops/teleconferences, email review (open to the public or closed to invited participants), verbal review at a conference BOF or workshop (with written notes), formal interproject reviews (e.g., a formal request/response between OSG and CDIGS). Ideally, a combination of methods will be used. For example, a reasonable plan might be to hold a formal review with OSG and TeraGrid supported by meetings and email, invited email review from several other projects, and a public request for comments on public mailing lists.
- Scope - Reviews may be done individually per project goal/dev.globus roadmap item or lumped together as collections of plans. It is strongly recommended, however, that the scope of the review not exceed the range of issues that a single user would find useful. I.e., no "whole CDIGS project" reviews, other than the annual NSF review and perhaps the External Advisory Committee. A data workshop, for example, might review all of the data-related plans. A meeting with another project (e.g., TeraGrid) might only be able to review one or two specific plans.
- Responses - To be successful, a review must obtain explicit positive community feedback. "Silent approval" is not sufficient. Negative feedback does not necessarily mean that the plan is not worthwhile, but it should be taken seriously and must be documented.
- Documentation - Feedback from communities and commitments to use the products of the plan must be written down. It is not sufficient to have the information in your head: you need to record it someplace where others can get at it as well.