Minutes of the RevMan Advisory Group Meeting held at the UK Cochrane Centre on 12 May 1998

Minutes approved on ?

Present: Mike Clarke, Jon Deeks, Monica Fischer (minutes), Sonja Henderson, Rasmus Moustgaard, Andy Oxman (Convenor), Reive Robb, Mark Starr, Lindsay Stead.

Apologies: Clive Adams, Chris Cates, Lelia Duley, Kate Oldfield, Veronica Yank.

Contents:
Welcome and introductions and apologies for absence
Approval of minutes of last meeting and matters arising therefrom
Planned changes for RevMan 4.0
Plans for release of RevMan 4.0
Criticism Manager
RevMan training and support
RAG membership and convenor
Next meeting
Any other business
Appendix A

Welcome and introductions and apologies for absence

Andy welcomed everyone to the meeting and apologised that he had not been able to find substitutes for the people who had been unable to attend. It was agreed that the minutes of the meeting should be circulated for comments to the full group before approving them.

Approval of minutes of last meeting and matters arising therefrom

1. The minutes of the previous meeting (27 November 1997) were approved.
2. Item 2.b: Karen Stamm will be asked to include details in Cochrane News which will indicate that a reviewer can identify the major publication for an included study by marking the relevant citation with a *. This advice will also be included in the Handbook. Action: Andy
3. The Style Group (convened by Update Software) was asked to advice on which style should be used when including references in the text of a review (e.g. should the reference be in square brackets?); and how a study ID should be formatted (e.g. based on country or author?). Andy needs this information by the end of May if it is to be included in the next edition of the Handbook. Action: Sonja, Mark
4. Item 2.c: The ability to classify the `data source' of each outcome for each study, rather than for a study as a whole, will not be implemented in RevMan 4.0. This and the suggestion that studies where unpublished data have been used for particular outcomes could be marked will be added to the RevMan 5.0 wish-list. Action: Rasmus
5. An ability to mark a review as submitted for inclusion in ModMan should be included in RevMan 4.0 Action: Rasmus, Monica
6. Report from Statistical Methods Working Group (SMWG): Jon had surveyed the members of the SMWG concerning statistical issues in RevMan 4.0. The recommendations of the SMWG were accepted by the RAG. See Appendix A below.

Action items from the Appendix of the minutes of the last meeting:

7. Item A.24 and A.25: Printing problems in MetaView may not be solved in RevMan 4.0. However, Update Software is investigating if they can create a program that will generate bitmaps of graphs in MetaView and thereby resolve some of the issues about printing. The wish-list items concerning printing should be transferred to the wish-list for RevMan 5.0. Action: Rasmus
8. Item C.12: RevMan 4.0 will be able to compare two reviews and indicate the sections in which they differ, but the program will not be able to highlight the differences within these sections. However, with the ability to export the entire text of a review to an ASCII file, it should be possible to compare sections word by word in a word processor program.
9. A.1: Mike will finalise the specification for the `What's New' section of a review and pass this to Rasmus. Action: Mike
10. D.3: Guidance about the importance of considering harmful effects of interventions should be included in the next edition of the Handbook. Action: Andy
11. B.10: Jon and Doug Altman are finalising a paper commenting on the use of different statistical methods. The paper will in the first instance be published in Statistics in Medicine. Jon will investigate if the journal will allow the paper to be subsequently included in the Handbook (as was done with the report of the Individual Patient Data Methods Working Group Stats Med 1995; 14: 2057-79). Action: Jon

Planned changes for RevMan 4.0

Rasmus demonstrated RevMan 4.0 in its present state (mainly the new tree structure and the menus). It was agreed that the new interface seemed to be very user friendly and that it should be easy for new as well as existing reviewers to familiarise themselves with the new software. After the demonstration, the document `Changes to RevMan 4.0' was discussed. Comments noted during the demonstration and during the discussion of this document are listed below:

1. It should be investigated if the full text of a review can be pasted from a word processor (e.g. for reviewers who prefer to prepare all their text in a word processor rather than in RevMan). This should also be an advantage for Macintosh users. Action: Rasmus
2. The Style Group should consider the format of an ID for citations. In the present demonstration version of RevMan 4.0, citations are numbered 1, 2, 3, etc. in the tree structure, but this makes it difficult for the reviewer to determine quickly which citation is which. Citations IDs should be included to allow references to specific publications of a study in the text of a review. Rasmus needs to add this to the structure of a Review, and needs feedback on the length of the citation ID from Sonja and Mark by 15 June. (Post hoc note: the length of a citation ID for other references is 20 characters.) Action: Sonja, Mark, Rasmus
3. An ability to view all citations in a review should be added. Action: Rasmus
4. An ability to explode all levels of the tree-structure should be added. Action: Rasmus
5. Both the USA and the UK dictionaries should be included in RevMan 4.0 with an option to select the one for RevMan to use. Action: Rasmus
6. A formal proposal will be submitted shortly by The Style Group that only one title should be included in a review (and not both a short title and a long title). Therefore, it was recommended and agreed that the sort option should be by long title and not short title. Action: Rasmus
7. It is still not resolved if funnel-plots can be published in The Cochrane Library, but if not, an intermediate solution would be to make them available through the Database of Unpublished Data which is being set up in Oslo. Jon recommended that only funnel-plots of major outcomes are included in a review, but will confirm this with the SMWG. To implement Jon's recommendation, a tick-box would need to be added to the outcome-level to indicate that a funnel-plot should be calculated. Action: Jon, Rasmus
8. When the delete button is selected, the confirmation box should specify what the reviewer is trying to delete. Action: Rasmus
9. A system to allow unique numbering of reviews is still needed, but Rasmus reported that a solution is very near. It was agreed that it should be possible to make copies of reviews. Action: Rasmus, Mark
10. If possible, an undo option should be included on the toolbar (which would work for more than just the text). Action: Rasmus
11. If there are suggestions to word the menu title `Action' differently, please let Rasmus know asap. Action: Everyone
12. It was noted that eventually a sub-set of co-reviewers' contact details should be published on the Cochrane Library (name, institution and country). However, for this to be implemented CRG modules should include contact details of co-reviewers as well as primary reviewers. Monica will pursue this. Action: Monica
13. Sonja mentioned a problem with the pick-list of countries under sources of support in the current version of RevMan. She will report further on this to Monica. Action: Sonja
14. An ability to allow word count and spell checking of a section of text that has been blocked should be added. Action: Rasmus
15. It was confirmed that the synopsis of a review will be mandatory for new as well as old reviews. Synopses of many reviews have already been prepared by Hilda Bastian in collaboration with the reviewers and their CRGs.
16. The heading `Previously published version of this review' should be renamed to `Other published versions of this review'. Action: Rasmus
17. The column-size for tables should be stored to aid display in the Cochrane Library. Action: Rasmus
18. A document describing what happens when exporting a review from the 32 bit version of RevMan and importing it into the 16 bit version of RevMan 4.0 (if this gets implemented) should be written. However, a 16 bit version of RevMan 4.0 will only be implemented if there seems to be a justified need for this. A survey will take place later in the year to determine this. Action: Rasmus, Monica
19. The Software Development Group is finalising a proposal to change the summary-screen, which will be circulated shortly to all entities. The proposal, along with feedback from the consultation, will be passed to the CCSG. Action: Rasmus, Monica
20. Jon informed the group that documentation for the statistical methods in MetaView was ready. The document is currently in Latex format, but Jacob Riis at Nordic CC will look into converting this to a more commonly used format. Action: Jon, Jacob Riis
21. It should be possible to select/deselect printing of notes which will not be published. It should also be possible to print just those sections which are relevant to protocols. Action: Rasmus
22. Guidance is needed in the Handbook about how to deal with outcomes with no data. For example, should they be included in the table of comparisons or simply noted in the text? Action: Andy, Mike
23. A suggestion has been made that `Conclusions' should be renamed `Author's conclusions'. Andy will follow-up on this with the CCSG. (Post hoc note: The suggestion was approved by the Executive CCSG on 14 May.) Action: Andy
24. The heading `Citations' should be removed from the tree-structure. Action: Rasmus
25. Guidance will be needed on how to cite different types of information (including unpublished data) in the new structured reference. Action: Mike

Plans for release of RevMan 4.0

It is planned that an alpha version of RevMan 4.0 will be available and demonstrated at the Colloquium in Baltimore. This Alpha-version should be tested by the members of the SDG and the RAG before the next RAG meeting on Monday 30 November. The beta-testing is planned to take place during the months of March, April and May. RevMan 4.0 should be released on 15 July 1999 (if all goes well!). Conversion of old reviews should take place during the last week of July (when it is planned to have a `hot line' to assist reviewers and review group co-ordinators [RGCs] to convert reviews).

The last submission date for CRG modules with reviews in the old format will be brought forward to 14 July 1999 (for publication in The Cochrane Library, Issue 4, October 1999) to avoid RGCs having to cope simultaneously with the preparation of their module submission and the arrival of RevMan 4.0. The submission date for CRG modules with reviews in the new format will then be 27 October 1999 (for publication in The Cochrane Library, Issue 1, January 2000). Thus, each CRG will need to have converted their module to the new format by this deadline if their module is to be published in the first issue of the year 2000. Update Software will not automatically convert modules for CRGs.

In conjunction with the next RAG meeting, a specially convened RevMan 4.0 Documentation group will also meet. As well as looking at documentation for RevMan 4.0, the group will map other sources of documentation which refer to RevMan that need amending (such as the Handbook). The initial membership of the Group is: Mike Clarke, Jon Deeks, Monica Fischer, Claire Glenton, Rasmus Moustgaard and Jacob Riis.

It was noted that it may be easier to distribute RevMan 4.0 on CDs rather than on floppy disks. Monica will look into this and investigate if we can find a sponsor to pay for the production of the CDs. Action: Monica

Criticism Manager

Andy reported that discussions were taking place to move forward the development of CritMan and the Criticisms Management System. In brief, the issues mainly concern complex versus simple systems. A teleconference is being planned with the San Francisco Cochrane Centre and the SDG.

RevMan training and support

FAQs for both software issues and methodological issues are now being maintained. Tor-Arne Bertheussen is looking into setting up discussions lists with archiving facilities.

A plan for training CRGs and reviewers in the use of RevMan 4.0 should be drawn up. Action: Mike, Monica

Reive commended Rasmus for his willingness and speediness in providing support to reviewers and CRGs.

RAG membership and convenor

Andy explained that he has accepted to become Chair of the Cochrane Collaboration from October 1998 and for this reason would not be able to continue as either convenor or a member of the RevMan Advisory Group. Andy thanked people for their contributions to the Group's work during his convenor-ship. He was thanked for good leadership and was wished all the best in his future involvement in the Collaboration. Mike was nominated to be the new convenor and was elected unanimously. It was agreed that Mike should investigate expanding the membership of the Group to ensure attendance by more people at future RAG meetings.

PLEASE PASS NOMINATIONS TO MIKE AS SOON AS POSSIBLE.

In the future, if a member is unable to attend a meeting s/he should suggest a substitute to Mike. The substitute could then be invited - at the discretion of the Convenor - to come to the meeting if there is likely to be low attendance. Action: Mike, Everyone

Rasmus Moustgaard, who is responsible for software implementation of RevMan, was accepted as a full member of the Group.

Next meeting

30 November 1998, 11.00 - 16.00 at the UKCC seminar room. The RevMan 4.0 Documentation Group will meet from 9.00-11.00 and again from 16.00-17.00 on that day.

Any other business

None.

Appendix A

Summary of the Software Development Survey of the Cochrane SMWG

Sixteen people responded to the request for comments concerning developments to the RevMan software when these results were compiled. Several people have responded since, all in a similar vein. As had been anticipated, the responses varied considerably.

(1) Data Checks

The comments received were notable for their lack of consistency! Remarks ranged from "the drug industry requires duplicate entry as routine, so should Cochrane", to "reviewers have to take some responsibility for the quality of the data entered- they should not rely on automated checks".

Overall views were
(a) that these matters are more an issue for education/awareness raising than for software.
(b) double checking of data is more important, and reviewers should be encouraged to work as a pair undertaking data extraction and data checking independently
(c) that the suggested checks are not foolproof, and may give a false sense of security
(d) that the checks serve to generate warnings rather than to identify definite errors
(e) that extra data entry will hack people off
(f) if checks are implemented, then there should be some way of switching them off
(g) for only one of the items (sd/se check) were more people in favour than against

Proposal to the Software Development Group
That most of these issues are issues which can best be dealt with in the Handbook and not in software. There presently is no mention of checks in the Handbook.
(a) that the SDG proposes that the Handbook is updated to include a section on data checking where the importance of duplicate data extraction and data checking is emphasised.
(b) that advice on the extraction of continuous data in included in the Handbook.
(c) that checks for continuous data are included in a new section of the Handbook, where their limitations and full implications can be best explained.
(d) that if checks are ever implemented in the software in the future they must be easily disabled.

(2) Calculation of NNT

Again, there were lots of inconsistent comments, and no great enthusiasm. General points were: (a) if NNTs were available they are likely to be inappropriately used
(b) lots of untested assumptions about constancy of effects are made in the calculation of NNTs (whichever method is used)
(c) research (presently ongoing) is needed to look at the constancy assumptions issues, and the effect of pooling trials of various lengths of follow-up
(d) many reviewers are currently not even recording the length of follow-up for their trials, so thinking about NNTs is presently dangerous
(e) NNT should be calculated on the basis of overall RR or OR
(f) NNT should be calculated on the basis of overall RD
(g) providing a range of estimates according to baseline risk likely to be more useful
(h) presentation of confidence intervals may give undeserved credibility to an NNT
(i) the best use of NNT is for the application of results to an individual patient and so it should not be calculated as part of a Cochrane review

Proposal to the Software Development Group
There is no consensus on the methods for calculation of NNTs in systematic reviews, or the validity of the NNT as a summary statistic in systematic reviews. These issues are currently being researched.
(a) It is too early to recommend inclusion of NNTs in the next version of RevMan.
(b) However, it would seem sensible to recommend that a section on the calculation of NNTs is included in the Handbook, where their limitations could be explained, and methods given for those who wish to compute them by hand.


(3) Calculation of the significance of between group analyses

Strong views were sounded that these tests were not correct in the many situations where residual heterogeneity was a problem, which could easily be the majority. Two alternative methods which incorporate the residual heterogeneity were mentioned, and require futher investigation to gauge the complexities of the algorithms needed to programme them.

Other general comments were:
(a) Its a bad idea to automate these tests- there should be no enticements to commit subgroup analysis!
(b) users should be aware that these tests have very low power

Proposal to the Software Development Group
The simple form of this test using heterogeneity statistics is wrong in situations where there is excess heterogeneity- a version which accounts for this is needed before any test is implemented.
(a) that we look into the feasibility of programming alternative methods before making a decision as to whether it should be included in RevMan 4.0. There is a strong possibility that it will require more complex programming than is currently used in the whole of MetaView.
(b) that if the test is implemented, its is only used on the request of the reviewer.
(c) that the Handbook includes guidance on the interpretation of the results of these tests, especially noting the problems of their low power

(4) Funnel plots
Nearly everybody agreed that 1/se is preferable. A few people wanted both 1/se and n.

Proposal to the Software Development Group
That 1/se is used for the vertical axis of the funnel plot.

(5) Other recommendations which will go along with this are:


• that P-values are presented in addition to the heterogeneity and effect statistic, and that they are clearly labelled on the MetaView plot.
• that the Mantel-Haenszel relative risk replaces the Peto odds ratio as the default method for dichotomous data
• that the summary screen is revised to show appropriate statistics (does not show weighted mean differences when only the standardised mean difference is valid, etc.)