Minutes of the Software Development Group's business and strategic planning meetings held at Pain Relief Unit (Churchill Hospital, Headington) on 3 December 1999

Minutes approved on 5 April 2000

Present: Chris Cates, Mike Clarke, Jon Deeks, Monica Fischer (Convenor), Peter Langhorne, Mark Starr, Phil Wiffen

Apologies: Sonja Henderson, Julian Higgins

In attendance: Jini Hetherington (minutes)

Contents:
Welcome and apologies
Potential conflict of interest
Approval of minutes of the last meeting
Action points from the last meeting
RevMan 4 - feedback from RAG meeting
Update of MetaView
ModMan 4.0 - feedback from MAG meeting
Update on status of the CritMan Software Advisory Group
Link between Comprehensive Meta-Analysis and RevMan
Web archive of SDG, RAG and MAG minutes and other relevant documents
Principles for software development and tests - experiences from the development of RevMan 4, MetaView 4 and ModMan 4
Next generation of Cochrane Information Management System (CIMS)
Remit and membership of the Software Development Group
Possible collaboration with the Campbell Collaboration
Any other business
Next meeting

BUSINESS MEETING

Welcome and apologies

Monica welcomed everyone to the meeting, in particular the new member Peter Langhorne (representing the Steering Group), and thanked Peter for agreeing to chair the strategic planning part of the meeting. Apologies were received from Sonja Henderson and Julian Higgins (new representative for the Statistical Methods Group); Jon Deeks attended the meeting in place of Julian.

Potential conflict of interest

Everyone was asked to declare their potential conflict of interest bearing in mind the general remit of the Software Development Group and the items on the day’s agenda.

• Chris Cates: none known
• Mike Clarke: as editor of the Reviewers’ Handbook
• Jon Deeks: involvement in the development of Metaxis (a statistical package developed by Update Software)
• Monica Fischer: involvement in the development of RevMan; funding of software development at the NCC dependent on decisions made by the SDG
• Peter Langhorne: none known
• Mark Starr: involvement in development of commercial software, e.g. Metaxis; as publisher of the Cochrane Library
• Phil Wiffen: none known

Approval of minutes of the last meeting

Minutes of the meeting held on 28 April 1999 was approved with two corrections relating to dates:

Item 5.b: change 3121 May to 31 May. Item 6b: change 156 June to 16 June.

Action points from the last meeting

Action points addressed elsewhere in these minutes or action points carried out without needing any further explanation are not mentioned below.

Item 4.f.: The issue of what do with old criticisms when a review is updated needs to be addressed. Should comments and criticisms remain with the review or be moved to a Web archive, for instance? Chris will follow up on this and liaise with Mike with respect to the Reviewers’ Handbook.

Action: Chris, Mike

Item 6.b.: The Nordic Cochrane Centre had been advised against submitting a proposal to the 5^th Framework. The proposed plans for the next generation of the Cochrane Information Management System would not be seen as innovative and ground breaking, as required by the relevant key action under the Framework.

Item 7.b.: There were concerns as to how the CRG glossaries would be seen as a useful addition to the Cochrane Library. In their current format, they would not be easily accessible to users. It was suggested that a central glossary for the Cochrane Library would be preferable to the individual ones. Mike and Mark agreed to look into this issue. Mark also raised concern about the usefulness of new, structured topics lists and gave an example of a topics list with 1000 topics with just 10 reviews attached.

Action: Mike, Mark

RevMan 4 - feedback from RAG meeting

The RAG had agreed to issue a letter of apology about the problems caused by the release of new software. It was agreed that the SDG, the NCC and Update Software should also sign the letter. Mike will draft the letter and circulate it for approval.

Action: Mike

There were some confusion about who submits synopses for publication. Do they come in via the reviews or can they be submitted as separate items? Mike will follow up on this issue.

Action: Mike

The RAG had gone through the latest wish list, ‘tidied’ this up and agreed on the things that should be implemented in RevMan 4.1. No further discussion was needed by the SDG.

Julian Higgins had done a survey of the Statistical Methods Group, which had resulted in a recommendation that Inverse-variance-weighted average should be available in RevMan. The RAG had endorsed this proposal (see also 12.1.a). Jon and Mike had agreed to prepare a paper addressing the issues surrounding including additional graphs in Cochrane Reviews that are not supported by MetaView.

Update Software had identified problems with links to Study IDs with identical reference IDs under additional references. For the time being it was agreed that a stronger warning should be added to RevMan. Mike will draft this and send to Rasmus Moustgaard. In the longer run, however, RevMan should not allow identical Ids. Jacob Riis should add this to the wish list for RevMan.

Action: Mike, Rasmus Moustgaard, Jacob Riis

The RAG endorsed the proposed changes to the funnel plot program.

1. Field testing of RevMan 4 and Meta View 4.1

   See item 6, below.

Update of MetaView

It was agreed that MetaView would be worked on further by Update Software and sent to the Nordic Cochrane Centre for testing. After any further necessary amendments have been made beta testing will take place: at least ten dedicated beta testers will take a full day to test the combined package of RevMan 4.1 and MetaView 4.1. This whole process should probably take about six months in total.

ModMan 4.0 - feedback from MAG meeting

Sonja and Monica will circulate the wish list items that the ModMan Advisory Group would like to see implemented, and after obtaining comments from the SDG, send this to Update Software and ask them to respond and estimate when a new version of ModMan could be available. The MAG should be advised that the general principles of software testing and release apply to the ModMan software as much as to the RevMan software. Communication between the MAG and Update Software should be channelled through the Software Development Group.

Action: Sonja, Monica

Update on status of the CritMan Software Advisory Group

Chris pointed out the duplication of effort between having an advisory group on software and also a group advising on implementation of the comments and criticisms system. For the time being, he will be monitoring the need for special, in-house software for managing received comments and criticisms through the Criteds mailing list. It had been suggested that the database held in San Francisco, which contains all submitted comments and criticisms and their replies, could be opened to all Comments and Criticisms Editors (not members of the public). Chris asked for the name of the person taking over from Mildred Cho as Convenor of the Criticism Management Group.


Action: Monica

Link between Comprehensive Meta-Analysis and RevMan

Monica had circulated a proposal from Michael Borenstein to offer users of CMA the ability to import reviews in RevMan format and then do additional analysis in this software. There were no objections to this proposal.

Web archive of SDG, RAG and MAG minutes and other relevant documents

It was agreed that the Nordic Cochrane Centre should create a web site archive to make this information publicly available on a web site linked to the Collaboration’s site. All group members must first approve minutes before they are passed to Jacob Riis at the NCC for publication. Monica will circulate a set of guidelines on how the minutes should be formatted (to prepare them for web publication) and on under which circumstances the minutes can be published.

Action: Monica

STRATEGIC PLANNING MEETING

Principles for software development and tests - experiences from the development of RevMan 4, MetaView 4 and ModMan 4

1. Input from various groups: RAG, MAG, SMG and others

   Input from several sources was summarised as follows:

   1. In future we should not work back from a deadline, rather we should work from when the updated software is fully tested and ready for release.
   2. During the implementation of new software individual reviewers could choose to work in two different versions of mandatory software. From the date of release of fully tested software they have one year to switch to the new software. CRGs should have longer to switch over, but they should bear in mind the problems inherent in doing so (this only applies to software in which structural changes in the document format are introduced).
   3. To recommend strongly that CRGs upgrade to keep in line with new and updated software releases, listing the advantages of doing so. The CRG would make the decision as to when to release the software containing structural changes to their reviewers. Software containing non-structural changes is optional as to when the CRGs should release it to their reviewers.
   4. Pre-release testing should include piloting.
   5. Funding should be set aside in future for testing and training. Implications for IT support within the CRG should be considered with each new release.
   6. Volunteers should be called for, for recruiting beta-testers to field test new software.
   7. As far as possible RevMan 4.1 would follow best practice for testing once appropriate beta-testers had been identified.
   8. Suggestions for new/changed statistical methods are not things that can be implemented quickly without consultation and discussion, preferably during Colloquia when complex statistical issues could be debated.
   9. Full documentation should be included before new software can be signed off for release.
   10. Specified individuals should be listed for the above process, people’s vested interests should be transparent and clear lines of accountability established.
   11. There should be a clear advance statement of milestones and target dates, and communication between all members of the project development team, especially if any slippage is foreseen. For the development of RevMan, for instance, named representatives of the Statistical Methods Group, the RevMan Advisory Group, the Handbook Group, Update Software, The Nordic Cochrane Centre and any other relevant groups, should all form part of the development team.
   12. All members should all sign off on software before it can be released.

Next generation of Cochrane Information Management System (CIMS)

1. Draft proposal for the next generation of the CIMS

   It was generally agreed that the Collaboration’s information management system needs to be rationalised for the future. Monica outlined a proposal prepared by the NCC to develop updated software. Mike reported that the RAG need further discussion and is prepared to meet again in February to discuss any proposal. It was cautioned that we need to find solutions, which work for both developing and developed countries. MAG had also asked for provision of central tracking software to be considered.

   There then followed a discussion about whether the next step should not first focus on assessment of user needs rather than proceeding to solutions. Mark asked what we can do to facilitate the exchange of information, i.e. agree on a standard data structure: we should come up with an agreed way of passing information around. Jon suggested that we should be asking reviewers how they do things and what tools they need to do them, rather than trying to decide for them. Peter asked whether the NCC could put together a funding proposal on a more prospective basis, focussing on functions needed rather than prescribing several different solutions. The users’/readers’ perspective should be of paramount importance.

   It was agreed that our approach to developing the next generation of software should include a period of reflection and examination of the problems before finding solutions. All the key players should be involved from the outset. The proposal should be for funding to undertake needs assessment for software for writing reviews (currently RevMan and MetaView) and for compiling modules of protocols, reviews and review group specific information (currently ModMan), and to develop and implement systems to accomplish specific tasks. It was recognised there are other groups in the Collaboration which are overlapped by the NCC grant proposal. It would be important to distinguish between software for which this group does and does not have responsibility. Comments and Criticisms, trial management, contact details management, the editorial process, and data archiving (raw data in a standard form, which is publicly documented) should be main components of the proposal. Monica should draft a proposal for the next generation of the Cochrane Information Management System based on the above discussion.

   Action: Monica

   There remained the need to consider interim solutions to existing problems:

2. Generic inverse variance

   Jon reported that the Statistical Methods Group had requested that the option to display generic inverse variance should be made possible in the software. This could improve several reviews, as the software is currently inadequate to the task. However, as this would require a structural change in RevMan, the current advice was to allow this information to be put into the text of the review. A proposal should go to the Steering Group to allow additional graphs to be imported, as recently agreed to in principle by the RevMan Advisory Group.

   Action: Jon, Mike

3. ModMan update

   It was agreed to plan for further development of ModMan and release of an update. The wish list would be circulated to the SDG to assess the importance of the items on it, before coming to a decision as to whether or what to implement from that list. The NCC proposed taking over the development of ModMan; however, it was recognised that the prime consideration is that ModMan should facilitate the easy incorporation of reviews into the Parent Database. More supporting information to back up the NCC’s proposal is needed before the SDG and Update Software can fully consider this proposal.

4. The future of the statistical analysis tool in RevMan

   The possibility was discussed that development of the software for analysing RevMan data should become the responsibility of the Nordic Cochrane Centre. This would happen after the revisions to MetaView 4.1 have been made. A firm proposal to this end should be submitted to the full Steering Group to make this decision. A written standard would be needed from Update Software, as to the existing display of the various statistical options.

5. New structure for ModMan for other entities

   Was not discussed.

6. Contact Database software

   The Norwegian branch of the NCC and the German Cochrane Centre have been addressing the fact that the Cochrane Databases, as maintained in HIREx, is not serving the the needs of the Collaboration. The working group from these two centres is planning to develop a new piece of software for this purpose, commencing with a pilot. The SDG flagged their concern that a needs assessment should be undertaken initially. Monica reported that Georg Koch had already undertaken a needs assessment surveying users in the Cochrane Collaboration, and reviewed existing commercial and freeware packages. Monica agreed to circulate the findings of Georg Koch’s survey on these issues.

   Action: Monica

Remit and membership of the Software Development Group

It was agreed to defer discussion to the next SDG meeting regarding altering its remit and expanding its membership. The reworked NCC proposal should be circulated to the SDG and the Informatics Group.

Action: Monica

Possible collaboration with the Campbell Collaboration

The SDG welcomes this collaboration in principle, recognising the need to avoid unnecessary duplication of effort.

Any other business

None.

Next meeting

The next SDG meeting should be linked to the February RevMan Advisory Group meeting, the date of this yet to be advised.

IMSG minutes | MinMan home