Minutes | Organizational Effectiveness Committee (OEC) Meeting 12 March 2017

OEC Attendees: Rinalia Abdul Rahim – Chair, Lito Ibarra, Khaled Koubaa, Markus Kummer and George Sadowsky

Invited Observers: Becky Burr, Chris Disspain, Asha Hemrajani and Leon Sanchez

Executive and Staff Attendees: Susanna Bennett, Michelle Bright, Xavier Calvez, Samantha Eisner, Larisa Gurnick, Lars Hoffmann, Aaron Jimenez, Melissa King, Margie Milam, Wendy Profit, Laena Rahim, Lisa Saulino and Charla Shambley

---

The following is a summary of discussions, decisions, and actions identified:

The Meeting was called to order at 4:11 p.m. local time in Copenhagen, Denmark.

1. Agenda – The Chair established the agenda for the meeting and gave an overview of items to be discussed. The Chair noted that the approval of preliminary reports and open action items have been excluded from the meeting agenda and the OEC will approve them via e-mail; so as to enable the OEC to focus specifically on review-related topics.

2. Ongoing Organizational Reviews Update – The OEC was provided with a brief update on several of the ongoing organizational reviews, for the OEC's information.

   GNSO Review: In implementation phase; no updates.

   At-Large Review: ITEMS International ("ITEMS"), the independent examiner, submitted their Draft Report for Public comment on 31 January 2017, after extensive consultation with the At-Large Review Working Party on the examiner's initial findings and recommendations. The public comment period closes on 24 March 2017. ITEMS will incorporate feedback from the community and will publish their Final Report in May 2017. There is also a public session scheduled during ICANN59 to discuss the details of the report.

   ASO Review: Review is in process; no new updates.

   Nominating Committee ("NomCom") Review: ICANN Org is in the final planning stages for the review. The Request for Proposal was launched on 19 January 2017 and closed on 17 February 2017. ICANN Org is requesting the OEC to review and confirm the scorecard, in preparation for reviewing and evaluating proposals and completing the procurement process. This agenda item was discussed in more detail under Item 4 below.

3. Organizational Review Streamlining and Recalibration – Organizational Reviews are central to ICANN's continuous improvement, accountability and transparency, and legitimacy of the multistakeholder model. An effective Organizational Review program provides important checks and balances. There have been community concerns raised about the effectiveness of the program, and as a result, the OEC received a report on several concerns raised by the community and the potential solutions to those concerns. These solutions may be appropriate to integrate now, or for consideration in the next review cycle. 

   Issue: Examiners - The community has voiced their concern that examiners do not understand the unique nature and nuanced balance of ICANN organizations (SO/ACs) and do not give credit and recognition to the extensive efforts of the volunteers from ICANN organizations. Interaction between the independent examiners and the community could be further improved to make sure that both parties understand each other; and each other's roles and responsibilities.

   Along with focusing on reviews as a value-add to the community that is undergoing the review, rather than a punitive activity, there could be further reliance on external standards/frameworks to help understand the quality of examiners and the quality of their work relative to a particular standard.

   One of the areas where the community is concerned about the work of the examiners is when recommendations are developed that are considered by the community as either not helpful, or as harmful. A potential solution is to divide the examiner's work into two phases, and separating the assessment from the recommendation phase. During the first phase, the examiner would focus on developing the problem statement and conducting an assessment and providing the community with their views on what the current state is and areas that require improvement. In the second phase, after testing and socializing the problem statement, the examiner could then focus on coming up with potential solutions (the recommendations). This staging of phases could be implemented in the current cycle of reviews, for any that have not reached a recommendation phase.

   ICANN Org reiterated that there is not expected to be a financial impact from this staging, as the same examiner would be used for both the assessment and recommendation phases. The OEC noted that this option provides an avenue to ensure that if there are issues of misunderstanding, they can be corrected halfway through the review, and guidance can also be provided by the independent examiner to the organization under review. The OEC agreed with this division in principle, and requested the development of a document that provides a clear path forward on how this two-phased review would work.

   Other potential solutions, which include the potential splitting of roles for assessment and recommendation phases, and examination of a model, similar to the EFQM excellence model, would each require longer-term feasibility assessments.

   Issue: Community Bandwidth - The Organizational Reviews of the Root Server System Advisory Committee (RSSAC), Security and Stability Advisory Committee (SSAC) and the Country Code Names Supporting Organization (ccNSO) are each scheduled to initiate by August 2017. The OEC discussed potential schedule modifications in light of community workload concerns and other feasibility concerns being raised.

   The OEC discussed that, based upon a July 2015 resolution of the ICANN Board, the SSAC and RSSAC reviews had already been postponed from their anticipated starting dates under the Bylaws. The OEC agreed that it would discuss with each of these three groups scheduled for an Organizational Review whether the group was ready to proceed with their review as scheduled, and if not, how to assure a schedule that would initiate the reviews within a set period of time. If any group that identified that a deferral would be preferable and proceeding with the scheduled review may not be feasible, public comment would be sought on the deferral, for Board consideration and decision. The OEC noted that, given the community workload and resourcing concerns under discussion across the ICANN community, it might be appropriate to defer some or all of these upcoming Organizational Reviews if the community agrees.

   • Action Items:

     • ICANN Org will provide an update on initial research into applying a widely accepted organizational excellence model and framework as a potential solution to assist with the streamlining and recalibration of Organizational Reviews, for the next cycle of Organizational Reviews.
     • In relation to the proposal of separating the assessment phase from the recommendation phase (to be applied to the next possible review i.e., NomCom Review), ICANN Org to prepare a comprehensive proposal, codifying the relevant steps involved to ensure that the proposed processes are set out in a clear manner.
     • OEC Chair to reach out to each of the three entities scheduled for an upcoming Organizational Review to determine whether the entity wishes for its review to proceed as scheduled.
     • For any entity that identifies that review is not feasible at this time, ICANN Org to seek public comment to inform later OEC recommendation and Board consideration.

4. Organizational Reviews: Reviewer Assessment and Scoring: The OEC received a briefing on the standard procurement scorecard used to evaluate proposals received in response to RFPs for independent examiners. ICANN's procurement process is in line with industry standards and best practices and ensures confidentiality, impartiality, fairness and discretion to the bidders and finalists.

   The OEC questioned how it is possible to evaluate an applicant's understanding of an assignment based upon a review of their response to an RFP. ICANN Org explained that this is measured by looking at the totality of information received from the bidders, including written responses to the questions in the RFP, the supplementary documents produced, interviews with the short-listed bidders, and how they present their proposed methodology.

   The OEC provided feedback to ICANN Org that "language" should be incorporated as a criterion in the scorecard. The OEC agreed with ICANN Org's proposed scorecard methodology, incorporating the recommended changes.

   • Action Items:

     • ICANN Org to prepare a proposal as to whether the scoring methodology for the selection of independent examiners should be modified or remain unchanged.
     • ICANN Org to incorporate "language" as a criterion under Item 4 of the evaluation scorecard for examiners.

5. Specific Reviews – Planning for OEC's Oversight Role: The OEC had a discussion to start planning for the expected approval of the expansion of its role into oversight of Specific Reviews under the ICANN Bylaws. ICANN Org identified a few issues that are pressing in terms of oversight and guidance. First, ICANN Org discussed the challenges faced in the new selection process for review teams, which has shifted to the SO/AC leaders, but there is no community agreement on the selection process. This results in a lengthy process to identify and select review teams. As part of its oversight role, the OEC also needs to consider that diversity goals are met – however, currently there is no process to deal with geographic and gender imbalance.

   Another issue that the OEC could provide some input on is the management of resources required for Specific Reviews. Given the fiscal impact that these Specific Reviews pose, with up to 21 Review Team members, potential travel needs and resource needs, as well as general ICANN Org support, there may be a need to move from fiscal-year based budgets to a multi-year project budget that will be managed by ICANN Org in collaboration with the relevant Review Team. ICANN Org will identify where the OEC might have some inputs or oversight assistance with this.

   Finally, ICANN Org highlighted the impact on the community posed by the review schedule, requiring intense participation from Review Team members as well as active engagement with the community. Therefore, to ensure that there is quality and sufficient participation in Review Teams, the OEC will need to consider ways to support the Review Teams to maintain a narrow focus on relevant issues.

   Given the overarching issues raised above, the OEC Chair emphasized the importance of finalizing the Operating Standards document. Currently, there is not a consolidated set of guidelines or standards for the Review Teams to adhere to. Establishing and documenting a set of concrete guidelines will provide some clarity to the community on the schedule for reviews and further assist them to understand the workload and processes involved.

   The OEC will review the strawman document on Operating Standards which will be drafted by ICANN Org and thereafter, will be shared with the community for their feedback and eventually be adopted as the standards to guide Specific Reviews.

   • Action Items:

     • The OEC will seek feedback from the relevant organizations to ascertain whether there is any inclination to defer their respective upcoming Organizational Reviews. If they are amenable to deferring their reviews, the next step for each would be opening a public comment to inform later Board consideration of such deferral.
     • ICANN Org to develop a strawman document on Operating Standards, for the OEC and community's feedback and input.

6. Specific Reviews: Briefing – Due to time constraints, this agenda item was not discussed during the meeting. ICANN Org requested the OEC to refer to the meeting materials for an update on the status of Specific Reviews underway or soon to be initiated, i.e. ATRT3, CCT, RDS and SSR2.

7. Any Other Business

   • ICANN Org to advise the OEC Chair whether a face-to-face meeting is necessary before and/or during the Board's upcoming workshop in Geneva, and the duration required for each meeting.
   • The OEC to work towards obtaining Board approval of the modification of the OEC Charter to include its oversight of Specific Reviews.

   The Chair called the meeting to a close at 05:31 p.m. local time in Copenhagen, Denmark.

Published on 04 May 2017