Third Accountability and Transparency Review

The Third Accountability and Transparency Review (ATRT3) was initiated in January 2017. By May 2021, the review was in the plan implementation phase. By April 2023, four of the 15 Board-approved ATRT3 recommendations have been completed, including improvements for collecting community feedback and suspending future RDS and SSR reviews until the next ATRT considers their futures. Ten out of the 15 recommendations are currently being worked on, including the re-examination of the ATRT3 implementation, which is expected to be completed by ICANN 77. The key recommendations at play include Recommendation 3.2, which proposes another CCT review, limited to one year, starting within two years after the next round of new gTLDs, with data collection beforehand; Recommendation 3.4 is that ATRT Reviews should continue suggesting terminations, amendments, or creations of periodic reviews. Recommendation 3.5 creates the ICANN Holistic Review, based on Operating Standards for Specific Reviews to improve SOs/ACs/the NomCom practices, collaboration mechanisms, and accountability. Recommendation 3.6 is that Organizational Reviews should evolve into ICANN Org-established improvement programs in each SO/AC/NC, including an annual satisfaction survey, regular assessment, and funding. This recommendation has not yet been started due to a dependency on an ongoing initiative, the Pilot Holistic Review, before developing the Pilot Continuous Improvement Program. Recommendations 3.2 and 3.4 require modifications to the ICANN Bylaws and are being worked on, while the implementation of Recommendations 3.5 and 3.6 may impact Recommendation 3.2. Work on the Pilot Holistic Review is ongoing and will include a second Public Comment period with a focus on the lack of agreement within the community. Recommendations 4.1-4.5 are being worked on to enhance ICANN's accountability and transparency through the evaluation of metrics and targeted outcomes. Finally, Recommendation 5 is being addressed through the Prioritization framework.

Background

Purpose and Origin of Specific Reviews

The Affirmation of Commitments, an agreement between ICANN and the United States Department of Commerce, establishes ICANN's obligations to perform its duties with specific commitments in mind. All of the commitments bear on public and consumer trust of the organization. ICANN is to perform its functions in a manner that:

• ensures accountability and transparency of decision-making;
• preserves the security, stability, and resiliency of the DNS;
• promotes competition, consumer trust, and consumer choice; and
• enables access to registration data.

It is also charged to periodically review and assess its performance through the lens of each of the above commitments.

ICANN's board enshrined these commitments (and the associated reviews) in its Bylaws in Article 1 (Mission, Commitments, and Core Values) and in Article 4 (Accountability and Review). Article 4.6 deals with "Specific Reviews," each of which is tied to one of the commitments in the Affirmation of Commitments.

The Organizational Effectiveness Committee of the board oversees the conduct of specific reviews. The ATRT is one such specific review.

Delays in Launching Substantive Work

The call for volunteers for ATRT3 was initially put out in January 2017. It was subsequently extended in April 2017. At the time of extension, the Board noted that multiple factors were impacting the timing of ATRT3. Because of the timing of ATRT2, the ICANN Bylaws required that the review be launched no later than October 2017. Meanwhile, the Cross Community Working Group on ICANN Accountability (CCWG-Accountability) notified the Board in 2016 that its work was overlapping substantially with the scope of ATRT reviews. The CCWG-Accountability co-chairs raised the possibility of crafting a narrow scope for ATRT3, to avoid duplication of work. At the same time, the IANA Transition was just wrapping up, and many voices within the ICANN community were speaking up about volunteer fatigue. As a result, in 2018 the Board submitted three separate papers for public comment: "Short-Term options to Adjust the Timeline for Specific Reviews," "Long-Term Options to Adjust the Timeline of Reviews," and a follow-up "Next Steps on Reviews" document.

In October 2018, the board approved a resolution to instruct ATRT3 to begin work, encouraging them to promptly submit their work plan and begin substantive work in January 2019. It also appointed Maarten Botterman to the ATRT3 team.

Review Team

The ATRT3 review team held its first in-person meeting in April 2019 to set the scope of the review and establish its work plan. Unlike previous ATRT reviews, the review team did not identify a need for an independent expert to consult. Instead, they identified four top-level issues for the review team's work parties to address:

1. Board governance and community interaction;
2. GAC interaction with the board and the broader community;
3. Review processes - how ICANN receives public input, as well as a review of the implementation of recommendations from ATRT2;
4. The impact of ICANN's decision-making process as it relates to the broader Internet community and the effectiveness of ICANN's policy development process in fostering cross-community participation.

Methodology

The ATRT3 team utilized a number of fact-finding methods in order to gain a picture of the current state of ICANN's accountability and transparency initiatives:

• Review and assessment of ATRT2 implementation;
• Within each topic area, the work parties devised questions for a survey that was issued to the ICANN community, and a survey specifically aimed at ICANN structure (ACs, SOs, GNSO constituencies, and [[Regional At-Large Organization|RALOs) participation;
• Conducted interviews at ICANN 65 and ICANN 66, as well as public sessions at ICANN 66;
• Received briefings from various groups, including ICANN org's Public Participation team and the NomCom2 implementation working group;
• Reviewed the ICANN Accountability Indicators in detail;
• Reviewed many ICANN documents; and
• Requested and received numerous clarifications and details from ICANN org and staff

The survey received 88 individual responses (with 50 of those responses completing the majority of the survey) and responses from 14 of the 17 specifically invited ICANN groups.

ICANN 66 Sessions

ATRT3's sessions at ICANN 66 took place in parallel with the wider community investigation of improvements for Article 4 reviews. ICANN staff facilitated a conversation around enhancing the effectiveness of review recommendations and their implementation, where Cheryl Langdon-Orr and Pat Kane from the ATRT3 review team presented early impressions of the team in relation to review process. At its meeting with the Board Caucus, the review team presented its findings, suggestions, and recommendations to date. At the time, the team was considering three options for the reform of Article 4 reviews:

• Constitute a single permanent entity in ICANN to coordinate reviews as they currently stand and independently assess implementation of recommendations;
• Replace all Specific Reviews with one review and all Organizational Reviews with one review; or
• Replace all Specific Reviews and all Organizational Reviews with one review.

The proposals generated conversation among board members and review team members regarding how best to implement the reforms required for each option. The review team also had provisional proposals regarding diversity and representation, public comment issues, and other issue areas identified in the draft report. There was fruitful conversation about each of those issues.

On the same day, ATRT3 presented an update on its progress to the GAC. The meeting included a similar presentation of findings and provisional recommendations related to the "GAC Interaction" work track.

Finally, the review team held an engagement session for the general community. The presentation focused on the five topic areas, and elaborated on the review team's impressions regarding specific and organizational reviews. The team suggested that multiple factors may be present in the near-universal dissatisfaction with organizational reviews, and the less intense, but still substantial dissatisfaction with specific reviews:

• Lack of coordination and overlap areas under review by different teams, sometimes resulting in conflicting recommendations;
• Too many reviews in total, and too many reviews going on at once;
• Competition for ICANN resources at the recommendation & implementation phase;
• Lack of time or resources to effectively conduct the reviews;
• Failure to properly implement some recommendations and report such implementation failures; and
• Structural and bylaws-specific obstacles to having a systemic and holistic viewpoint

Draft Report

The four work parties presented findings that were consolidated into a draft report, published for public comment on December 14, 2019. The draft report proposed substantial changes to ICANN's processes around specific and organizational reviews, prioritization of improvements, and public participation.

Findings

The draft report contained a number of findings with regard to the four topic areas described above, as well as its review of ATRT2 implementation. The report's executive summary listed several notable findings:

ATRT2 Implementation - Contrary to ICANN's reporting on its website, which identified implementation as 100% complete, the ATRT3 team found that: 60% of the recommendations in the ATRT2 report had been fully implemented; 23% of the recommendations had been partially implemented; and 17% of the recommendations had not been implemented.

Article 4 Reviews

• Only 16% of the ICANN Structures responding to the survey found specific reviews (Article 4.6 of the ICANN Bylaws to be "effective" or "very effective;"
• 78% of individuals and 91% of structures agreed that specific reviews should be reconsidered or amended;
• 46% of structures found organizational reviews (Article 4.4 of the ICANN Bylaws) to be "effective" or "very effective;"
• 85% of individuals and 83% of structures agreed that organizational reviews should be reconsidered or amended.

Public Participation & Community Interaction

• An overwhelming majority (100% structures, 82% individuals) believe that the information compiled on ICANN.org should be better structured and organize to improve searchability and access;
• 50% of individuals thought that public comment proceedings were "effective" or "very effective" at gathering community input, and 88% of individuals were in favor of re-examining the concept of public comments.

Board Decision-Making and Accountability

• 54% of the structures were unaware of the existence of ICANN Accountability Indicators, and 67% of the responding structures found the Accountability Indicators "somewhat ineffective;"
• A large majority of the community believed that the ATRT3 team should make recommendations regarding prioritization;
• There was a similar broad agreement that there should be a mechanism for retiring recommendations if they become moot or do not seem likely to be implemented;
• There was near-unanimous agreement that the ICANN community should have a role as a decisional participant in any mechanism or process that makes recommendations regarding the prioritization of work.

GAC Interaction with the Board and Community Although not included in the executive summary as "key findings," the draft report noted a strong consensus among the structures responding to the survey that GAC interaction, transparency, and collaboration had improved significantly. Individuals had a less favorable view, but the draft report concluded that there might be closer attention to recent improvements in the GAC's communication strategies among leaders of SOs, ACs, constituency groups and RALOs.

Recommendations and Suggestions

The draft report made a distinction between recommendations and suggestions:

The ATRT3 is focused on ensuring that its recommendations meet the requirements as set out in the Operating Standards for Specific Reviews while suggestions may not necessarily meet this standard.5 The ATRT3 does not consider suggestions to be less important than recommendations. The determination if an item is a suggestion or a recommendation will be finalized in ATRT3’s final report.

The report's preamble highlighted five topics that it considered to be high priority for both community discussion and feedback, and implementation of recommendations or suggestions.

1. ATRT2 Implementation Issues - A number of suggestions specifically address the failure to implement recommendations from ATRT2. In addition, the report recommends that in-progress specific reviews appoint and empower implementation shepherds as recommended by the Operating Standards for Specific Reviews. In addition, the report recommends ensuring that implementation is actively tracked on the ICANN website, and that if any changes are made to the process of tracking and reporting implementation, that legacy web pages and other resources are updated to reflect the changes and provide the location of the new processes.
2. Prioritization of Work - The ATRT3 team took note of a variety of parallel efforts to improve prioritization and rationalization of work, and concluded that "only a community-led process can legitimately develop a system for prioritizing the implementation of reviews, CWG, and CCWG recommendations." The draft report provided suggestions regarding the creation of a development team that would create a prioritization process, as well as baseline requirements for that process.
3. Article 4 Reviews - Noting that it is best to approach the reformation of the review process "holistically," the draft report provided two possible options for addressing the myriad issues relating to specific and organzational reviews:
   1. Maintain the current set of specific and organizational reviews in combination with a new oversight mechanism to manage reviews and the implementation of recommendations. The new mechanism should be housed in an Independent Accountability Office (similar to the ICANN Ombudsman), which would have responsibility for SO/AC accountability as well as the job of coordinating the timing of reviews and the implementation of recommendations; or
   2. Maintain the spirit of Article 4.4 organizational reviews but conduct them as three- to five-day leadership workshops focused on self-assessment in a context of a continuous improvement process. The workshops would occur at least once every three years, or more often as desired by each organization subject to Article 4.4 review. Outcomes and recommendations from these workshops would be publically posted, and implementation would be tracked and reported on as well. The reports from these workshop reviews would feed into a new holistic organizational review of all SOs, ACs, and the NomCom. The holistic review would take place every seven years and have a maximum duration of twelve to eighteen months. The time between holistic reviews allows at least two cycles of self-assessment and improvement for each organization, as well as sufficient time to implement recommendations from the holistic review. For the Article 4.6 reviews, consolidate ATRT and the "accountability and transparency" directives of the CCT and RDS reviews into a single Accountability and Transparency review, to be conducted every seven years with a maximum duration of twelve to eighteen months. Conduct the SSR review as a three- to five-day workshop, or maintain the current review process for SSR reviews.
4. Public comment - Expand public comment activities to not only seek general input on entire documents. Clearly identify the "intended audience" for a public comment proceeding ("general community, technical community, legal experts, etc."), so that anyone may respond but specific communities are called upon to respond. Provide a summary of key questions in plain language, and include any responses to these key questions in the staff report on the public comment proceeding. Where appropriate and feasible, provide translations of a summary of the document, as well as key questions, and accept responses from speakers of official ICANN languages.
5. Accountability Indicators - Foster public awareness of the ICANN Accountability Indicators, including presentation of the indicators at an ICANN Meeting. The draft report "strongly suggest[ed]" that ICANN "rapidly undertake a serious review of its Accountability Indicators" to ensure: that they met their stated objectives; that they provided useful data; that they provided data that could inform a decision maker; and that the data for each indicator is current and up to date.

Public Comment on the Draft Report

The draft report received comments from many ICANN bodies, as well as the board. There was general support for the suggestions regarding the implementation of ATRT2 recommendations, the improvement and expansion of public comment modes and processes, and the need for prioritization. However, the creation of a community-led prioritization authority was met with skepticism and concern. The Board suggested that existing community mechanisms might be leveraged to provide ICANN's community with a voice in prioritization discussions. RySG, IPC, and other commenters feared the insularity of such an organization, as well as the perception that "ICANN insiders" were making decisions for the entire community. The two options regarding the reform of reviews each received many comments. While there was a general consensus that Option 1 was a non-starter (the ICANN Board, for example, stated that it "does not consider Option 1 (as described in the Draft Report) to be viable"), the responses to Option 2 varied widely in enthusiasm for, and faith in, such an option. Many commenters stated that the proposal would need considerable refinement, and some (including the board) suggested that the task may be too big for ATRT3's limited timeline. There was also pushback from the ICANN Board and others regarding the review team's decision to offer both "recommendations" and "suggestions."

The review team analyzed and incorporated the comments into the final version of its report, which was delivered to the board in May 2020.

Final Report

In its prologue, the ATRT3 final report highlighted several items of concern that occurred during the process of the review that the ATRT3, due to time constraints, was unable to substantially address. All of the events had some bearing on the accountability or transparency of ICANN's actions and leadership. The review team expressed hope that the events would be the subject of a future holistic review or ATRT review cycle. The events included the proposed change of ownership for the .org registry, the Expedited Policy Development Process on the Temporary Specification for gTLD Registration Data (EPDP), accountability and transparency issues related to DNS Abuse enforcement and policies, and COVID-19 "emergency" shortening of review cycles for budget and planning documents.

The report's findings were largely the same. The largest changes were seen in the recommendations made by the review team. The "suggestions" were removed, and the review team offered five recommendations that adhered to the "Operating Standards for Specific Reviews" model and methods. The five recommendations contain a large amount of detail; high-level summaries follow, but it is advisable for readers wishing to fully understand the rationales and desired outcomes for each recommendation to consult the final report:

High Priority Recommendations

1. Amending Article 4 Reviews:

• Specific Reviews should be handled as follows:
  • Suspend the RDS/WHOIS review until the EPDP on the Temporary Specification is complete;
  • Conduct one further CCT review, initiated within two years of the addition to the root of new gTLDs from a new application round. Limit the timeframe of the review to one year, and ensure that a framework for data collection is already in place and that a complete data set is available prior to the initiation of work;
  • Upon the completion of SSR2, suspend any further SSR2 reviews until the next ATRT review (or any new review that takes over the obligations of the ATRT) makes a determination whether SSR reviews should be amended, terminated, or kept as is. The ICANN Board will have the authority to launch an SSR review if it deems it necessary; and
  • ATRT reviews to continue, with recommended improvements, and with the next ATRT review starting no later than two years after the approval of the board of any recommendation from the first Holistic Review (see below)
• ICANN should establish and perform a new "Holistic Review" no later than one year after board approval of any recommendation from ATRT3. After that, alternate the timing of ATRT and Holistic Reviews, so that each review begins no later than twenty-four (ATRT) or thirty months after the approval of the first recommendation contained in the other review. (As in the draft report, this is designed to allow at least two "continuous improvement" cycles to occur within organizations between Holistic Reviews.)
  • No other reviews should be launched during the pendency of a Holistic Review;
  • Holistic Reviews shall last a maximum of eighteen months and operate in accordance with the Operating Standards for Specific Reviews;
  • Holistic Reviews will focus on: 1) continuous improvement efforts of SOs, ACs, and the NomCom; 2) Inter-organizational communication and collaboration; 3) accountability of the SOs, ACs, or constituent parts to their members and constituencies; and 4) reviewing the entirety of the ICANN organizational structure to determine if all the constituent parts still serve a purpose, or if there are opportunities to alter the structure(s) and operations of ICANN to better represent the community and improve the effectiveness
• Organizational Reviews should be transformed into "continuous improvement programs" within each structure currently subject to an Article 4.4 review.
  • Allowing for some flexibility so that each organization can optimize its approach to continuous improvement, each program will include annual satisfaction surveys of members/participants, annual assessment of the improvement program (which may either be self-directed or facilitated), and budgetary and staff support that at a minimum matches the operational support for the current organizational review process.
  • Results of the annual surveys and assessments will be published and will be reviewed during the next Holistic Review.

2. Prioritization of Review and CCWG Recommendations: Create an annual process of prioritization that gathers representatives from ICANN Board, ICANN org, and any SOs or ACs that wish to participate (one representative from each). This panel would operate by consensus, and would integrate into existing financial and strategic planning processes. The panel would be charged to establish a priority order for recommendations resulting from reviews, cross-community working groups, and any other community-sourced process the Board deems appropriate. Similar to the Operating Standards for Specific Reviews, the prioritization process shall employ the following decision-making guidance:

• Relevance to ICANN’s mission, commitments, core values, and strategic objectives;
• Value and impact of implementation;
• Cost of implementation and budget availability;
• Complexity and time to implement;
• Prerequisites and dependencies with other recommendations; and
• Relevant information from implementation shepherds (or equivalents)

Medium Priority Recommendations

3. Accountability Indicators & Strategic and Operational Planning Transparency: Scrap "Accountability Indicators" and instead provide plain language descriptions of the Board's strategic objectives, goals, and operating plans. Measure the outcomes of such plans and publish results in annual reports. Publish a summary report at the end of each five-year strategic planning cycle, with a description of objectives and goals, initiatives intended to meet those goals, and the success of such initiatives.

Low Priority Recommendations

4. Public Comment: Improve public comment proceedings as recommended in the draft report: provide a plain-language summary of the document being published; list key questions raised by the document and for which answers are sought; where appropriate and feasible, translate the summary and key questions into the official ICANN languages; and if relevant, identify the intended audience of the document or the key questions.

5. ATRT2 Implementation: Review the implementation of ATRT2 Recommendations in light of ATRT3’s assessment of these and complete their implementation subject to prioritization.

Minority Statements of ATRT3 Members

Three separate minority statements were submitted to an Annex of the final report. All three expressed concern regarding the proposed changes to Article 4 reviews, as well as the accountability and transparency of the changes between the draft report to final report. The concerns expressed in these statements were the subject of a meeting between the ATRT3 implementation shepherds and a board caucus. In the meeting, the co-chairs of ATRT3, as well as board liaison Leon Sanchez, member Sebastien Bachollet, and member Vanda Scartezini, all made similar points about the minority statements being more tied to the constituencies "behind" the authors, rather than the personal beliefs of the ATRT3 members themselves. The board addressed these statements, and the conversation between the implementation and the board caucus, in its deliberation of the final report.

Public Comment on the Final Report

In addition to publishing the final report for public comment, the Board specifically requested feedback regarding a section of the transmission letter from ATRT3:

...ATRT3 is proposing significant changes to Organizational Reviews and Specific Reviews. ATRT3 strongly suggests that the ICANN Board implement a moratorium on launching any new Organizational and Specific Reviews until it has made a decision on this recommendation.

The report generated a number of detailed public comments. The IPC, BC, and ISPCP comments echoed (and sometimes replicated) the statements made in the minority statements presented by their appointees to the ATRT3 review team. The GNSO Council also expressed concern regarding the removal of important external accountability measures contained within the Article 4 reviews. Most of the other respondents, including the ALAC and the ccNSO, agreed with the recommendations, with some taking issue with various specifics in the proposed implementation.

Board Action and Implementation

Following the public comment period, in November 2020, the ICANN Board approved the recommendations from the final report for action and consideration, with the caveat that implementation was "subject to prioritization" and, in some cases, would require additional community input and buy-in. In the attached recommendation scorecard, the Board acknowledged and approved the following recommendations, identified dependencies for each, and identified next steps for each:

Recommendation	Dependencies	Resources/Costs	Other Considerations	Action Items	Implementation Progress
Public Comment: 1.1 Plain language summaries, key questions, translation of same; 1.2 Explore alternate methods of receiving feedback and systematize the use of such methods; publish an updated public comment guidelines document	ICANN organization plans to launch certain features of its Information Transparency Initiative (ITI) by the start of Fiscal Year (FY) 2022, i.e. by July 1 2021. ICANN org’s Public Comment and ITI teams will provide training to all appropriate ICANN org functions to ensure readiness for the launch of the new Public Comment feature and to stress the importance of all guidelines on public comments.	Costs are already assumed in the operating budgets for ITI and Public Comment staff. Any additional costs to be assessed before including in a resourcing plan.	ICANN sought public feedback on the ITI initiative and new public comment systems. ICANN org’s Public Comment team also met with the ATRT3 team in late 2019 to provide an overview of the new Public Comment features. The new Public Comment features to be launched with ITI will enable improved tracking of those initiatives for which alternative feedback mechanisms were used.	Board approves these recommendations and notes that the ITI covers many of the action items contained in the recommendations; thus, implementation of recommendations should be subject to the ITI timeline. 1.1 and 1.2 may need to be separately tracked as they involve distinct work efforts.
ATRT2 Implementation: Implement incomplete recommendations from ATRT2	Further work and coordination are necessary between ICANN org and the ATRT3 Implementation Shepherds to understand more clearly what can be done to consider the ATRT2 recommendations fully implemented.	An understanding of the full scope of the implementation steps is needed, in order to estimate anticipated resources/costs.		The Board directs ICANN org to undertake a thorough analysis of the ATRT3’s suggestions pertaining to the implementation of ATRT2 recommendations, and to engage with the ATRT3 Implementation Shepherds regarding those suggestions to identify resource-effective means, where appropriate, to complete the implementation of the ATRT2 recommendations discussed in the ATRT3 assessment.
Specific Reviews: 3.1 Suspend any further RDS Reviews until the next ATRT Review can consider the future of RDS Reviews in light of the final EPDP report recommendations, the results of the Board’s consideration of these, as well as any other developments which affect Directory Services; 3.2 One additional CCT Review after the next round of new gTLD applications; 3.3 Suspend any further SSR Reviews until the next ATRT Review (or similar), which shall decide whether to terminate, amend, or maintain SSR Reviews as-is; 3.4 Continue ATRT Reviews with recommended improvements	These recommendations require amendments to the ICANN Bylaws, which require broad community support for the proposed amendments. There needs to be a set of objective, uniform criteria in place for a future ATRT team to utilize regarding decisions to terminate or amend reviews	Conducting a CCT Review: $2.2MM	This recommendation addresses some, but not all, of the issues with the current review mechanism, as identified by collaborative efforts of the board, org, and community. Aspects of the recommended implementation timeline may not be feasible, given the need to amend bylaws.	The Board approves Recommendations 3.1, 3.2, 3.3, and 3.4, subject to community agreement to the Bylaws change. When deemed appropriate through the prioritization process, the Board directs ICANN org to begin the process to make the appropriate Bylaw amendments, but if the Empowered Community rejects the Bylaws changes, further ICANN community discussion would be required before implementation. Further, the Board notes that there may be a need to track the implementation of Recommendations 3.1, 3.2, 3.3, and 3.4 separately due to the distinct work efforts and implementation steps required.
Holistic Review 3.5 Create a "Holistic Review"" in accordance with ATRT3 recommendations and existing ICANN guidelines for specific reviews"	As above, this will require amendments to the ICANN Bylaws, and thus requires broad community support for those amendments. ICANN Board also requires more information regarding specifics, including resourcing needs. However, the Holistic Review could be run as a pilot program, alleviating the need to amend the Bylaws, and also providing some practical experience regarding best practices, resources required, and pitfalls to avoid. Any pilot would still need widespread buy-in from the ICANN community. The Holistic Review concept has multiple potential dependencies and cross-over with other ICANN projects, including Work Stream 2, Board efforts to streamline the review process, and other recommendations from the ATRT3 team. Objective criteria will also need to be developed regarding the usefulness of this review process and whether to continue this or any other review.	Many specifics need to be nailed down to understand potential costs as well as resource support.	Simultaneous launching of continuous improvement mechanisms within ICANN structures and the Holistic Review will be challenging. Recommended timeframes for full implementation of these processes may be unrealistic. The framework and standards for continuous improvement efforts across structures need consideration and development to ensure apples-to-apples comparisons and scoring.	Subject to prioritization and available resources, the Board directs ICANN org to initiate the first Holistic Review as a pilot, and operated pursuant to community-agreed Terms of Reference and relevant elements of the Operating Standards for Specific Reviews.
Organizational Reviews 3.6 Evolve the content of Organizational Reviews into continuous improvement programs in each SO/AC/NC	Similar to Holistic Review: Bylaws amendment required, but a pilot Continuous Improvement Program could be run without amendment to Bylaws. The pilot would still require broad community buy-in. Also, pursuant to ATRT3's recommendation that org reviews be suspended, GNSO3's start date is an issue to be resolved.	Estimated cost of planning for and implementing Continuous Improvement Programs for all SO/AC/NCs (could include developing periodic surveys, advising on methodology, and changes over time) in terms of possible external consultants (plus ICANN org time TBD): $150,000-200,000 (one time cost). The estimated cost of supporting Continuous Improvement Programs’ annual review: $35,000-50,000 per year per structure (provided they elect to hire independent examiners).	Similar issues as above - simultaneous launch of evolution process and Holistic Review may be challenging. The recommended timeframes may not be realistic, and the transition period may involve some overlap with status quo efforts to improve the MSM, review process, and other projects. Objective standards for assessing continuous improvement progress need to be developed. The recommendation addresses some, but not all, of the problems with review processes that have been identified through a collaborative effort between the ICANN Board, org, and community.	The Board approves Recommendation 3.6 with the caveat that more information is required to better understand how to operationalize the Continuous Improvement Program to ensure it yields the outcomes intended by the ATRT3 before a Bylaws amendment is completed. When deemed appropriate through the prioritization process, the Board directs ICANN org to initiate the development of a project plan to implement a pilot Continuous Improvement Program in alignment with ATRT3 intent, and in parallel with the views of ICANN structures based on their unique needs and interests, and taking into account any ongoing improvement processes by the ICANN structures.
Transparency and Accountability 4.1 Plain language description of how each goal, outcome, or operating initiative meets the objectives of a stated goal in the strategic plan; 4.2 Plain language definitions of success & evaluation criteria for strategic or operational plans and goals; 4.3 "Retrofit"" the 2021-2025 strategic plan and 2021 operating plan with a supplement that employs the processes of Recommendations 4.1 and 4.2 to establish clear goals, objectives, and criteria for success; 4.4 Annual strategic objectives report; 4.5 Over-arching strategy and objectives report for each five-year strategic plan, including 2016-2020		Recommendations 4.1 and 4.2 are already within the scope of the budget; 4.3-4.5 would involve additional costs, which would either need to be separately resourced or which may involve trade-offs in implementation between the recommendations.	Recommendation timelines may be unreasonable given additional costs and the scope of work involved, particularly in Recommendation 4.3. ICANN already does a lot of this (hence the existing budget for Recommendations 4.1 and 4.2). Implementation should build on and leverage current reporting methods and approaches.	The Board approves Recommendations 4.1, 4.2, 4.3, 4.4, and 4.5 subject to prioritization, noting however that the timing requirement stipulated in Recommendation 4.3 (that a supplementary document be produced within six months of approving this recommendation) is not feasible within the specified timeline.
Prioritization Create an annual process of prioritization that gathers representatives from the ICANN Board, ICANN org, and any SOs or ACs that wish to participate (one representative from each). This panel would operate by consensus, and would integrate into existing financial and strategic planning processes. The panel would be charged to establish a priority order for recommendations resulting from reviews, cross-community working groups, and any other community-sourced process the Board deems appropriate.	The Board notes the community, Board, and org’s ongoing efforts to prioritize ICANN’s work as part of the planning process. Implementation of R5 will be incorporated as part of the prioritization of all of ICANN’s work, using a framework for prioritization is developed. The work on prioritization will include the work plans for the Planning at ICANN Operating Initiative as included in ICANN’s FY21-25 Operating Plan.	Estimated cost of planning for and implementing a community-led prioritization process and running a pilot of such a process could include possible external consultants (plus ICANN org time TBD): $75,000-150,000 (one-time cost).	The Board's submitted public comment on the draft report stated that existing prioritization efforts and processes should be leveraged, instead of creating an entirely new entity. The design of a community-led prioritization process will need to take into account, and/or be complementary to the efforts to prioritize the ongoing work of all of ICANNs as part of the Planning at ICANN operating initiative as included in the FY21-25 Operating Plan. The Board's fiduciary duty to make decisions in the public interest and the interests of a secure and stable DNS cannot be waived or superseded by a community process.	The Board approves this recommendation and directs ICANN org to proceed to implementation. The Board acknowledges that the implementation of this recommendation is dependent upon the need to prioritize all of ICANN’s work through the annual planning cycle, and the need for the development of a framework in collaboration with the community and ICANN org. The Board directs ICANN org to develop a framework of prioritization taking into account community groupings, mechanisms, and processes. The Board directs ICANN org to facilitate efforts to develop a framework toward achieving an agreed-upon definition of what it would mean for the prioritization process to “operate by consensus of the individual SO/ACs, Board, and org members that are participating in the prioritization process.”

Implementation Progress

ATRT3 is, as of July 2021, in the implementation phase. In a blog post after the Board action in November 2020, Maarten Botterman wrote that the prioritization process would commence immediately, and noted that wide public participation would be necessary to inform that process, as well as to ensure that the proposed Holistic Review and continuous improvement program pilots were successful.

The Implementation Shepherds for ATRT3 are Adetola Sogbesan, Cheryl Langdon-Orr, Daniel Nanghaka, Pat Kane, Sebastien Bachollet, and Vanda Scartezini.

Impact on Organizational Reviews

In June 2021, the board agreed to defer GNSO3, in large part because of the recommendations of the ATRT3 final report. The board also effectively placed a hold on all pending organizational reviews until such time as the ATRT3 recommendations were integrated into an implementation plan:

Resolved,...the Board determines that it is not feasible to initiate the GNSO3 Review at this time. The Board is deferring the initiation of the GNSO3 Review until such time as the Board, community and org better understand the impact of the ATRT3 Recommendations on the next Organizational Review cycle, as well as in consideration of the current community workload. Resolved,...the Board directs the ICANN President and CEO, or his designee, to develop a comprehensive plan for the timing and conduct of the next Organizational Review cycle, taking into consideration the timing of the implementation of the ATRT3 recommendations. This plan should be developed in consultation with the ICANN community, and ICANN org shall provide periodic updates to the Board on progress towards such a plan.