10.3 Methods

This report focuses on the aspects of management undertaken to manage particular environmental topics in the ACT. It comprises a qualitative assessment of performance against all six elements of the IUCN Management Effectiveness Framework (context, planning, inputs, processes, outputs and outcomes).

Five management topics are considered:

• air quality
• land
• water
• biodiversity
• heritage.

Criteria were developed under each framework element with a total of 24 indicators across the 6 elements. Information relevant to assessing performance against each of the indicators was assembled by OCSE staff, and provided to the independent assessors for review. The assessors sought additional information from relevant research papers and other source documents, stakeholder workshops and discussions. The assessors then rated performance, providing a justification for the rating and a documentation of the main evidence they considered in reaching this judgement.

A four-point rating scale commonly used in management effectiveness evaluation systems was adopted (Table 10.3).

Table 10.3 Description of the ratings

Rating	Score	Interpretation
4: Effective	76–100% of optimal condition	A score of 4 indicates that, in general, management is effective, although there is still room for improvement
3: Mostly effective	51–75% of optimal condition	A score of 3 indicates that management is mostly effective, but requires improvement. A score of 3 indicates that management effectiveness is better than 50%
2: Partially effective	26–50% of optimal condition	A score of 2 indicates that management is only partially effective and requires significant improvement. A score of 2 indicates that the system is operating at less than 50% effectiveness, and this would be demonstrated by systems failing, with likely clear evidence of adverse changes in management or condition of the environment
1: Not effective	0–25% of optimal condition	A score of 1 indicates that management is completely absent or very seriously deficient

Six indicators were used to assess the effectiveness of management for each management topic:

• Context
  • Managers know and understand the environmental values and attributes that management is seeking to maintain or enhance relevant to the topic.
  • Managers understand the threats and pressures (direct, indirect and cumulative) affecting the topic.
  • Managers are aware of and understand the broader regional-, national- and international- (if relevant) level influences relevant to the topic.
  • Managers understand the stakeholders’ perspective about the topic.
• Planning
  • There is a planning system and/or policy framework in place that effectively addresses the topic.
  • Clear, measurable and appropriate objectives, outputs and outcomes for management of the topic have been documented.
  • The main stakeholders and/or the local community are appropriately engaged in planning to address the topic.
• Financial, staffing and information inputs
  • Financial resources are sufficient to meet objectives.
  • The skills and expertise are adequate to inform and implement management decisions.
  • Adequate information (eg biophysical, socioeconomic, heritage) is available to inform management decisions.
• Management systems and processes
  • The main stakeholders, and/or industry(ies) and/or community are effectively engaged in the ongoing management of the topic.
  • An appropriate governance system is in place to address the topic.
  • Performance monitoring of plans or programs is effective and timely.
  • Impacts (direct, indirect and cumulative) of the threats and pressures associated with the topic are appropriately considered.
  • The most current and relevant (available) information (eg biophysical, research and/or monitoring; socioeconomic; heritage) is applied appropriately to make relevant management decisions regarding the topic.
  • Approaches and methods used in managing the topic are appropriate and in line with best practice.
  • Relevant standards and targets are identified and are being met for the topic.
• Delivery of outputs
  • The implementation/operation plans have progressed appropriately (time and budget).
  • The results have achieved their stated management objectives for the topic.
  • The implementation/operation plans have progressed appropriately (time and budget).
  • The results have achieved their stated management objectives for the topic.
• Achievement of outcomes
  • The outputs are on track to achieve stated outcomes.
  • The outputs are reducing the major risks and the threats for the topic.
  • The condition of values is within the acceptable range.

An example of the assessment process for biodiversity protection outcomes is shown in Table 10.4. Ratings were assigned for a management topic against each indicator; these individual ratings were added and then scaled to produce an overall rating of ‘effective’, ‘mostly effective’, ‘partially effective’ or ‘ineffective’.

Table 10.4 Example assessment of management effectiveness for outcome indicators for biodiversity protection

Topic: Biodiversity Context	Score	Evidence	Source
Managers know and understand the environmental values and attributes that management is seeking to maintain or enhance relevant to the topic	4.00	The ACT Nature Conservation Strategy, plans of management, business plans, works plans, weed operational plans, threatened species action plans, biodiversity offset management plans provide good knowledge and understanding of values and attributes The Parks and Conservation Service establishes strategic partnerships with third parties to assist in the preservation of biodiversity, including with Conservation Volunteers Australia, the Australian National University Fenner School and the University of Canberra. These relationships yield scientific information, formal and informal, which are applied in biodiversity conservation The Nature Conservation Act 2014 and the Environment Protection and Biodiversity Conservation Act 1999 (Cwlth) provide the statutory basis for management and regulatory response Knowledge of species and distributions is at a level of detail that is rare in other jurisdictions Considerable community and consultant data, but largely unverified	Nature Conservation Act 2014 Environment Protection and Biodiversity Conservation Act 1999 (Cwlth) Threatened Species Recovery Plans Kangaroo Management Plan Namadgi National Park Plan of Management Tidbinbilla Plan of Management ACT Nature Conservation Strategy ACT Pest Animal Management Strategy

ACT = Australian Capital Territory

Scores for each element of the IUCN framework were scaled to provide a total score out of 40 and a rating system was developed to convert scores to a rating of management:

• If the total score is 32.6–40.0, then the overall grading statement for that element is ‘effective’.
• If the total score is 25.1–32.5, then the overall grading statement for that element is ‘mostly effective’.
• If the total score is 17.6–25.0, then the overall grading statement for that element is ‘partially effective’.
• If the total score is 10.0–17.5, then the overall grading statement for that element is ‘ineffective’.

The refined assessment system was reviewed in a workshop attended by key staff from the ACT. Staff provided the independent assessors with data sources and supporting evidence relevant to making a judgement about performance for each of the 24 indicators for each management topic.

The independent assessors reviewed evidence and assigned an initial rating to each of the indicators. The rating was agreed to by consensus between the assessors following discussion of the available evidence. The rating and the reasons for assigning the rating (eg key points of evidence or other considerations relating to the rating) were noted in a standard proforma. The ratings and reasons were subsequently discussed with staff from OCSE and relevant ACT agencies to ensure all relevant supporting evidence had been considered. Based on this open and iterative process of discussion and review, the assessors adjusted a number of assessments where improved knowledge and understanding indicated that the original ratings were either too high or too low, and the list of evidence supporting the assessment was refined as necessary.

A summary of the scores is shown in Appendix 1. Detailed scores, justification and sources of information for each indicator for each topic are shown in Appendix 2.