If Overseer is to be suitable for use in regulation to help clean up New Zealand’s rivers and lakes then a large measure of transparency is needed, says Parliamentary Commissioner for the Environment, Simon Upton.

Delivering his report titled ‘Overseer and Regulatory Oversight’ on the widely-used farm management tool last month, Peter advised: “It is time to open up Overseer”.

Originally developed as a farm management tool to calculate nutrient loss, Overseer is increasingly being used by councils in regulation, due to excessive nutrient run-off from farms impacting on the health of waterways.

Peter says to ensure cleaner water, farmers and regional councils need to be confident that Overseer’s outputs are reliable.

“To help build confidence there needs to be more transparency around how the model operates. Any model operates with a measure of uncertainty. That’s normal. The question is whether the level of uncertainty is an acceptable one?

“Confidence in Overseer can only be improved by opening up its workings to greater scrutiny.”

Peter says it will take time to improve Overseer and provide transparency around how it operates. “In the meantime, regional councils can continue to use it but they need to be aware of its limitations.” 

The report found important elements of the model are not open for review, and some gaps and shortcomings need to be addressed.

Hence, Peter recommends that if the Government wants to see Overseer used as a regulatory tool, then it needs to address a number of issues.

These include: commissioning a comprehensive evaluation to ensure the Overseer model is independently peer reviewed, and is subject to sensitivity and uncertainty analysis; providing greater transparency around how the model works; aligning Overseer’s ownership, governance and funding arrangements with the transparency required for it to be used as a regulatory tool; and providing official guidance on how Overseer should be used by regional councils.

Reaction to the report has been mostly supportive of the PCE recommendations. But a range of experts from the agricultural science and freshwater management fields also suggest different ways to move forward with Overseer. See page 28 for more information.

Feds back recommendations

Federated Farmers is backing the call from the PCE for more work to be done to improve the accuracy and transparency of Overseer.

FF environment spokesperson Chris Allen says the PCE report highlights many shortcomings with Overseer – particularly for use as a regulatory tool – pinpointing issues that his organisation “has been raising for some time”.

“Even for types of farming systems and geography within models that have been calibrated, actual results for nitrogen losses can be 25-30 per cent off the mark – in either direction. Outside these calibration ranges, results can be up to 50 per cent inaccurate.

“The significant inherent inaccuracies in the Overseer model means that is very unfair when the model is used to regulate farming activity central to farmers’ livelihoods, and even more importantly to mount prosecutions,” says Chris.

Chris says the PCE has found uncertainty with Overseer data input standards, a series of issues with how to handle Overseer upgrades/version changes where a farmer’s modelled nutrient losses can jump around by up to 80 per cent, and the inability of the modelling tool to represent farm systems in particular regions.

“Federated Farmers agrees with the PCE that Overseer needs more government investment, third-party peer review, greater transparency; and most particularly that Overseer should not be used to assign absolute limits to discharges on farm activities, that can then be traded.”

Chris says Overseer’s other shortcomings also need to be recognised. “Overseer simply models, and not that accurately, nutrients lost from the farming system – it does not model what happened to the nutrients after that and certainly does not accurately model the effect on the surrounding environment.”

Refinement needed

Professor Jenny Webster-Brown, of the Waterways Centre for Freshwater Management, Lincoln University and University of Canterbury, says there are a great many numerical models used in the management of our natural resources, all of which have their worth as well as their limitations. 

“Overseer is an established, accessible, well-supported numerical model. However, it is increasingly being used for purposes other than those it was designed and validated for; mainly by modellers who do not have access to details of the modelling processes and their limitations,” says Jenny.

“If Overseer is to be the model of choice for national and regional scale nutrient loss and load modelling, and widely accepted and adopted across multiple sectors, it needs to be openly reviewed and refined for its use in this context.

“The modelling process also needs to become transparent. The PCE’s report recommends such a review, as well as a role for central government to provide national consistency and support for a comprehensive, independent review process.”

Legal challenge

University of Otago legal and scientific researcher Dr Julie Everett-Hincks with NZ facing significant issues in degraded water quality, the National Policy Statement for Freshwater has prompted some regional councils to adopt Overseer as a tool to encourage compliance and enforce nitrogen limits on farmers.

“Overseer would not likely withstand legal challenge, but more importantly is it right to burden farmers with regulatory compliance when the tool used cannot reasonably measure nutrient losses? In its current form and governance structure, Overseer is not fit to be a regulatory tool.”

Julie says Overseer was never designed to estimate limits and enforce compliance. “Its original design was a nutrient budgeting tool for farmers. Overseer has been adapted over the last four decades, both in its governance structure and its capacity to meet regional council demands. 

“Overseer Ltd’s strategy is making Overseer ‘the trusted on-farm strategic management tool for achieving optimal nutrient use for increased profitability and managing within environmental limits’. “However, to be trusted it must first become transparent.”

Julie says the PCE recommends a comprehensive and well-resourced evaluation of Overseer for it to be considered a regulatory tool – and also recommends Overseer adopts an open-source model, which conflicts with the current business model.

“To regulate water quality in Aotearoa, I believe that a purpose-built model is required.”

Only model

Professor Troy Baisden, BOPRC chair in Lake and Freshwater Science, University of Waikato, says Overseer was and is intended to be NZ’s main computer tool developed to support farmers’ efforts to minimise nutrient losses.

“Over a dozen years, use has shifted beyond this intention and the model is now required by many regional councils setting limits on nitrogen losses from farming to maintain water quality.

“Overseer’s path from helpful calculator to regulatory tool has been bumpy for both councils and farmers, and that is the issue the PCE has investigated.

“On the upside, Overseer is well used and reflects some of our farming systems well. That would be perfect if Overseer was still mainly a calculator to improve farm nutrient management.

“But, when used to enforce regulation, Overseer lacks the openness and transparency needed for scientists to review model results or develop improvements.

“The report concludes Overseer has achieved a safe monopoly on regulatory use. Essentially, Overseer is the best model we have, because it is the only model we have.

“As a result, there are reasons to recommend the Government address issues of openness, enabling the science community to do more to check and improve Overseer.”

Inevitable limitations

Dr Suzi Kerr, senior fellow, Motu Economic and Public Policy Research, says the report is a really thorough, useful and long overdue evaluation of the Overseer model as a regulatory tool.

“It identifies and explores key fundamental issues such as governance, transparency and quality control; addressing these with some of the suggestions in the report could significantly improve the quality and acceptability of the model.

“At the same time it makes clear the inevitable limitations of any model, and hence our need to accept inaccuracy if we want to have flexible policy.

“Reassuringly, many of the modelling challenges it identifies are specific to water quality where, for example, leaching rates depend heavily on local farm conditions so are data heavy and where the importance of leaching from a specific farm for water quality goals depends on catchment-wide processes.”

Suzi believes creating a strong version of Overseer, or a similar tool, to estimate greenhouse gases is a significantly easier task “though one that is still in progress”.

Connect to FEPs

Our Land and Water National Science Challenge’s chief scientist Professor Richard McDowell says catchment water quality is driven by action at the farm scale.

“Overseer is ‘good enough’ to model most complex farm systems and estimate nitrogen and phosphorus loss in response to farm practices. While parts of the model can be improved there’s a risk that focusing on uncertainty delays action on farms,” says Richard, who was one of the report’s reviewers and also a contributor to the model’s development.

“My opinion is the best use of Overseer is to connect it to Farm Environment Plans, and I’d like to have seen this included as a recommendation.

“FEPs should fall under the same national guidance recommended by the Commissioner for Overseer, to ensure plans are effective. Currently, there is potential for significant variation in quality between 16 regional councils.

“Beyond this connection to FEPs, Overseer needs to evolve so it is able to map and target critical source areas of nutrient loss within a farm (spatial variation) and measure the impact of day-to-day decisions on the farm (temporal variation).

“The next step would see Overseer and other models used to help farmers and growers identify the most suitable land uses for areas most prone to nutrient loss.”

Clear path forward

University of Waikato’s Faculty of Science and Engineering Professor Louis Schipper says Overseer is a computer model that, in part, provides information on nutrient losses from agricultural practices below the root zone that can pollute waterways.

“The model can be used in a regulatory fashion, where Overseer predicts nutrient losses based on farmer practices. Consequently, Overseer can create operating boundaries for farmers and there is considerable interest in the model’s accuracy and transparency, which is the focus of the PCE report. “Other countries are struggling with similar issues and have regulated inputs which, as the report suggests, may not be welcomed by NZ farmers as being more constraining.”

Louis says it’s an important conclusion is that Overseer is not open enough to external inspection and critique and the level of uncertainty of the model outputs is not well understood.

“In my opinion, the PCE’s overarching conclusions and recommendations provide a clear path forward and need to be addressed comprehensively and with urgency.

“It is critical that any review is carried out in a constructive fashion identifying strengths and weaknesses of Overseer and identifies a way to enhance its performance so that farmers, regulators, and the public have confidence in the use of the model to protect water bodies while also providing long-term certainty for land managers.

“A major challenge is that as the model continues to improve, users might end up with different predictions for the same management practices. The community will need a degree of understanding and common agreement on how these changes are then used when making decisions.”