External review of MLA MBfP and MMfS programs 2009

Summary

​GHD Hassall was contracted by Meat & Livestock Australia (MLA) to conduct an independent review of
the More Beef from Pastures (MBfP) and Making More from Sheep (MMfS) programs. The purpose of
the review was to:Evaluate the programs' impacts on the beef, and lamb and sheepmeat industries;Assess how successful the programs have been in achieving stated key performance criteria; andProvide recommendations as to how the respective programs should (in the event of further investment) be structured, targeted, monitored and evaluated in order to improve and measure industry impact.
 
MLA and Australian Wool Innovation (AWI) have developed and funded the delivery of the programs to provide best management practice packages of information, tools and learning opportunities for
Australian sheep and beef producers to assist them to increase the productivity and profitability of their enterprises.
In order to achieve the objectives of the review, GHD Hassall developed an Evaluation Framework and Program Logic for the programs to identify the information required, including data gaps. Data was obtained via: a desktop review of information provided by MLA and AWI; questionnaire surveys of beef and sheep producer participants; and interviews of key informants involved in the delivery of the programs.
Questionnaire surveys were mailed to 500 participants in each program with a response rate of approximately 20%. This response rate was acceptable for a survey of this type and provided sufficient confidence that the results can be used to inform conclusions about the programs as a whole.
The draft Program Logic identified that the goals and key performance indicators (KPIs) adopted by MLA and AWI related mainly to: awareness of the programs; participation in program education activities such as field days and workshops; and intention to change practices on farm and actual practice change. For these KPIs, evaluation sheets completed by producer participants following program activities indicated that the programs have mostly succeeded in meeting the targets (MBfP) or have exceeded the targets (MMfS).
Independent market research also indicated that practice change KPIs had largely been met. For example, 50% of MBfP course participants and 57% (n=200) of MMfS participants were influenced to change management practices as a result of course attendance (Logan 2009). In another survey, 46% (N=301) of MMfS participants stated they had made a practice change as a result of their participation in the program (GHD Hassall 2009).​
The producer surveys for this project confirmed that practice changes had occurred and that these changes were thought to have contributed to enterprise productivity and profitability increases. For example, improved grazing management was nominated by approximately 60% of producers in both programs to have led to management improvements. Forty one percent of MBfP producers estimated production increases ranging from 0-5% to greater than 20%, while 28% estimated profitability increases ranging from 0-5% to 15-20%.
For MMfS producers, 44% estimated production increases ranging from 0-5% to 15-20%, while 39% estimated profitability increases ranging from 0-5% to 15-20%.
A benefit cost analysis (BCA) was completed for each program using the following assumptions: 20% of program participants experienced an increase in income; increase in income of 5%; discount rate of 7%.
The benefit cost ratio (BCR) was calculated as 4.35 for the MBfP program and 3.9 for the MMfS program. A sensitivity analysis indicated the relative insensitivity of the BCRs to changes in key assumptions. For example, the percentage of farmers experiencing an increase in income would need to fall to 5% of participants for both programs to break even (BCR = 1).
The estimated BCRs for both programs are consistent with other cost benefit analyses undertaken by
GHD Hassall to assess the impact of agricultural research and development investments. For example an evaluation of MLA's Sustainable Grazing System Harvest Year provided a BCR of 2.9, and an ex-post assessment of the Grains Research & Development Corporation's plant breeding research provided a BCR of 4.9.
Producers in both programs also claimed that activities had produced benefits in natural resource management (NRM) and in a range of social indicators related to knowledge, aspirations, skills and attitudes (KASA). These benefits were not monetarised for inclusion in the BCA.
Although the programs had met their awareness and education targets as envisaged, and MLA's and
AWI's investment had achieved positive impacts on the beef, lamb and sheepmeat industries as demonstrated by the BCA, the review found that key informants were supportive of a new approach to program delivery that focussed on adoption of practices at farm level. This can be achieved by:Aligning program objectives and structure to maximise impact;Targeting producers and practices;Focusing program delivery and KPIs on practice change;Improving brand management to enhance recognition and attribution; andEnhancing monitoring, evaluation and reporting (MER) to improve program performance and reporting.
 
A profiling framework was provided as a process for selecting appropriate producer targets based on consideration of enterprise goals and characteristics as well as critical drivers of productivity, profitability and sustainability of the potential audience. The learning preferences of the target audience could then be explored to determine roles and responsibilities of service deliverers. The draft program logic developed for this project was used to provide an example of a detailed monitoring, evaluation and reporting (MER) framework in order to measure industry impacts.
It was considered that a similar management and delivery system should be retained for the future (including the reinstatement of a national coordinator for MBfP) although roles and responsibilities will change with this more targeted approach. These roles and responsibilities should be clearly defined by the Program Logic and MER frameworks.