2.4 Accountability and performance

Improving performance information

The availability of good information on the performance of government programmes and activities is crucial to ensuring taxpayer funds are well spent and government held to account. High quality information is essential to answering basic questions like what was the money used for? What was the policy objective? Was it achieved? 

As well as a means of accountability to citizens and Parliament, performance information is also essential in:

  • evidence-based policy making – to strengthen the information base available to the government in the consideration of budget and strategic policy options through better delivery of evidence on programme and policy impacts;
  • enhancing budget and strategic policy agility – access to information enables government to assess the relative and absolute merits of its programmes and to consider new spending proposals consistent with its fiscal targets; and
  • demonstrating policy delivery credentials of government agencies – through supporting and demonstrating the use of performance information by agencies to monitor the performance and on-going relevance of programmes for which they are responsible.

The Commission has been asked to identify options for improving the assessment of government activities and the reporting of performance with a view to increasing transparency and accountability. These are the hallmarks of responsible government.

The quality and usefulness of current reporting of performance through Budget-related documents such as portfolio budget statements varies markedly.

While a large volume of performance information is available, the Commonwealth lacks a coherent performance framework that it is easy to comprehend. Current arrangements make it difficult for the community to determine whether money is being well spent, whether spending programmes are delivering on their objectives and how efficiently and effectively the public sector is performing.

Funding Commonwealth activities

The Constitution requires that all revenues or monies raised or received by the Executive Government of the Commonwealth be placed in one Consolidated Revenue Fund.

The Constitution also provides that there must be an appropriation, made by law, for the purposes of the Commonwealth, before money can be drawn down from the Consolidated Revenue Fund.

Under current arrangements departments and agencies receive funding through the annual Commonwealth Appropriation Acts which provide funds to deliver specified ‘outcomes’ the government is seeking to achieve.

Contributing to these outcomes are specific government programmes. Often several programmes will contribute to the achievement of one outcome. A summary of the outcomes and programmes framework is outlined in Figure 2.1 below.

Figure 2.1: Outcomes and programmes framework

This figure shows the Outcomes and Programmes framework, with funds being appropriated from the Consolidated Revenue Fund at the outcome level before flowing onto the programme and programme Expenses levels.

Source: National Commission of Audit.

Outcomes are high-level statements as shown in Table 2.3. In this example, the appropriation for the Department of Human Services (some $4 billion of expenditure) is described in a single outcome.

Table 2.3: Department of Human Services – Appropriation Act
 

Departmental
$'000

Administered
$'000

Total
$'000

Outcome 1 -

Support individuals, families and communities to achieve greater self-sufficiency; through the delivery of policy advice and high quality accessible social, health and child support services and other payments; and support providers and businesses through convenient and efficient service delivery

4,008,936 16,958 4,025,984
Total 4,008,936 16,958 4,025,894

Source: Australian Government, 2013.

Similarly, the majority of the Department of Defence’s appropriation funding (some $22.9 billion from a total of $23.8 billion) is provided for under Outcome 1:

The protection and advancement of Australia’s national interests through the provision of military capabilities and the promotion of security and stability.

Department of Defence, 2013.

While there are many instances of high-level outcomes, there are also examples of more meaningful outcome statements. For example, Outcome 2 (Access to Pharmaceutical Services) for the Department of Health and Ageing provides:

Access to cost-effective medicines, including through the Pharmaceutical Benefits Scheme and related subsidies, and assistance for medication management through industry partnerships.

Department of Health and Ageing, 2013.

Outcome 2 for the Department of Families, Housing, Community Services and Indigenous Affairs also provides an example of a clear description on what the funding is intended for:

Access to affordable, safe housing through: payments and support services; and rental subsidies to low and moderate income households.

Department of Families, Housing, Community Services and Indigenous Affairs, 2013.

A list of all agencies’ outcomes, as published in the 2013-14 portfolio additional estimates statements, is included at Attachment 2.3.

Presented together with the Appropriation Acts as Budget-related documents, portfolio budget statements are intended to provide additional information, both of a financial and non-financial nature, at the portfolio and agency level.

As outlined by the Australian National Audit Office (2011), information on each programme as contained in the portfolio budget statements should outline:

  • associated deliverables (for example, the benefits or services intended to be provided, or the transfer payments to be made); and
  • annual performance reporting on the delivery of programmes and achievements against a set of key performance indicators.

Evolution of performance reporting

Since the mid‐1980s, budgeting and reporting mechanisms introduced by the Commonwealth have been designed to, among other things, allow citizens to better understand government operations and how public funds are being used to achieve policy objectives.

The trend has seen a move from an approach that was heavily oriented to inputs such as wages and other running costs, to models designed to provide citizens and Parliament with a stronger focus on outcomes, giving a more useful picture of the impact sought by government for the funds appropriated. These models use performance indicators as a way of measuring how well the deliverables have contributed to an outcome (Australian National Audit Office, 2011).

Outcomes and outputs framework

The outcomes and outputs framework was introduced as part of the 1999–2000 Commonwealth Budget. Under this framework, entities were required to specify intended outcomes and to measure and report on actual performance. Outcomes were the results or impacts of government policy measures on the Australian community.

At the time, entities were required to identify the various output groups that contributed to government outcomes during the Budget period. The portfolio budget statements included entities’ outcomes by output groups, specifying the indicators and targets used to assess and monitor performance (Australian National Audit Office, 2011).

Operation Sunlight and the Murray Review

In 2005, while in opposition, the Labor Party released a paper titled Operation Sunlight to initiate discussion around Budget transparency and accountability. The paper was re-released in 2006.

In 2008, former Senator Andrew Murray was commissioned to undertake a review of Budget transparency and to make recommendations as part of the then Government’s Operation Sunlight reforms.

In late 2008, the then Government released the Murray Review, its response to the recommendations of the Murray Review and an update to the original Operation Sunlight paper, titled Operation Sunlight: Enhancing Budget Transparency, which outlined six key objective of Operation Sunlight. They were:

  • tightening the outcomes and outputs framework;
  • changing Budget papers to improve their readability and usefulness;
  • improving the transparency of estimates;
  • expanding the reach of Budget reporting;
  • improving intergenerational reporting; and
  • improving the financial framework.

Outcomes and programmes framework

The 2009-10 Budget process implemented a number of initiatives in response to Operation Sunlight, such as the enhanced reporting of programme level data, descriptions and key performance indicators in portfolio budget statements, improved resource statements to allow readers to follow an item from an agency level view of the appropriation bills into the financial statements in the portfolio budget statements, and greater detail on the operation of special accounts.

Under this framework, entities are required to measure the intended results, impacts or consequences of actions by government on the Australian community. A central element of this framework is that entities are required to identify and report against the programmes that contribute to government outcomes over the Budget and forward years, rather than the output group that contributed to government outcomes (Australian National Audit Office, 2011).

While the move to an outcomes and programmes framework has improved transparency and reporting at the programme level, the level of programme detail currently provided in portfolio budget statements varies considerably. Some entities provide only high level information, while others include information that is more useful and relevant to gauging performance for policy purposes.

For example, Figure 2.2 shows the level of information presented for programme 3.1 ‘Employment Services’ in the Department of Education, Employment and Workplace Relations portfolio budget statement.

Figure 2.2: Department of Education, Employment and Workplace Relations – programme 3.1 outline

This figure shows Programme 3.1 -Employment Services, of the Department of Employment and Workplace Relations within the Outcomes and Programmes framework.

Source: Department of Education, Employment and Workplace Relations, 2013.

The Employment Services programme is intended to contribute to achieving the outcome of:

Enhanced employability and acquisition of labour market skills and knowledge and participation in society through direct financial support and funding of employment and training services.

Department of Education, Employment and Workplace Relations, 2013.

In this example, $1.36 billion is projected to be spent in 2013-14 across six ‘programme expenses’ areas within the employment services programme.

Information useful in decision-making, such as whether to expand, cease or reduce a programme is not apparent until the programme expenses level, which includes for example, Job Services Australia, the Pacific Seasonal Workers Programme and the Productivity Education and Training Fund.

Relevant key performance indicators are also outlined. In terms of information that is available to assist in determining how successful the government has been in achieving its objectives, the quality of key performance indicators is relatively high in this example.

They include indicators such as costs per employed person and the proportion of job seekers off benefits at intervals of three and 12 months following participation in employment services.

However, the key performance indicators only relate to the performance of Job Services Australia and none of the other aspects that make up the programme, such as the Pacific Seasonal Workers Programme and the Productive Ageing Package. It is therefore difficult to determine whether these programmes are contributing effectively to the government’s objectives in this area.

Comprehensive performance measurements against all programme expenses would assist in both measuring the effectiveness of a programme in meeting its objectives but also decision-making.

In contrast, the information presented to Parliament in the portfolio budget statements for Austrade’s Export Market Development Grants Scheme (Figure 2.3) is of limited usefulness in answering basic questions like what was the money used for? What was the policy objective? Was it achieved?

The information provided simply outlines that $125 million is budgeted to be spent. The key performance indicators are simply the number of grant applications and the number of grant recipients. There is no information on the results of this spending.

Figure 2.3: Austrade – programme 1.2 outline

This Figure shows Programme 1.2 - Trade development schemes - Export Market Development Grant Scheme, of Austrade within the Outcomes and Programmes framework.

Source: Austrade, 2013.

Performance reporting

The change in the Budget reporting framework from 2009-10 to a ‘programme’ focus was intended to more clearly demonstrate achievements against pre-defined programme objectives.

Commonwealth entities currently report against some 650 government programmes that are measured by some 3,500 key performance indicators (ANAO, 2013).

However, as illustrated above, it is often difficult to get an accurate and comprehensive picture of government programmes and what they are achieving. The level of information available at the ‘programme’ level is inadequate to provide useful policy insights.

The Australian National Audit Office has undertaken several assessments of entity performance measurement and reporting. It has identified that entities continue to experience challenges in developing and implementing meaningful key performance indicators, and that the administrative framework supporting the development and auditing of key performance indicators remains problematic (ANAO, 2013).

The challenges in ensuring sufficient transparency and accountability of government expenditure are demonstrated in the Department of Foreign Affairs and Trade portfolio budget statement.

The objective of Programme 1.2 (see Figure 2.4 below) is:

To advance Australia’s foreign, trade and economic, and security interests through participation in international organisations.

Department of Foreign Affairs and Trade, 2013.

Figure 2.4: Department of Foreign Affairs and Trade – programme 1.2 outline

This figure shows Programme 1.2 - Payments to International Organisations, of the Department of Foreign Affairs and Trade within the Outcomes and Programmes framework.

Source: Department of Foreign Affairs and Trade, 2013.

The level of detail provided for $258 million of associated expenditure is reported in one line. Similar reporting occurs in the Department of the Treasury’s portfolio budget statement for payments to international financial institutions.

While it is recognised that the Department of Foreign Affairs and Trade and the Department of the Treasury do not have prime accountability for expenditure of these funds, which are ultimately deployed by international entities, it is reasonable to expect that more information be included in the portfolio budget statements as the prime accountability document.

The inclusion of detail such as to whom the funds were provided, for what purpose and what outcome, would provide more relevant information. In some cases, more detail is provided elsewhere, for example in the annual report, which lists the names of all organisations receiving funds.

The key performance indicators used to determine if the programme’s objectives were met are that:

Payments to international organisations are timely and within budget.

Department of Foreign Affairs and Trade, 2013.

This reflects the current focus on financial rather than policy performance.

A similar situation occurs in the portfolio budget statements for the Department of Health. As outlined in Figure 2.5, which relates to Programme 1.3 – Drug Strategy, the level of detail provided at the Programme and Programme Expenses level gives no information to inform the reader of the types of activities the government is undertaking.

While the funding relates to activities such as tobacco plain packaging, drug and alcohol treatment services, the Substance Misuse Prevention and Service Improvement Grants Fund, and the National Drugs Campaign, no information is provided on how these activities contribute to the ‘total programme expenses’ of $259 million budgeted for in 2013-14.

Figure 2.5: Department of Health – programme 1.3 outline

This figure shows Programme 1.3 - Drug Strategy, of the Department of Heath and Ageing within the Outcomes and Programmes framework.

Source: Department of Health and Ageing, 2013.

In terms of the key performance indicators for this programme, there is no recognition of the individual performance of the components that make up the programme. Due to this there is no way for assessing the success of, for example, the National Drugs Campaign in comparison to that of the Substance Misuse Prevention and Service Improvement Grants Fund.

It is recognised that the broad range of activities within government mean that some entities’ programmes suit the development of straightforward key performance indicators more than others. Despite these challenges, the Commission considers that more meaningful and measureable key performance indicators should be developed and maintained.

Commonwealth Financial Accountability Review

Recently a review was undertaken of the Commonwealth’s financial framework to identify options for reform that would support high quality resource management and performance.

The Commonwealth Financial Accountability Review was initiated by the former Government in 2010 with the then Department of Finance and Deregulation releasing an associated discussion paper in March 2012: Is Less More? Towards Better Commonwealth Performance.

A position paper was consequently released in November 2012 (Sharpening the Focus, A Framework for Improving Commonwealth Performance) by the former Government which recommended greater transparency and accountability in the Commonwealth financial framework but with a greater focus on government effectiveness.

The purpose of the Commonwealth Financial Accountability Review was to create a financial framework that is simple, adds value and is easy to use. The review concentrated on seven key directions for reform:       

  • enhancing transparency and accountability;
  • more effective governance arrangements;
  • improving performance;
  • engaging with risk;
  • building capability and culture;
  • simplifying requirements; and
  • clarifying obligations.

‘Improving performance’ included the integration of the performance framework. The Review outlined that an effective financial framework should be able to identify the purpose for which public money is being allocated; how and where it is being spent; and what the expenditure has achieved.

The Sharpening the Focus position paper that followed noted that the current financial framework does not draw links to planning, budgeting or evaluation activities. Furthermore, more emphasis should be placed on the availability of good quality performance information to support policy and budget processes. This need for additional information stems from the current position where the most readily available information on the public sector tends to be financial in nature, which does not in itself provide insights into whether publicly-funded programmes and activities are achieving their policy objectives and outcomes.

Better performance management information would provide a means for both designing new initiatives and increasing the level of scrutiny of ‘base funding’, not routinely examined as part of the Budget process.

Public Governance, Performance and Accountability Act 2013

The culmination of the Commonwealth Financial Accountability Review was the enactment of the Public Governance, Performance and Accountability Act 2013 (PGPA Act) in June 2013.

The PGPA Act will replace both the Financial Management and Accountability Act 1997 and the Commonwealth Authorities and Companies Act 1997 with a piece of legislation governing the management of public resources and the performance of Commonwealth entities.

The PGPA Act is based on four guiding principles, that:

  • government should operate as a coherent whole;
  • a uniform set of duties should apply to all resources handled by Commonwealth entities;
  • performance of the public sector is more than financial; and
  • engaging with risk is a necessary step in improving performance.

Guided by these principles, particularly that performance of the public sector is more than financial, the PGPA Act places a requirement on entities to properly measure, record and assess their performance and to report this annually in performance statements, that will form part of annual reports.

The annual performance statements will provide a comparison of actual performance against planned performance as outlined in an entity’s corporate plan. Furthermore, these annual performance statements may be examined and reported on by the Auditor-General.

The requirements for annual performance statements is intended to rebalance the focus of entity reporting between financial and non-financial performance information to provide a more complete explanation of performance to determine what they have achieved (Department of Finance, 2013).

The policy and principles underlying the PGPA Act are a positive step in enhancing accountability and transparency. The PGPA Act aims to establish the means to provide ‘a clear cycle of planning, measuring, evaluating and reporting of results to the Parliament, ministers and the public’ by:

  • explicitly recognising the high-level stages of the resource management cycle;
  • recognising the value of clearly articulating key priorities and objectives;
  • requiring every Commonwealth entity to develop corporate plans;
  • introducing a framework for measuring and assessing performance, including requiring effective monitoring and evaluation; and
  • maintaining the rigorous audit arrangements currently in place.

However, as much of the detail on the workings of the PGPA Act is yet to be settled in the related Rules, which will replace the regulations that were previously made under the Financial Management and Accountability Act 1997 and the Commonwealth Authorities and Companies Act 1997 as well as guidance material, the implications for entities at the operational level are still to be codified.

These implications include the level performance measures are expected to examine, and how the new requirements will relate to other reporting mechanisms such as the portfolio budget statements. Similarly, it is not clear at this stage how these requirements will ‘close the loop’ and flow through to influence decision-making, including through the Budget process.

Programme evaluation

Evaluation is the process of measuring and assessing the impact and merits of government policies, strategies and programmes. It provides the means of determining the appropriateness, effectiveness and efficiency of government policies and programmes:

  • Appropriateness – the extent to which there is a sound basis for government’s intervention into a particular area of activity, that is, should we do this?
  • Effectiveness – the extent to which stated objectives have been met by government’s intervention, that is, how did the actual outcomes compare to the desired outcomes?
  • Efficiency – the extent to which resources have been used productively in delivering government activities?

The previous section outlined the benefits of improved programme reporting and performance assessment, as part of the day-to-day operations of government.

Programme Evaluations can take a broader view to examine questions such as: is this an appropriate role for government?; does the current activity align with current government priorities?; and can the programme be better integrated/coordinated with broader government activities?

Evaluations can also provide for a more comprehensive assessment of a programme’s efficiency and effectiveness.

The ACT Government Evaluation Guidelines (ACT Government, 2010) suggest that evaluation can help answer questions such as:

  • Is the policy producing the intended outcomes or any unintended outcomes?
  • Has the policy achieved its stated objectives?
  • Are there better ways of achieving these outcomes and objectives?
  • Is the policy still aligned with government priorities, particularly in light of changing circumstances?
  • Should the current programme be expanded, contracted or discontinued?
  • Is there a case to establish new programmes?
  • Can resources be allocated more efficiently by modifying a particular programme or the mix of programmes.

Rigorous and systematic evaluation assists policy design and review, supports Budget decision-making, assists in ongoing programme management and strengthens government accountability.

Current Commonwealth Evaluation Activity

Most existing Commonwealth review activity is focused on assurance processes, designed to assist government decision-making around new policy proposals before a policy decision is made.

The lack of a systematic Commonwealth evaluation process is not to say that no reviews take place. As well as Productivity Commission inquiries, there are regular mechanisms like statutory reviews, parliamentary inquiries and one off policy reviews. Some large scale cross portfolio policy initiatives of recent years, such as Building Australia’s Future Workforce aimed at improving workforce participation, have also included evaluation strategies into their design. However, instances such as this are the exception rather than the rule.

Table 2.4 provides a summary of existing evaluation and review mechanisms at different stages of the programme life cycle. This shows that most of these existing processes are concentrated prior to Budget decision-making, with post decision-making review being largely of an ad hoc nature.

Once a policy or programme is agreed and funding committed, existing evaluation activity across government is dependent on the priority given to evaluation by portfolio ministers or individual agencies.

For example, the Australian Aid Programme has included comprehensive evaluation for a number of years. Other agencies also undertake evaluations, but much of this information is held in-house and is not made widely available.

Across government, the quality of evaluations is variable, with evaluation activity on an ad hoc rather than systematic basis.

Conducting evaluations is not an end in itself. While evaluations are useful for ongoing programme management, the limited visibility of evaluation at the centre of government decision-making, the Budget process, indicates that even the evaluation currently being undertaken is not being used to its potential.

The Commonwealth is currently out of step with international practice and also some other Australian jurisdictions. Both New South Wales and the Australian Capital Territory have recently introduced comprehensive evaluation frameworks. The New South Wales system was introduced following the recommendations of the 2012 NSW Commission of Audit.

Internationally, there has been a lot of recent activity, possibly associated with the more difficult economic conditions now facing many countries (Robinson, 2013). Examples of countries with well developed evaluation frameworks include the United Kingdom, Canada, New Zealand, South Africa, South Korea and Chile.

Table 2.4: Evaluation and review mechanisms
Mechanisms Mechanisms

Prior to decision making

During programme implementation

Implementation Readiness Assessments (IRAs):
identifies implementation risks related to new policy proposals and consequences if risks eventuate.

Gateway Review:
additional assurance mechanism used across the life-cycle of a programme, but primarily during implementation, where a costly proposal has been agreed but significant implementation risks have been identified.

Pre-decision Gateway reviews:
conducted across the policy, programme development and implementation lifecycle, which may last for several years – gate 0 and First Stage reviews are conducted prior to government decision.

Cabinet Implementation Unit:

tracks, assesses and reports to the Cabinet on the implementation and delivery progress of the government’s highest priorities.
ICT/Property Two Pass Review processes:
ICT/property specific assurance process used to guide agencies in the development of robust business cases before government approves large scale projects.

Evaluating programmes

Strategic Reviews:
Government-commissioned Finance department reviews of discrete areas of existing government expenditure (now curtailed).

Risk Potential Assessment Tool (RPAT):
a pre decision assurance mechanism used to measure the relative risk associated with new policy proposals – feeds into consideration by Cabinet.

Agency Capability Reviews:
impartial assessments of agencies’ capability to meet current and future challenges against a common capability framework.

Regulatory Impact Statements:
measures the regulatory impact of new policy proposals.

Statutory Reviews:
are specific reviews required by legislation or regulation.

Implementation Plan:
an implementation plan is a management tool for a specific policy measure, or package of measures, designed to assist agencies to plan for, manage and monitor implementation effectively.

Parliamentary reviews (including Senate, Joint Committee of Public Accounts and Audit):
the Parliament and the committees that it establishes can initiate specific inquiries into government activity.

Productivity Commission:
an independent government agency that can be commissioned by government to conduct public inquiries and studies into a broad range of issues prior to government decision making.

Productivity Commission:
An independent government agency that can be commissioned by government to conduct an inquiry into government programmes or activities;

Cabinet Implementation Unit:
PM&C function, whose primary role is to assess the implementation risks associated with government submissions.

Australian National Audit Office:
independent government agency that provides audit and assurance advice on the financial performance of existing government bodies.

 

Royal Commissions:
commissioned at the discretion of the Parliament to undertake broad tasks detailed in terms of reference.

 

Other ad-hoc evaluations and reviews:
commissioned by the government and may be undertaken by the agency or an external party.

References

Australian Capital Territory Government 2010, ACT Government Evaluation Policy and Guidelines, Canberra.

Australian National Audit Office 2000, Agencies’ Performance Monitoring of Commonwealth Government Business Enterprises, Canberra.

Australian National Audit Office 2011, Development and Implementation of Key Performance Indicators to Support the Outcomes and Programs Framework, Canberra.

Australian National Audit Office 2013, The Australian Government Performance Measurement and Reporting Framework – Pilot Project to Audit key performance indicators, Canberra.

Blöndal J, Bergvall D, Hawkesworth I & Deighton-Smith R 2008, Budgeting in Australia, OECD Journal on Budgeting, Vol 8/2.

Department of Defence 2013, Defence Portfolio Budget Statement 2013-14, Canberra.

Department of Education, Employment and Workplace Relations 2013, Education, Employment and Workplace Relations Portfolio Budget Statements 2013-14, Canberra.

Department of Families, Housing, Community Services and Indigenous Affairs 2013, Families, Housing, Community Services and Indigenous Affairs Portfolio Budget Statement 2013-14, Canberra.

Department of Finance 2013a, Public Governance, Performance and Accountability Rule – Draft Annual Performance Statement Rule, Canberra.

Department of Finance 2013b, Centrally Commissioned Processes and Reviews for the Australian Government – High Level Overview, Canberra.

Department of Finance and Deregulation 2011, Commonwealth Government Business Enterprise Governance and Oversight Guidelines, Canberra.

Department of Finance and Deregulation 2012a, Sharpening the Focus – A Framework for Improving Commonwealth Performance, Canberra.

Department of Finance and Deregulation 2012b, Guidance for the Preparation of the 2012-13 Portfolio Budget Statements, Canberra.

Department of Finance and Deregulation 2012c, Is Less More? Towards Better Commonwealth Performance, Canberra.

Department of Foreign Affairs and Trade 2013, Foreign Affairs and Trade Portfolio Budget Statements 2013-14, Canberra.

Department of Health and Ageing 2013, Health and Ageing Portfolio Budget Statements 2013-14, Canberra.

Joint Committee of Public Accounts and Audit 2013, Report 438 – Advisory Report on the Public Governance, Performance and Accountability Bill 2013, Canberra.

Mackay K 2011, The Performance Framework of the Australian Government, 1987 to 2011, OECD Journal on Budgeting, Vol 11/3, OECD Publishing, Paris.

New South Wales Commission of Audit 2012, NSW Commission of Audit Final Report

Government Expenditure, 4 May 2012, Sydney.

New South Wales Government 2013, NSW Government Evaluation Framework, Sydney.

New Zealand Government, Better Public Services Results Targets, viewed January 2014, <http://www.ssc.govt.nz/better-public-services>.

OECD 2013, Strengthening Budget Institutions in OECD Countries – Results of the 2012 Budget Practices and Procedures Survey, OECD Conference Centre, Paris, 3-4 June 2013.

Robinson M 2013, Spending Review, in Annual Meeting of OECD Senior Budget Officials, OECD Conference Centre, Paris, 3-4 June 2013.

Scottish Government, Scotland Performs, viewed January 2014, <http://www.scotland.gov.uk/About/Performance/scotPerforms>.

Victorian Auditor-General 2010, Performance Reporting by Departments, Melbourne.