A paper for the 2nd Annual Developing, Managing & Measuring Outcomes-Based Policy in the Public Sector Conference
John Bishop
Social Commentator

Wellington
31st March and 1st April 2004

Introduction

Ensuring any kind of policies mean something throughout the organisation is always a challenge for senior management teams and particularly for Chief Executives and their communications people.

So on the face of it, ensuring that "outcomes-based policies are meaningful throughout the organisation's hierarchy", is simply other internal communications issue. From another perspective it's one more challenge for the HR people, the planners and senior management to get people in the organisation to sit up and take notice about something. And if that's all there was to say about the matter, I'd stop now and we could have an early finish to the day.

So is there something new here?

I think there is certainly something going on, and this afternoon I will try and unravel it for you and then seek to answer the four specific questions that are posed in the brief.

First of all what is Managing for Outcomes?

According to the SSC website Managing for Outcomes in the Public Service is "an outcome-based approach to departments' planning, management and reporting. Its aim is to improve the performance of the Public Service and requires departments to adopt a strategic and outcome-focused approach to planning, management and reporting while focusing on delivering outputs."

So what's the big deal about it?

In preparing this presentation I went to the State Services Commission and talked to Deputy Commissioner Tony Hartevelt.

It is clear that the Commission feels itself under pressure to improve the use of public funds. He made a number of points to me on this matter:

1. Departments have developed sophisticated systems for bidding for new money. The difficult part is assessing the effectiveness of baseline money. If you've got it, you tend to keep it, even in an environment where funding is scarce. Public sector processes for evaluating and recommending changes in the baseline lag well behind the processes for getting new money.
2. Efforts to measure outcomes over a shorter period have provided too difficult. Results take half a decade or longer to emerge and often it's hard to prove cause and effect.
3. Chief executives have to have the courage and the information to go to Ministers and say that they ought to do less of this and more of something else. Getting them to do that has proved very difficult. That is what Managing for Outcomes is all about.
4. We want them to specify the "vital few things" that they really have to get right.
5. Build an explanation that they can support with evidence. We call it "intervention logic".
6. We talk co-operation and co-ordination but we don't practice collaboration. The extent to which we have collaborated is despite the system not because of it. Where it has happened it's because of determined professionals on the ground made it happen
7. Collaboration is an agency saying we'll stop doing this and you (another agency) can do it instead. Success will be when the whole system is greater than the sum of its parts.

Mr Hartevelt explained that last year was the first time that all 35 departments had been required to do a Statement of Intent, and that the shortcomings in the SOIs were only to be expected, predictable, and would be remedied as departments gained more experience.

He might be right, but it is worth looking at the report of the team that evaluated what the departments did.

The team was led by Tim Blackmore of the Treasury and was published in August last year. It's available on the SSC website. So what did the report say. Here's their summary.

• Departmental capacity to undertake meaningful organisational capability appraisal appears limited. Departments do not have a clear picture of the current state or future capability requirements, or access to common capability appraisal metrics.
• Measurement of outcomes - Outcome indicators or performance measures are virtually non-existent in the majority of 2003/04 SOIs. Most departments have identified this as an area for future development.
• Intervention logic - The quality of intervention logic is generally not good. The SOIs have rationale statements but the question is whether these are well founded. In some cases, the causal links between outcomes and outputs are not clear.
• Identification of risk and risk management is very limited across the SOI set. Demonstration of an understanding of risk, and the link between risk and operating environment, is non-existent.
• Audience assessment is weak. Currently departments focus too much on meeting central agency expectations in their tabled SOIs, greater referencing or hypertexting to other lower level planning documents and processes would be welcome, as would more simply worded and structured SOIs.
• Management of shared outcome development and sectoral reporting could be improved. At the moment departments tend to assert linkages to each others' outcome sets rather than describe how interdepartmental collaboration or shared outcomes contribution will work. OAG officials suggest that effective sectoral reporting is dependent on designated single agency leadership, and clarity in how accountability for outputs can be approached in a context of shared contribution to outcomes.

It's hard to see that as much more than a C+, must try harder, could do a lot better, kind of a report card.

And the SSC has made it clear that it does want and expect more, and it's put in place various measures to help departments to do.

Mr Hartevelt said "The SSC is now highlighting that it is not going to let up on MfO and getting SOIs done well. We will be looking for demonstrable progress in the next round. Fair enough.

Survey of Departments

Departments would typically start their planning and preparation for their SOIs in February/March so I decided to run a small survey to see whether the matter was being taken more seriously than last year.

I developed a short questionnaire and asked 10 departments to take part. The questions were about what happened last year, and what was happening this year, and the survey also asked about internal communications and internal leadership.

Respondents could complete the survey anonymously and online. I got seven replies, and these are attached to the paper.

Looking first at what happened last year; the picture that emerges is that

• The language of Managing for Outcomes was important - five of the seven Chief Executives used it a lot or sometimes
• The approach was very important or reasonably important for all seven agencies
• And this year its likely to be more important than last year for four of the seven agencies and about the same for the other three

I was also interested in how they had communicated to or involved staff last year.

• Only one of the seven had prepared a communications plan
• Six put stuff on the intranet, and five sent staff a message of some kind
• Only three sought staff input in any formal way but
• All seven advised stakeholders and key external audiences

The conclusion I draw is that the external audiences are seen as more important than the internal audience, and while there were efforts made to inform staff the efforts to engage or involve staff were rather less.

So were things going to change this year? What would people do differently this time round? There were a variety of answers

• more input from third tier managers
• Outputs and performance measures have been reviewed.
• We have started earlier to allow plenty of time for consultation and thinking. A significant amount more time is being placed into ensuring robustness of outcomes, definitions and measurement.
• Engaged Minister on a number of occasions Focus of senior staff day to raise awareness
• and understanding. More formal and frequent briefings of staff
• With a new Act and a new strategic plan and hence new goals our planned outcomes will be different

These are constructive and sensible steps to take.

So were there greater efforts being made in terms of time, resources and priority?

Four of the seven were putting in more effort on the SOI and three were putting more effort into the communications as well. The others were doing about the same as last year on both counts.

One of the key features of the SOI/Managing for Outcomes process is what the SSC calls "the intervention logic", the chain of reasoning that links outcomes to outputs to the activities of the agency. How well are those linkages understood by the staff, and for that matter by the leadership.

The responses from the seven agencies indicated that they thought that staff understood the relationship

between activities and outputs	very well	3
	quite well	4

but the relationship between outputs and outcomes were much less well understood

between outputs and outcomes	very well	1
	quite well	1
	only somewhat	4
	not very well at all	1

Here's the interesting bit. I asked the agencies the same question about how well the leadership understood these matters.

between activities and outputs	very well	6
	quite well	1

but the relationship between outputs and outcomes were much less well understood

between outputs and outcomes	very well	3
	quite well	3
	only somewhat	1
	not very well at all	0

Survey Conclusions

In short the two main findings are:

• The leadership understand both sets of relationships better than the staff, and
• Understanding of the relationship between activities and outputs is higher for both staff and leadership than it is of the relationship between outputs and outcomes.

I find that rather disturbing because it suggests that departments and agencies know how to carry out their activities and can see how this advances the department's outputs, but after that it becomes rather bleary.

As Tony Hartevelt put it to me.... "the penny has not dropped that (the SOI) is a document that is about changing the way of managing public resources based on evidence of effectiveness. This is only slowly emerging."

So final question, is the leadership doing anything about this. Are they clearly explaining those relationships to staff. Three are, and three do so sometimes. One respondent checked the box saying 'they try'.

(Full results are in Appendix One)

Answering the Questions

So let's answer the four questions or matters that were raised in the brief.

They were

• Implementing a strategic focus on communicating outcomes based management throughout the organization

The answers I think are the same answers as for another other organisation wide programme. One, the change has to be led, endorsed and modeled by the senior management.

Two, they have to mean it, and be serious and credible about it and make it a big deal. If necessary back it with sanctions.

Three, you have to show that the new ways of doing things are important to the organisation and to the staff at all levels.

Four, you have to show staff the connection between your public service management planning model and their tasks. Just as the SSC is looking for intervention logic between the outcomes, outputs and the activities of the department, so too spell out the internal intervention logic between the activities of the department and the tasks any one person or group carries out. Answer the question: what does this mean for me as a staff member, what do I have to do differently?

Five, measure and evaluate, and six, celebrate. The message is clear, what gets measured gets done, and what is celebrated is what is important. You want this stuff taken seriously: measure it and celebrate it.

• Establishing a strategy for making outcomes accessible, purposeful and of value to different departments and levels of staff

I think I have answered that question already. The key to me is in the breaking down of the activities into tasks performed by individuals and groups.

One point that does emerge here. I said earlier that you needed to be able to answer the questions: what does this mean for me, what do I have to do differently?

For a lot of staff a lot of the time, the answer will be that they don't have to do things differently. The organisation might see the activity differently but if the tasks don't change then the shift in thinking doesn't mean much to the people doing the work.

That's important because it makes it even harder to engage them in the planning cycle, and the Managing for Outcomes model because their tasks remain disconnected from it.

• Improving outcomes based policy development by integrating staff feedback

Several agencies in my survey said that they were making greater efforts to involve either third level managers or staff in key front line areas in the planning round, but it was proving very difficult.

At least part of the reason is what I was talking about before: no change in their jobs means little incentive to engage.

However I would like to make two suggestions:

One is that staff feedback need not be feedback about the plan or statement of intent, which is a hard thing to get your brain around.

Staff feedback can and ought to be a continuous process. Why not designate one person on the planning team to bring together all the feedback formal and informal that has recorded during the year and feed that into the planning round.

Not only should that help inform the planners, the communications people will get excited, because you have created an opportunity to show staff that their feedback has been heard, taken into account and is actually making a difference. That's quite a powerful message, which will encourage more feedback in future.

Of course you do have to have feedback gathered during the year, and if you haven't got any, you are probably planning in a vacuum anyway. My point is that staff feedback does not have to be feedback about the plan itself; it could be feedback about their activities and experiences, which is then reflected in the content of the plan.

The second suggestion is that you don't necessarily have to get feedback about all of the plan. Why not select say three or four "flashpoints" if you like and seek feedback on those.

By flashpoints I mean issues on which there is going to be keenly held views, so there is something to talk about and debate. Few people will be motivated to feedback anything about routine tasks adequately performed, particularly where the tasks are not either mission critical or inherently interesting in themselves.

They are much more likely to feedback on points where there is controversy, uncertainty, differing expectations, variations in performance and the like.

So organise some sessions where each of these points is discussed by those who are interested in doing so, and record the suggestions, comments, and reactions. Use a facilitator to probe and explore and get more than just top of mind views. Make the groups think. There will be merit in some of their suggestions, and that makes the exercise worthwhile.

• Using outcomes to drive motivation towards achieving strategic objectives

This seems to me to reverse the usual direction of planning logic. The conventional approach is to set outcomes, derive outputs from those and work down to a set of objectives and activities for the agency. What this appears to suggest is that the desire for outcomes can be used to motivate people to achieve the objectives of the agency.

For example the desire for a healthier New Zealand can be used to motivate health professionals to do more immunization, because immunization makes a direct contribution to a healthier New Zealand.

I am not at all sure that I agree with the logic and certainly I do not know of any empirical evidence that shows that the logic works in that way.

It might, and I am not saying it doesn't. I am saying I don't know.

On the face of it the evidence is mixed. Doctors, teachers and police officers may well all share a desire to make society a better place...and good on them for that...but that doesn't seem to make recruitment to Kaitaia, south Auckland or the West Coast of the South Island any easier.

On the other hand a passionate belief in a healthy New Zealand clearly underlies the activities of those who campaign for a smoke free New Zealand or to reduce obesity, or to remove the stigma of mental illness. So perhaps believing in ends does drive people to work harder on the intermediate goals.

Despite the lofty goals of greater efficiency to which the SSC wishes departments to aspire and preferably reach, the reality is somewhat different.

One agency said ...it is a challenge to engage non-managerial people in public sector management frameworks. The key is face to face briefings at all levels - this time commitment can be a big ask when pressing day to day realities compete for the attention of managers.

And this really brings me back to the central problem. If Managing for Outcomes and its most tangible symbol the Statement of Intent are to become significant tools of logic justifying state intervention and the use of public money, then departments really have to do better than this.

Secondly, if departments and agencies are to become well focused on delivering their activities which advance outputs, which are a step to outcomes then the whole organisation needs to understand that bottom to top and down again. The evidence would seem to be that they don't - or at least not yet anyway.

Finally, saying that it is hard to make planning frameworks meaningful is really to say that management hasn't yet found ways to make the work understandable and interesting enough that staff will find it valuable to take an interest.

And if management can't motivate staff, achieve goals that contribute to good social outcomes, and if they can't prove that their intervention helps people live better lives in some way, or make a difference to the quality of our society, you have got to ask why are we allowing them to be in charge of a significant chunk of the government's and our nation's resources.

I think it is a very simple proposition at the end of the day: if you can't prove it makes a positive difference, you can't spend it. That rule, rigorously applied, would certainly focus a few minds and might, in turn, improve the quality of government expenditure.

John Bishop
Social Commentator
1 April 2004

32 Disley Street
Highbury
Wellington
Tel 04 475 8650
Mob 0274 482 247
Web www.johnbishop.co.nz
Email john@johnbishop.co.nz

Data Summary: Managing for Outcomes/Statements of Intent

John Bishop - Communicator for a paper to the "Second Annual Developing, Managing and Measuring Outcomes Based Policy in the Public Sector" Conference
Wellington 31 March & 1 April 2004