Perceptions and reality

By Karen Poutasi, CE, New Zealand Qualifications Authority

22 September 2006

Download the speech

Network of Public Sector Communicators Conference, Wellington

Thank you for that welcome.

This is a gathering of public sector communications practitioners. You work for government departments or crown agencies and your task is explaining government policy and services. In other words, you are at the sharp end of how the public sector is perceived. Of course, this is not an easy task. If you worked in the private sector, I think your job would be easier. That's because your employers would have an elegant empiric by which to judge success or failure; the KPI being profit or loss.

Another take on the private sector is that for a listed company complete success is when it meets all its stakeholders' expectations. So if the shareholders, the customers, the employees and the regulators are all happy, then all is right with the world. Indeed, this is a practical goal for a company making and selling widgets.

Can we say the same in the public sector? I think not. In the public sector, although we have a range of stakeholders, life is more complex and difficult. Often we have to implement government policy in the full knowledge that intrinsically some of those impacted will not be pleased, this just being the way of the world. Having said that I also think being a public servant is a singular honour and working in the public sector is rewarding in many ways.

But firstly, who am I?

As you possibly know, I come from a medical background. I specialised in public health medicine, then moved into management, working across the country in senior management jobs and became Director General of Health in 1995. In May this year I moved to be take up the position I hold now as CE of the Qualifications Authority, my first working experience outside health. I must say, so far I am thoroughly enjoying it, getting my head around a new set of policy issues, but also seeing many parallels, including in the communications area.

And what does my new organisation do?

Based on the headlines, you would think NZQA and NCEA are the same things. Not so. Running secondary school exams is actually a relatively small part of the New Zealand Qualifications Authority's work. Let me show you what we do with this overhead [ see the Table opposite ].

Actually I wouldn't even bother trying to read this now. It won't make a lot of sense without defining terms, and it's an example of less than crystal clear communications.

Broadly, and put simply, we manage qualifications and undertake a number of quality assurance functions within the education sector. And yes, somewhere in there, we run national senior secondary school examinations. It's the second bullet point.

As you can see, we do a lot, most of which rightly generates no media interest. We employ about 360 staff and large numbers of part time experts to carry out specialist functions, such as writing and marking exams. We do all this alongside our sister agencies in the education sector - the Ministry, the Tertiary Education Commission, the Education Review Office and Career Services. We have a Communications Unit with five full-time staff and one part-timer. They manage our media interface, external and internal communications, our website and our publications.

So what are the communications challenges that the Qualifications Authority faces? We have a strategic agenda in terms of how we communicate what we do - more of that proactive focus later. But first, let's look at reactive work.

Typically, we have two hot spots - when we close failed Private Training Establishments and secondary exams - NCEA and Scholarship.

When a Private Training Establishment is in trouble, we often face a dilemma in terms of communications. At least up till now, our house rule has been that for the provider concerned, this is a commercial matter and it is not for us to litigate the issue in public. This is especially so because a provider in trouble financially may survive, except that publicity may see enrolments dry up, forcing the business into bankruptcy. This type of tension is common in public sector communications.

But, often, as a provider approaches liquidation, the students or the staff go to the media, and we then get a reporter asking "What's going on?" We try as best we can to answer that question honestly without making things worse for the provider, which is difficult. If the provider shuts down, then our task, if international students are involved, is to find them another education provider, so their studies can continue.

One of the issues we have considered is should we insist on placing our audits of Private Training Establishments on our website. At the moment, this is voluntary. Private Training Establishments choose whether they want our audits of them available on line. If they decline, the website records that fact also. But, for the protection of the NZ Education Inc brand, especially internationally, should publication be mandatory? A nice puzzle and again typical of public sector challenges, where we want to err on the side of transparency, but need to persuade others that such an approach is in their best interests.

In some past years Private Training Establishments in strife have been more newsworthy than secondary exams. But mostly its NCEA and Scholarship that have our Communications Unit running off in all directions. The reasons are obvious. Lots of young people in every part of the country are involved, the results have consequences for their lives, and every adult went through the schooling system, so everyone has an opinion. And also, school zoning influences property prices in our bigger cities.

So, in our second hot spot of secondary assessment, what are the communications challenges? The answer is many, some obvious and some subtle, which lend themselves to an analysis of the linkage of strategic communications to reactive communications. If we can get a strategic edge of understanding, we assist the hot spot management.

I think the problem is summed up well by this illustration.

The text reads "When faced with a totally new situation, we tend always to attach ourselves to the objects, to the flavour of the most recent past. We look at the present through a rear-view mirror. We march backwards into the future. Suburbia lives imaginatively in Bonanza-land." (Note 1)

There's truth in that. Let's see how it works in detail.

At the sharp end, the criticism of NCEA often hinges on critics thinking that a 'standard' is the same as a traditional 'subject', whereas most assuredly it is not.

The assessment system that you and I and all parents experienced when we were young had us sitting exams in subjects. But nowadays, for assessment purposes, subjects are broken down into much smaller units, called standards. Each standard is roughly analogous to a topic or even a single question in a traditional exam.

The virtue of breaking subjects down into smaller units is that this frees schools to mix and match what they teach to better meet their students' particular needs. Indeed, there is evidence now that this diversification is starting to happen, which is exciting.

In terms of logic, you would expect, all being well, these much smaller units of assessment to produce very different patterns of results from a much larger aggregated subject. Yet, when this becomes apparent, so often the critics pounce, claiming that, in comparison with the past, all is now chaos.

We all know that the smaller the sample in any statistical measurement the greater the margin of error. For each standard, typically an exam has one question. Depending on the wording, some years the question is going to be harder than other years and this will show up in the results. With the best will in the world, this will always occur to some degree, although, be assured, we are working to reduce year on year variability.

Likewise, the numbers of students attempting each standard is smaller than the numbers attempting subjects as a whole. The year before last we had one standard with only one candidate. Last year for the same standard there were 18 candidates. However, when we aggregate the results for standards back into subjects we actually have more stable results than in the last few years of School Certificate when that qualification wasn't scaled.

However, this point is consistently lost in the coverage, despite having been explained many times. Too much we are still looking in the rear vision mirror. The communications challenge - especially for a Chief Executive - is how to shift these views, especially when they are deeply held or widely experienced.

In addition, an assumption consistently heard is that, and I'm putting this very baldly, "For some to succeed, others must fail". A variant is "Failure is a fact of life and the fact that no-one fails anymore at school is a bad preparation for life."

This perspective sees education as like this illustration - a farmer sorting stock.

In the past, the 'sorting' involved about half the students failing School Certificate, those 'failures' then being cast for a particular role in life. More culls would take place and finally a small elite would head off to university. Then the school system's contribution to the management of employment hierarchy was done, at least till next year.

The problem is the world that generated that education system no longer exists. Globalisation has created a multi-layered, complex and constantly-changing economy. The demand for unskilled labour is minimal and the demand for skilled workers immense. Job security is rare and people's careers constantly change. This rewards countries, companies and individuals who are fast changing and quick learning. The slogan "learning for life" is not a cute daydream - it's necessary for social and economic success, individually and collectively. In this economy labour units are too valuable to waste, and some kind of training is necessary for almost all jobs.

Putting this another way, requiring people to fail doesn't work anymore - it isn't useful. Gaining a comforting social status from others being permanently in the bottom slot is a luxury we can't afford.

New Zealand has done pioneering work developing a National Qualifications Framework to match the requirements of this new economy. As a result we have a multi-layered offering of qualifications and courses. This means anybody, no matter their social or economic status, can find a way of advancing themselves and their lives, if they have ability and make the effort. That's a great thing. Learners still fail - but no one is required to fail.

The brightest and best don't need others to fail for their brilliance to be recognised. Yet, sadly, time and again, this imagined need for failure to define success keeps coming back in the rhetoric.

Now we are moving into strategic messages as part of the problem I've been discussing. We need to understand what the modern economy is and what it requires. The link between the modern economy and the Qualifications Authority's work needs to be emphasised again and again. We intend for it to be a key theme in our external communications.

So instead of a farmer culling sheep, we now have a qualifications system that for the learner is like this [click for Slide] a workshop in which the learner, by dint of effort, chooses the tools that he or she deems to be of value. In this world, the learner decides what's appropriate to his or her life, assisted by excellent teachers and tutors.

A more general problem of perceptions can be seen also, and it's not unique to the Qualifications Authority or the Education Sector. For want of a better name, we could call it the 'expectation of certainty' problem.

For example, in health we would sometimes get into strife with something like a mass screening for a particular illness. We'd say to the public something like "This procedure will get it right 70 per cent of the time. For the rest, half the remainder will be told they are ill when they are not and the other half will be told they are well when they are ill". And people would react with astonishment, saying how can you countenance something so sloppy? We would reply that in terms of net benefit to those who are ill and will have their illness detected and treated, this is a huge step forward. But for the public the uncertainty and imprecision of such a programme, despite its worth, would be troubling.

Or the reverse could happen. The public would reach for a test, expecting it to be 100 per cent specific and 100 per cent sensitive and not want to hear of the risks arising from it not being 100 per cent in either regard.

The Reserve Bank has had a similar problem with commentators criticising slight gaps between its published economic projections and actual economic results. The Reserve Bank sought to solve this by publishing historical numbers in percentages and forward projections as fractions. So they would say last year the economy grew at say 3.19 per cent and this year it will grow at about 3 and a quarter per cent. The quarter was supposed to communicate how approximate any projection has to be. The reporters all just laughed and changed the fraction back to 3.25 per cent. Their excuse was their software couldn't print fractions.

In education, people find the intrinsic uncertainty of assessment hard to accept and impute false meaning into assessment results.

Assessment by its nature is inexact. For example, our assessment specialists have estimated (Note 2) that in an average secondary school 3-hour examination a student has about a 50 per cent chance of scoring within 4 per cent of his or her ability level. In other words, in an exam a candidate with an ability level of 55 per cent has a 50 per cent chance of scoring below 52 per cent or above 58 per cent. That's in a traditional single 3 hour exam spanning a whole subject. The variables here include day-to-day fluctuations in the student's abilities and performance and fluctuations in the assessment, such as differences in individual questions and the way each question is marked.

This is a challenge - it requires work to reduce variability where possible and to explain variability where it is inherent.

You may have noticed that internal assessment in schools has come in for some stick in the media recently. The odd thing is, on average over time internal assessment is more accurate that one-shot end-of-year exams, mainly because of multiple sampling.

So where does that leave us?

First of all, people like me have to be realistic about what communications people like you can achieve. Complexity is difficult to sell. We need all the tools and wisdom at our disposal to assist us. You are critical to that, and behalf of CEs, I thank you.

However, that doesn't mean one should be give up and be silent. Good policies need to be explained. It's been said elsewhere that two basic rules should apply to external communications. They are "Tell the truth" and "Stomp every lie".

"Stomp every lie" is obvious. Where misinformation is placed in the public domain, always correct it - be vigilant and don't let things slip. A falsehood tolerated becomes accepted truth.

"Telling the truth" is the more interesting mantra. To tell the truth, you have to know the truth. So the first and most telling thing in the development of any communications plan is to work out your key messages. I think in too many communications plans there are too many key messages. This reflects a failure to work out what really matters or a fudge to cover up doctrinal contradictions. You should be able to count your key messages on one hand. Indeed, the ideal would be to have only one - one simple and very direct proposition.

Let me try this. The New Zealand Qualifications Authority ensures that New Zealand's qualifications are accepted as credible and robust here in New Zealand and internationally - a simple version of what we do.

Then external communications benefit from is repetition - repeating those key messages at every opportunity, which requires discipline and focus. One of the things I've observed is the risk of getting bored with your own message. You've said it 500 times already, but don't forget - it's new for the person who hasn't heard your story before. Likewise, you need to get the strategic context for your work understood.

Let me conclude with a nice tale about perceptions (Note 3).

Apparently during World War Two there was a famous mathematician in Moscow. He caused quite a stir when he published an article in Pravda saying that the good citizens of Moscow who were going to the air raid shelters were being silly. The mathematician said that he had calculated the size of the German Air Force coming over Moscow, the tonnage of bombs being dropped and the size of the city. The compelling conclusion, he said, was that the chance of being hit by a bomb was so remote that enduring the discomfort of an air raid shelter was ridiculous.

Well, the citizenry kept going to the air raid shelters and the good professor kept sleeping comfortably in his nice warm bed.

Then a couple of months later, during a raid, the famous mathematician was seen running into one of the air raid shelters. The people there said "Hold on - aren't you the comrade who said we are all being silly coming down here?" The mathematician was heard to reply "I know, I know, but I've had to rethink. There's only ever been one elephant at the Moscow Zoo, and last night the Luftwaffe got it."

Marshall McLuhan, The Medium is the Massage, p74, 1967

Hood & Strachan, Formal valuing of human competence: Effects and effectiveness,1996

Peter Bertsein, Against the Gods: The Remarkable Story of Risk

Page updated: 22 September 2006