The RAHK Public Opinion Poll
Here, I will translate from Mo's View:
In recent years, the Hong Kong media loved to report on the public opinion polls from a certain organization known as the Research Association of Hong Kong.
It is a good thing for another organization to break the monopoly of Robert Chung's Hong Kong University Public Opinion Programme. But the problem is that I do not appreciate the background of the Research Association of Hong Kong and its method of announcing the public opinion data.
The reason that Robert Chung's studies are trustworthy is that he conducted public opinion reserach for many years and his reporting format and data meet the standards of acceptance by the international scientific community. The accuracy and reliability levels are disclosed to the public. You can question Robert Chung's analysis or questionnaire design, but his data are authoritative for purposes of news reporting.
The Research Association of Hong Kong does not disclose the academic backgrounds of its members so that outsiders do not know if they are qualified to conduct public opinion polls. Worse yet, their press releases do not show accuracy and reliability levels like Robert Chung. They do not eliminate the problematic survey issues (for example, if the interviewee does not know nine out of ten politicians, then their ratings should not count). So how did their research come about? The media do not ask why or how, and they just quote the data which are academically suspect. As such, the media are misleading the public.
This post then goes on to quote information about the individuals associated with the Research Association of Hong Kong, hinting that they are politically biased and pro-Chinese Communist. The summary postiion is this:
If the Hong Kong media are responsible, their editors should figure about if these data are suspect before they decide to report these public opinion poll results.
That is a fair requirement.
Now I cannot fully answer that question. I will only look at the information as disclosed by the Research Association of Hong Kong, and it is more than what is suggested by Mo's View but still less than full disclosure.
For the purpose of discussion, I will use the latest RAHK poll about the support of the major Hong Kong major political parties among Hong Kong citizens. I will refer to this pdf. document.
The polling period is September 24-29, 2005. There were 811 completed surveys that were obtained through a probablity sample of telephone numbers in Hong Kong. This is the same mode of interview and slightly smaller sample size than the HK POP studies, but I have no further information on the quality of the interviewing. For example, RAHK does not explain how respondents are selected within a household (is it the person who picked up the phone? or is it randomly selected from all qualified persons?). Professionally, it is expected to mention the response rates (note: the HK POP responses rates are usually between 60% and 70%). In addition, there should be some mention of the standard problems about telephone sampling in Hong Kong (see, for example, Justin Mitchell's A Question of Survey Abuse in The Standard). I won't revisit those issues here.
In this RAHK survey, the respondents were asked to rank a list of Hong Kong political parties on a five-point scale: 1=Very unsupportive; 2=Unsupportive; 3=so-so; 4: Supportive; 5=Very supportive. The respondents can say that they don't know the political parties or they have no opinion, in which cases they won't provide any scores. The mean score would be based upon only those who knew the party and had an opinion.
For example, the top scoring political party is the Article 45 Concern Group with a mean score of 2.95 (8% did not recognize them and 3% had no opinion). The lowest scoring political party is the April Fifth Movement with a mean score of 2.35 (5% did not recognize them and 2% had no opinion).
With respect to the specific technical complaint in Mo's View, the Research Association of Hong Kong deals with the problem that some people do not recognize a specific political party or else has no opinion about it. However, I have a great deal of personal difficulty accepting the levels in the RAHK data. According to the data table, 100% of the respondents could recognize the Democratic Party of Hong Kong and the lowest recognition rate was 90% for the General Alliance (泛聯盟). Going back to what I told Justin Mitchell:
"Four people live there, three within the target universe of an adult. One is my mom who can't talk anymore. Another is a 79-year-old domestic helper who only speaks peasant dialect Shanghainese. So how's she going to do a interview about the chief executive? She has neither the verbal skills, knowledge nor interest. A Filipina maid is the third person. What are you going to ask her? I'm the fourth one. If you say you'll take 'anyone' who is willing to talk and I'm willing to talk, I'm that person. But it's misleading for polls to say they project the total adult population of Hong Kong. They don't."
Now even I am having a hard time keeping track of the Hong Kong Federation of Trade Unions (工聯會) versus the Hong Kong Confederation of Trade Unions (職工盟), or the Article 45 Concern Group (四十五條關注組) versus the April Fifth Action (四五行動). I very much doubt how anyone can keep track of these political parties. Certainly, nobody else in my household has heard of any of these parties.
Now we come to the second page of the document. This shows the demographic composition of the sample. I have quickly looked up some Hong Kong Census data and made these comparisons:
RAHK: 16% age under 18; 45% age 18-40; 34% age 41-65; 5% age 65+
HK Census: 15% age under 15; 29% age 15-34; 45% age 35-64; 12% age 65+
RAHK: 10% elementary school or less; 48% secondary school; 42% tertiary or higher.
HK Census: 26.5% elemenary school or less; 51.9% secondary school; 26.7% tertiary or higher.
RAHK: 16% Hong Kong Island; 26% Kowloon East; 18% Kowloon West; 18% New Territories East; 22% New Territories West.
HK Census: 18% Hong Kong Island; 15% Kowloon East; 14% Kowloon West; 24% New Territories East: 29% New Territories West.
Anyone can see that the demographic distributions are different, with the standard biases in telephone-based sample surveys for a political survey in Hong Kong (not enough younger and older people and too well educated). From RAHK's pdf document, it looked like they went through the political parties first before asking the four demographic questions. It is possible that they just interviewed the person who picked up the telephone and this may sometimes be a minor.
When a completed sample looks democraphically skewed, it is common to weight the sample back to the census figures but the RAHK makes no mention of such a procedure. It is also peculiar that the RAHK should have 16% of its respondents under the age of 18. What is the minimum age for the survey? 15? 12? Given that this is a political study, one would think that they should have interviewed only adults. If the least known political party has a recognition rate of 90% among a sample with 16% of the people being under the age of 18, then there must be some pretty politically savvy youngsters out there.
At this point, I can say this to the Hong Kong media. If such poll results were sent to the US media, it would have been tossed out. The reason is that they know that some of their readers are smart enough to recognize the deficiencies and will give them hell for it. In Hong Kong, the media does not know enough and their readers do not know enough to reject this type of survey. I don't think necessarily think that the RAHK poll results are totally wrong and the truth may be quite close. But there are some doubts. For example, if the RAHK study over-represents well-educated people in their sample, then maybe the Article 45 Concern Group are not as popular as often believed. I don't think that RAHK is cheating, but they really ought to try to meet professional standards in terms of rigor and disclosure.