How Not To Rank Universities

In some countries, it is obsessive-compulsive to rank universities.  Who is ranked number one in the country in college football? in college basketball? in college baseball in the United States?  For even something as simple as college basketball, the ranking methodology can be controversial.  After all, not all teams play against all other teams.  So a college basketball team with a 28-0 record may not be as 'good' as one with a 26-2 team on account of the fact the first college played a 'softer' schedule against weak teams.  So it stands to reason that it is much harder and more controversial to come up with an overall ranking of all universities in a country.

In Hong Kong, there is not much interest in ranking universities by football or basketball.  However, there is an annual ranking of universities that is always the subject of much controversy.  The particular ranking is conducted by Education18.com (the 2005 report is here) and here is this year's rankings:

1. The University of Hong Kong
2. The Chinese University of Hong Kong
3. The Hong Kong University of Science and Technology
4. The Polytechnic University of Hong Kong
5. The Hong Kong Baptist University
6. The City University of Hong Kong
7. The Hong Kong Institute of Education
8. The Lingnan University

For two days in a row, the Vice-Chancellor of Lingnan University (ranked at the bottom this year) has published two opinion essays in Sing Tao and another one in Ming Pao (see InMediaHK).  Sour grapes?  No, more like the victim of some bad methodology that is positioned as being 'useful for the purpose of consultation.'

How did this ranking come about?  There are seven components to the overall score.

20%: public ranking of local universities
20%: admissions grades
20%: research performance
10%: performance of graduates
10%: research output
10%: lecturer and student ratio
10%: library collections among the universities

This is a hodgepodge of information: the public ranking is based upon a public opinion poll commissioned by Education18.com; the performance of graduates is based upon a survey of employers done by Education18.com; the rest of the information is based upon publicly available data, some of which are outdated.  Apart from the public opinion poll, the other pieces are there because they can be had cheaply.  Is this the ideal bundle of information?  Far from it.  This is convenient, and that does not make it useful or informative.

In the following, I discuss each component separately.

Public ranking of local universities

This is based upon a public opinion survey conducted by the Hong Kong University Public Opinion Programme.  Yes, read the name of the organization again -- it is the Hong Kong University Public Opinion Programme that ranked Hong Kong University as number 1.  Can you spell 'conflict of interest'?  Education18.com is paying for the survey and they really should have used anybody else but a university-affiliated organization.

In the HKU POP final report, it is written:

The researcher is aware that the POP team is part of the University of Hong Kong, which is one of the universities rated by the respondents.  As a precaution to eliminate any possible bias due to desirability effect, all respondents were explicitly told at the beginning of the interview that the POP team was an independent research team, and the respondents should simply report honestly what they felt.

Well, that's kind of na´ve to assume this little speech up front is going to 'eliminate any possible bias due to desirability effect.'  It is entirely possible and quite likely that this little speech actually highlighted Hong Kong University and created even greater bias.

The next question is just how the public is going to rank the universities, taking into account "the university's local and international reputations, facilities, campus environment, qualification of its teaching staff, academic research performance, conduct and quality of its students, its learning atmosphere, as well as the diversification and degree of recognition"?  Please remember these qualifications are read over the telephone during the interview.  Are you overwhelmed by the requirements yet?

Remember that the survey respondents are drawn from the general population.  Thus, I am a potential respondent.  So I start thinking about how I might come up with a ranking in response to the question while considering all the aspects that I was just told to consider.  Here are some random thoughts that I jotted down.

Given my lack of knowledge, why would anyone be interested in my opinion?  For one, I wouldn't even value my own opinion.  I'd love to sit down with the people who designed this survey and pepper them with specific questions: "Dr. Chung, what do you know about the degree of recognition for the translation diploma programme at the Chinese University of Hong Kong?  the size and qualifications of the staff at the nursing programme at the Hong Kong Baptist University?  the availability of swimming pools and their operating hours at these universities?  is it easy or hard to find parking spaces?  the quality of the food at the school cafeteria?  ..." and watch them hem-and-haw when they don't know.

Admissions Grades

This is the average admissions grade on two A-level subjects across all programmes offered at the university.  This is very much a function of the programmes, then.  One would have liked to see some consideration of basic skills such as Chinese and English language competence.

Earmarked grants for research

This is a function of the success rates of the grand applications and the total amount of grants.  Well, it seems that if some tycoon comes along and drops a HK$1 billion endowment, then the faculty could get on with the real work of study and research as opposed to spending a lot of time writing grants to everyone to scrounge for relatively small grants that come with loads of administrative paperwork.

Performance of graduates

The methodology is unexplained, probably because the details would be quite embarrassing.  It seems that Education18.com (and this was not done by HKU POP) spoke to some employers (and the big technical hurdle is to find a representative sample instead of a convenience sample of personal acquaintances).  I infer from the table that there were about 250 employers in the sample.  It also looked as if the employers were asked about their most preferred university graduates, together with a list of reasons.  This limits the kind of employers that can be asked.  For example, the owner of a street food stall would have no need for university-educated workers, so that eliminates a lot of small enterprises.  Very large employers are also excluded, because HSBC hires by the hundreds and the human resource manager cannot say that they prefer this or that university; rather their decision ought to be based upon the specific attributes of the applicants as matched up against the job requirements (e.g. you might prefer University A for merger and acquisitions, but University B for human resource management due to the existence of specialized programs).

Let us look at a more detailed hypothetical example.  The employer is a newspaper editor, whose reasoning may go something like: "During the past three years that I have worked here, I hired ten reporters.  Of the five that came from University A, one was really good but the other four were horrible.  The one that came from University B is my star reporter.  The two from University C are disasters ...  And I have never hired anyone from Universities E and F, so I can't say."  How would you like this newspaper editor to state his preference of university graduates?  And what would any answer from him mean anyway?  This is just junk survey data passed off as science.

Research output

This is based upon counting "scholarly books, monographs and chapters; journal publications; conference papers; creative & literary works, consulting reports and case studies; patents, agreement, assignment and companies; other output."  That's cute -- as all items in the same classification are counted the same as each other; so a keynote conference address at the highly esteemed American Psychological Association would be treated the same as a poster session at an obscure conference, and so on.  And it would seem that a university can boost its ranking by filing to incorporate a large number of shell companies that don't have to do anything.

Lecturer and student ratio.

The Hong Kong Institute of Education has the lowest number of 1:8 while Lingnan University is highest at 1:16.  What does this mean anyway?  In the United States, the game is played this way -- you can spend a lot of money to hire one big-name full professor or you can take that same money and hire six adjunct faculty members who only teach classes, hold other jobs and has no tenure or employee benefits.  So, are you better off with one or six faculty members?

Library collections

So Hong Kong University has 2,102,442 books, 45,670 journals, 6,614 electronic databases and 34,711 electronic journals for a total of 2,189,437 items.  Divided by 8,835 students, this gives a ratio of 248:1.  By contrast, the Hong Kong University of Science and Technology only has a ratio of 113:1.  Hmmm ... but aren't we forgetting that this is a 'science and technology' place, and would a student who wants to major in science/technology have more resources here than at HKU (whose 248:1 ratio includes many non-major-related resources)?

One of the stated goals of this research was to provide a set of reference materials to assist students who are about to choose a university.  The factors above are not necessarily unimportant, but the prospective student ought to check out the details (such as lecturer-student ratio, library resources, classroom equipments, dormitories, social life, etc.) of the exact program that he/she wants.  Meanwhile, this particular general ranking based mostly upon public data available as a matter of convenience and not by relevance is not doing anyone any favors.