What does Generative AI make of University Reputation?

Just as SEO (search engine optimisation) boosts websites to achieve higher positions in traditional search engines, GEO (generative engine optimisation) seeks to improve the responses to AI queries by tailoring content that appeals to target audiences and resonates with AI engines.

But, before launching into that process, we thought it would be timely to take a very basic ‘layperson’ view of how one of the AL chatbots, ChatGPT, is presenting various universities. We selected 100 universities across the UK and Ireland. We have worked with some of them and know many of them. It was an interesting exercise to ask AI what it thinks of each of them.

And, as AI engorges data from multiple sources and regurgitates impressions and opinions, we form a picture of the user journey – whether potential student, academic peer, industry partner or prospective funder – that presents a new topic of discussion on how and where universities place their content and seek to influence their stakeholders.

We simply asked: ‘What is the reputation of …… university?”. In order to keep the answers comparable and fair, we chose not to adopt a ‘persona’ as say a prospective student or employee. The results were revealing.

We found that the overwhelming focus of the answers about a university’s reputation was on positive factors, as if Chat GPT was seeking to find something good to say about each university, whatever its quality and standing. Even where weaknesses were identified, they tended to be described as ‘challenges’ balanced with some wider context.

It generally told us about the location, heritage, academic standards, student lifestyle, research highlights and national and international standing of the university. BUT the answers and the format were not consistent for every university. Some responses contained images (flattering and off-putting), whilst others were pure text; some referenced position in national and international rankings, whilst others gave a vague positioning; some contained scathing reviews from student forums and Reddit, whilst others referenced glowing press releases issued by the university’s own press office.

ChatGPT arranged the answers with some logical headings, but the headings weren’t the same for every university. The answer for the reputation of the University of Aberdeen covers:

• General Reputation

• Academic Strengths

• Rankings

• International Perspective

• Pros and Cons

The general reputation is fairly factual, giving date of establishment, type of university, ‘public research university’ and location (yawn for those in the UK but perhaps useful if you are in China or the United States). Pros and Cons are similarly banal, and we don’t learn much by finding out that it has ‘Solid student satisfaction scores in the UK’. Academic strengths are much more interesting, and reveal some true differentiating subjects: Medicine and Health Sciences, Engineering (especially petroleum and energy), Geosciences, Law, Divinity and Theology.

And Aberdeen starts to show ‘brand blue water’ when you read that it is considered a good choice “for postgraduate study in niche areas like energy law or offshore engineering”.

Does this help us compare Aberdeen with, say, the University of Glasgow? The ChatGPT headings for the University of Glasgow are different to Aberdeen: for example, academic strengths for the latter are hidden under the heading Global Ranking and Research Excellence, where ChatGPT says it does well in THE Impact Rankings in Law, Medicine, Psychology, and Social Sciences.

But you can just ask ChatGPT to “compare University of Aberdeen with University of Glasgow” and you will be fed comparative information under six headings: Academic Reputation & Rankings, Courses & Specialties, Location & Environment, Campus & Facilities, Student Life & International Outlook, Tuition & Scholarships. These are followed by a summary table giving its Best Option and a final Who Should Choose Which? category that attempts to match each university to student type.

With such a confident voice, what’s not to like? Well, as the control of information fed becomes more and more remote from the university, the worry is that it may be inaccurate or inadequate. So, we decided to ask ChatGPT where it sourced its information.

It told us that AI would probably use several methods to compile its responses, including:

• Data Aggregation - from academic rankings, student reviews, and employment outcomes.

• Sentiment Analysis - of online content, including social media and forums, to gauge public perception.

• Custom Search Tools - summarising relevant articles, reports, and statistics.

• Predictive Insights - using tools to predict future trends in university performance based on historical data.

It even boasted that “Using these tools, AI can provide a comprehensive view of a university's reputation from diverse and updated sources”.

The problem revealed from our exercise is that these responses are not consistent, fair or comparable. We noticed too that the information was not necessarily up-to-date and consequently could be inaccurate. And, of course, when probed further, AI has a habit of hallucinating – making up answers until it is corrected.

In general, ChatGPT used university-generated material - sometimes sourced directly and sometimes linked through other channels such as Times of India, Wikipedia or Study In UK. It relied heavily on rankings from US News, QS and THE and in some cases, the Academic Ranking of World Universities. National league tables such as The Times/ Sunday Times Good University Guide, The Guardian University Guide and Complete University Guide were used for UK universities. Where it could, it seemed to feed media articles on topics such as funding, accommodation and social issues affecting students from established outlets such as The Guardian, The Times and The Scottish Sun. Interestingly, the most common bulletin board used was Reddit, with all the rabbit holes aligned.

Comparing the universities, it seems to us that the less well-known a university is, the more ChatGPT relies on university-generated material. If the universities themselves are providing the general text that accompanies these official sources, e.g. universities normally write their own rankings profiles, then we can assume much of ChatGPT’s editorial answers originate from the universities themselves. But it doesn’t appear that some universities are putting their brand strengths at the core of what ChatGPT is scraping. The University of Manchester’s answer had no mention of its pre-eminence in graphene, for example, on which it has built much of its reputation recently.

So the learnings are, from asking ChatGPT, "What is the reputation of x university?"

• ChatGPT (and other generative AI bots) is useful for very quickly assembling public domain information on university reputation (rankings and historical dates) and providing rankings data.

• The answers were not very strategic, distinctive or interesting. Taken together, most universities sounded very similar.

• The results were focused on the sort of questions and information that is typically used for student choice of universities, indicating that the majority of information that ChatGPT is using has that focus. But, of course, universities have a wide range of other stakeholders.

• Responses to questions are not consistent across universities. They can be out of date and inaccurate.

• Although ChatGPT relies heavily on university-generated information, there is little evidence of universities strategically crafting and placing their competitive advantages before their audiences.

• Some universities are better at showcasing their academic strengths than others, and where they do this, they could be attracting funding, and discerning students who value academic expertise over student experience.

• Universities clearly need to be aware of how their key audiences are accessing information about them. Our answers from ChatGPT may be useful in helping to dial up messaging that is being missed by AI tools, or to amplify points that resonate.

• But above all, the exercise reminded us of the need for expert – human – interpretation of AI data, just as for desk or market research.

That leads to the question of how colleagues involved in university reputation management can influence the sources available to open-source, Large Language Model (LLM) tools like ChatGPT.

What next?

Our work with universities focuses on finding and articulating key drivers that can be communicated to target audiences through specific channels, in order to influence their perceptions. These can cover a host of factors, including first impressions, student experience, research achievement, internationalisation, city location, partnerships, news coverage and institutional brand.

If you would like to know more about what AI chatbots are saying about your university compared with your peers, and the extent to which this aligns with your strategy, reputation and brand proposition, then get in touch.

[1] https://www.statista.com/statistics/1471959/use-of-chatgpt-by-age/