eye square’s research methods can of course also be used for analyzing AI-generated contents or services, no matter if it is texts, ads, videos or websites. When we started the AIX Lab, however, we wanted to work on two areas of expertise: Understanding how LLMs and Gen-AI can be used for our research work and how they might change our company and the industry. And secondly, always eye square’s central mission: To dive into the experience of users and consumers when it comes to AI.

eye square has been using different AI tools for data processing and analytics for years and is constantly working on dedicated tools. Of course, the news around ChatGPT and other Gen-AI models fanned our curiosity about possible productivity gains in research, be it formulation of study guidelines, summing up qualitative insights from interviews or coding open answers from surveys.
 

The results have been mixed so far. While the tools can be helpful in summarizing long texts e.g. research papers, they are sometimes unreliable with even rather simple tasks such as counting in larger data sets – e.g. results differ, even with same prompts and same data sets.
 

Regarding qualitative results, LLMs are not very good in interpreting and not really able to differentiate what is important and what is not. Instead of finishing tasks autonomously, the usage of AI tools up to now means extra work such as finding different ways of prompting and doing evaluations manually in parallel to double-check and improve quality of results.

The most important questions for a human experience research company like us is how humans experience the interaction with AI and how AI might impact user behavior in the nearer future.
 

Together with Strategic Labs, a Berlin-based foresight agency, we try to figure out how online behavior and media usage might change over the next years, to be prepared to offer the best fitting research methods. We consider the whole range of AI-based applications and their possible impact on user behavior, from conversational interfaces like bots and voice assistants to the possibly ubiquitous spread of AI-assistants in everyday life (e.g. replacing Google search or dedicated apps). And of course, this also includes understanding the impact of AI-generated content on human digital experience: How well do campaigns work that have been created with the help of Generative AI? How does AI impact users’ attitude towards digital media in terms of trust and digital brand-building – and how does this impact current business models?
 

In terms of user experience, an important aspect for us is how consumers experience the interaction with AI. Based on in-depth interviews, our AIX Lab has developed a model and scales to describe and measure AI Experience.

As proof-of-concept we surveyed how users experience ChatGPT. Results of our study for Germany show that there is a big group of people who are worried about the potential implications which ChatGPT could have for their lives. Many try to avoid the topic as much as possible and feel overwhelmed by the developments. Half of our sample had already used ChatGPT, and the majority of them were either confident of being able to cope well with the developments or were even excited about the possibilities that LLMs will bring as an enabler for them.
 

When developing and launching new AI-based features, services and content, companies need to be aware of the very diverse states of mind and mixed emotions of their target audiences - and measure, analyze and consider them wherever possible. eye square’s research approaches will help to find a user-centric way into implementing AI.