Memex jr. – eye square’s customer dialogue with LLM-based knowledge

“MEMEX jr.” is the first eye square project primarily concerned with knowledge management, developed by a team of data scientists and artists (Frederik Neitzel, Daniel Döhringer, Moritz Mattern, dennis Rudolph and Dr. Julia Nitschke) that had previously also cooperated in eye square’s extensive augmented reality work.

eye square has launched “MEMEX jr.”, a LLM-based AI-Avatar that draws on the know-how and knowledge of its top scientists and consultants. Named in honor of Vanevar Bush’s visionary 1945 article “As we may think”, this disruptive new tool will be made available to selected customer groups. The backbone of “MEMEX jr.” is fed with a heterogeneous set of the proprietary e2-knowledge derived from articles, videos, sound files and other content.

eye square’s CEO Michael Schiessl commented: “I am impressed by how far the combination of vector data banks and LLMs carries. The avatar explains some of the insights very well, such as our unique framework model of neurosemiotics, definitions of key concepts such as attention, experience or meaning. It does this at least as well as our senior management would – given the obvious fact that much of the externalized knowledge is instantly available. Unlike a human executive it is available 24/7 – beyond time zones, times of day, schedules, and off-times. This major step frees up time and resources for our management. The outcome is scientifically very precise, gives reasonable strategic outlines and overall has big creative power. I believe clients who want to have a fast eye square view on broader questions will definitely benefit from it. “MEMEX jr.” also serves as a great platform for our internal knowledge distribution.”

eye square’s CTO Andreas Thoelke adds: “MEMEX jr. perfectly implements our credo of combining science, technology, and art. It is a proof of concept for our AI strategy which foresees the application of AI primarily in knowledge management, for easier data analysis for accelerating studies and projects and finally “semantic eye tracking”. The latter is especially promising as it integrates major areas of our neurosemiotic model, especially semantics and system 0.“

Meet “MEMEX jr.”

“MEMEX jr.” System and Function

“MEMEX jr.” is the first eye square project that deals primarily with knowledge management and was developed by a team of data sciencists and artists (Frederik Neitzel, Daniel Döhringer, Moritz Mattern, Dennis Rudolph, and Dr. Julia Nitschke) that had previously also cooperated in eye square’s extensive augmented reality work. With a keyboard or voice-activated interactions, users can seamlessly engage the system to capture queries, which are then run through a comprehensive knowledge database. The integration of Retrieval Augmented Generation (RAG) ensures unparalleled precision and relevance in information extraction, since semantic retrieval, aided by vector databases, refines the accuracy and richness of the responses.

These curated responses then undergo another layer of optimization as they are channeled through a Large Language Model. Employing nuanced prompt engineering, the LLM has been tailored to assume ‘Michael Schiessl’s persona’, ensuring that the responses are both personalized and refrain from venturing into speculative territory. This approach addresses any knowledge gaps in LLMs and guarantees the delivery of current and contextually appropriate information. To encapsulate the experience, the feedback is synthesized using a text-to-speech model, echoing the distinct cadence of Michael’s voice. With the audio prepared, our Unity-driven VR/AR environment detects and retrieves these audio files. As a result,  Michael’s Avatar articulates the responses, offering users an engaging and realistic conversation experience.

“MEMEX jr.” Exemplary Chat

Question: What is neurosemiotics?

Answer: “Neurosemiotics is eye square’s integrated analysis model for understanding human experience, specifically in relation to brands, advertising and products. It combines theories and insights from neuromarketing, deep psychology, and social cognition to create a descriptive model that merges qualitative and quantitative methods. Neurosemiotics zooms in on the central levels of psychological representation, encompassing perception, feeling, and thinking. At each of these levels, dedicated methods, and measurement procedures like eye-tracking, electrodermal activity (EDA), and electroencephalography (EEG) are employed. The goal is a comprehensive grasp of consumer preferences and market supply. For marketing and market research enthusiasts, neurosemiotics offers a conceptually sound and empirically measurable avenue to delineate the impact of brands, campaigns, and products.”

Question: What is system 0?

Answer: “System 0, as described by eye square, refers to the perceptive-sensory experience, the real perception. It is the most fundamental and quickest type of system, often referred to as “sensory perception.” It captures the pure physical-neurological perception, where neither cognitive nor implicit/emotional centers are engaged. This model of perception goes beyond implicit and explicit explanations and is based on extensive attention data collected by eye square over the last ten years. The receptive quality of System 0 can help build longer relationships and is valuable in understanding the advertising impact.”