As part of our research, we ask users to do a side-by-side comparison of responses to the same prompt from different platforms. After users pick their preferred response, we ask them why they chose it.
After reviewing 1,135 reasons why users prefer certain responses on certain platforms, we color coded them to see how much the reasons vary by platform.
As you can see, there are some marked differences.
Why users prefer Bard:
- Formatting (33%): The presentation of data with bullets, bold phrases, and tabular information helps users process information more effectively.
- Clarity (23%): Information is well-explained, broken down, and contains the right level of detail.
- Examples (16%): Providing examples helps users better understand the responses.
“I liked the spacing, the shorter length of the sentences. With a mathematical question, I personally would like to see more numbers and less words in the answer, though it is good to have some words. Having a space between each line, and not too much text made it easier to concentrate on each step instead of concentrating on reading and trying to pick numbers out of a long sentence.”
Why users prefer GPT-3.5:
- Well-explained (31%): The information is detailed, well-explained, and provides a holistic view, making it easy for users to understand.
- Accuracy (13%): The answers provided by GPT 3.5 are perceived as accurate, and users find them actionable for making decisions.
“It explained it and just made me believe it was the right answer.”
Why users prefer GPT-4:
- Concise (24%): Information provided is just the right amount, avoiding cognitive overload.
- Easy to Understand (14%): Complex concepts are broken down into simple language, making it easier for users to comprehend.
- Well-formatted (13%): Information is visually presented in an organized manner with bold headings, bullets, and a clear structure.
“I just really preferred the way the AI bolded each topic - it makes it so much easier visually to read everything and stay engaged.”
Why users prefer Claude:
- Concise (27%): The responses are of the right length and use simple language that is easy to understand
- Well-formatted (16%): Users appreciate the bullet point format that presents information in a clear and organized manner
“I like that it's broken-down into bullet points which makes it easier to understand and follow.”
Why users prefer Bing:
- Sources (38%): Users appreciate the inclusion of links for additional information, providing validation
- Concise (37%): Users like receiving short and quick responses
“This question was one that needed a quick response, which it provided, but also offered links to other information if the user wanted to explore further.”
Implications
- AI PLATFORMS ARE DIFFERENT: This sounds obvious but when we compare why people like responses and what words they use to describe their preference, we see real differences emerge (just look at the colors)! Companies should consider this when looking to leverage certain platforms and carefully research if their type of content matches well to what the platform excels at. For companies that want to cite sources, Bing is clearly appreciated for the ability to show a source. For companies that want their more nuanced topics concepts well explained, Bard and Claude are well-regarded for that.
- TRUST AND CREDIBILITY: Bing's ability to cite sources is one of the biggest drivers of preference we saw across platforms and topics. Users are reassured about the quality and accuracy of the information they receive from AI when there are links to websites. This helps assure readers that the information is coming from a valid source and is current information. Long-term, AI platforms will need to carefully consider how they "show their work" to users. Those that give users the option to dive deeper into source content may stand out from the rest.
- WELL SAID: AI platforms win when they present content in an easy-to-follow format using tools like bullets and concise paragraphs. In today's fast-paced “scrolling” world, AI responses that favor skimmable, shorter content may get more traction. That said, users should also be presented with the option for more thoughtful answers where context and more detail would be important. This idea of optionality of content length seems well suited for the deliberative nature of AI.
