Culverhouse Conversations:
Keeping Pace with AI

The findings suggested that the text summarization tools performed at a moderately satisfactory level. However, the results also indicated that the AI-generated summaries replicated only about 20% of the important content covered in human-written summaries, with human authors clearly outperforming the automated systems. When they revisited the study years later with modern generative AI, the results were significantly different.

Now, AI writes better than humans on most linguistic dimensions. It does a great job with clear structure, proper tone, professional language, and overall effectiveness. The only place AI still struggles is in conveying emotion. It can sound polished and professional, but it doesn’t truly capture human feeling or authenticity.

In the updated study, the researchers employed business prompts that required responses from the perspective of a CEO. Each prompt was answered by a human executive, ChatGPT, and Gemini, allowing for a direct comparison of how each “writer” approached the identical communication task.

To evaluate the effectiveness of each generated response as a form of leadership communication, the researchers applied a taxonomy derived from prior literature. Approximately 500 participants assessed the responses using the expanded 18-dimension evaluation framework. “In every category, AI surpassed humans. In every category,” Naidoo said.

Venugopal asked how the evaluations worked. “The 500 were evaluating all three at the same time? Is there overlap of evaluators for the same document?”

Naidoo said yes, “all 500 participants evaluated each document”.

Venugopal then inquired about the aggregate score, and the efficacy rate, which had been 20% before.

“This time it's showing us that in every case, in 100% of the cases, AI surpassed humans,” Naidoo said. “Except for emotion.”

Emotion is central to leadership communication. “Leaders that are most successful are ones that leverage empathy and emotion,” Naidoo said, pointing to ongoing research on employee support and retention.

Venugopal asked if they gave a sample essay of the CEO and then asked AI to write it, but in a different voice, because maybe then the emotion could be matched.

Even then, the outcome didn’t change.

What wasn’t surprising was everything else. “Forceful… decisiveness… it was winning each time,” Naidoo said. In category after category, AI didn’t just compete. It led.

Dulek jumps in with the simple question, “Is it better than the analysts?”

Venugopal’s says it’s “surprisingly horrible,” and giving AI more information makes it worse. The models start overthinking and ignore what they’ve learned and rely too heavily on the new information.

“Their training is being overridden by the context that we give,” Venugopal said.

Naidoo connects this back to his own research, noting that AI tends to assume the worst. AI doesn’t like uncertainty, so it resolves it by making things sound definite and often negative.  That same forcefulness shows up again.

Mobbs wanted to know whether giving more history, like multiple earnings calls, would help. Venugopal admits they tried variations, but limits like context size get in the way.

When they remove identifying details, like company names, products, the AI struggles badly, especially with smaller firms. But for big, well-known companies, it still performs reasonably well. Somehow, it can infer what company it’s looking at based on patterns.

The takeaway from the exchange is that AI isn’t a neutral tool that summarizes information. It reshapes it and makes language more certain, forceful, and sometimes less accurate. It can recall facts, but struggles with nuance, probability, and context, which is important in research and decision-making.

AI and disclosure bias

Doxey’s research is on how people and AI judge what information companies should disclose, and found AI shows a clear bias. He and Dr. Chez Sealy designed a study where both investors and auditors rated how important different company disclosures are.

Ideally, the auditors would match the ratings of the investors, but they did not. In their results, users wanted more disclosure and auditors consistently rated disclosures as less important.

The expectation was that AI, which can take on different roles, might better match the user perspective. Instead, AI was even more extreme and rated almost everything as highly important to disclose. It showed a strong conservative bias.

“So auditors were more bullish?” Naidoo asks.

“Auditors might say something on a seven-point scale and have a two or a three to disclose. The users might have been a four or a five, and the LLM would be like a six or a seven,” said Doxey. “If they just sort of blindly take what it says, they’ll be much more conservative. But if they actually use it… it actually brings them very close to where the users are.”

Venugopal suggests, “Can you nudge the LLM to move up?” by prompting it to reconsider its answers. Doxey acknowledges the idea but emphasizes, “there seems to be a conservative bias. It’s going to err on the side of disclosure.”

The conversation shifts to practice. Why aren’t auditors fully using AI? Doxey points to constraints of client confidentiality, regulatory oversight, and repeatability. “LLMs just don’t work that way. There’s still that element of we don’t know exactly what we’re going to get out of it this time,” Doxey said.

Naidoo asks “Is it better to be more conservative from a risk perspective?” Doxey responds, “The problem with being too conservative is you are telling companies to disclose a whole bunch of information that… is actually immaterial.”

Doxey reframes accounting and auditing as solutions to human interaction problems like trust, interpretation, and decision-making. But in a world where “the company’s AI system is putting out real economic data points” and another AI consumes them, “materiality goes out the window.”

Dulek lists a key insight, saying “It doesn’t deal well with ambiguity.” Doxey agrees, translating it into accounting terms. “We would call that a conservative bias… it’s kind of losing the nuance.”

It comes back to training. Can AI be tuned to behave differently? Naidoo is optimistic but Venugopal says, “Training is not going to cover all scenarios… it’s very tough for them to move faster” in new environments.