English Español Français Deutsch
← Back to the list
Foundational figures Methodologists Directly upstream of ai100's methodology

David Jurgens

Whether language models that handle factual questions cleanly can also handle the kind of social knowledge that determines what humans actually mean when they say things.

The 2023 paper "Do LLMs Understand Social Knowledge?", with Jurgens as senior co-author, ran modern LLMs through a battery of tests built from sociolinguistics research — implicature, indirect speech acts, conventionalized politeness markers, the social inferences a competent human listener makes without thinking about them. The results were mixed enough to matter: LLMs that scored high on factual QA benchmarks made systematic errors on social inferences that any reasonably socialized human gets right, especially when the social context was non-American or non-mainstream. For ai100, this connects to a question we have to think about — when an engine recommends one brand instead of another, what social signal is the engine reading from the query, and would a human reading the same query interpret that signal the same way.

Worth following when
you want LLM evaluation grounded in sociolinguistics — testing whether the model understands what humans mean, not just what they literally say.
Topics
sociolinguistic evaluation of LLMs; computational social science with language models; the gap between factual competence and social competence in AI systems.
Key works
"Do LLMs Understand Social Knowledge?" (2023, senior co-author); body of work on sociolinguistics and computational social science; U Michigan publications connecting NLP and social science research.