Projects
Here is a couple of long-term projects I’m working on right now!
NoCha
Large language models have made huge strides, expanding from handling up to 2k tokens to even 2M tokens. However, evaluating them remains a challenge. It's tough to assess book-length inputs and avoid testing on training data. NoCha addresses this by involving readers who have read recently published novels for entertainment. These readers create true and false claims about the books they read, writing pairs of statements about the same event or character, differing just by the false information included in the false claim. Models then validate these claims using the book as context. The idea is simple, if you know that “Despite her skills as an Apoth, Nusis is unable to reverse engineer the type of portal opened by the reagents key found in Rona's wooden chest.” is true, then you know that “By using her skills as an Apoth, Nusis is able to reverse engineer the type of portal opened by the reagents key found in Rona's wooden chest.” is false. Yet, this task is tough for current models, and you can see their performance on the NoCha leaderboard.
LitMT
In LitMT we're looking into how machine translation can help translators and bring new stories to readers. Machine translation can make translators' work easier by reducing their cognitive load (O'Brien, 2012). But there's another side to it—the readers. We've build a website to share novels that have been translated by machines, including some that haven’t been available in other languages before. This project is as much about understanding how readers feel about these translations as it is about the translation process itself. By gathering feedback directly from readers, we hope to make machine translations better and more enjoyable to read. It's an exciting area to explore, and I believe that by getting readers' insight on what works and what doesn't, we can make literature more accessible to everyone, regardless of the language they speak.
HCI-MT
In this project, we're looking into how NLP systems can help translators work better and faster. It's clear these tools have a lot of potential, but figuring out how to fit them into the everyday workflow of translators isn’t straightforward. My work involves working with literary translators to study their use of these technologies. We aim to understand how to use these systems most effectively to support their efforts in producing high-quality translations more efficiently.