Publications

While long-context large language models (LLMs) can technically summarize book-length documents (>100K tokens), the length and complexity of the documents have so far prohibited evaluations of input-dependent aspects like faithfulness. In this paper, we conduct the first large-scale human evaluation of faithfulness and content selection on LLM-generated summaries of fictional books. Our study mitigates the issue of data contamination by focusing on summaries of books published in 2023 or 2024, and we hire annotators who have fully read each book prior to the annotation task to minimize cost and cognitive burden. We collect FABLES, a dataset of annotations on 3,158 claims made in LLM-generated summaries of 26 books, at a cost of $5.2K USD, which allows us to rank LLM summarizers based on faithfulness: Claude-3-Opus significantly outperforms all closed-source LLMs, while the open-source Mixtral is on par with GPT-3.5-Turbo. An analysis of the annotations reveals that most unfaithful claims relate to events and character states, and they generally require indirect reasoning over the narrative to invalidate. While LLM-based auto-raters have proven reliable for factuality and coherence in other settings, we implement several LLM raters of faithfulness and find that none correlates strongly with human annotations, especially with regard to detecting unfaithful claims. Our experiments suggest that detecting unfaithful claims is an important future direction not only for summarization evaluation but also as a testbed for long-context understanding. Finally, we move beyond faithfulness by exploring content selection errors in book-length summarization: we develop a typology of omission errors related to crucial narrative elements and also identify a systematic over-emphasis on events occurring towards the end of the book.

Pretrained language models underpin several AI applications, but their high computational cost for training limits accessibility. Initiatives such as BLOOM and StarCoder aim to democratize access to pretrained models for collaborative community development. However, such existing models face challenges: limited multilingual capabilities, continual pretraining causing catastrophic forgetting, whereas pretraining from scratch is computationally expensive, and compliance with AI safety and development laws. This paper presents Aurora-M, a 15B parameter multilingual open-source model trained on English, Finnish, Hindi, Japanese, Vietnamese, and code. Continually pretrained from StarCoderPlus on 435 billion additional tokens, Aurora-M surpasses 2 trillion tokens in total training token count. It is the first open-source multilingual model fine-tuned on human-reviewed safety instructions, thus aligning its development not only with conventional red-teaming considerations, but also with the specific concerns articulated in the Biden-Harris Executive Order on the Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence. Aurora-M is rigorously evaluated across various tasks and languages, demonstrating robustness against catastrophic forgetting and outperforming alternatives in multilingual settings, particularly in safety evaluations. To promote responsible open-source LLM development, Aurora-M and its variants are released.

For the past decade, temporal annotation has been sparse: only a small portion of event pairs in a text was annotated. We present NarrativeTime, the first timeline-based annotation framework that achieves full coverage of all possible TLinks. To compare with the previous SOTA in dense temporal annotation, we perform full re-annotation of TimeBankDense corpus, which shows comparable agreement with a significant increase in density. We contribute TimeBankNT corpus (with each text fully annotated by two expert annotators), extensive annotation guidelines, open-source tools for annotation and conversion to TimeML format, baseline results, as well as quantitative and qualitative analysis of inter-annotator agreement.

Large language models (LLMs) are competitive with the state of the art on a wide range of sentence-level translation datasets. However, their ability to translate paragraphs and documents remains unexplored because evaluation in these settings is costly and difficult. We show through a rigorous human evaluation that asking the GPT-3.5 (text-davinci-003) LLM to translate an entire literary paragraph (e.g., from a novel) at once results in higher-quality translations than standard sentence-by-sentence translation across 18 linguistically-diverse language pairs (e.g., translating into and out of Japanese, Polish, and English). Our evaluation, which took approximately 350 hours of effort for annotation and analysis, is conducted by hiring translators fluent in both the source and target language and asking them to provide both span-level error annotations as well as preference judgments of which system’s translations are better. We observe that discourse-level LLM translators commit fewer mistranslations, grammar errors, and stylistic inconsistencies than sentence-level approaches. With that said, critical errors still abound, including occasional content omissions, and a human translator’s intervention remains necessary to ensure that the author’s voice remains intact. We publicly release our dataset and error annotations to spur future research on the evaluation of document-level literary translation.

The rise in malicious usage of large language models, such as fake content creation and academic plagiarism, has motivated the development of approaches that identify AI-generated text, including those based on watermarking or outlier detection. However, the robustness of these detection algorithms to paraphrases of AI-generated text remains unclear. To stress test these detectors, we build a 11B parameter paraphrase generation model (DIPPER) that can paraphrase paragraphs, condition on surrounding context, and control lexical diversity and content reordering. Using DIPPER to paraphrase text generated by three large language models (including GPT3.5-davinci-003) successfully evades several detectors, including watermarking, GPTZero, DetectGPT, and OpenAI's text classifier. For example, DIPPER drops detection accuracy of DetectGPT from 70.3% to 4.6% (at a constant false positive rate of 1%), without appreciably modifying the input semantics. To increase the robustness of AI-generated text detection to paraphrase attacks, we introduce a simple defense that relies on retrieving semantically-similar generations and must be maintained by a language model API provider. Given a candidate text, our algorithm searches a database of sequences previously generated by the API, looking for sequences that match the candidate text within a certain threshold. We empirically verify our defense using a database of 15M generations from a fine-tuned T5-XXL model and find that it can detect 80% to 97% of paraphrased generations across different settings while only classifying 1% of human-written sequences as AI-generated. We open-source our models, code and data.

We present a summary of the efforts to improve conference peer review that were implemented at ACL’23. This includes work with the goal of improving review quality, clearer workflow and decision support for the area chairs, as well as our efforts to improve paper-reviewer matching for various kinds of non- mainstream NLP work, and improve the overall incentives for all participants of the peer review process. We present analysis of the factors affecting peer review, identify the most problematic issues that the authors complained about, and provide suggestions for the future chairs. We hope that publishing such reports would (a) improve transparency in decision-making, (b) help the people new to the field to understand how the *ACL conferences work, (c) provide useful data for the future chairs and workshop organizers, and also academic work on peer review, and (d) provide useful context for the final program, as a source of information for meta-research on the structure and trajectory of the field of NLP.

While machine translation evaluation metrics based on string overlap (e.g., BLEU) have their limitations, their computations are transparent: the BLEU score assigned to a particular candidate translation can be traced back to the presence or absence of certain words. The operations of newer learned metrics (e.g., BLEURT, COMET), which leverage pretrained language models to achieve higher correlations with human quality judgments than BLEU, are opaque in comparison. In this paper, we shed light on the behavior of these learned metrics by creating DEMETR, a diagnostic dataset with 31K English examples (translated from 10 source languages) for evaluating the sensitivity of MT evaluation metrics to 35 different linguistic perturbations spanning semantic, syntactic, and morphological error categories. All perturbations were carefully designed to form minimal pairs with the actual translation (i.e., differ in only one aspect). We find that learned metrics perform substantially better than string-based metrics on DEMETR. Additionally, learned metrics differ in their sensitivity to various phenomena (e.g., BERTScore is sensitive to untranslated words but relatively insensitive to gender manipulation, while COMET is much more sensitive to word repetition than to aspectual changes). We publicly release DEMETR to spur more informed future development of machine translation evaluation metrics.

Literary translation is a culturally significant task, but it is bottlenecked by the small number of qualified literary translators relative to the many untranslated works published around the world. Machine translation (MT) holds potential to complement the work of human translators by improving both training procedures and their overall efficiency. Literary translation is less constrained than more traditional MT settings since translators must balance meaning equivalence, readability, and critical interpretability in the target language. This property, along with the complex discourse-level context present in literary texts, also makes literary MT more challenging to computationally model and evaluate. To explore this task, we collect a dataset (Par3) of non-English language novels in the public domain, each aligned at the paragraph level to both human and automatic English translations. Using Par3, we discover that expert literary translators prefer reference human translations over machine-translated paragraphs at a rate of 84%, while state-of-the-art automatic MT metrics do not correlate with those preferences. The experts note that MT outputs contain not only mistranslations, but also discourse-disrupting errors and stylistic inconsistencies. To address these problems, we train a post-editing model whose output is preferred over normal MT output at a rate of 69% by experts. We publicly release Par3 to spur future research into literary MT.

Large-scale, high-quality corpora are critical for advancing research in coreference resolution. However, existing datasets vary in their definition of coreferences and have been collected via complex and lengthy guidelines that are curated for linguistic experts. These concerns have sparked a growing interest among researchers to curate a unified set of guidelines suitable for annotators with various backgrounds. In this work, we develop a crowdsourcing-friendly coreference annotation methodology, ezCoref, consisting of an annotation tool and an interactive tutorial. We use ezCoref to re-annotate 240 passages from seven existing English coreference datasets (spanning fiction, news, and multiple other domains) while teaching annotators only cases that are treated similarly across these datasets. Surprisingly, we find that reasonable quality annotations were already achievable (>90% agreement between the crowd and expert annotations) even without extensive training. On carefully analyzing the remaining disagreements, we identify the presence of linguistic cases that our annotators unanimously agree upon but lack unified treatments (e.g., generic pronouns, appositives) in existing datasets. We propose the research community should revisit these phenomena when curating future unified annotation guidelines.

This article revisits statistical relationships across Romance cognates between lexical semantic shift and six intra-linguistic variables, such as frequency and polysemy. Cognates are words that are derived from a common etymon, in this case, a Latin ancestor. Despite their shared etymology, some cognate pairs have experienced semantic shift. The degree of semantic shift is quantified using cosine distance between the cognates’ corresponding word embeddings. In the previous literature, frequency and polysemy have been reported to be correlated with semantic shift; however, the understanding of their effects needs revision because of various methodological defects. In the present study, we perform regression analysis under improved experimental conditions, and demonstrate a genuine negative effect of frequency and positive effect of polysemy on semantic shift. Furthermore, we reveal that morphologically complex etyma are more resistant to semantic shift and that the cognates that have been in use over a longer timespan are prone to greater shift in meaning. These findings add to our understanding of the historical process of semantic change.

Recent text generation research has increasingly focused on open-ended domains such as story and poetry generation. Because models built for such tasks are difficult to evaluate automatically, most researchers in the space justify their modeling choices by collecting crowdsourced human judgments of text quality (e.g., Likert scores of coherence or grammaticality) from Amazon Mechanical Turk (AMT). In this paper, we first conduct a survey of 45 open-ended text generation papers and find that the vast majority of them fail to report crucial details about their AMT tasks, hindering reproducibility. We then run a series of story evaluation experiments with both AMT workers and English teachers and discover that even with strict qualification filters, AMT workers (unlike teachers) fail to distinguish between model-generated text and human-generated references. We show that AMT worker judgments improve when they are shown model-generated output alongside human-generated references, which enables the workers to better calibrate their ratings. Finally, interviews with the English teachers provide deeper insights into the challenges of the evaluation process, particularly when rating model-generated text.

Speech perception involves the integration of acoustic features with socioindexical information (e.g., speaker’s ethnicity). English speakers, for instance, may rate English utterances produced by a native speaker as more accented if they believe the speaker to be Asian than if they believe the speaker to be Caucasian. This study investigates whether Japanese speakers will similarly rate English utterances as more accented when made to believe that the speaker was Asian-looking. Forty-eight participants rated the accentedness (9-point Likert-scale) of a set of sentences spoken by native English speakers. Sixteen listeners were made to believe the speakers were Caucasian, sixteen that the speakers were Asian, and sixteen were assigned to the audio-only condition. The results indicate no effect of ethnicity on accent rating by the non-native raters. Hence, non-native (Japanese) speakers may be less influenced by the speaker’s perceived ethnicity when evaluating the level of English accent as spoken by native speakers.

In the past two decades, the constant development and popularization of the internet, as well as the emergence of smartphones and other mobile devices, triggered the emergence of new ways of communication. Among these, we can distinguish (1) instant messaging (IM)—a tool enabling individuals to communicate using their phones, laptops, or other devices—and (2) social media—various platforms facilitating communication between individuals or between an individual and a larger group. This shift to digital communication created a need for nonverbal (or nontextual) cues to overcome limitations of computer-mediated communication (CMC). Park, Baek, and Cha (2014, 466) point out that such nonverbal cues account for up to 93% of everyday communication. Although on-paper communication (e.g., letters or postcards) has existed for centuries, it has never required a constant, dialogue-like exchange of utterances. Thus, one could convey one’s emotions only by writing about them. In contrast, communication in the digital age requires a quick and effective exchange of information, leaving little space for lengthy messages—but plenty of room for misinterpretation. Moreover, the emergence of internet forums and social networking services created a previously nonexistent opportunity to talk to people one would not have encountered otherwise. This makes any interpretation of textual messages increasingly difficult, since it is based on previous knowledge about the respective interlocutor. Digital pictograms and ideograms like emoticons, and later emoji (encompassing smileys as well as ideograms), were a natural answer to the basic human need for instantaneous, confusion-free, effective communication, since these new forms of digital exchange turned out to be largely bereft of traditional nonverbal cues. Ptaszynski et al. (2010, 46) explain that emoticons “are virtual representations of body language and their main function is similar—namely to convey information about the speaker’s emotional state”. Accordingly, emoticons are considered representations of body language in a textual form. Sanderson described the very first emoticons (also called ‘smileys’) as “a sequence of ordinary characters you can find on your computer keyboard” (Sanderson 1993, 1).

Among the many controversies that have raged about English education in Japan (Kawasumi, 1978), one of the most fundamental has been about whether the teaching of English in schools should be for practical purposes or for fostering the students’ intellect, character, and general knowledge. Advocates of practical applications, called jitsuyō in Japanese, emphasize English’s usefulness in education, work, and personal life, and they point to the language’s growing role as an international lingua franca. Supporters of the second approach, while not denying the practical uses of the language for some, say that, in Japanese educational contexts, most students would be better served if their instruction in English is positioned as part of a broad, liberal education, or kyōyō. This controversy is not new. As described below and in Gally (2018), this confl ict between jitsuyō and kyōyō can be seen in disputes about the need for English education that fl ared in the 1920s and 1970s. Although this debate has not been completely resolved, there has been a widespread perception, probably justifi ed, that kyōyō has largely given way to jitsuyō as the primary motivation for teaching and learning English in Japan. This increased emphasis on practical applications of the language can be seen in several areas, including the reduced focus on literature in university language classes and the adoption of “communication” as a central concept in government-directed Englishlanguage curricula. In areas other than English language education, however, kyōyō has maintained a strong presence, as can be seen by the incorporation of ethics education (dōtoku kyōiku) into standard school curricula and the many books and magazines published in recent years that promise to foster the kyōyō of adults. And despite the disbanding of liberal arts departments at many public universities in the early 1990s, several continue to have major divisions with kyōyō in their names, including Saitama University and the University of Tokyo. Interestingly, Akita International University and the School of International Liberal Studies at Waseda University, both established in 2004, include kyōyō in their Japanese names while teaching all of their classes in English. Nevertheless, while the term kyōyō has been frequently used in different educational contexts, the continuity in the different meanings of kyōyō has rarely been discussed. The disputes about English education, even the one between jitsuyō and kyōyō, have tended to be discussed fragmentally, as if the meaning of kyōyō is unique to a specifi c time and context. In this paper, drawing on Foucault’s concept of discours, which we here call “discourse,” we attempt to explore how the meanings of kyōyō have been used to justify statements about who needs to learn English in Japan from a broader perspective. Our focus is on how the evolving meanings of kyōyō share a certain consistency and similarity. In the process, we reveal that the discourse of kyōyō functions in part as resistance to the profit - and efficiency-driven perspectives of policies at the national level.

Languages with logographic writing systems present a difficulty for traditional character-level models. Leveraging the subcharacter information was recently shown to be beneficial for a number of intrinsic and extrinsic tasks in Chinese. We examine whether the same strategies could be applied for Japanese, and contribute a new analogy dataset for this language.

Native English listeners from North America rely primarily on changes in formants, not vowel duration, when perceiving the vowel contrast in the minimal pair bit and beat manipulated from a Canadian English sample. In this paper, we evaluated which cue do native English listeners from other regions use when perceiving the same North American vowel contrast. For this purpose, we used the same task and stimuli as in the study with North American listeners. Our results indicate that listeners from the UK, New Zealand, Ireland and Singapore used primarily changes in formants, a pattern similar to listeners from North America. Australian listeners, however, relied primarily on vowel duration rather than formants. The reaction time results suggest that the difference between Australian listeners and other listeners may be due to differences in the characteristics of vowels in Australian English versus North American English.

ロマンス語とは,ラテン語から派生した姉妹言語の総称である。ロマンス諸語には,語源を共有する同源語が数多く存在するが,意味や用法が異なるものもある。本稿では,ロマンス語のうち仏伊西の動詞に焦点を絞り,意味変化の大きさと,言語内要因の関係を統計的に分析した。意味変化の大きさは単語の分散表現で測定した。分析の結果,意味変化の大きさには,ロマンス語の頻度,ラテン語の多義性,ロマンス語の単語長が影響することが判明した。