essay in japanese period

Japanese History: Edo Period Essay

• To find inspiration for your paper and overcome writer’s block
• As a source of information (ensure proper referencing)
• As a template for you assignment

Introduction

Bibliography

The Edo period also known as the Tokugawa period is the period between 1603-1868 in the Japanese history when Japan was under the Tokugawa Shogunate rule who had divided the country into 300 regions known as Daimyos. Tokugawa leyasu officially opened the era on March, 24, 1603 while Tokugawa yoshinobu resigned on May, 3 1868 after the Meiji restoration. The Tokugawa family ruled Japan from their base in Edo (currently Tokyo). The post of Emperor was more ceremonial during the Edo era (Patricia 60).

Tokugawa leyasu supported foreign trade but he was also suspicious of the influence of the outsiders during the pre-Edo period; Japan underwent the Nanban trade era during which the intense interaction with the European powers took place, namely, economic and religious. Trade restrictions, Christian missionary execution and Spanish expulsion were some of the restrictions that were enforced. The Closed Country Edict in 1635 was the climax of all the restrictions because of the following:

• Set highly strict regulations to minimize the movement of people into and out of the Japanese territory; death penalty was the consequence.
• Catholicism and all Christian practices were forbidden; Missionaries were also barred from entering Japan, and harsh sentences were drawn for those who entered.
• Trade restrictions were set; trade along ports was consequently limited. Portuguese relations with Japan were completely cut off (Alfred 138).

The Edo period was marked by the urban culture in Japan, for instance, Edo became the largest city on earth during those times with a population of 1.2 million residents as compared to the second largest place, London, with 800,000 residents. The period also experienced the rise of entertainment culture such as theaters or humorous novels.

Ordinary residents were also able to gain access to print media following the polychrome woodblocks development. People were also interested in learning more about Europe and all its sciences, commonly known as “Dutch learning” despite the minimal contact between Japan and the Western world (Alfred 100).

Arrival of Matthew Calbraith Perry and his four-ship fleet along the Edo Bay in July 1853 marked the end of the seclusion period in Japan. Japan finally accepted Perry’s demands to ending seclusion and opening up to foreign trade, consequently, the Treaty of Kanagawa that opened-up two ports (Hakodate and the port of Shimoda) to foreign American ships was signed (Administration, United States. National Archives and Records 1-4).

Five years after the Treaty of Kanagawa, the Harris treaty was signed between Japan and the US. The Kanagawa treaty became a catalyst factor of internal conflicts, which were only solved after the Tokugawa shogunate’s fall; similar agreements were negotiated by European powers such as Russia, the United Kingdom and France. (William 4)

After 250 years rule over Japan, the Tokugawa Shogunate turned the Japanese nation into a united cohesive nation with the mushrooming of many urban centers across Japan, for example, Edo became the largest and most populated city on earth with 1.2 million residents; Japan also experienced some artistic as well as intellectual development during this period.

On the other hand, the seclusion policy undermined all the good things that are associated with the rule; it is a policy that consistently haunted the Tokugawa rule, and finally led to its fall as the Japanese opted for a more open Meiji restoration that allowed all forms of Western culture to freely penetrate into Japan without necessarily having to restrict them.

Administration, United States. National Archives and Records. The Treaty of Kanagawa: Setting the Stage for Japanese-American Relations. New York City: National Archives and Records Administration, 2003. Print.

Alfred J. Andrea, and James H. Overfield. The Human Record, Volume II: Sources of Global History: Since 1500. London: Cengage Learning, 2011. Print.

Patricia Buckley Ebrey, Anne Walthall, and James Palais. East Asia: A Cultural, Social, and Political History. Tokyo: Cengage Learning, 2008. Print.

William C. Middlebrooks. Beyond Pacifism: Why Japan Must Become a Normal Nation: Why Japan Must Become a Normal Nation. New York City: ABC-CLIO, 2008. Print.

• Hiroshima: Rising from the Ashes of Nuclear Destruction
• Tokugawa Settlements and Rule of Meiji in Japan
• Kitagawa and Gainsborough Artworks
• World History: A Peace to End All Peace by David Fromkin
• The Adventures of Ibn Battuta
• The Effects of the Korea Division on South Korea After the Korean War
• The Tale of a Great Journey: "The Rihla" by Ibn Battuta
• The Boxer Rebellion
• Chicago (A-D)
• Chicago (N-B)

IvyPanda. (2018, November 20). Japanese History: Edo Period. https://ivypanda.com/essays/edo-tokugawa-japan/

"Japanese History: Edo Period." IvyPanda , 20 Nov. 2018, ivypanda.com/essays/edo-tokugawa-japan/.

IvyPanda . (2018) 'Japanese History: Edo Period'. 20 November.

IvyPanda . 2018. "Japanese History: Edo Period." November 20, 2018. https://ivypanda.com/essays/edo-tokugawa-japan/.

1. IvyPanda . "Japanese History: Edo Period." November 20, 2018. https://ivypanda.com/essays/edo-tokugawa-japan/.

IvyPanda . "Japanese History: Edo Period." November 20, 2018. https://ivypanda.com/essays/edo-tokugawa-japan/.

Site Content

The columbia anthology of japanese essays.

Zuihitsu from the Tenth to the Twenty-First Century

Edited and translated by Steven D. Carter

Columbia University Press

Pub Date: October 2014

ISBN: 9780231167710

Format: Paperback

List Price: $45.00 £38.00

Shipping Options

Purchasing options are not available in this country.

ISBN: 9780231167703

Format: Hardcover

List Price: $135.00 £113.00

ISBN: 9780231537551

Format: E-book

List Price: $44.99 £38.00

• EPUB via the Columbia UP App
• PDF via the Columbia UP App

The focused ramble of the traditional Japanese essay format called zuihitsu (literally, 'following the brush') has appealed to writers of both genders, all ages, and every class in Japanese society. Highly personal, these essays contain dollops of philosophy, odd anecdotes, quiet reflection, and pronouncements on taste. In running alongside the main tracks of Japanese literature, this broad collection of zuihitsu brims with idiosyncratic interest. Liza Dalby, author of The Tale of Murasaki and East Wind Melts the Ice: A Memoir Through the Seasons

Savor a copy of The Columbia Anthology of Japanese Essays , and take a contemplative walk through the Japanese mind, full of poetic turns and pithy longings, ribald humor and lofty aspirations. Kris Kosaka, The Japan Times

Rich and highly enjoyable.... This evocative selection serves both as an excellent introduction to the genre for the English-speaking world and as a reminder that, no matter how distant or seemingly different the society, people's individual struggles, aspirations and aesthetics transcend their own times. Morgan Giles, Times Literary Supplement

Winner, 2016 2015-2016 Japan-United States Friendship Commission Prize for the Translation of Japanese Literature

About the Author

• Asian Fiction and Literature
• Asian Literature in Translation
• Asian Studies
• Asian Studies: Arts and Culture
• Asian Studies: Fiction and Literature
• Fiction and Literature
• Literary Studies
• Asian Studies: East Asian History
• History: East Asian History

• Environment
• Globalization
• Japanese Language
• Social Issues

After the Meiji Light: The Transition to Taisho, 1905-1912

Editor’s Note: It will be particularly helpful to have access to either Gordon, Andrew, A Modern History of Japan or McClain, James , Japan : A Modern History to conduct this lesson.

With the revision of the Unequal Treaties, acknowledgement by the West of Japan’s great power status, and its acquisition of a colonial empire, Japan’s wars against the Chinese and Russians seemed to represent the realization of the Meiji dream.  Instead, Japan’s leaders recognized early just how fragile their new great power status was abroad and how precarious popular support for their government was at home.

• Students will compare and contrast the benefits and obstacles brought about in Japan by the Sino-Japanese and Russo-Japanese wars;
• Students will understand the different reasons why the transition from the Meiji to the Taisho periods can be called a time of uncertainty about Japanese society, Japan's leadership, and Japan's place in the world; and
• Students will describe reasons for increased political consciousness and dissatisfaction with governmental policies and actions among the Japanese populace during this time period.

Common Core Standards College and Career Readiness Anchor Standards for Reading

• Standard 1.  Read closely to determine what the text says explicitly and to make logical inferences from it; cite specific textual evidence when writing or speaking to support conclusions drawn from the text.
• Standard 7.   Integrate and evaluate content presented in diverse formats and media, including visually and quantitatively, as well as in words.

College and Career Readiness Anchor Standards for Writing

• Standard 2.   Write informative/explanatory texts to examine and convey complex ideas and information clearly and accurately through the effective selection, organization, and analysis of content.

College and Career Readiness Anchor Standards for Speaking and Listenin g

• Standard 4.  Present information, findings, and supporting evidence such that listeners can follow the line of reasoning and the organization, development, and style are appropriate to task, purpose, and audience.
• McRel Standard 4 .  Gathers and uses information for research purposes .
• McRel Standard 5 .  Uses the general skills and strategies of the reading process .
• McRel Standard 7 .  Uses reading skills and strategies to understand and interpret a variety of informational texts .
• McRel Standard 8 .  Uses listening and speaking strategies for different purposes .

World History

• McRel Standard 36 .  Understands patterns of global change in the era of Western military and economic dominance from 1800 to 1914 .
• McRel Standard 37 .  Understands major global trends from 1750 to 1914 .
• McRel Standard 38 .  Understands reform, revolution, and social change in the world economy of the early 20th century .
• McRel Standard 39 .  Understands the causes and global consequences of World War I .

The transition from Meiji to Taisho Japan was a period of reflection and uncertainty for the Japanese people and their political leaders.

In what ways did the Sino-Japanese and Russo-Japanese Wars impact Japanese society and Japan’s economy?

• What set of expectations arose among the Japanese populace during these wars and why was the Japanese government unable to live up to its own wartime propaganda?
• What changes to Japanese society led intellectuals to be both proud of its nation’s accomplishments, yet also to be extremely uncertain of the costs of progress during the transition from Meiji to Taisho?

Divide students into small groups and ask them to formulate a response to the following question: In contemporary American society, what recourse does the populace have to express grievances about governmental policies and actions?

• Geography Quiz : (3-5 minutes) Prepare either a list of ten place names and ask the students to locate them correctly on maps of Japan and East Asia, or one page of ten place names with blank spaces provided and two maps with the places marked on the maps.   Include dots, arrows pointing to islands, and so forth and label A-J for a matching exercise.
• Lecture : Use an image of the Japanese people meeting in Hibiya Park on September 5, 1905, after the government made public the terms of the Treaty of Portsmouth, as a starting point for a discussion of the significance of the Hibiya Riots.  (During the riots, around 30,000 protesters gathered to call for the government to renegotiate the treaty with Russia.  As the crowd got out of hand and began to destroy government property, the police responded with force, leaving seventeen of the rioters dead.)
• Reading : The Hibiya Riots signified the growth of a mass political consciousness and the emergence of a nationalist chauvinism among the populace as well as marking the beginning of popular movements against the government.  In short, the Japanese people had reached their limit of endurance and willingness to sacrifice for the government uncritically.  The chapter entitled “New Awakenings, New Modernities” in James McClain’s Japan: A Modern History , is good background reading for this unit, while Andrew Gordon’s subsection “The Era of Popular Protest” (pages 131-35) from his A Modern History of Japan provides more specific information on the Hibiya Riots themselves.
• What main themes do you see running through Soseki’s take on modern life?
• Why would Soseki lament the isolation and alienation of the individual in urban Japanese society?
• Most Japanese peopledid not want to go back to life under Tokugawa rule, but what does Soseki tell us is being lost in Japan’s headlong rush to modernize?
• On the basis of these passages, how would you describe the way the protagonist (Joji) sees Naomi?
• What is Tanizaki implicitly criticizing in these passages?
• Do you see these passages as allegorical?
• After studying the biographies of General Nogi and the Taisho Emperor, what general sense of things to come do you think the Japanese people experienced with the end of the Meiji period in 1912?
• Do you think that they felt confident about the future of their nation?  Why or why not?
• Critique: Using Soseki’s prose as a model, have each student write a short comment on a controversial issue in modern American popular thought, describing this issue’s positive and negative ramifications in contemporary culture.
• Japan's new international position;
• The weakening of the old Meiji leadership;
• The development of political parties;
• The women's movement;
• Economic and social modernization and the gap between rich and poor;
• The development of mass media; and
• The growing pace of public protest.
• Analysis - Essay and/or Discussion : Ask students to study the table in A Modern History of Japan: From Tokugawa Times to the Present (page 132), and to describe the various opinions, trends and policies that they perceive in these events and their outcomes during the late Meiji and the Taisho periods.

Receive Website Updates

Please complete the following to receive notification when new materials are added to the website.

Late Heian Period (ca. 900-1185)

• Nara and Heian Japan (710 AD - 1185 AD) [About Japan: A Teacher's Resource] An overview of Japan's Nara and Heian periods. Discusses the Fujiwara family, their private estates, and the rise of the warrior.

• Heiji Monogatari Emaki (Tale of the Heiji Rebellion) Scrolls - "A Night Attack on the Sanjo Palace" [Princeton University] The Heiji disturbance, which occurred late in 1159, represents a brief armed skirmish in the capital. One faction, led by Fujiwara Nobuyori, in alliance with the warrior Minamoto Yoshitomo, staged a coup. In the scene depicted here, they surrounded the palace, captured the sovereign, placed him in a cart and then consigned the structure to the flames. Even though Nobuyori and Yoshitomo were triumphant here, they later suffered defeat and death at the hands of their rival Kiyomori...After the Heiji disturbance, Taira Kiyomori gained influence as a trusted advisor to the retired emperor, Go-Shirakawa. He launched his own coup some twenty years later, which unleashed a civil war, known commonly as the Genpei Wars (1180-85). One of Yoshitomo's sons, Minamoto Yoritomo, triumphed in this campaign, and consigned Taira Kiyomori's relatives to death or exile. Yoritomo established the Kamakura bakufu, which provided judicial and policing authority for its followers, known as housemen (gokenin) from 1185 until 1333. // The Heiji scrolls date from the thirteenth century and represent a masterpiece of "Yamato" style painting. They can be documented as being treasured artifacts in the fifteenth century, when nobles mention viewing them, but they now only survive in fragmentary form. The scene appearing here, entitled "A Night valuable depiction of Japanese armor as it was worn during the early Kamakura era (1185-1333). By contrast, most surviving picture scrolls showing warriors date from the fourteenth century and show later styles of armor...The scrolls read from right to left, and all action flows to the left. A few people hurrying flow into a confused throng of warriors and nobles, epitomized by a wayward bystander being crushed by an ox cart. Out of the confusion, attention shifts to the palace...Continued on site.

• The Legends of Hachiman [Smith College Museum of Art] From protector of the imperial house, to protector of the Minamoto military house, to protector of the nation, the legend of the Shinto deity, Hachiman, evolved throughout Japanese history... During the Heian period, Hachiman became the protector of the Minamoto military house when the clan adopted Hachiman as their clan deity ( ujigami ) ...Minamoto Yoritomo (1147-1199), who would defeat the Taira clan in the Gempei Wars (1180-1185), a victory that was attributed in no small part to Hachiman's divine protection. The appropriation of Hachiman by the Minamoto clan is seen in multiple instances in the Japanese war epic that describes the Gempei War, Tales of Heike . The site provides background on the scrolls, suggestions for viewing a handscroll, and questions for discussion .

Kamakura Period (1185-1333); Muromachi Period (1333-1568)

• The Age of the Samurai (1185-1868) [Asia for Educators] A guide to the samurai governments that ruled Japan from 1185 to 1868. With discussion questions.

• Kamakura and Nanbokcho Periods (1185-1392) [Timeline of Art History, The Metropolitan Museum of Art] A short introduction, with images of seven artworks in the museum's collection.

• Muromachi Period (1392-1573) [Timeline of Art History, The Metropolitan Museum of Art] A short introduction, with images of six artworks in the museum's collection.

• Japan's Medieval Age: The Kamakura & Muromachi Periods [About Japan: A Teacher's Resource] An in-depth look at political, economic, cultural, and religious life during the Kamakura and Muromachi periods.

Lesson Plan • A Case Study of Medieval Japan through Art: Samurai Life in Medieval Japan [Program for Teaching East Asia, Center for Asian Studies, University of Colorado] "The samurai warrior has come to symbolize Japan's medieval period of social and political unrest that lasted from the late twelfth to late sixteenth centuries. Working with artistic renderings of the samurai as well as cultural artifacts of samurai life, students recognize the complex, complementary aspects of the samurai culture that developed during this period. Students consider this more nuanced view of the samurai as they take on the role of advisors to a director hoping to make an authentic film about Medieval Japan." An in-depth introductory essay and lesson plan, with images, focusing on the Kamakura (1185-1333), the Muromachi (1336-1573), and the Momoyama (1573-1603) shogunates.

The Mongols' Failed Naval Campaigns Against Japan, 1274 and 1281

• "Relics of the Kamikaze" [Archaeology] An excellent article about the discovery and excavation of Khubilai Khan's 13th-century invasion fleet off the coast of Takashima. With a map and several images. From the January/February 2003 issue of Archaeology magazine.

• Scrolls of the Mongol Invasions (Annotated) [Princeton University] This site allows you to view individual scenes depicting the Mongol Invasions of Japan. Takezaki Suenaga, a warrior who fought against the Mongols in both 1274 and 1281, commissioned scrolls recounting his actions. This unique record of the invasions, and important eyewitness account, was heavily damaged in the ensuing centuries – according to lore they were even once dropped into the ocean! By the time of their rediscovery in the eighteenth century, the scenes and text of the scrolls were scattered into separate sheets. See also the partner site Mongol Invasions of Japan - 1274 and 1281 - this web site is devoted to understanding the Mongol Invasions of Japan in 1274 and 1281. The failure of the invasions gave rise to the notion of the "divine wind" or Kamikaze, although an exploration of the invasions reveals that the Japanese defeated the Mongols with little need of divine, or meteorological intervention.

• Mongol Invasions of Northeast Asia: Korea and Japan [PDF] [Association for Asian Studies] With images and maps, topics include: Kamikaze, the 'Divine Wind,' The Mongol Continental Vision Turns Maritime, Korea's Historic Place in Asian Geopolitics, Mongol Invasions of Japan, Reflections on the Mongol Maritime Experience.

• The Legends of Hachiman [Smith College Museum of Art] "This particular pair of lavishly ornamented handscrolls illustrates the legends of the Shinto deity Hachiman [whose 'popularity ... increased after the thirteenth century when Japan was attacked by Mongol forces in 1274 and 1281']. The paintings, which date to the mid-seventeenth century, are rendered in the yamato-e style favored by the members of the Tosa school to which they are attributed. Both the painting and the calligraphy exemplify the highly refined styles favored by the court at the start of the Edo period (1615-1868)." With background information on Shinto and Hachiman and viewing a handscroll.

Also see the Video Unit on Medieval Japan in the History-Archaeology section (Kamakura and Muromachi Periods) for more about the Mongol invasions of Japan .

New Sects in Buddhism

Shinran, 1173-1263, founder of the Jodo Shinshu (The True Teaching of the Pure Land) Primary Source w/DBQs • Shinran's Lamentation and Self-Reflection [PDF] [Asia for Educators]

• Ox-Herding: Stages of Zen Practice [ExEAS, Columbia University] The ten ox-herding pictures and commentaries presented here depict the stages of practice leading to the enlightenment at which Zen (Chan) Buddhism aims. The story of the ox and oxherd is an old Taoist story, updated and modified by a twelfth-century Chinese Buddhist master to explain the path to enlightenment.

Dōgen Zenji, 1200-1253, founder of the Soto Zen sect Primary Source w/DBQs • Dōgen's How to Practice Buddhism (Bendōwa) [PDF] [Asia for Educators]

Nichiren, 1222-1282, founder of the Nichiren sect Primary Source w/DBQs • Nichiren's Rectification for the Peace of the Nation (Risshō Ankoku Ron) [PDF] [Asia for Educators]

Also see the Video Unit on Medieval Japan in the History-Archaeology section (Kamakura and Muromachi Periods) for more about the Buddhist sects in medieval Japan .

Maintaining Order through Political Transition

Minamoto Yoritomo, 1147-1199, and the Kamakura Bakufu Primary Source w/DBQs • Selected Documents of the Kamakura Bakufu [PDF] [Asia for Educators]

Ashikaga Takauji, 1305-1358 Primary Source w/DBQs • The Kemmu Shikimoku (Kemmu Code) [PDF] [Asia for Educators]

Imagawa Sadayo (Imagawa Ryōshun), 1325-1420 Primary Source w/DBQs • Articles of Admonition by Imagawa Ryōshun to His Son Nakaaki [PDF] [Asia for Educators]

Asakura Toshikage, 1428-14851 Primary Source w/DBQs • The Seventeen-Article Injunction of Asakura Toshikage [PDF] [Asia for Educators]

The Tale of the Heike

Primary Source • The Tale of the Heike [Asia for Educators] The Tale of the Heike recounts the struggle for power between the Taira (or Heike) and Minamoto (or Genji) houses in the late twelfth century. With the Taira's defeat in 1185 and the establishment of a new warrior government by the victorious Minamoto, the medieval age began. From this war tale, we can learn much about life in Japan during this transitional period and about warrior culture. With discussion questions.

Also see the "War Tales" section of the Video Unit on Medieval Japan in the History-Archaeology section (Kamakura and Muromachi Periods) for more about the Tale of the Heike .

For The Pillow Book (ca. 1002) and The Tale of Genji (ca. 1021) please see the Literature section of Time Period 600-1000 CE . For The Tale of the Heike please see the Military and Defense section , above.

Kamo no Chōmei (ca. 1153-1216)

Primary Source • An Account of My Hut [Asia for Educators] Excerpts from this famous essay written in 1212, in which the author, Kamo no Chōmei describes his own road to becoming a Buddhist monk. With discussion questions.

Video Unit • An Account of My Hut [Asia for Educators] A video unit on the famous 13th-century essay introduced above. Featuring Columbia University professor Donald Keene and Asia Society President Emeritus Robert Oxnam.

Yoshida Kenkō (1283-1350)

Primary Source • Essays in Idleness [Asia for Educators] Short excerpts from Essays in Idleness . With a brief historical introduction and exercises for students.

Video Unit • Kenkō's Essays in Idleness and Japanese Aesthetics [Asia for Educators] This video unit on Yoshida Kenkō's 14th-century literary work discusses the Japanese aesthetic of simplicity and impermanence. Featuring Columbia University professor Donald Keene and Asia Society President Emeritus Robert Oxnam.

Noh and Kyōgen

• The Forms of Japanese Drama [Asia for Educators] A brief description of the four major dramatic forms that came out of Japan's medieval period: Noh, Kyôgen, Kabuki, and Bunraku. Followed by a classroom exercise for students.

• Noh Drama [Asia for Educators] This unit begins with a short introduction to Noh, the oldest surviving form of Japanese theater. Also includes a description of two recommended play ("Atsumori" and "Sotoba Komachi"), followed by classroom exercises for students.

• Noh Costume [The Metropolitan Museum of Art] An overview of the development of Noh costumes during the Muromachi and Momoyama periods. With ten examples from the museum's collection.

Video Unit • An Introduction to Noh [Asia for Educators] This video unit on Noh, a dramatic form that originated in Medieval Japan, discusses Noh's history and basic structure, Noh masks, the aesthetics of Noh, and Noh theater today. Featuring Columbia University professors Donald Keene and Haruo Shirane, and Asia Society President Emeritus Robert Oxnam.

• Kyōgen [Asia for Educators] A short introduction to Kyōgen, the comedic counterpart to Noh. Also includes a description of a recommended play ("Busu"), followed by classroom exercises for students.

• The Way of Tea [Five College Center for East Asian Studies] Tea Ceremony, or Chado (茶道), is one of the Japanese traditional arts involving ritualistic preparation of tea. Cha (茶) means tea, and do (道) means the way or the path. Thus, Chado is translated as the Way of Tea.… The Way of Tea is composed of a series of acts such as building a fire in the hearth, boiling the water, whisking the green tea powder in a tea bowl, and serving it along with some sweets. Simply put, it is an act in which the host invites the guest to share a bowl of tea together. Indeed, it began as a simple act of making and drinking tea. Over the centuries, however, it was influenced by Zen Buddhism philosophy and became a highly stylized form of art.

• Steeped in History: The Art of Tea [PDF] [UCLA Fowler Museum]

• Tea Traditions [TeachJapan] A portal for units on tea in Japan developed by Asia Art museums.

• What is Teachable about Japanese Tea Practice? [Education about Asia] Download PDF on page.

Note to Teachers • The journal Education about Asia has many excellent teaching resources on-line on all topics related to East, South and SE Asia.

Scroll Painting

• Emakimono [Asia Society] "During the 11th to 16th centuries, painted handscrolls, called emakimono, flourished as an art form in Japan, depicting battles, romance, religion, folktales, and even stories of the supernatural world." A short background essay with a suggested activity for students.

Lesson Plan • A Case Study of Heian Japan through Art: Japan's Four Great Emaki [Program for Teaching East Asia, Center for Asian Studies, University of Colorado] " Emakimono or emaki , narrative picture scrolls, developed into a distinctly Japanese art form in the Heian period, 794-1185 CE. In this lesson, students examine four emaki masterpieces to analyze the highly refined court culture, politics, and religion in the late Heian period. Working in groups, they then create preview posters for a museum exhibit featuring the four emaki , providing their interpretation of the facets of Heian culture they believe exhibit-goers should learn." Introductory essay and lesson plan with images of picture scrolls from the period.

• Takezaki Suenaga's Scrolls of the Mongol Invasions of Japan [Princeton] An excellent interactive website with several versions of the recovered 13th-century scrolls commissioned by the Kyushu warrior Takezaki Suenaga, who fought against the Mongols during the invasions of 1274 and 1281. Viewers can compare the "original" (reassembled) 13th-century version to 18th- and 19th-century copies and also see a 21st-century reconstruction of the 13th-century version. Also features an illustrated glossary.

• The Kitano Tenjin Engi Emaki [Timeline of Art History, The Metropolitan Museum of Art] A multimedia learning website about a 13th-century Japanese handscroll that illustrates the legends of the Kitano Shrine (Kitano Tenjin Engi). Included are a short introduction to the Kitano Tenjin Engi Emaki and images of the scroll.

Find more art-related resources for Japan, 1000-1450 CE at OMuRAA (Online Museum Resources on Asian Art)

Related Timelines from Other Websites

World History for Us All Big Era 5: 300 - 1500 CE

The Metropolitan Museum of Art World Regions: 1000 - 1400 AD World Regions: 1400 - 1600 AD

Hyperhistory.com 1000 - 1500

| Index of Topics for All Time Periods |

Skip to Content

Other ways to search:

• Events Calendar
• Voices of Modern Japanese Literature

Lesson (pdf) Handout 1 Handout 1 Key Handout 2 Handout 2 Key Handout 3 Handout 3 Key Handout 4 Handout 5 Handout 6 Handout 6 Key Handout 7 Handout 7 Key Handout 8 Handout 8 Key

Sarah Campbell, Ketchikan High School, Ketchikan, AK

Introduction

Modern writers like Ernest Hemingway, F. Scott Fitzgerald, William Faulkner, and John Steinbeck are well known in American literary circles. These writers are often included in high school English and social studies curricula because of their artistic commentary on the way in which people viewed themselves and the world during the American modern period (1915-1945).  Through these authors’ voices, readers are able, for example, to consider how World War I challenged American optimism, explore how the Great Depression left many with a feeling of uncertainty, and contemplate how World War II furthered feelings of disjointedness and disillusionment in 20th-century life.  Through their varied approaches, techniques, and styles, modern American writers echoed the strong sense of isolation, alienation, and uncertainty felt by many Americans of the modern period.

The modernist movement was not exclusive to the United States; this literary movement extended around the world, including Japan. “ Modanizumu , as the term “modernism” was rendered into Japanese in the late 1920s, was a powerful intellectual idea, mode of artistic expression, and source of popular fashion in Japan from approximately 1910-1940,” explains William J. Tyler (2009) in his book Modanizumu: Modernist Fiction from Japan, 1913-1938 .  Similar to American writers, Japanese writers and artists of the Modern period also broke from the authority and traditions of the past by attempting new styles, subjects, and themes. Rapid industrialization, women’s and men’s suffrage movements, education reforms, Taishō democracy, and nationalism provided rich topics for late Meiji and Taishō writers.  Japanese Modern writers artistically commented upon the lifestyle, political, and socioeconomic changes of the early 20th century. Thus, their works provide rich sources for American high school curricula on Asian and world history and literature.

The Modern period in Japan overlaps the reigns of three Emperors: Meiji (1868-1912), Taishō (1912-1926), and early Shōwa (1926-1945). Throughout this lesson, “Modern Japan,” “Japan’s Modern Period,” and “Modern Literature” refer to this period of rapid modernization from the late 1800s through the late 1920s.

In this lesson, students read Meiji-Taishō literary works in their historical, cultural, biographical, and literary contexts, considering how individuals reacted to the process of modernization in Japan during the 20th century. During the lesson’s first day, students identify some basic characteristics of modernization in the late 19th and early 20th centuries in Japan, based on prior study and reading. Then, they make observations and inferences about this period, drawing on visual images from the late 1800s to the early 20th century, as well as Kambara Ariake’s poem “The Oyster Shell.” Through discussion, they define the characteristics of modern Japanese literature and the proletarian movement of the early 20th century in Japanese literature.

On Day 2, the class reads and discusses Mori Ogai’s “The Dancing Girl” (1890). They use the “Shared Inquiry” discussion format to explore the purpose and message of Ogai’s work and complete a reflective writing assignment considering what the short story reflects about modernization at the turn of the 20th century in Japan. On Days 3 and 4, students read Shimizu Shikin’s “The Broken Ring” (1891) and Kuroshima Denji’s “The Telegram” (1923), engaging in discussion and reflective writing using the strategies employed on Day 2.

Grade Level/Subject Area: Secondary/Asian Literature, World Literature, World History

Time Required: 4-5 class periods plus optional homework

For Students:

• Meiji-Taishō Background Essay ; this essay can be copied or students can access it online.
• Handout 1: Background Essay Reading Guide
• Handout 2: Visual Analysis Worksheet (you will need four copies for each student or student pair unless you plan to have students record their answers on separate sheets of paper)
• Handout 3: The Oyster Shell
• Handout 4: Characteristics of Modern Japanese Literature
• Handout 5: Reading for Tone and Mood
• Handout 6: Post-Reading Worksheet for “The Dancing Girl”
• Handout 7: Post-Reading Worksheet for “The Broken Ring”
• Handout 8: Post-Reading Worksheet for “The Telegram

For Teachers:

Computer and projector for showing the following images to students for analysis:

• September 1931 NAPF (Nippona Artista Proleta Federacio) and October 1931 NAPF (Nippona Artista Proleta Federacio) magazine covers
• Tokyo Construction Fair 1935
• Women in the 1920s: Woman Curling Hair and Ainu Woman Carrying Wood
• Handouts 1-3 and 6-8 Answer Keys
• Mori Ogai.  “The Dancing Girl.” Youth and Other Stories .  Ed. & trans. J. Thomas Rimer.  Honolulu: University of Hawaii Press, 1994.  6-24. Downloadable full text.
• Shimizu Shikin. “The Broken Ring.” Trans. Rebecca Jennison. The Modern Murasaki: Writing by Women of Meiji Japan .  Eds. Rebecca L. Copeland and Melek Ortabasi.  New York:  Columbia University Press, 2006.  232-239.
• Kuroshima Denji.  “The Telegram.” A Flock of Swirling Crows and Other Proletarian Writings.  Trans. Zeljko Cipris.  Honolulu: University of Hawaii Press, 2005.  17-24.

At the conclusion of this lesson, students will be better able to:

• Identify and discuss characteristics of modern Japanese literature, drawing from selected examples of Meiji and Taishō era poetry and short stories.
• Identify and compare multiple perspectives on modernization as reflected in modern literature of the Meiji-Taishō literature.

Essential Questions

• In what ways did the events of modern Japan influence writers of that period?
• What perspectives on modernization are reflected in literature produced during the Meiji and Taishō periods?

Teacher Background

For an overview of the Meiji-Taishō period, please see the Meiji-Taishō Background Essay for this curriculum project by Ethan Segal, Michigan State University.

A note on name order and special naming of Japanese authors :

In Japanese, the family name comes first and given name last. In the short biographies below, family names appear first and in capital letters. However, when referring to Japanese authors and poets, the Japanese follow special rules based on the writer’s time period, his or her use of a pen name, or genre of literature. So, for the four writers below, the Japanese often refer to Mori Ogai, Kambara Ariake, and Shimizu Shikin by just their given names or pen names. Kuroshima Denji, as a proletarian writer, is referred to by his family name, Kuroshima. This lesson follows the Japanese convention for these writers.

KAMBARA Ariake (1876-1952)

Kambara Ariake is actually a pen name for a writer known for his poetry, biographical novel, and narratives.  Ariake, as the poet is known, was born into an elite family, well-educated and well-travelled. His poetry earned him early success. In writing sonnets, a form rarely used at the time in Japan, he acquired a reputation as a symbolist poet. Failing to respond to new trends (e.g., embracing free verse) marked the end of Ariake’s writing career in 1947. 

MORI Ogai (1862-1922)

Mori Ogai had a rich career beginning as a Japanese army surgeon, then as a translator, and later as a novelist and poet. He was born into an elite Japanese family and was therefore afforded a strong education. He was well trained in Confucian classics, Chinese poetry, Western thought and medicine, and the German and Dutch languages. While in the army, he spent time living in Germany, where he developed an interest in European literature. His writings can be divided into four phases. First, Mori translated European poetry and plays into Japanese; next, he spent a period writing about his personal experiences. Later in life, from 1912-1916, he wrote historical stories. His final writing period, from 1916 until his death, was characterized by biographies of late Edo period doctors. Mori is considered one of the great writers of modern Japan.

SHIMIZU Shikin (1868-1933)

Raised in Kyoto, Shimizu Shikin was highly educated.  She was active in the women’s rights movement in Japan and frequently published in magazines.  Scholars credit Shimizu Shikin as one of the first writers to adopt a new narrative style of Japanese writing in the early 20th century—the I Novel, a confessional genre in which the literary events reflect the writer’s life, all written from a first-person point of view. This writing style was popularized by Natsume Sōseki. A pioneering woman in many respects, Shikin married for love. She travelled with her husband to Europe, later ending her writing career to be a wife and mother. 

KUROSHIMA Denji (1898-1943)

Kuroshima Denji was born into a poor farming family on Shodo Island in the Inland Sea. He later headed to Tokyo to work and study. In 1919 he was conscripted into the Japanese army and sent to fight in an Allied forces anti-revolutionary war against the new Soviet Union. His military service and war experiences were to influence much of Kuroshima’s writing. After the war, Kuroshima began writing and joined the emerging proletarian literature movement. This early 20th-century movement, in Japan and internationally, produced literature that focused on the harsh lives of peasants, workers, and other groups adversely affected by modernization or political repression and called attention to the need for social, economic, and political change. Kuroshima’s writing included stories of the experiences of Japanese soldiers who had served in the anti-Soviet war in Siberia as well as stories that focused on the struggles of Japanese peasants and workers. His collected works are a major contribution to the Japanese and international proletarian literature movement.

Preparing to Teach the Lesson

• This lesson uses several pedagogical techniques. Day 1 uses visual literacy questioning techniques to guide students in thoughtful and thorough analysis of visual source materials.  Days 2 through 5 employ “ Great Books’ Shared Inquiry ” methods for analyzing and discussing print sources. If you are not familiar with these techniques, you may wish to learn more by exploring the above links, which are also listed in the References section at the end of the lesson.
• Prior to the lesson, review the Teacher Background , the Historical Background Essay by Ethan Segal, the images to be examined on Day 1, student materials, and answer keys. Preparation time will vary according to teacher familiarity with the characteristics of Modern Literature and with the Meiji and Taishō periods in Japanese history.
• Obtain master copies of and read in advance the three short stories used in the lesson.
• Duplicate copies of visuals, stories, and handouts for student use.

Lesson Plan: Step-by-Step Procedure

Prior to beginning the lesson.

This lesson assumes students have studied modernization in Meiji and 20th-century Japan. If not, assign the Historical Background Essay and Handout 1, Background Essay Reading Guide , provided with this lesson, or assign students the appropriate chapter in their history text.

• Japan’s rapid modernization when faced with challenges from Western countries after Commodore Perry “opened” Japan to Western trade in the 1850s.
• The Meiji (1868-1911) government’s goals of industrializing and modernizing Japan.
• Rapid industrialization, the growth of critical industries upon which Japan’s modern economy was built.
• Ask students how these changes compare to changes in U.S. society around the same period. What do they know about the positive and negative aspects of the many changes that came with modernization in societies in the early 20th century?
• Next, introduce students to the Essential Questions provided at the beginning of this lesson plan. In what ways did the rapid modernization that characterized Japan of the 1880s-1920s influence writers of that period? What perspectives on modernization are reflected in literature produced during the Meiji and Taishō periods? Explain to students that they will be exploring the link between modernization and literature as reflected in the “Modern Literature” movement of this period in Japan.
• Distribute Handout 2, Visual Analysis Worksheet , to individual students or pairs; you will need four copies per student or pair or may direct students to write their answers on separate sheets of paper. Explain that the class is going to explore some images of early 20th-century Japanese society to form impressions of life and modernization in Japan at that time. Go over the Visual Analysis Worksheet so students understand the analysis process for each image they see.
• The first image is a photograph taken in Tokyo in 1934.
• September 1931 NAPF (Nippona Artista Proleta Federacio) magazine cover  
• October 1931 NAPF (Nippona Artista Proleta Federacio) magazine cover
• Women in the 1920s: Japanese women curling hair and Japanese women carrying wood
• Who was this image intended for? What do you see that makes you think this?
• Why do you think the artist or photographer chose to capture this event?
• What is the tone of the photo or illustration? That is, what is the artist or photographer’s attitude about the subject? On what visual clues do you base your answer?
• What is the mood of the photo or illustration? That is, what atmosphere does the image create for the viewer? What do you see that makes you know this?
• Taken together, how did the images you viewed help you understand Japan during this period?
• Next, distribute Handout 3 , “The Oyster Shell.” Read the excerpt from Kambara Ariake’s poem aloud to the students. Then ask students to read the poem again silently—this time using active reading strategies (line-by-line analysis of diction, images, symbols, form, and syntax).
• Lead students in a discussion of the poem using the questions on Handout 3 . An Answer Key with additional background for teachers is provided.
• To conclude Day 1, re-visit the images students viewed earlier in the class. Ask students to articulate characteristics the visuals of Modern Japan and Kambara’s poem share. Write these on the board.
• Distribute Handout 4 , Characteristics of Modern Japanese Literature. Review the handout. If time allows, conduct a class discussion comparing the list of characteristics students generated in Step 8 with the list on the handout. Otherwise, assign students to compare the two lists for homework.
• Tell students that today they will be reading a story titled “The Dancing Girl.” Published in 1890, “The Dancing Girl” came out at a time when Japan had been undergoing rapid industrialization, modernization, and social change for several decades during the late 1800s, the late Meiji period.
• Alert students to the Japanese convention of author names as outlined in the Teacher Background . In Japan, family name comes first, and given name last. However, in Japanese literature, well-established writers are known by their given names. So, in the case of Mori Ogai, whose story the class will read today, the author’s family name in Mori, and his given name is Ogai, but he is referred to as Ogai. Remind students of this convention as they move on to other stories in this unit.
• Remind students of the two Essential Questions for this lesson: In what ways did the events of modern Japan influence writers of that period? What perspectives on modernization are reflected in literature produced during the Meiji and Taishō periods? Ask students to keep these questions in mind as they read and discuss this story.
• How do you or those you know deal with adversity? Do you agree/disagree with these reactions? Why or why not?
• If you know something that might make a person unhappy, should you tell them anyway
• Should we only tell happy stories? Why or why not?
• Read Ogai’s “Dancing Girl” aloud to students or have students read independently. When the reading is complete, distribute Handout 5 , Reading for Tone and Mood; review the handout with students and ask them to complete the chart as they actively reread the story.
• At the end of the story, we are told that “friends like Aizawa Kenkichi are rare indeed, and yet to this very day there remains a part of me that curses him” (24). What did the narrator mean by this?
• Why were his countrymen so critical of his decision? If Elise’s madness had not removed the choice, how do you think the narrator would have resolved his dilemma?
• Who/what might Elise symbolize?
• Do you find the narrator’s confusion to be sincere? Why or why not?
• To conclude Day 2, have students complete Handout 6 , Post-Reading Worksheet for “The Broken Ring.” An Answer Key is provided. 
• Imagine that you are one of the characters from “The Dancing Girl.” What would your feelings be after the incident? Compose a letter adopting the persona of one of the main characters and expressing your reactions to the incident.
• Ask student volunteers to restate the Essential Questions for this lesson. Ask additional volunteers to review how the first story the class read, “The Dancing Girl,” addressed these essential questions.
• Set the stage for today’s story—“The Broken Ring,” by Shimizu Shikin—by alerting students that this story, written about the same time (late Meiji, 1891) as “The Dancing Girl,” explores another dimension of the changes of modernization, focusing on relationships, marriage, and expectations in modern society.
• If you know something that might make a person unhappy, should you tell them anyway?
• Read “The Broken Ring” aloud to students or have students read independently.
• Ask students to actively reread the story. As students read the text for the second time, ask them to mark their copy of the text as follows: Mark places in the story “ ” when the main character is happy. Mark places in the story “ ” when the main character is unhappy.
• What does the story seem to reveal about female identity and female duty?
• At the end of the story, we are told that “my only remaining hope is that this broken ring may somehow be restored to its perfect form by the hand that gave it to me. But I know, of course, that such a thing is not yet…”(239).  What did the narrator mean by this? Why end the statement with an ellipsis? What is the effect of this punctuation?
• The narrator reported that her father “has now come to have great sympathy for my long years of suffering” (239). How might the father’s change of heart serve as a symbol? What might this reveal about social attitudes in Japan at the turn of the 20th century?
• What do you think this story says about women in modern Japan at the turn of the century?
• The narrator asked, “Ah, will it take a hundred years before even a few will come to understand the precious value of this ring?” ( 232). Do you find the narrator’s confusion to be sincere? Why or why not? What do you think she hoped people would realize?
• Why did the narrator never tell us her name? What was achieved in her anonymity?
• To conclude Day 3, have students complete Handout 7 , Post-Reading Worksheet for “The Broken Ring.” An Answer Key is provided.
• As an optional extension, assign the following personal reflection writing assignment: Imagine that you are one of the characters from “The Broken Ring.” What would your feelings be after the incident? Compose a letter adopting the persona of one of these main characters and expressing your reactions to the incident.
• Ask students to volunteer ways in which the second story, “The Broken Ring,” addressed the Essential Questions.
• Explain to students that the final example of Modernist literature from Japan was written somewhat later than the first two stories, in the early 1920s. By this time, Japan had experienced over four decades of rapid change and modernization and was a fully industrialized society very much like England or the United States, and with many of the same social issues. The story focuses on another dimension of this rapid change—poor rural Japanese who experienced the changes of modernization in a different way than the middle class people of the previous stories. Thus, this story, “The Telegram,” presents yet another dimension of modernization.
• Provide students with a brief definition of proletarian literature of this time period, drawing from the paragraph about Kuroshima Denji in the Teacher Background . Ask what differences or new dimensions students might expect to see reflected in this story representing proletarian literature.
• Read “The Telegram” aloud in class or have students complete the reading independently.
• Why didn’t Gensaku allow his son to return to school? What statements or descriptions did you expect to hear but didn’t? How do you account for these omissions?
• What surprised you about the Kuroshimas’ story?
• How did the son react to his father’s decision? Why do you think he reacted the way he did?
• How did the choice of words affect you?
• How did the son live his life after his father’s decision? How do you think he felt about this?
• Why do you think the author told this story?
• Refer to Handout 4 and our discussion of the characteristics of modern Japanese literature from yesterday. What characteristics of modern Japanese literature are reflected in this story?
• What perspectives on modernization does this story illuminate?
• How do you or those you know deal with adversity? Do you agree/disagree with these reactions? Why or why not
• To conclude the day, have students complete Handout 8 , the Post-Reading Worksheet for “The Telegram.”
• As an optional extension, assign the following personal reflection writing assignment: Imagine that you are one of the characters from “The Telegram.” What would your feelings be after the incident? Compose a letter adopting the persona of one of these main characters and expressing your reactions to the incident.

As an optional assessment, students may be assigned the following essay:

How do you or those you know deal with adversity? How did Japanese people react to the challenges in their lives as a result of the demands of modernization? Do you agree/disagree with these reactions and/or the method in which they voiced their sentiments? Why or why not? Write a five-paragraph essay in which you articulate your understanding of the social landscape in the late 19 th and early 20 th centuries, Japan’s Modern Period in literature.

Standards Alignment

Common core :.

• CCSS.ELA-Literacy.RL.11-12.6:  Analyze a case in which grasping a point of view requires distinguishing what is directly stated in a text from what is really meant.
• CCSS.ELA-Literacy.W.11-12.1:  Write arguments to support claims in an analysis of substantive topics or texts, using valid reasoning and relevant and sufficient evidence.
• CCSS.ELA-Literacy.W.11-12.2:  Write informative/explanatory texts to examine and convey complex ideas and information clearly and accurately through effective selection, organization, and analysis of content.
• CCSS.ELA-Literacy.RH.11-12.7:  Integrate and evaluate content presented in diverse media and formats, including visually and quantitatively, as well as in words in order to address or solve a problem.
• CCSS.ELA-Literacy.WHST.11-12.7:  Conduct short as well as more sustained research projects to answer a question (including a self-generated question) or solve a problem; narrow or broaden the inquiry when appropriate; synthesize multiple sources on the subject, demonstrating understanding of the subject under investigation.

Great Books Foundation .

Keene, Donald. Dawn to the West: Japanese Literature of the Modern Era. New York: Columbia University Press,  1998.

Tyler, William J. Modanizumu: Modernist Fiction from Japan, 1913-1938. Honolulu: University of Hawaii Press, 2008. 

Baker, Frank W. “Visual Literacy.” Media Literacy in the K-12 Classroom.   International Society for Technology in Education.

Additional Resources

Japanese authors and literature of the modern period.

“A Short History of Japanese Literature: Part 5” and “Part 6.” Japan Kaleidoskop.  July 5, 2013. 

Meiji-Taishō History

Christensen, Maria. “The Meiji Era and the Modernization of Japan.” The Samurai Archives Japanese History Page. Cunningham, Mark E., and Lawrence J. Zwier. The End of the Shoguns and the Birth of Modern Japan.  Pivotal Moments in History series .  Minneapolis, MN: Twenty-First Century Books, 2009.

Created 2015 Program for Teaching East Asia

Becoming Modern

• Meiji and Taishō Japan: An Introductory Essay
• Voices from the Past: The Human Cost of Japan’s Modernization, 1880s-1930s
• The Nature of Sovereignty in Japan, 1870s-1920s
• A Window into Modern Japan: Using Sugoroku Games
• Moga , Factory Girls, Mothers, and Wives
• Inventing Modern Japanese Man
• Negotiating Relationships: United States and Japan, 1905-1933

Terms of Use:  Permission is given to reproduce this module for classroom use only. Other reproduction is prohibited without written permission from the Program for Teaching East Asia.

「『The Art Of、 Japanese Punctuation〜。』」！？ What Periods, Commas, Quotation Marks and Brackets Look Like in Japanese

March 21, 2016 • words written by Koichi and Kristen Dexter • Art by Aya Francisco

When you're sitting there writing something, you may take the little things for granted… little things like periods , commas , and quotation marks . That's cool— they only bind together everything a sentence holds dear . If you didn't have these little things, this "punctuation" if you will, the fabric of sentence time would tear apart, creating some kind of super-black hole. (Ironically, it would just look like a period.)

And, wouldn't you know it, punctuation exists in Japanese as well! It's not that much different from English punctuation, but there are definitely a few things to keep in mind if you want to read Japanese more easily or one day get into Japanese translation . In this article, I'm going to cover pretty much all the Japanese punctuation you'll run into. In order to learn it, it'll only take a quick read. Feel free to use this article as reference!

Let's get started with some backstory.

Japanese Punctuation Before the West

This may be shocking, but before the Meiji era there was no punctuation in Japanese. Their version of the modern-day period ( 。) was introduced from China centuries earlier. But of course, it was ignored. When it was used, it was put just about anywhere to mean just about anything.

Thanks to Emperor Meiji's love for Western literature, punctuation like the period and comma ( 、) eventually made its way into written Japanese. In 1946, some years after the Meiji Restoration, the Ministry of Education passed a bill, letting people know how they were supposed to use them. Luckily for us English speakers, this means that a lot of Japanese punctuation symbols are nice and familiar! Unless of course you're trying to read anything pre-WWII, in which case the punctuation is weird and/or nonexistent.

Full-Width Spacing

One thing that really stands out to me in Japanese writing is the spacing. While it differs between operating systems, handwriting style, and your Japanese IME, Japanese typography tends to be something known as "full-width." English, on the other hand, is "half-width." Can you see the difference?

• nandedarou?

While you can type in half-width spaces in Japanese, it looks crowded compared to text you'll see everywhere else. The Japanese language was made to be nice and spread out. And that carries over to their punctuation, as well. There are technically no spaces between letters or words in Japanese. The only place you will find "extra" space is after punctuation, where they are automatically included. This saves anyone typing in Japanese from having to hit the space bar unnecessarily, especially since it's done so infrequently otherwise.

To sum things up, you don't usually have to worry about adding spaces between sentences. Punctuation has you covered. For example:

皆さんこんにちは、トウフグのコウイチでございます。ハロー！

Find the comma and the period. There's a little half-width (normal width in English) space after them, even though I didn't add them in. All I did was type the comma and period themselves— it all counts as one "letter", even when you try to highlight it (go ahead, try and highlight the above sentence).

Now that you know all about empty space in Japanese writing, what about learning all the (main) Japanese punctuation available to you? Let's do it!

Japanese Punctuation Marks

Because Japanese punctuation is so similar to English punctuation, there is a lot of overlap. As I mentioned earlier, however, there also tend to be a lot of subtle differences, which I'll go over below.

• 。 句点 (くてん) or 丸 (まる)

The Japanese period is used much the same as the English period. It marks a full-stop, or end to a sentence. In vertical writing, it sits at the bottom right, below the character before it. If the sentence is on its own or has quotes, however, the Japanese period is omitted most of the time. Japanese periods look like this:

The period itself is a small circle, and not a dot. This character is used the majority of the time in written Japanese, though, occasionally, you will see Western-style periods when a sentence ends with an English word.

• 、 読点 (とうてん) or 点 (てん)

The Japanese comma, like the Japanese period, is used in much the same way as the English one. It's put in the same place as the period (bottom right after the word) in vertical writing, as well.

Comma usage in Japanese is incredibly liberal compared to English. You can stick it pretty much wherever you want a break or pause in your sentence. Just don't abuse the power, please, it, is, irritating.

• 「」 鈎括弧 (かぎかっこ)
• 「」 Single Quotation Marks

Instead of things that look like "this" for quotation marks, which would get confusing because of dakuten (more on that later), the Japanese use little half-brackets to indicate quotes. Although these are called "single quotation marks" or "single quotes", which might make you think of 'this', they are the most common style of quote to use in Japanese. Almost any time you need to use a marker for quotes, you'll use single quotes.

• 『』 二重鉤括弧 (にじゅうかぎかっこ) or 白括弧 (しろかっこ)
• 『』 Double Quotation Marks

Double quotes are a lot less common than single quotes, but they have one good purpose. You know when you have to quote something that's quoting something else? In English, that usually looks like this: "The dog said 'woof' and ran away."

In Japanese punctuation, double quotes go inside single quotes when you're quoting text within text. It's the same rules as in British English punctuation (single first, double second).

Sometimes people will use these double quotes alone as if they are single quotes, but that's a stylistic choice on their part.

• 〜 波線 (なみせん) or 波ダッシュ (なみだっしゅ)
• 〜 Wave Dash

The wave dash isn't really similar to the Western (straight) dash in use. But it's likely the wave dash became popular because straight-line-dashes are already used in katakana to show a long vowel, and not differentiating it here would be confusing.

There are some uses that are like the Western dash, like showing a range of something (４〜５, ９時〜１０時, etc), but there are some Japanese-only uses of this punctuation, including drawing out and changing the pitch of a vowel sound (そうだね〜), showing where something is from (アメリカ〜), and marking subtitles (〜こんにちは〜).

• ・ 中黒 (なかぐろ) or 中点 (なかてん) or 中ポツ (なかぽつ) or 黒丸 (くろまる)
• ・ Interpunct

The interpunct is a dot that aligns with the vertical or horizontal center (depending on writing direction) with the words next to it. It's typically used to break up words that go together. You see this most often when you have multiple words written in katakana, like foreign names.

It can be used with Japanese words, as well, though the use is more specialized in those cases. Some Japanese words, when placed side by side, can be ambiguous because combinations of kanji can mean different things. And if you have too many kanji next to each other it can get confusing.

Finally, the interpunct is used to break up lists, act as decimal points when writing numbers in kanji (why would you do that, please don't do that), and separate anything else that needs clarification. For example:

• ？ クエスチョンマーク or はてなマーク or 疑問符 (ぎもんふ) or 耳垂れ (みみだれ)
• ？ Question Mark

You'd think the Japanese question mark would be self explanatory, but there's a thing or two you ought to know about it. Just like its Western-style counterpart it indicates a question— that's simple enough. Thing is, though that Japanese already has a grammar-based marker (か) to show that you're making an inquiry, rendering any further punctuation redundant most of the time. As such, you won't see question marks in formal writing. Casual writing is a different story, because 1) casual writing has different rules in most languages and 2) Japanese speakers will often drop か in conversation in exchange for a questioning tone of voice, which is hard to convey without a question mark.

• ！ 感嘆符 (かんたんふ) or ビックリマーク or 雨垂れ (あまだれ) or エクスクラメーションマーク
• ！ Exclamation Point

The Japanese exclamation mark is used just like the Western one. It shows volume or emotion or both. You won't see exclamation marks in formal Japanese, though it's really common everywhere else, especially on Twitter, email, and text.

• （） 丸括弧 (まるかっこ)
• （） Parentheses

These look like English parentheses, but they have the extra spaces I mentioned when I covered full-width spacing. They're often used to show the kana readings of kanji words— for example:

They're also used an awful lot online in Japanese dictionaries and other educational resources ( like dusty paper books ). And, of course, they're used for annotations (like this) within a sentence.

• 【】 隅付き括弧 (すみつきかっこ) or 太亀甲 (ふときっこう) or 黒亀甲 (くろきっこう) or 墨付き括弧 (すみつきかっこ)
• 【】 Thick Brackets

Finally! Some Japanese punctuation we don't have in English! Sure, we have [] brackets, called 角括弧 （ かくかっこ ） in Japanese, but look at these dark ones! Brackets like this don't have a singular use, and they can really be used for anything; showing emphasis, listing items, or just making your brackets stand out more.

• ｛｝ 波括弧 (なみかっこ)
• ｛｝ Brackets

Just like the thick brackets, there is no specific use for these curly braces either. Often, though, you'll see them in inside normal brackets［｛｝］and in mathematical equations, too. I could have added about ten other bracket variations to this list. Seriously, there are way too many bracket types in Japanese.

• … 三点リーダー (さんてんりーだー)

Unlike the English ellipsis, the Japanese version typically hovers around the vertical middle of the line, instead of sitting at the bottom (though they can be formatted that way, as well). There can be as few as two ‥ or as many as six or more …… . They can symbolize the passing of time, silence, or a pause. They also convey silent emotion, which you'll recognize if you read a lot of anime and manga. Finally, you may also see them in text to symbolize long vowels or an omission or missing content.

Japanese Phonetic Marks

These aren't technically punctuation, but they're important symbols you'll see in Japanese and you should know what they mean, too.

• ゛ 濁点 (だくてん) or 点々 (てんてん)
• ゛ Dakuten or Tenten

These are the little marks you see next certain kana to make them "voiced." What that means, basically, is that your vocal cords vibrate when you say a them. They look like English quotations marks, which is probably why the Japanese version was created and is used way more often. They look like this when they're attached to kana:

And, just like the extra space that's added automatically between characters when you type in Japanese, you don't have to add these dakuten manually. Thanks to romaji you just type things how they sound— for example, "ga" for が— and the correct dakuten are added to the hiragana or katakana without any extra effort on your part. Thanks, technology!

• ゜ 半濁音 (はんだくおん) or 丸 (まる)
• ゜ Handakuten or Maru

The handakuten is similar to dakuten, but this little open circle means that the consonant it's attached to is "half" voiced. There are only a few of these in Japanese and they all make the "p" sound.

• っ 促音 (そくおん) or つまる音 (つまるおと)
• っ Small Tsu or Double Consonant

If you see this smaller version of the hiragana つ, it is not pronounced "tsu" (ever!). If you see it in the middle of a word, before a consonant, it means that the consonant after it is a "double" consonant. If you see it at the end of a word (before the particle と in many onomatopoeia) then it's a glottal stop. That means it's kind of like a constricted sound in your throat (that's your glottis in there, thus the name). The katakana version looks like this ッ.

• ー 長音符 (ちょうおんぷ) or 音引き (おんびき) or 棒引き (ぼうびき) or 伸ばし棒 (のばしぼう)
• ー Long Vowel Mark

Long vowel marks mark long vowels. So, instead of スウパア, you'd write スーパー. Simple right?

You'll mostly see these in katakana, hardly ever in hiragana. The only time you'll see them with hiragana is at the end of a sentence or after a drawn out particle or interjection. When it's used like that, it's interchangeable with 〜.

Bonus Symbols

While we're at it, let's look at some other symbols you're bound to see in Japanese.

• 々 踊り字 (おどりじ) or 躍り字 (おどりじ)
• 々 Iteration Mark

This neat-looking kanji is something called an iteration mark. That's a fancy way of saying it is a "repeater", i.e. any kanji it follows is repeated. You've probably seen it in words like 人々 （ ひとびと ） (people),　 時々 （ ときどき ） (sometimes), and even place names like 代々木 （ よよぎ ） (Yoyogi [Park]). There used to be repeaters for kana too, but they're hardly ever used nowadays. They look like this:

• Hiragana unvoiced: ゝ
• Katakana unvoiced: ヽ
• Hiragana voiced: ゞ
• Katakana voiced: ヾ

ヶ: 箇 & 个 Replacement

This may look like a small katakana ケ (and it is), but it's also used as a replacement for the counter 箇 (か), especially in months: ヶ月 (かげつ). See how it isn't read け, but か? So when you come across ５ヶ月, you read it as ごかげつ, or five months. You'll also see it pop up in place names like Chigasaki 茅ヶ崎市 （ ちがさきし ） , and Sekigahara 関ケ原町 （ せきがはら ） . But instead of か, it's pronounced が because rendaku . Totally not confusing, right?

• ￥ 円記号 (えんきごう)
• ￥ Yen Symbol

The yen symbol is used just like the dollar sign $ in English. You put it before the numbers it's referencing. You'll see this anywhere money is involved like receipts, price tags, online stores. But make sure you don't accidentally write this: ￥１００円. 円 is the kanji for yen. You need to pick! It's either ￥１００ or １００円.

• 〒〶 郵便記号 (ゆうびんきごう) or 郵便マーク (ゆうびんまーく)
• 〒〶 Postal Mark

This postal mark is used on addresses to indicate the postal code. That's pretty important if you have a Japanese pen-pal or if you're going to be mailing things in Japan. The one in the circle is usually on maps for post offices, so if you need to find the post office, look for this symbol. They're on mailboxes too, usually in red and white, unlike the American blue you may be used to.

There are plenty of other punctuation marks in Japanese , but these are the main ones (or the ones that I thought were important to learn). You'll also see a bunch of different brackets, colons, and so on in Japanese. But it should be pretty simple to understand how they're used and what they're doing there, now that you've learned the rules I've laid out here.

That does bring me to one last thing, which I think is pretty interesting, and that is:

Kaomoji As Japanese Punctuation

Kaomoji 顔文字 （ かおもじ ） , which translates to "Face Letters", is using text to draw little faces which show some kind of emotion. They're basically Japanese emoticons. While kaomoji will probably never be officially considered punctuation, I feel like it is a sort of new wave post-modern neo-punctuation.

When put together, they are characters that represent strong emotion, like the exclamation mark. They can also represent confusion or a questioning tone, like a question mark. On top of that, there are probably 20-30 different "feelings" they can represent that add to your sentences or paragraphs or phrases. While they aren't a single character (neither is an ellipsis, so take that punctuation snobs!), they do represent something which adds feeling to the sentence. That's basically what punctuation does, so why not kaomoji too?

If kaomoji can indeed be considered punctuation, there'd be a lot of them— too many to add to this list. Good thing we have a big kaomoji guide .

In terms of using kaomoji in Japanese, they usually go at the end of sentences or phrases. Think of them as periods that also convey emotion. Take that period! Go back to your soulless home in the country of boring-ville ヾ(♛;益;♛)ノ

Anyways, there you have it. I hope you learned something new, and thought about kaomoji a little bit, too. There really isn't a lot to learn when it comes to Japanese punctuation because you have most of the concepts down already (assuming you're not reading this as a tiny baby). It's really the subtleties that are interesting, I think, so enjoy them but don't get too hung up on them.

Main Navigation

• Contact NeurIPS
• Code of Ethics
• Code of Conduct
• Create Profile
• Journal To Conference Track
• Diversity & Inclusion
• Proceedings
• Future Meetings
• Exhibitor Information
• Privacy Policy

NeurIPS 2024

Conference Dates: (In person) 9 December - 15 December, 2024

Homepage: https://neurips.cc/Conferences/2024/

Call For Papers 

Abstract submission deadline: May 15, 2024

Full paper submission deadline, including technical appendices and supplemental material (all authors must have an OpenReview profile when submitting): May 22, 2024

Author notification: Sep 25, 2024

Camera-ready, poster, and video submission: Oct 30, 2024 AOE

Submit at: https://openreview.net/group?id=NeurIPS.cc/2024/Conference  

The site will start accepting submissions on Apr 22, 2024 

Subscribe to these and other dates on the 2024 dates page .

The Thirty-Eighth Annual Conference on Neural Information Processing Systems (NeurIPS 2024) is an interdisciplinary conference that brings together researchers in machine learning, neuroscience, statistics, optimization, computer vision, natural language processing, life sciences, natural sciences, social sciences, and other adjacent fields. We invite submissions presenting new and original research on topics including but not limited to the following:

• Applications (e.g., vision, language, speech and audio, Creative AI)
• Deep learning (e.g., architectures, generative models, optimization for deep networks, foundation models, LLMs)
• Evaluation (e.g., methodology, meta studies, replicability and validity, human-in-the-loop)
• General machine learning (supervised, unsupervised, online, active, etc.)
• Infrastructure (e.g., libraries, improved implementation and scalability, distributed solutions)
• Machine learning for sciences (e.g. climate, health, life sciences, physics, social sciences)
• Neuroscience and cognitive science (e.g., neural coding, brain-computer interfaces)
• Optimization (e.g., convex and non-convex, stochastic, robust)
• Probabilistic methods (e.g., variational inference, causal inference, Gaussian processes)
• Reinforcement learning (e.g., decision and control, planning, hierarchical RL, robotics)
• Social and economic aspects of machine learning (e.g., fairness, interpretability, human-AI interaction, privacy, safety, strategic behavior)
• Theory (e.g., control theory, learning theory, algorithmic game theory)

Machine learning is a rapidly evolving field, and so we welcome interdisciplinary submissions that do not fit neatly into existing categories.

Authors are asked to confirm that their submissions accord with the NeurIPS code of conduct .

Formatting instructions:   All submissions must be in PDF format, and in a single PDF file include, in this order:

• The submitted paper
• Technical appendices that support the paper with additional proofs, derivations, or results 
• The NeurIPS paper checklist  

Other supplementary materials such as data and code can be uploaded as a ZIP file

The main text of a submitted paper is limited to nine content pages , including all figures and tables. Additional pages containing references don’t count as content pages. If your submission is accepted, you will be allowed an additional content page for the camera-ready version.

The main text and references may be followed by technical appendices, for which there is no page limit.

The maximum file size for a full submission, which includes technical appendices, is 50MB.

Authors are encouraged to submit a separate ZIP file that contains further supplementary material like data or source code, when applicable.

You must format your submission using the NeurIPS 2024 LaTeX style file which includes a “preprint” option for non-anonymous preprints posted online. Submissions that violate the NeurIPS style (e.g., by decreasing margins or font sizes) or page limits may be rejected without further review. Papers may be rejected without consideration of their merits if they fail to meet the submission requirements, as described in this document. 

Paper checklist: In order to improve the rigor and transparency of research submitted to and published at NeurIPS, authors are required to complete a paper checklist . The paper checklist is intended to help authors reflect on a wide variety of issues relating to responsible machine learning research, including reproducibility, transparency, research ethics, and societal impact. The checklist forms part of the paper submission, but does not count towards the page limit.

Please join the NeurIPS 2024 Checklist Assistant Study that will provide you with free verification of your checklist performed by an LLM here . Please see details in our  blog

Supplementary material: While all technical appendices should be included as part of the main paper submission PDF, authors may submit up to 100MB of supplementary material, such as data, or source code in a ZIP format. Supplementary material should be material created by the authors that directly supports the submission content. Like submissions, supplementary material must be anonymized. Looking at supplementary material is at the discretion of the reviewers.

We encourage authors to upload their code and data as part of their supplementary material in order to help reviewers assess the quality of the work. Check the policy as well as code submission guidelines and templates for further details.

Use of Large Language Models (LLMs): We welcome authors to use any tool that is suitable for preparing high-quality papers and research. However, we ask authors to keep in mind two important criteria. First, we expect papers to fully describe their methodology, and any tool that is important to that methodology, including the use of LLMs, should be described also. For example, authors should mention tools (including LLMs) that were used for data processing or filtering, visualization, facilitating or running experiments, and proving theorems. It may also be advisable to describe the use of LLMs in implementing the method (if this corresponds to an important, original, or non-standard component of the approach). Second, authors are responsible for the entire content of the paper, including all text and figures, so while authors are welcome to use any tool they wish for writing the paper, they must ensure that all text is correct and original.

Double-blind reviewing:   All submissions must be anonymized and may not contain any identifying information that may violate the double-blind reviewing policy.  This policy applies to any supplementary or linked material as well, including code.  If you are including links to any external material, it is your responsibility to guarantee anonymous browsing.  Please do not include acknowledgements at submission time. If you need to cite one of your own papers, you should do so with adequate anonymization to preserve double-blind reviewing.  For instance, write “In the previous work of Smith et al. [1]…” rather than “In our previous work [1]...”). If you need to cite one of your own papers that is in submission to NeurIPS and not available as a non-anonymous preprint, then include a copy of the cited anonymized submission in the supplementary material and write “Anonymous et al. [1] concurrently show...”). Any papers found to be violating this policy will be rejected.

OpenReview: We are using OpenReview to manage submissions. The reviews and author responses will not be public initially (but may be made public later, see below). As in previous years, submissions under review will be visible only to their assigned program committee. We will not be soliciting comments from the general public during the reviewing process. Anyone who plans to submit a paper as an author or a co-author will need to create (or update) their OpenReview profile by the full paper submission deadline. Your OpenReview profile can be edited by logging in and clicking on your name in https://openreview.net/ . This takes you to a URL "https://openreview.net/profile?id=~[Firstname]_[Lastname][n]" where the last part is your profile name, e.g., ~Wei_Zhang1. The OpenReview profiles must be up to date, with all publications by the authors, and their current affiliations. The easiest way to import publications is through DBLP but it is not required, see FAQ . Submissions without updated OpenReview profiles will be desk rejected. The information entered in the profile is critical for ensuring that conflicts of interest and reviewer matching are handled properly. Because of the rapid growth of NeurIPS, we request that all authors help with reviewing papers, if asked to do so. We need everyone’s help in maintaining the high scientific quality of NeurIPS.  

Please be aware that OpenReview has a moderation policy for newly created profiles: New profiles created without an institutional email will go through a moderation process that can take up to two weeks. New profiles created with an institutional email will be activated automatically.

Venue home page: https://openreview.net/group?id=NeurIPS.cc/2024/Conference

If you have any questions, please refer to the FAQ: https://openreview.net/faq

Abstract Submission: There is a mandatory abstract submission deadline on May 15, 2024, six days before full paper submissions are due. While it will be possible to edit the title and abstract until the full paper submission deadline, submissions with “placeholder” abstracts that are rewritten for the full submission risk being removed without consideration. This includes titles and abstracts that either provide little or no semantic information (e.g., "We provide a new semi-supervised learning method.") or describe a substantively different claimed contribution.  The author list cannot be changed after the abstract deadline. After that, authors may be reordered, but any additions or removals must be justified in writing and approved on a case-by-case basis by the program chairs only in exceptional circumstances. 

Ethics review: Reviewers and ACs may flag submissions for ethics review . Flagged submissions will be sent to an ethics review committee for comments. Comments from ethics reviewers will be considered by the primary reviewers and AC as part of their deliberation. They will also be visible to authors, who will have an opportunity to respond.  Ethics reviewers do not have the authority to reject papers, but in extreme cases papers may be rejected by the program chairs on ethical grounds, regardless of scientific quality or contribution.  

Preprints: The existence of non-anonymous preprints (on arXiv or other online repositories, personal websites, social media) will not result in rejection. If you choose to use the NeurIPS style for the preprint version, you must use the “preprint” option rather than the “final” option. Reviewers will be instructed not to actively look for such preprints, but encountering them will not constitute a conflict of interest. Authors may submit anonymized work to NeurIPS that is already available as a preprint (e.g., on arXiv) without citing it. Note that public versions of the submission should not say "Under review at NeurIPS" or similar.

Dual submissions: Submissions that are substantially similar to papers that the authors have previously published or submitted in parallel to other peer-reviewed venues with proceedings or journals may not be submitted to NeurIPS. Papers previously presented at workshops are permitted, so long as they did not appear in a conference proceedings (e.g., CVPRW proceedings), a journal or a book.  NeurIPS coordinates with other conferences to identify dual submissions.  The NeurIPS policy on dual submissions applies for the entire duration of the reviewing process.  Slicing contributions too thinly is discouraged.  The reviewing process will treat any other submission by an overlapping set of authors as prior work. If publishing one would render the other too incremental, both may be rejected.

Anti-collusion: NeurIPS does not tolerate any collusion whereby authors secretly cooperate with reviewers, ACs or SACs to obtain favorable reviews. 

Author responses:   Authors will have one week to view and respond to initial reviews. Author responses may not contain any identifying information that may violate the double-blind reviewing policy. Authors may not submit revisions of their paper or supplemental material, but may post their responses as a discussion in OpenReview. This is to reduce the burden on authors to have to revise their paper in a rush during the short rebuttal period.

After the initial response period, authors will be able to respond to any further reviewer/AC questions and comments by posting on the submission’s forum page. The program chairs reserve the right to solicit additional reviews after the initial author response period.  These reviews will become visible to the authors as they are added to OpenReview, and authors will have a chance to respond to them.

After the notification deadline, accepted and opted-in rejected papers will be made public and open for non-anonymous public commenting. Their anonymous reviews, meta-reviews, author responses and reviewer responses will also be made public. Authors of rejected papers will have two weeks after the notification deadline to opt in to make their deanonymized rejected papers public in OpenReview.  These papers are not counted as NeurIPS publications and will be shown as rejected in OpenReview.

Publication of accepted submissions:   Reviews, meta-reviews, and any discussion with the authors will be made public for accepted papers (but reviewer, area chair, and senior area chair identities will remain anonymous). Camera-ready papers will be due in advance of the conference. All camera-ready papers must include a funding disclosure . We strongly encourage accompanying code and data to be submitted with accepted papers when appropriate, as per the code submission policy . Authors will be allowed to make minor changes for a short period of time after the conference.

Contemporaneous Work: For the purpose of the reviewing process, papers that appeared online within two months of a submission will generally be considered "contemporaneous" in the sense that the submission will not be rejected on the basis of the comparison to contemporaneous work. Authors are still expected to cite and discuss contemporaneous work and perform empirical comparisons to the degree feasible. Any paper that influenced the submission is considered prior work and must be cited and discussed as such. Submissions that are very similar to contemporaneous work will undergo additional scrutiny to prevent cases of plagiarism and missing credit to prior work.

Plagiarism is prohibited by the NeurIPS Code of Conduct .

Other Tracks: Similarly to earlier years, we will host multiple tracks, such as datasets, competitions, tutorials as well as workshops, in addition to the main track for which this call for papers is intended. See the conference homepage for updates and calls for participation in these tracks. 

Experiments: As in past years, the program chairs will be measuring the quality and effectiveness of the review process via randomized controlled experiments. All experiments are independently reviewed and approved by an Institutional Review Board (IRB).

Financial Aid: Each paper may designate up to one (1) NeurIPS.cc account email address of a corresponding student author who confirms that they would need the support to attend the conference, and agrees to volunteer if they get selected. To be considered for Financial the student will also need to fill out the Financial Aid application when it becomes available.

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• My Account Login
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Open access
• Published: 03 June 2024

Applying large language models for automated essay scoring for non-native Japanese

• Wenchao Li 1 &
• Haitao Liu 2  

Humanities and Social Sciences Communications volume  11 , Article number:  723 ( 2024 ) Cite this article

129 Accesses

1 Altmetric

Metrics details

• Language and linguistics

Recent advancements in artificial intelligence (AI) have led to an increased use of large language models (LLMs) for language assessment tasks such as automated essay scoring (AES), automated listening tests, and automated oral proficiency assessments. The application of LLMs for AES in the context of non-native Japanese, however, remains limited. This study explores the potential of LLM-based AES by comparing the efficiency of different models, i.e. two conventional machine training technology-based methods (Jess and JWriter), two LLMs (GPT and BERT), and one Japanese local LLM (Open-Calm large model). To conduct the evaluation, a dataset consisting of 1400 story-writing scripts authored by learners with 12 different first languages was used. Statistical analysis revealed that GPT-4 outperforms Jess and JWriter, BERT, and the Japanese language-specific trained Open-Calm large model in terms of annotation accuracy and predicting learning levels. Furthermore, by comparing 18 different models that utilize various prompts, the study emphasized the significance of prompts in achieving accurate and reliable evaluations using LLMs.

Similar content being viewed by others

Accurate structure prediction of biomolecular interactions with AlphaFold 3

Testing theory of mind in large language models and humans

Highly accurate protein structure prediction with AlphaFold

Conventional machine learning technology in aes.

AES has experienced significant growth with the advancement of machine learning technologies in recent decades. In the earlier stages of AES development, conventional machine learning-based approaches were commonly used. These approaches involved the following procedures: a) feeding the machine with a dataset. In this step, a dataset of essays is provided to the machine learning system. The dataset serves as the basis for training the model and establishing patterns and correlations between linguistic features and human ratings. b) the machine learning model is trained using linguistic features that best represent human ratings and can effectively discriminate learners’ writing proficiency. These features include lexical richness (Lu, 2012 ; Kyle and Crossley, 2015 ; Kyle et al. 2021 ), syntactic complexity (Lu, 2010 ; Liu, 2008 ), text cohesion (Crossley and McNamara, 2016 ), and among others. Conventional machine learning approaches in AES require human intervention, such as manual correction and annotation of essays. This human involvement was necessary to create a labeled dataset for training the model. Several AES systems have been developed using conventional machine learning technologies. These include the Intelligent Essay Assessor (Landauer et al. 2003 ), the e-rater engine by Educational Testing Service (Attali and Burstein, 2006 ; Burstein, 2003 ), MyAccess with the InterlliMetric scoring engine by Vantage Learning (Elliot, 2003 ), and the Bayesian Essay Test Scoring system (Rudner and Liang, 2002 ). These systems have played a significant role in automating the essay scoring process and providing quick and consistent feedback to learners. However, as touched upon earlier, conventional machine learning approaches rely on predetermined linguistic features and often require manual intervention, making them less flexible and potentially limiting their generalizability to different contexts.

In the context of the Japanese language, conventional machine learning-incorporated AES tools include Jess (Ishioka and Kameda, 2006 ) and JWriter (Lee and Hasebe, 2017 ). Jess assesses essays by deducting points from the perfect score, utilizing the Mainichi Daily News newspaper as a database. The evaluation criteria employed by Jess encompass various aspects, such as rhetorical elements (e.g., reading comprehension, vocabulary diversity, percentage of complex words, and percentage of passive sentences), organizational structures (e.g., forward and reverse connection structures), and content analysis (e.g., latent semantic indexing). JWriter employs linear regression analysis to assign weights to various measurement indices, such as average sentence length and total number of characters. These weights are then combined to derive the overall score. A pilot study involving the Jess model was conducted on 1320 essays at different proficiency levels, including primary, intermediate, and advanced. However, the results indicated that the Jess model failed to significantly distinguish between these essay levels. Out of the 16 measures used, four measures, namely median sentence length, median clause length, median number of phrases, and maximum number of phrases, did not show statistically significant differences between the levels. Additionally, two measures exhibited between-level differences but lacked linear progression: the number of attributives declined words and the Kanji/kana ratio. On the other hand, the remaining measures, including maximum sentence length, maximum clause length, number of attributive conjugated words, maximum number of consecutive infinitive forms, maximum number of conjunctive-particle clauses, k characteristic value, percentage of big words, and percentage of passive sentences, demonstrated statistically significant between-level differences and displayed linear progression.

Both Jess and JWriter exhibit notable limitations, including the manual selection of feature parameters and weights, which can introduce biases into the scoring process. The reliance on human annotators to label non-native language essays also introduces potential noise and variability in the scoring. Furthermore, an important concern is the possibility of system manipulation and cheating by learners who are aware of the regression equation utilized by the models (Hirao et al. 2020 ). These limitations emphasize the need for further advancements in AES systems to address these challenges.

Deep learning technology in AES

Deep learning has emerged as one of the approaches for improving the accuracy and effectiveness of AES. Deep learning-based AES methods utilize artificial neural networks that mimic the human brain’s functioning through layered algorithms and computational units. Unlike conventional machine learning, deep learning autonomously learns from the environment and past errors without human intervention. This enables deep learning models to establish nonlinear correlations, resulting in higher accuracy. Recent advancements in deep learning have led to the development of transformers, which are particularly effective in learning text representations. Noteworthy examples include bidirectional encoder representations from transformers (BERT) (Devlin et al. 2019 ) and the generative pretrained transformer (GPT) (OpenAI).

BERT is a linguistic representation model that utilizes a transformer architecture and is trained on two tasks: masked linguistic modeling and next-sentence prediction (Hirao et al. 2020 ; Vaswani et al. 2017 ). In the context of AES, BERT follows specific procedures, as illustrated in Fig. 1 : (a) the tokenized prompts and essays are taken as input; (b) special tokens, such as [CLS] and [SEP], are added to mark the beginning and separation of prompts and essays; (c) the transformer encoder processes the prompt and essay sequences, resulting in hidden layer sequences; (d) the hidden layers corresponding to the [CLS] tokens (T[CLS]) represent distributed representations of the prompts and essays; and (e) a multilayer perceptron uses these distributed representations as input to obtain the final score (Hirao et al. 2020 ).

AES system with BERT (Hirao et al. 2020 ).

The training of BERT using a substantial amount of sentence data through the Masked Language Model (MLM) allows it to capture contextual information within the hidden layers. Consequently, BERT is expected to be capable of identifying artificial essays as invalid and assigning them lower scores (Mizumoto and Eguchi, 2023 ). In the context of AES for nonnative Japanese learners, Hirao et al. ( 2020 ) combined the long short-term memory (LSTM) model proposed by Hochreiter and Schmidhuber ( 1997 ) with BERT to develop a tailored automated Essay Scoring System. The findings of their study revealed that the BERT model outperformed both the conventional machine learning approach utilizing character-type features such as “kanji” and “hiragana”, as well as the standalone LSTM model. Takeuchi et al. ( 2021 ) presented an approach to Japanese AES that eliminates the requirement for pre-scored essays by relying solely on reference texts or a model answer for the essay task. They investigated multiple similarity evaluation methods, including frequency of morphemes, idf values calculated on Wikipedia, LSI, LDA, word-embedding vectors, and document vectors produced by BERT. The experimental findings revealed that the method utilizing the frequency of morphemes with idf values exhibited the strongest correlation with human-annotated scores across different essay tasks. The utilization of BERT in AES encounters several limitations. Firstly, essays often exceed the model’s maximum length limit. Second, only score labels are available for training, which restricts access to additional information.

Mizumoto and Eguchi ( 2023 ) were pioneers in employing the GPT model for AES in non-native English writing. Their study focused on evaluating the accuracy and reliability of AES using the GPT-3 text-davinci-003 model, analyzing a dataset of 12,100 essays from the corpus of nonnative written English (TOEFL11). The findings indicated that AES utilizing the GPT-3 model exhibited a certain degree of accuracy and reliability. They suggest that GPT-3-based AES systems hold the potential to provide support for human ratings. However, applying GPT model to AES presents a unique natural language processing (NLP) task that involves considerations such as nonnative language proficiency, the influence of the learner’s first language on the output in the target language, and identifying linguistic features that best indicate writing quality in a specific language. These linguistic features may differ morphologically or syntactically from those present in the learners’ first language, as observed in (1)–(3).

我-送了-他-一本-书

Wǒ-sòngle-tā-yī běn-shū

1 sg .-give. past- him-one .cl- book

“I gave him a book.”

Agglutinative

彼-に-本-を-あげ-まし-た

Kare-ni-hon-o-age-mashi-ta

3 sg .- dat -hon- acc- give.honorification. past

Inflectional

give, give-s, gave, given, giving

Additionally, the morphological agglutination and subject-object-verb (SOV) order in Japanese, along with its idiomatic expressions, pose additional challenges for applying language models in AES tasks (4).

足-が 棒-に なり-ました

Ashi-ga bo-ni nar-mashita

leg- nom stick- dat become- past

“My leg became like a stick (I am extremely tired).”

The example sentence provided demonstrates the morpho-syntactic structure of Japanese and the presence of an idiomatic expression. In this sentence, the verb “なる” (naru), meaning “to become”, appears at the end of the sentence. The verb stem “なり” (nari) is attached with morphemes indicating honorification (“ます” - mashu) and tense (“た” - ta), showcasing agglutination. While the sentence can be literally translated as “my leg became like a stick”, it carries an idiomatic interpretation that implies “I am extremely tired”.

To overcome this issue, CyberAgent Inc. ( 2023 ) has developed the Open-Calm series of language models specifically designed for Japanese. Open-Calm consists of pre-trained models available in various sizes, such as Small, Medium, Large, and 7b. Figure 2 depicts the fundamental structure of the Open-Calm model. A key feature of this architecture is the incorporation of the Lora Adapter and GPT-NeoX frameworks, which can enhance its language processing capabilities.

GPT-NeoX Model Architecture (Okgetheng and Takeuchi 2024 ).

In a recent study conducted by Okgetheng and Takeuchi ( 2024 ), they assessed the efficacy of Open-Calm language models in grading Japanese essays. The research utilized a dataset of approximately 300 essays, which were annotated by native Japanese educators. The findings of the study demonstrate the considerable potential of Open-Calm language models in automated Japanese essay scoring. Specifically, among the Open-Calm family, the Open-Calm Large model (referred to as OCLL) exhibited the highest performance. However, it is important to note that, as of the current date, the Open-Calm Large model does not offer public access to its server. Consequently, users are required to independently deploy and operate the environment for OCLL. In order to utilize OCLL, users must have a PC equipped with an NVIDIA GeForce RTX 3060 (8 or 12 GB VRAM).

In summary, while the potential of LLMs in automated scoring of nonnative Japanese essays has been demonstrated in two studies—BERT-driven AES (Hirao et al. 2020 ) and OCLL-based AES (Okgetheng and Takeuchi, 2024 )—the number of research efforts in this area remains limited.

Another significant challenge in applying LLMs to AES lies in prompt engineering and ensuring its reliability and effectiveness (Brown et al. 2020 ; Rae et al. 2021 ; Zhang et al. 2021 ). Various prompting strategies have been proposed, such as the zero-shot chain of thought (CoT) approach (Kojima et al. 2022 ), which involves manually crafting diverse and effective examples. However, manual efforts can lead to mistakes. To address this, Zhang et al. ( 2021 ) introduced an automatic CoT prompting method called Auto-CoT, which demonstrates matching or superior performance compared to the CoT paradigm. Another prompt framework is trees of thoughts, enabling a model to self-evaluate its progress at intermediate stages of problem-solving through deliberate reasoning (Yao et al. 2023 ).

Beyond linguistic studies, there has been a noticeable increase in the number of foreign workers in Japan and Japanese learners worldwide (Ministry of Health, Labor, and Welfare of Japan, 2022 ; Japan Foundation, 2021 ). However, existing assessment methods, such as the Japanese Language Proficiency Test (JLPT), J-CAT, and TTBJ Footnote 1 , primarily focus on reading, listening, vocabulary, and grammar skills, neglecting the evaluation of writing proficiency. As the number of workers and language learners continues to grow, there is a rising demand for an efficient AES system that can reduce costs and time for raters and be utilized for employment, examinations, and self-study purposes.

This study aims to explore the potential of LLM-based AES by comparing the effectiveness of five models: two LLMs (GPT Footnote 2 and BERT), one Japanese local LLM (OCLL), and two conventional machine learning-based methods (linguistic feature-based scoring tools - Jess and JWriter).

The research questions addressed in this study are as follows:

To what extent do the LLM-driven AES and linguistic feature-based AES, when used as automated tools to support human rating, accurately reflect test takers’ actual performance?

What influence does the prompt have on the accuracy and performance of LLM-based AES methods?

The subsequent sections of the manuscript cover the methodology, including the assessment measures for nonnative Japanese writing proficiency, criteria for prompts, and the dataset. The evaluation section focuses on the analysis of annotations and rating scores generated by LLM-driven and linguistic feature-based AES methods.

Methodology

The dataset utilized in this study was obtained from the International Corpus of Japanese as a Second Language (I-JAS) Footnote 3 . This corpus consisted of 1000 participants who represented 12 different first languages. For the study, the participants were given a story-writing task on a personal computer. They were required to write two stories based on the 4-panel illustrations titled “Picnic” and “The key” (see Appendix A). Background information for the participants was provided by the corpus, including their Japanese language proficiency levels assessed through two online tests: J-CAT and SPOT. These tests evaluated their reading, listening, vocabulary, and grammar abilities. The learners’ proficiency levels were categorized into six levels aligned with the Common European Framework of Reference for Languages (CEFR) and the Reference Framework for Japanese Language Education (RFJLE): A1, A2, B1, B2, C1, and C2. According to Lee et al. ( 2015 ), there is a high level of agreement (r = 0.86) between the J-CAT and SPOT assessments, indicating that the proficiency certifications provided by J-CAT are consistent with those of SPOT. However, it is important to note that the scores of J-CAT and SPOT do not have a one-to-one correspondence. In this study, the J-CAT scores were used as a benchmark to differentiate learners of different proficiency levels. A total of 1400 essays were utilized, representing the beginner (aligned with A1), A2, B1, B2, C1, and C2 levels based on the J-CAT scores. Table 1 provides information about the learners’ proficiency levels and their corresponding J-CAT and SPOT scores.

A dataset comprising a total of 1400 essays from the story writing tasks was collected. Among these, 714 essays were utilized to evaluate the reliability of the LLM-based AES method, while the remaining 686 essays were designated as development data to assess the LLM-based AES’s capability to distinguish participants with varying proficiency levels. The GPT 4 API was used in this study. A detailed explanation of the prompt-assessment criteria is provided in Section Prompt . All essays were sent to the model for measurement and scoring.

Measures of writing proficiency for nonnative Japanese

Japanese exhibits a morphologically agglutinative structure where morphemes are attached to the word stem to convey grammatical functions such as tense, aspect, voice, and honorifics, e.g. (5).

食べ-させ-られ-まし-た-か

tabe-sase-rare-mashi-ta-ka

[eat (stem)-causative-passive voice-honorification-tense. past-question marker]

Japanese employs nine case particles to indicate grammatical functions: the nominative case particle が (ga), the accusative case particle を (o), the genitive case particle の (no), the dative case particle に (ni), the locative/instrumental case particle で (de), the ablative case particle から (kara), the directional case particle へ (e), and the comitative case particle と (to). The agglutinative nature of the language, combined with the case particle system, provides an efficient means of distinguishing between active and passive voice, either through morphemes or case particles, e.g. 食べる taberu “eat concusive . ” (active voice); 食べられる taberareru “eat concusive . ” (passive voice). In the active voice, “パン を 食べる” (pan o taberu) translates to “to eat bread”. On the other hand, in the passive voice, it becomes “パン が 食べられた” (pan ga taberareta), which means “(the) bread was eaten”. Additionally, it is important to note that different conjugations of the same lemma are considered as one type in order to ensure a comprehensive assessment of the language features. For example, e.g., 食べる taberu “eat concusive . ”; 食べている tabeteiru “eat progress .”; 食べた tabeta “eat past . ” as one type.

To incorporate these features, previous research (Suzuki, 1999 ; Watanabe et al. 1988 ; Ishioka, 2001 ; Ishioka and Kameda, 2006 ; Hirao et al. 2020 ) has identified complexity, fluency, and accuracy as crucial factors for evaluating writing quality. These criteria are assessed through various aspects, including lexical richness (lexical density, diversity, and sophistication), syntactic complexity, and cohesion (Kyle et al. 2021 ; Mizumoto and Eguchi, 2023 ; Ure, 1971 ; Halliday, 1985 ; Barkaoui and Hadidi, 2020 ; Zenker and Kyle, 2021 ; Kim et al. 2018 ; Lu, 2017 ; Ortega, 2015 ). Therefore, this study proposes five scoring categories: lexical richness, syntactic complexity, cohesion, content elaboration, and grammatical accuracy. A total of 16 measures were employed to capture these categories. The calculation process and specific details of these measures can be found in Table 2 .

T-unit, first introduced by Hunt ( 1966 ), is a measure used for evaluating speech and composition. It serves as an indicator of syntactic development and represents the shortest units into which a piece of discourse can be divided without leaving any sentence fragments. In the context of Japanese language assessment, Sakoda and Hosoi ( 2020 ) utilized T-unit as the basic unit to assess the accuracy and complexity of Japanese learners’ speaking and storytelling. The calculation of T-units in Japanese follows the following principles:

A single main clause constitutes 1 T-unit, regardless of the presence or absence of dependent clauses, e.g. (6).

ケンとマリはピクニックに行きました (main clause): 1 T-unit.

If a sentence contains a main clause along with subclauses, each subclause is considered part of the same T-unit, e.g. (7).

天気が良かった の で (subclause)、ケンとマリはピクニックに行きました (main clause): 1 T-unit.

In the case of coordinate clauses, where multiple clauses are connected, each coordinated clause is counted separately. Thus, a sentence with coordinate clauses may have 2 T-units or more, e.g. (8).

ケンは地図で場所を探して (coordinate clause)、マリはサンドイッチを作りました (coordinate clause): 2 T-units.

Lexical diversity refers to the range of words used within a text (Engber, 1995 ; Kyle et al. 2021 ) and is considered a useful measure of the breadth of vocabulary in L n production (Jarvis, 2013a , 2013b ).

The type/token ratio (TTR) is widely recognized as a straightforward measure for calculating lexical diversity and has been employed in numerous studies. These studies have demonstrated a strong correlation between TTR and other methods of measuring lexical diversity (e.g., Bentz et al. 2016 ; Čech and Miroslav, 2018 ; Çöltekin and Taraka, 2018 ). TTR is computed by considering both the number of unique words (types) and the total number of words (tokens) in a given text. Given that the length of learners’ writing texts can vary, this study employs the moving average type-token ratio (MATTR) to mitigate the influence of text length. MATTR is calculated using a 50-word moving window. Initially, a TTR is determined for words 1–50 in an essay, followed by words 2–51, 3–52, and so on until the end of the essay is reached (Díez-Ortega and Kyle, 2023 ). The final MATTR scores were obtained by averaging the TTR scores for all 50-word windows. The following formula was employed to derive MATTR:

\({\rm{MATTR}}({\rm{W}})=\frac{{\sum }_{{\rm{i}}=1}^{{\rm{N}}-{\rm{W}}+1}{{\rm{F}}}_{{\rm{i}}}}{{\rm{W}}({\rm{N}}-{\rm{W}}+1)}\)

Here, N refers to the number of tokens in the corpus. W is the randomly selected token size (W < N). \({F}_{i}\) is the number of types in each window. The \({\rm{MATTR}}({\rm{W}})\) is the mean of a series of type-token ratios (TTRs) based on the word form for all windows. It is expected that individuals with higher language proficiency will produce texts with greater lexical diversity, as indicated by higher MATTR scores.

Lexical density was captured by the ratio of the number of lexical words to the total number of words (Lu, 2012 ). Lexical sophistication refers to the utilization of advanced vocabulary, often evaluated through word frequency indices (Crossley et al. 2013 ; Haberman, 2008 ; Kyle and Crossley, 2015 ; Laufer and Nation, 1995 ; Lu, 2012 ; Read, 2000 ). In line of writing, lexical sophistication can be interpreted as vocabulary breadth, which entails the appropriate usage of vocabulary items across various lexicon-grammatical contexts and registers (Garner et al. 2019 ; Kim et al. 2018 ; Kyle et al. 2018 ). In Japanese specifically, words are considered lexically sophisticated if they are not included in the “Japanese Education Vocabulary List Ver 1.0”. Footnote 4 Consequently, lexical sophistication was calculated by determining the number of sophisticated word types relative to the total number of words per essay. Furthermore, it has been suggested that, in Japanese writing, sentences should ideally have a length of no more than 40 to 50 characters, as this promotes readability. Therefore, the median and maximum sentence length can be considered as useful indices for assessment (Ishioka and Kameda, 2006 ).

Syntactic complexity was assessed based on several measures, including the mean length of clauses, verb phrases per T-unit, clauses per T-unit, dependent clauses per T-unit, complex nominals per clause, adverbial clauses per clause, coordinate phrases per clause, and mean dependency distance (MDD). The MDD reflects the distance between the governor and dependent positions in a sentence. A larger dependency distance indicates a higher cognitive load and greater complexity in syntactic processing (Liu, 2008 ; Liu et al. 2017 ). The MDD has been established as an efficient metric for measuring syntactic complexity (Jiang, Quyang, and Liu, 2019 ; Li and Yan, 2021 ). To calculate the MDD, the position numbers of the governor and dependent are subtracted, assuming that words in a sentence are assigned in a linear order, such as W1 … Wi … Wn. In any dependency relationship between words Wa and Wb, Wa is the governor and Wb is the dependent. The MDD of the entire sentence was obtained by taking the absolute value of governor – dependent:

MDD = \(\frac{1}{n}{\sum }_{i=1}^{n}|{\rm{D}}{{\rm{D}}}_{i}|\)

In this formula, \(n\) represents the number of words in the sentence, and \({DD}i\) is the dependency distance of the \({i}^{{th}}\) dependency relationship of a sentence. Building on this, the annotation of sentence ‘Mary-ga-John-ni-keshigomu-o-watashita was [Mary- top -John- dat -eraser- acc -give- past] ’. The sentence’s MDD would be 2. Table 3 provides the CSV file as a prompt for GPT 4.

Cohesion (semantic similarity) and content elaboration aim to capture the ideas presented in test taker’s essays. Cohesion was assessed using three measures: Synonym overlap/paragraph (topic), Synonym overlap/paragraph (keywords), and word2vec cosine similarity. Content elaboration and development were measured as the number of metadiscourse markers (type)/number of words. To capture content closely, this study proposed a novel-distance based representation, by encoding the cosine distance between the essay (by learner) and essay task’s (topic and keyword) i -vectors. The learner’s essay is decoded into a word sequence, and aligned to the essay task’ topic and keyword for log-likelihood measurement. The cosine distance reveals the content elaboration score in the leaners’ essay. The mathematical equation of cosine similarity between target-reference vectors is shown in (11), assuming there are i essays and ( L i , …. L n ) and ( N i , …. N n ) are the vectors representing the learner and task’s topic and keyword respectively. The content elaboration distance between L i and N i was calculated as follows:

\(\cos \left(\theta \right)=\frac{{\rm{L}}\,\cdot\, {\rm{N}}}{\left|{\rm{L}}\right|{\rm{|N|}}}=\frac{\mathop{\sum }\nolimits_{i=1}^{n}{L}_{i}{N}_{i}}{\sqrt{\mathop{\sum }\nolimits_{i=1}^{n}{L}_{i}^{2}}\sqrt{\mathop{\sum }\nolimits_{i=1}^{n}{N}_{i}^{2}}}\)

A high similarity value indicates a low difference between the two recognition outcomes, which in turn suggests a high level of proficiency in content elaboration.

To evaluate the effectiveness of the proposed measures in distinguishing different proficiency levels among nonnative Japanese speakers’ writing, we conducted a multi-faceted Rasch measurement analysis (Linacre, 1994 ). This approach applies measurement models to thoroughly analyze various factors that can influence test outcomes, including test takers’ proficiency, item difficulty, and rater severity, among others. The underlying principles and functionality of multi-faceted Rasch measurement are illustrated in (12).

\(\log \left(\frac{{P}_{{nijk}}}{{P}_{{nij}(k-1)}}\right)={B}_{n}-{D}_{i}-{C}_{j}-{F}_{k}\)

(12) defines the logarithmic transformation of the probability ratio ( P nijk /P nij(k-1) )) as a function of multiple parameters. Here, n represents the test taker, i denotes a writing proficiency measure, j corresponds to the human rater, and k represents the proficiency score. The parameter B n signifies the proficiency level of test taker n (where n ranges from 1 to N). D j represents the difficulty parameter of test item i (where i ranges from 1 to L), while C j represents the severity of rater j (where j ranges from 1 to J). Additionally, F k represents the step difficulty for a test taker to move from score ‘k-1’ to k . P nijk refers to the probability of rater j assigning score k to test taker n for test item i . P nij(k-1) represents the likelihood of test taker n being assigned score ‘k-1’ by rater j for test item i . Each facet within the test is treated as an independent parameter and estimated within the same reference framework. To evaluate the consistency of scores obtained through both human and computer analysis, we utilized the Infit mean-square statistic. This statistic is a chi-square measure divided by the degrees of freedom and is weighted with information. It demonstrates higher sensitivity to unexpected patterns in responses to items near a person’s proficiency level (Linacre, 2002 ). Fit statistics are assessed based on predefined thresholds for acceptable fit. For the Infit MNSQ, which has a mean of 1.00, different thresholds have been suggested. Some propose stricter thresholds ranging from 0.7 to 1.3 (Bond et al. 2021 ), while others suggest more lenient thresholds ranging from 0.5 to 1.5 (Eckes, 2009 ). In this study, we adopted the criterion of 0.70–1.30 for the Infit MNSQ.

Moving forward, we can now proceed to assess the effectiveness of the 16 proposed measures based on five criteria for accurately distinguishing various levels of writing proficiency among non-native Japanese speakers. To conduct this evaluation, we utilized the development dataset from the I-JAS corpus, as described in Section Dataset . Table 4 provides a measurement report that presents the performance details of the 14 metrics under consideration. The measure separation was found to be 4.02, indicating a clear differentiation among the measures. The reliability index for the measure separation was 0.891, suggesting consistency in the measurement. Similarly, the person separation reliability index was 0.802, indicating the accuracy of the assessment in distinguishing between individuals. All 16 measures demonstrated Infit mean squares within a reasonable range, ranging from 0.76 to 1.28. The Synonym overlap/paragraph (topic) measure exhibited a relatively high outfit mean square of 1.46, although the Infit mean square falls within an acceptable range. The standard error for the measures ranged from 0.13 to 0.28, indicating the precision of the estimates.

Table 5 further illustrated the weights assigned to different linguistic measures for score prediction, with higher weights indicating stronger correlations between those measures and higher scores. Specifically, the following measures exhibited higher weights compared to others: moving average type token ratio per essay has a weight of 0.0391. Mean dependency distance had a weight of 0.0388. Mean length of clause, calculated by dividing the number of words by the number of clauses, had a weight of 0.0374. Complex nominals per T-unit, calculated by dividing the number of complex nominals by the number of T-units, had a weight of 0.0379. Coordinate phrases rate, calculated by dividing the number of coordinate phrases by the number of clauses, had a weight of 0.0325. Grammatical error rate, representing the number of errors per essay, had a weight of 0.0322.

Criteria (output indicator)

The criteria used to evaluate the writing ability in this study were based on CEFR, which follows a six-point scale ranging from A1 to C2. To assess the quality of Japanese writing, the scoring criteria from Table 6 were utilized. These criteria were derived from the IELTS writing standards and served as assessment guidelines and prompts for the written output.

A prompt is a question or detailed instruction that is provided to the model to obtain a proper response. After several pilot experiments, we decided to provide the measures (Section Measures of writing proficiency for nonnative Japanese ) as the input prompt and use the criteria (Section Criteria (output indicator) ) as the output indicator. Regarding the prompt language, considering that the LLM was tasked with rating Japanese essays, would prompt in Japanese works better Footnote 5 ? We conducted experiments comparing the performance of GPT-4 using both English and Japanese prompts. Additionally, we utilized the Japanese local model OCLL with Japanese prompts. Multiple trials were conducted using the same sample. Regardless of the prompt language used, we consistently obtained the same grading results with GPT-4, which assigned a grade of B1 to the writing sample. This suggested that GPT-4 is reliable and capable of producing consistent ratings regardless of the prompt language. On the other hand, when we used Japanese prompts with the Japanese local model “OCLL”, we encountered inconsistent grading results. Out of 10 attempts with OCLL, only 6 yielded consistent grading results (B1), while the remaining 4 showed different outcomes, including A1 and B2 grades. These findings indicated that the language of the prompt was not the determining factor for reliable AES. Instead, the size of the training data and the model parameters played crucial roles in achieving consistent and reliable AES results for the language model.

The following is the utilized prompt, which details all measures and requires the LLM to score the essays using holistic and trait scores.

Please evaluate Japanese essays written by Japanese learners and assign a score to each essay on a six-point scale, ranging from A1, A2, B1, B2, C1 to C2. Additionally, please provide trait scores and display the calculation process for each trait score. The scoring should be based on the following criteria:

Moving average type-token ratio.

Number of lexical words (token) divided by the total number of words per essay.

Number of sophisticated word types divided by the total number of words per essay.

Mean length of clause.

Verb phrases per T-unit.

Clauses per T-unit.

Dependent clauses per T-unit.

Complex nominals per clause.

Adverbial clauses per clause.

Coordinate phrases per clause.

Mean dependency distance.

Synonym overlap paragraph (topic and keywords).

Word2vec cosine similarity.

Connectives per essay.

Conjunctions per essay.

Number of metadiscourse markers (types) divided by the total number of words.

Number of errors per essay.

Japanese essay text

出かける前に二人が地図を見ている間に、サンドイッチを入れたバスケットに犬が入ってしまいました。それに気づかずに二人は楽しそうに出かけて行きました。やがて突然犬がバスケットから飛び出し、二人は驚きました。バスケット の 中を見ると、食べ物はすべて犬に食べられていて、二人は困ってしまいました。(ID_JJJ01_SW1)

The score of the example above was B1. Figure 3 provides an example of holistic and trait scores provided by GPT-4 (with a prompt indicating all measures) via Bing Footnote 6 .

Example of GPT-4 AES and feedback (with a prompt indicating all measures).

Statistical analysis

The aim of this study is to investigate the potential use of LLM for nonnative Japanese AES. It seeks to compare the scoring outcomes obtained from feature-based AES tools, which rely on conventional machine learning technology (i.e. Jess, JWriter), with those generated by AI-driven AES tools utilizing deep learning technology (BERT, GPT, OCLL). To assess the reliability of a computer-assisted annotation tool, the study initially established human-human agreement as the benchmark measure. Subsequently, the performance of the LLM-based method was evaluated by comparing it to human-human agreement.

To assess annotation agreement, the study employed standard measures such as precision, recall, and F-score (Brants 2000 ; Lu 2010 ), along with the quadratically weighted kappa (QWK) to evaluate the consistency and agreement in the annotation process. Assume A and B represent human annotators. When comparing the annotations of the two annotators, the following results are obtained. The evaluation of precision, recall, and F-score metrics was illustrated in equations (13) to (15).

\({\rm{Recall}}(A,B)=\frac{{\rm{Number}}\,{\rm{of}}\,{\rm{identical}}\,{\rm{nodes}}\,{\rm{in}}\,A\,{\rm{and}}\,B}{{\rm{Number}}\,{\rm{of}}\,{\rm{nodes}}\,{\rm{in}}\,A}\)

\({\rm{Precision}}(A,\,B)=\frac{{\rm{Number}}\,{\rm{of}}\,{\rm{identical}}\,{\rm{nodes}}\,{\rm{in}}\,A\,{\rm{and}}\,B}{{\rm{Number}}\,{\rm{of}}\,{\rm{nodes}}\,{\rm{in}}\,B}\)

The F-score is the harmonic mean of recall and precision:

\({\rm{F}}-{\rm{score}}=\frac{2* ({\rm{Precision}}* {\rm{Recall}})}{{\rm{Precision}}+{\rm{Recall}}}\)

The highest possible value of an F-score is 1.0, indicating perfect precision and recall, and the lowest possible value is 0, if either precision or recall are zero.

In accordance with Taghipour and Ng ( 2016 ), the calculation of QWK involves two steps:

Step 1: Construct a weight matrix W as follows:

\({W}_{{ij}}=\frac{{(i-j)}^{2}}{{(N-1)}^{2}}\)

i represents the annotation made by the tool, while j represents the annotation made by a human rater. N denotes the total number of possible annotations. Matrix O is subsequently computed, where O_( i, j ) represents the count of data annotated by the tool ( i ) and the human annotator ( j ). On the other hand, E refers to the expected count matrix, which undergoes normalization to ensure that the sum of elements in E matches the sum of elements in O.

Step 2: With matrices O and E, the QWK is obtained as follows:

K = 1- \(\frac{\sum i,j{W}_{i,j}\,{O}_{i,j}}{\sum i,j{W}_{i,j}\,{E}_{i,j}}\)

The value of the quadratic weighted kappa increases as the level of agreement improves. Further, to assess the accuracy of LLM scoring, the proportional reductive mean square error (PRMSE) was employed. The PRMSE approach takes into account the variability observed in human ratings to estimate the rater error, which is then subtracted from the variance of the human labels. This calculation provides an overall measure of agreement between the automated scores and true scores (Haberman et al. 2015 ; Loukina et al. 2020 ; Taghipour and Ng, 2016 ). The computation of PRMSE involves the following steps:

Step 1: Calculate the mean squared errors (MSEs) for the scoring outcomes of the computer-assisted tool (MSE tool) and the human scoring outcomes (MSE human).

Step 2: Determine the PRMSE by comparing the MSE of the computer-assisted tool (MSE tool) with the MSE from human raters (MSE human), using the following formula:

\({\rm{PRMSE}}=1-\frac{({\rm{MSE}}\,{\rm{tool}})\,}{({\rm{MSE}}\,{\rm{human}})\,}=1-\,\frac{{\sum }_{i}^{n}=1{({{\rm{y}}}_{i}-{\hat{{\rm{y}}}}_{{\rm{i}}})}^{2}}{{\sum }_{i}^{n}=1{({{\rm{y}}}_{i}-\hat{{\rm{y}}})}^{2}}\)

In the numerator, ŷi represents the scoring outcome predicted by a specific LLM-driven AES system for a given sample. The term y i − ŷ i represents the difference between this predicted outcome and the mean value of all LLM-driven AES systems’ scoring outcomes. It quantifies the deviation of the specific LLM-driven AES system’s prediction from the average prediction of all LLM-driven AES systems. In the denominator, y i − ŷ represents the difference between the scoring outcome provided by a specific human rater for a given sample and the mean value of all human raters’ scoring outcomes. It measures the discrepancy between the specific human rater’s score and the average score given by all human raters. The PRMSE is then calculated by subtracting the ratio of the MSE tool to the MSE human from 1. PRMSE falls within the range of 0 to 1, with larger values indicating reduced errors in LLM’s scoring compared to those of human raters. In other words, a higher PRMSE implies that LLM’s scoring demonstrates greater accuracy in predicting the true scores (Loukina et al. 2020 ). The interpretation of kappa values, ranging from 0 to 1, is based on the work of Landis and Koch ( 1977 ). Specifically, the following categories are assigned to different ranges of kappa values: −1 indicates complete inconsistency, 0 indicates random agreement, 0.0 ~ 0.20 indicates extremely low level of agreement (slight), 0.21 ~ 0.40 indicates moderate level of agreement (fair), 0.41 ~ 0.60 indicates medium level of agreement (moderate), 0.61 ~ 0.80 indicates high level of agreement (substantial), 0.81 ~ 1 indicates almost perfect level of agreement. All statistical analyses were executed using Python script.

Results and discussion

Annotation reliability of the llm.

This section focuses on assessing the reliability of the LLM’s annotation and scoring capabilities. To evaluate the reliability, several tests were conducted simultaneously, aiming to achieve the following objectives:

Assess the LLM’s ability to differentiate between test takers with varying levels of oral proficiency.

Determine the level of agreement between the annotations and scoring performed by the LLM and those done by human raters.

The evaluation of the results encompassed several metrics, including: precision, recall, F-Score, quadratically-weighted kappa, proportional reduction of mean squared error, Pearson correlation, and multi-faceted Rasch measurement.

Inter-annotator agreement (human–human annotator agreement)

We started with an agreement test of the two human annotators. Two trained annotators were recruited to determine the writing task data measures. A total of 714 scripts, as the test data, was utilized. Each analysis lasted 300–360 min. Inter-annotator agreement was evaluated using the standard measures of precision, recall, and F-score and QWK. Table 7 presents the inter-annotator agreement for the various indicators. As shown, the inter-annotator agreement was fairly high, with F-scores ranging from 1.0 for sentence and word number to 0.666 for grammatical errors.

The findings from the QWK analysis provided further confirmation of the inter-annotator agreement. The QWK values covered a range from 0.950 ( p  = 0.000) for sentence and word number to 0.695 for synonym overlap number (keyword) and grammatical errors ( p  = 0.001).

Agreement of annotation outcomes between human and LLM

To evaluate the consistency between human annotators and LLM annotators (BERT, GPT, OCLL) across the indices, the same test was conducted. The results of the inter-annotator agreement (F-score) between LLM and human annotation are provided in Appendix B-D. The F-scores ranged from 0.706 for Grammatical error # for OCLL-human to a perfect 1.000 for GPT-human, for sentences, clauses, T-units, and words. These findings were further supported by the QWK analysis, which showed agreement levels ranging from 0.807 ( p  = 0.001) for metadiscourse markers for OCLL-human to 0.962 for words ( p  = 0.000) for GPT-human. The findings demonstrated that the LLM annotation achieved a significant level of accuracy in identifying measurement units and counts.

Reliability of LLM-driven AES’s scoring and discriminating proficiency levels

This section examines the reliability of the LLM-driven AES scoring through a comparison of the scoring outcomes produced by human raters and the LLM ( Reliability of LLM-driven AES scoring ). It also assesses the effectiveness of the LLM-based AES system in differentiating participants with varying proficiency levels ( Reliability of LLM-driven AES discriminating proficiency levels ).

Reliability of LLM-driven AES scoring

Table 8 summarizes the QWK coefficient analysis between the scores computed by the human raters and the GPT-4 for the individual essays from I-JAS Footnote 7 . As shown, the QWK of all measures ranged from k  = 0.819 for lexical density (number of lexical words (tokens)/number of words per essay) to k  = 0.644 for word2vec cosine similarity. Table 9 further presents the Pearson correlations between the 16 writing proficiency measures scored by human raters and GPT 4 for the individual essays. The correlations ranged from 0.672 for syntactic complexity to 0.734 for grammatical accuracy. The correlations between the writing proficiency scores assigned by human raters and the BERT-based AES system were found to range from 0.661 for syntactic complexity to 0.713 for grammatical accuracy. The correlations between the writing proficiency scores given by human raters and the OCLL-based AES system ranged from 0.654 for cohesion to 0.721 for grammatical accuracy. These findings indicated an alignment between the assessments made by human raters and both the BERT-based and OCLL-based AES systems in terms of various aspects of writing proficiency.

Reliability of LLM-driven AES discriminating proficiency levels

After validating the reliability of the LLM’s annotation and scoring, the subsequent objective was to evaluate its ability to distinguish between various proficiency levels. For this analysis, a dataset of 686 individual essays was utilized. Table 10 presents a sample of the results, summarizing the means, standard deviations, and the outcomes of the one-way ANOVAs based on the measures assessed by the GPT-4 model. A post hoc multiple comparison test, specifically the Bonferroni test, was conducted to identify any potential differences between pairs of levels.

As the results reveal, seven measures presented linear upward or downward progress across the three proficiency levels. These were marked in bold in Table 10 and comprise one measure of lexical richness, i.e. MATTR (lexical diversity); four measures of syntactic complexity, i.e. MDD (mean dependency distance), MLC (mean length of clause), CNT (complex nominals per T-unit), CPC (coordinate phrases rate); one cohesion measure, i.e. word2vec cosine similarity and GER (grammatical error rate). Regarding the ability of the sixteen measures to distinguish adjacent proficiency levels, the Bonferroni tests indicated that statistically significant differences exist between the primary level and the intermediate level for MLC and GER. One measure of lexical richness, namely LD, along with three measures of syntactic complexity (VPT, CT, DCT, ACC), two measures of cohesion (SOPT, SOPK), and one measure of content elaboration (IMM), exhibited statistically significant differences between proficiency levels. However, these differences did not demonstrate a linear progression between adjacent proficiency levels. No significant difference was observed in lexical sophistication between proficiency levels.

To summarize, our study aimed to evaluate the reliability and differentiation capabilities of the LLM-driven AES method. For the first objective, we assessed the LLM’s ability to differentiate between test takers with varying levels of oral proficiency using precision, recall, F-Score, and quadratically-weighted kappa. Regarding the second objective, we compared the scoring outcomes generated by human raters and the LLM to determine the level of agreement. We employed quadratically-weighted kappa and Pearson correlations to compare the 16 writing proficiency measures for the individual essays. The results confirmed the feasibility of using the LLM for annotation and scoring in AES for nonnative Japanese. As a result, Research Question 1 has been addressed.

Comparison of BERT-, GPT-, OCLL-based AES, and linguistic-feature-based computation methods

This section aims to compare the effectiveness of five AES methods for nonnative Japanese writing, i.e. LLM-driven approaches utilizing BERT, GPT, and OCLL, linguistic feature-based approaches using Jess and JWriter. The comparison was conducted by comparing the ratings obtained from each approach with human ratings. All ratings were derived from the dataset introduced in Dataset . To facilitate the comparison, the agreement between the automated methods and human ratings was assessed using QWK and PRMSE. The performance of each approach was summarized in Table 11 .

The QWK coefficient values indicate that LLMs (GPT, BERT, OCLL) and human rating outcomes demonstrated higher agreement compared to feature-based AES methods (Jess and JWriter) in assessing writing proficiency criteria, including lexical richness, syntactic complexity, content, and grammatical accuracy. Among the LLMs, the GPT-4 driven AES and human rating outcomes showed the highest agreement in all criteria, except for syntactic complexity. The PRMSE values suggest that the GPT-based method outperformed linguistic feature-based methods and other LLM-based approaches. Moreover, an interesting finding emerged during the study: the agreement coefficient between GPT-4 and human scoring was even higher than the agreement between different human raters themselves. This discovery highlights the advantage of GPT-based AES over human rating. Ratings involve a series of processes, including reading the learners’ writing, evaluating the content and language, and assigning scores. Within this chain of processes, various biases can be introduced, stemming from factors such as rater biases, test design, and rating scales. These biases can impact the consistency and objectivity of human ratings. GPT-based AES may benefit from its ability to apply consistent and objective evaluation criteria. By prompting the GPT model with detailed writing scoring rubrics and linguistic features, potential biases in human ratings can be mitigated. The model follows a predefined set of guidelines and does not possess the same subjective biases that human raters may exhibit. This standardization in the evaluation process contributes to the higher agreement observed between GPT-4 and human scoring. Section Prompt strategy of the study delves further into the role of prompts in the application of LLMs to AES. It explores how the choice and implementation of prompts can impact the performance and reliability of LLM-based AES methods. Furthermore, it is important to acknowledge the strengths of the local model, i.e. the Japanese local model OCLL, which excels in processing certain idiomatic expressions. Nevertheless, our analysis indicated that GPT-4 surpasses local models in AES. This superior performance can be attributed to the larger parameter size of GPT-4, estimated to be between 500 billion and 1 trillion, which exceeds the sizes of both BERT and the local model OCLL.

Prompt strategy

In the context of prompt strategy, Mizumoto and Eguchi ( 2023 ) conducted a study where they applied the GPT-3 model to automatically score English essays in the TOEFL test. They found that the accuracy of the GPT model alone was moderate to fair. However, when they incorporated linguistic measures such as cohesion, syntactic complexity, and lexical features alongside the GPT model, the accuracy significantly improved. This highlights the importance of prompt engineering and providing the model with specific instructions to enhance its performance. In this study, a similar approach was taken to optimize the performance of LLMs. GPT-4, which outperformed BERT and OCLL, was selected as the candidate model. Model 1 was used as the baseline, representing GPT-4 without any additional prompting. Model 2, on the other hand, involved GPT-4 prompted with 16 measures that included scoring criteria, efficient linguistic features for writing assessment, and detailed measurement units and calculation formulas. The remaining models (Models 3 to 18) utilized GPT-4 prompted with individual measures. The performance of these 18 different models was assessed using the output indicators described in Section Criteria (output indicator) . By comparing the performances of these models, the study aimed to understand the impact of prompt engineering on the accuracy and effectiveness of GPT-4 in AES tasks.

Based on the PRMSE scores presented in Fig. 4 , it was observed that Model 1, representing GPT-4 without any additional prompting, achieved a fair level of performance. However, Model 2, which utilized GPT-4 prompted with all measures, outperformed all other models in terms of PRMSE score, achieving a score of 0.681. These results indicate that the inclusion of specific measures and prompts significantly enhanced the performance of GPT-4 in AES. Among the measures, syntactic complexity was found to play a particularly significant role in improving the accuracy of GPT-4 in assessing writing quality. Following that, lexical diversity emerged as another important factor contributing to the model’s effectiveness. The study suggests that a well-prompted GPT-4 can serve as a valuable tool to support human assessors in evaluating writing quality. By utilizing GPT-4 as an automated scoring tool, the evaluation biases associated with human raters can be minimized. This has the potential to empower teachers by allowing them to focus on designing writing tasks and guiding writing strategies, while leveraging the capabilities of GPT-4 for efficient and reliable scoring.

PRMSE scores of the 18 AES models.

This study aimed to investigate two main research questions: the feasibility of utilizing LLMs for AES and the impact of prompt engineering on the application of LLMs in AES.

To address the first objective, the study compared the effectiveness of five different models: GPT, BERT, the Japanese local LLM (OCLL), and two conventional machine learning-based AES tools (Jess and JWriter). The PRMSE values indicated that the GPT-4-based method outperformed other LLMs (BERT, OCLL) and linguistic feature-based computational methods (Jess and JWriter) across various writing proficiency criteria. Furthermore, the agreement coefficient between GPT-4 and human scoring surpassed the agreement among human raters themselves, highlighting the potential of using the GPT-4 tool to enhance AES by reducing biases and subjectivity, saving time, labor, and cost, and providing valuable feedback for self-study. Regarding the second goal, the role of prompt design was investigated by comparing 18 models, including a baseline model, a model prompted with all measures, and 16 models prompted with one measure at a time. GPT-4, which outperformed BERT and OCLL, was selected as the candidate model. The PRMSE scores of the models showed that GPT-4 prompted with all measures achieved the best performance, surpassing the baseline and other models.

In conclusion, this study has demonstrated the potential of LLMs in supporting human rating in assessments. By incorporating automation, we can save time and resources while reducing biases and subjectivity inherent in human rating processes. Automated language assessments offer the advantage of accessibility, providing equal opportunities and economic feasibility for individuals who lack access to traditional assessment centers or necessary resources. LLM-based language assessments provide valuable feedback and support to learners, aiding in the enhancement of their language proficiency and the achievement of their goals. This personalized feedback can cater to individual learner needs, facilitating a more tailored and effective language-learning experience.

There are three important areas that merit further exploration. First, prompt engineering requires attention to ensure optimal performance of LLM-based AES across different language types. This study revealed that GPT-4, when prompted with all measures, outperformed models prompted with fewer measures. Therefore, investigating and refining prompt strategies can enhance the effectiveness of LLMs in automated language assessments. Second, it is crucial to explore the application of LLMs in second-language assessment and learning for oral proficiency, as well as their potential in under-resourced languages. Recent advancements in self-supervised machine learning techniques have significantly improved automatic speech recognition (ASR) systems, opening up new possibilities for creating reliable ASR systems, particularly for under-resourced languages with limited data. However, challenges persist in the field of ASR. First, ASR assumes correct word pronunciation for automatic pronunciation evaluation, which proves challenging for learners in the early stages of language acquisition due to diverse accents influenced by their native languages. Accurately segmenting short words becomes problematic in such cases. Second, developing precise audio-text transcriptions for languages with non-native accented speech poses a formidable task. Last, assessing oral proficiency levels involves capturing various linguistic features, including fluency, pronunciation, accuracy, and complexity, which are not easily captured by current NLP technology.

Data availability

The dataset utilized was obtained from the International Corpus of Japanese as a Second Language (I-JAS). The data URLs: [ https://www2.ninjal.ac.jp/jll/lsaj/ihome2.html ].

J-CAT and TTBJ are two computerized adaptive tests used to assess Japanese language proficiency.

SPOT is a specific component of the TTBJ test.

J-CAT: https://www.j-cat2.org/html/ja/pages/interpret.html

SPOT: https://ttbj.cegloc.tsukuba.ac.jp/p1.html#SPOT .

The study utilized a prompt-based GPT-4 model, developed by OpenAI, which has an impressive architecture with 1.8 trillion parameters across 120 layers. GPT-4 was trained on a vast dataset of 13 trillion tokens, using two stages: initial training on internet text datasets to predict the next token, and subsequent fine-tuning through reinforcement learning from human feedback.

https://www2.ninjal.ac.jp/jll/lsaj/ihome2-en.html .

http://jhlee.sakura.ne.jp/JEV/ by Japanese Learning Dictionary Support Group 2015.

We express our sincere gratitude to the reviewer for bringing this matter to our attention.

On February 7, 2023, Microsoft began rolling out a major overhaul to Bing that included a new chatbot feature based on OpenAI’s GPT-4 (Bing.com).

Appendix E-F present the analysis results of the QWK coefficient between the scores computed by the human raters and the BERT, OCLL models.

Attali Y, Burstein J (2006) Automated essay scoring with e-rater® V.2. J. Technol., Learn. Assess., 4

Barkaoui K, Hadidi A (2020) Assessing Change in English Second Language Writing Performance (1st ed.). Routledge, New York. https://doi.org/10.4324/9781003092346

Bentz C, Tatyana R, Koplenig A, Tanja S (2016) A comparison between morphological complexity. measures: Typological data vs. language corpora. In Proceedings of the workshop on computational linguistics for linguistic complexity (CL4LC), 142–153. Osaka, Japan: The COLING 2016 Organizing Committee

Bond TG, Yan Z, Heene M (2021) Applying the Rasch model: Fundamental measurement in the human sciences (4th ed). Routledge

Brants T (2000) Inter-annotator agreement for a German newspaper corpus. Proceedings of the Second International Conference on Language Resources and Evaluation (LREC’00), Athens, Greece, 31 May-2 June, European Language Resources Association

Brown TB, Mann B, Ryder N, et al. (2020) Language models are few-shot learners. Advances in Neural Information Processing Systems, Online, 6–12 December, Curran Associates, Inc., Red Hook, NY

Burstein J (2003) The E-rater scoring engine: Automated essay scoring with natural language processing. In Shermis MD and Burstein JC (ed) Automated Essay Scoring: A Cross-Disciplinary Perspective. Lawrence Erlbaum Associates, Mahwah, NJ

Čech R, Miroslav K (2018) Morphological richness of text. In Masako F, Václav C (ed) Taming the corpus: From inflection and lexis to interpretation, 63–77. Cham, Switzerland: Springer Nature

Çöltekin Ç, Taraka, R (2018) Exploiting Universal Dependencies treebanks for measuring morphosyntactic complexity. In Aleksandrs B, Christian B (ed), Proceedings of first workshop on measuring language complexity, 1–7. Torun, Poland

Crossley SA, Cobb T, McNamara DS (2013) Comparing count-based and band-based indices of word frequency: Implications for active vocabulary research and pedagogical applications. System 41:965–981. https://doi.org/10.1016/j.system.2013.08.002

Article   Google Scholar  

Crossley SA, McNamara DS (2016) Say more and be more coherent: How text elaboration and cohesion can increase writing quality. J. Writ. Res. 7:351–370

CyberAgent Inc (2023) Open-Calm series of Japanese language models. Retrieved from: https://www.cyberagent.co.jp/news/detail/id=28817

Devlin J, Chang MW, Lee K, Toutanova K (2019) BERT: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, Minneapolis, Minnesota, 2–7 June, pp. 4171–4186. Association for Computational Linguistics

Diez-Ortega M, Kyle K (2023) Measuring the development of lexical richness of L2 Spanish: a longitudinal learner corpus study. Studies in Second Language Acquisition 1-31

Eckes T (2009) On common ground? How raters perceive scoring criteria in oral proficiency testing. In Brown A, Hill K (ed) Language testing and evaluation 13: Tasks and criteria in performance assessment (pp. 43–73). Peter Lang Publishing

Elliot S (2003) IntelliMetric: from here to validity. In: Shermis MD, Burstein JC (ed) Automated Essay Scoring: A Cross-Disciplinary Perspective. Lawrence Erlbaum Associates, Mahwah, NJ

Google Scholar  

Engber CA (1995) The relationship of lexical proficiency to the quality of ESL compositions. J. Second Lang. Writ. 4:139–155

Garner J, Crossley SA, Kyle K (2019) N-gram measures and L2 writing proficiency. System 80:176–187. https://doi.org/10.1016/j.system.2018.12.001

Haberman SJ (2008) When can subscores have value? J. Educat. Behav. Stat., 33:204–229

Haberman SJ, Yao L, Sinharay S (2015) Prediction of true test scores from observed item scores and ancillary data. Brit. J. Math. Stat. Psychol. 68:363–385

Halliday MAK (1985) Spoken and Written Language. Deakin University Press, Melbourne, Australia

Hirao R, Arai M, Shimanaka H et al. (2020) Automated essay scoring system for nonnative Japanese learners. Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pp. 1250–1257. European Language Resources Association

Hunt KW (1966) Recent Measures in Syntactic Development. Elementary English, 43(7), 732–739. http://www.jstor.org/stable/41386067

Ishioka T (2001) About e-rater, a computer-based automatic scoring system for essays [Konpyūta ni yoru essei no jidō saiten shisutemu e − rater ni tsuite]. University Entrance Examination. Forum [Daigaku nyūshi fōramu] 24:71–76

Hochreiter S, Schmidhuber J (1997) Long short- term memory. Neural Comput. 9(8):1735–1780

Article   CAS   PubMed   Google Scholar  

Ishioka T, Kameda M (2006) Automated Japanese essay scoring system based on articles written by experts. Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, Sydney, Australia, 17–18 July 2006, pp. 233-240. Association for Computational Linguistics, USA

Japan Foundation (2021) Retrieved from: https://www.jpf.gp.jp/j/project/japanese/survey/result/dl/survey2021/all.pdf

Jarvis S (2013a) Defining and measuring lexical diversity. In Jarvis S, Daller M (ed) Vocabulary knowledge: Human ratings and automated measures (Vol. 47, pp. 13–44). John Benjamins. https://doi.org/10.1075/sibil.47.03ch1

Jarvis S (2013b) Capturing the diversity in lexical diversity. Lang. Learn. 63:87–106. https://doi.org/10.1111/j.1467-9922.2012.00739.x

Jiang J, Quyang J, Liu H (2019) Interlanguage: A perspective of quantitative linguistic typology. Lang. Sci. 74:85–97

Kim M, Crossley SA, Kyle K (2018) Lexical sophistication as a multidimensional phenomenon: Relations to second language lexical proficiency, development, and writing quality. Mod. Lang. J. 102(1):120–141. https://doi.org/10.1111/modl.12447

Kojima T, Gu S, Reid M et al. (2022) Large language models are zero-shot reasoners. Advances in Neural Information Processing Systems, New Orleans, LA, 29 November-1 December, Curran Associates, Inc., Red Hook, NY

Kyle K, Crossley SA (2015) Automatically assessing lexical sophistication: Indices, tools, findings, and application. TESOL Q 49:757–786

Kyle K, Crossley SA, Berger CM (2018) The tool for the automatic analysis of lexical sophistication (TAALES): Version 2.0. Behav. Res. Methods 50:1030–1046. https://doi.org/10.3758/s13428-017-0924-4

Article   PubMed   Google Scholar  

Kyle K, Crossley SA, Jarvis S (2021) Assessing the validity of lexical diversity using direct judgements. Lang. Assess. Q. 18:154–170. https://doi.org/10.1080/15434303.2020.1844205

Landauer TK, Laham D, Foltz PW (2003) Automated essay scoring and annotation of essays with the Intelligent Essay Assessor. In Shermis MD, Burstein JC (ed), Automated Essay Scoring: A Cross-Disciplinary Perspective. Lawrence Erlbaum Associates, Mahwah, NJ

Landis JR, Koch GG (1977) The measurement of observer agreement for categorical data. Biometrics 159–174

Laufer B, Nation P (1995) Vocabulary size and use: Lexical richness in L2 written production. Appl. Linguist. 16:307–322. https://doi.org/10.1093/applin/16.3.307

Lee J, Hasebe Y (2017) jWriter Learner Text Evaluator, URL: https://jreadability.net/jwriter/

Lee J, Kobayashi N, Sakai T, Sakota K (2015) A Comparison of SPOT and J-CAT Based on Test Analysis [Tesuto bunseki ni motozuku ‘SPOT’ to ‘J-CAT’ no hikaku]. Research on the Acquisition of Second Language Japanese [Dainigengo to shite no nihongo no shūtoku kenkyū] (18) 53–69

Li W, Yan J (2021) Probability distribution of dependency distance based on a Treebank of. Japanese EFL Learners’ Interlanguage. J. Quant. Linguist. 28(2):172–186. https://doi.org/10.1080/09296174.2020.1754611

Article   MathSciNet   Google Scholar  

Linacre JM (2002) Optimizing rating scale category effectiveness. J. Appl. Meas. 3(1):85–106

PubMed   Google Scholar  

Linacre JM (1994) Constructing measurement with a Many-Facet Rasch Model. In Wilson M (ed) Objective measurement: Theory into practice, Volume 2 (pp. 129–144). Norwood, NJ: Ablex

Liu H (2008) Dependency distance as a metric of language comprehension difficulty. J. Cognitive Sci. 9:159–191

Liu H, Xu C, Liang J (2017) Dependency distance: A new perspective on syntactic patterns in natural languages. Phys. Life Rev. 21. https://doi.org/10.1016/j.plrev.2017.03.002

Loukina A, Madnani N, Cahill A, et al. (2020) Using PRMSE to evaluate automated scoring systems in the presence of label noise. Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications, Seattle, WA, USA → Online, 10 July, pp. 18–29. Association for Computational Linguistics

Lu X (2010) Automatic analysis of syntactic complexity in second language writing. Int. J. Corpus Linguist. 15:474–496

Lu X (2012) The relationship of lexical richness to the quality of ESL learners’ oral narratives. Mod. Lang. J. 96:190–208

Lu X (2017) Automated measurement of syntactic complexity in corpus-based L2 writing research and implications for writing assessment. Lang. Test. 34:493–511

Lu X, Hu R (2022) Sense-aware lexical sophistication indices and their relationship to second language writing quality. Behav. Res. Method. 54:1444–1460. https://doi.org/10.3758/s13428-021-01675-6

Ministry of Health, Labor, and Welfare of Japan (2022) Retrieved from: https://www.mhlw.go.jp/stf/newpage_30367.html

Mizumoto A, Eguchi M (2023) Exploring the potential of using an AI language model for automated essay scoring. Res. Methods Appl. Linguist. 3:100050

Okgetheng B, Takeuchi K (2024) Estimating Japanese Essay Grading Scores with Large Language Models. Proceedings of 30th Annual Conference of the Language Processing Society in Japan, March 2024

Ortega L (2015) Second language learning explained? SLA across 10 contemporary theories. In VanPatten B, Williams J (ed) Theories in Second Language Acquisition: An Introduction

Rae JW, Borgeaud S, Cai T, et al. (2021) Scaling Language Models: Methods, Analysis & Insights from Training Gopher. ArXiv, abs/2112.11446

Read J (2000) Assessing vocabulary. Cambridge University Press. https://doi.org/10.1017/CBO9780511732942

Rudner LM, Liang T (2002) Automated Essay Scoring Using Bayes’ Theorem. J. Technol., Learning and Assessment, 1 (2)

Sakoda K, Hosoi Y (2020) Accuracy and complexity of Japanese Language usage by SLA learners in different learning environments based on the analysis of I-JAS, a learners’ corpus of Japanese as L2. Math. Linguist. 32(7):403–418. https://doi.org/10.24701/mathling.32.7_403

Suzuki N (1999) Summary of survey results regarding comprehensive essay questions. Final report of “Joint Research on Comprehensive Examinations for the Aim of Evaluating Applicability to Each Specialized Field of Universities” for 1996-2000 [shōronbun sōgō mondai ni kansuru chōsa kekka no gaiyō. Heisei 8 - Heisei 12-nendo daigaku no kaku senmon bun’ya e no tekisei no hyōka o mokuteki to suru sōgō shiken no arikata ni kansuru kyōdō kenkyū’ saishū hōkoku-sho]. University Entrance Examination Section Center Research and Development Department [Daigaku nyūshi sentā kenkyū kaihatsubu], 21–32

Taghipour K, Ng HT (2016) A neural approach to automated essay scoring. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Austin, Texas, 1–5 November, pp. 1882–1891. Association for Computational Linguistics

Takeuchi K, Ohno M, Motojin K, Taguchi M, Inada Y, Iizuka M, Abo T, Ueda H (2021) Development of essay scoring methods based on reference texts with construction of research-available Japanese essay data. In IPSJ J 62(9):1586–1604

Ure J (1971) Lexical density: A computational technique and some findings. In Coultard M (ed) Talking about Text. English Language Research, University of Birmingham, Birmingham, England

Vaswani A, Shazeer N, Parmar N, et al. (2017) Attention is all you need. In Advances in Neural Information Processing Systems, Long Beach, CA, 4–7 December, pp. 5998–6008, Curran Associates, Inc., Red Hook, NY

Watanabe H, Taira Y, Inoue Y (1988) Analysis of essay evaluation data [Shōronbun hyōka dēta no kaiseki]. Bulletin of the Faculty of Education, University of Tokyo [Tōkyōdaigaku kyōiku gakubu kiyō], Vol. 28, 143–164

Yao S, Yu D, Zhao J, et al. (2023) Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36

Zenker F, Kyle K (2021) Investigating minimum text lengths for lexical diversity indices. Assess. Writ. 47:100505. https://doi.org/10.1016/j.asw.2020.100505

Zhang Y, Warstadt A, Li X, et al. (2021) When do you need billions of words of pretraining data? Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, Online, pp. 1112-1125. Association for Computational Linguistics. https://doi.org/10.18653/v1/2021.acl-long.90

Download references

This research was funded by National Foundation of Social Sciences (22BYY186) to Wenchao Li.

Author information

Authors and affiliations.

Department of Japanese Studies, Zhejiang University, Hangzhou, China

Department of Linguistics and Applied Linguistics, Zhejiang University, Hangzhou, China

You can also search for this author in PubMed   Google Scholar

Contributions

Wenchao Li is in charge of conceptualization, validation, formal analysis, investigation, data curation, visualization and writing the draft. Haitao Liu is in charge of supervision.

Corresponding author

Correspondence to Wenchao Li .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Ethical approval

Ethical approval was not required as the study did not involve human participants.

Informed consent

This article does not contain any studies with human participants performed by any of the authors.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplemental material file #1, rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Li, W., Liu, H. Applying large language models for automated essay scoring for non-native Japanese. Humanit Soc Sci Commun 11 , 723 (2024). https://doi.org/10.1057/s41599-024-03209-9

Download citation

Received : 02 February 2024

Accepted : 16 May 2024

Published : 03 June 2024

DOI : https://doi.org/10.1057/s41599-024-03209-9

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

‘Shogun,’ ‘Ripley’ Bring Period Styles From Feudal Japan and Italy to Emmys Race

By Jazz Tangcay

Jazz Tangcay

Artisans Editor

• From ‘The Devil Wears Prada’ to ‘Kramer vs. Kramer’, See How Meryl Streep’s Career Inspired Her ‘Only Murders in the Building’ Character  55 mins ago
• Indie Film ‘Paper Flowers’ Based on Viral HuffPost Article Unveils Trailer and Sets Festival Release Dates – Film News in Brief 1 day ago
• ‘I’m A Virgo’s’ Jharrel Jerome on Playing a Giant, Working with Boots Riley and His Music Career 2 days ago

There is a wealth of period costumes cross the categories for Emmy voters to think about this season.

In limited series, Oscar-winner Colleen Atwood built 200 to 300 leather jackets for AppleTV+’s World War II drama “Masters of the Air.” “It took six months to get them built,” Atwood says. Procuring the right elements was a global process that included getting era-specific zippers from Japan and sheepskin from Scotland and England.

Related Stories

With queen catalog deal, sony would rule cooled music publishing market, dolly parton announces plans for broadway musical based on her life, 'hello, i'm dolly', popular on variety.

Rosario found himself limited by what primary sources were available, but he found paintings insightful to his research process. One of the costumes for Lady Ochiba (Fumi Nikaido) was based on a painting of the “Lady Ochiba of that period,” he says. Rosario notes that the first time we see the character, she’s wearing a beautiful uchikake that was made from scratch. “All those different layers — she’s wearing six or seven different layers — are actually all based on that painting. We studied the patterns of that painting, the layers of that painting, the meaning of those patterns, and then we reproduced that costume.”

“The New Look” costume designer Karen Serreau worked closely with fashion house Dior for the Apple TV+ series about the lives of designers Christian Dior and Coco Chanel during and after World War II.

From sharing swatches and sketches to fabrics, Dior’s collaboration helped Serreau’s work come together, especially when she needed to build certain outfits. The leopard-print fabric that she used to construct a dress came directly from Dior.

When he travels to Rome, Ripley’s clothes are more tailored. The designers’ biggest challenge was presenting a wardrobe that would reflect the character and his social status in black-and-white. Their solution? Enhancing fabric texture, patterns and even silhouettes in greater detail.

Brie Larson’s Elizabeth Zott does not conform to society’s standards in Apple TV+’s “Lessons in Chemistry,” set in the 1950s and ’60s. Costume designer Mirren Gordon-Crozier portrayed Elizabeth’s growth and journey from lab tech to eventual cooking show host. At the lab, Gordon-Crozier dressed her in teals, salmon pink and neutral colors. Elizabeth’s wardrobe staple was her lab coat, which had pockets. Gordon-Crozier used a different style lab coat for every episode. Even as a TV show host, Elizabeth wore a lab coat because she considered aprons silly, and the lab coat covered more surface area.

Ultimately, Gordon-Crozier wanted to show a woman of that time who wasn’t oppressed by her fashion and didn’t dress for society, unlike other women who surrounded her.

Annie Symons leaned into the opulence and flamboyance of the Jacobean era (1603-1625) for Starz’s “Mary & George,” a period rarely seen on TV. Symons turned to portraits for inspiration. When audiences are first introduced to Mary, she’s not well-off. Her palette is depleted. But as the show continues, she trades sex and clothes for power. Bows, feathered hats and collars were the norm for Mary. When she marries, she marries into wealth and obtains the power she seeks — and lace, pearls and elaborate sleeves become the norm. Similarly, pearls on Nicholas Galitzine were the norm for George since that was common for that era.

Other contenders in the costume design category include “Fargo,” “True Detective: North Country,” “Expats,” “Feud: Capote vs. the Swans” “All the Light We Cannot See” and “We Were the Lucky Ones.”

More from Variety

Genre-bending nigerian thriller ‘the weekend’ drops first-look clip ahead of tribeca premiere (exclusive), ai could contribute to virtual production sector growth, siriusxm to launch ‘mediaite’s press club’ in bid to woo media-news crowd, ashton kutcher says soon ‘you’ll be able to render a whole movie’ using ai: ‘the bar is going to have to go way up’ in hollywood, playstation state of play underscores muted release year ahead of summer game fest, ‘mean girls’ broadway star erika henningsen joins tina fey netflix series ‘four seasons’ (exclusive), more from our brands, ‘all things metal,’ doc about forging metal work with heavy metal, to premiere at 2024 tribeca film festival, can italy’s lake garda finally compete with como—or will it become a victim of its own success, premier league approves trial of new salary rules, keeping var, the best loofahs and body scrubbers, according to dermatologists, vanna white pays tearful tribute to pat sajak ahead of his wheel of fortune farewell — watch video, verify it's you, please log in.

An official website of the United States government

Here’s how you know

Official websites use .gov A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS A lock ( Lock A locked padlock ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

Japan: Japan Opens Public Comment Period for Revised Wood Product Standards

Link to report:, database source:.

Global Agricultural Information Network (GAIN)

Related reports, canada: ontario fast tracks allowing sales of wine and beer in retail stores, australia: exporter guide annual, india: grain and feed update, get reports, news, and more delivered straight to your inbox.

• Election 2024
• Entertainment
• Newsletters
• Photography
• Personal Finance
• AP Investigations
• AP Buyline Personal Finance
• AP Buyline Shopping
• Press Releases
• Israel-Hamas War
• Russia-Ukraine War
• Global elections
• Asia Pacific
• Latin America
• Middle East
• Election Results
• Delegate Tracker
• AP & Elections
• Auto Racing
• 2024 Paris Olympic Games
• Movie reviews
• Book reviews
• Personal finance
• Financial Markets
• Business Highlights
• Financial wellness
• Artificial Intelligence
• Social Media

Yuka Saso makes headlines in Japan as special edition papers mark her US Open win

A staff distributes an extra edition of the Yomiuri Shimbun newspaper reporting on Japanese golfer Yuka Saso winning the U.S. Women’s Open golf tournament Monday, June 3, 2024, in Tokyo. The Japanese title reads as “Saso won second major.” (AP Photo/Eugene Hoshiko)

Yuka Saso, of Japan, holds the tournament trophy after winning the U.S. Women’s Open golf tournament at Lancaster Country Club, Sunday, June 2, 2024, in Lancaster, Pa. (AP Photo/Matt Slocum)

• Copy Link copied

TOKYO (AP) — Commuters at Tokyo’s Shimbashi Station were greeted with a special edition of the Yomiuri newspaper on Monday.

The big news was the victory of Yuka Saso at the U.S. Open on Sunday, the second time she has won title. Japanese papers still print special editions to mark such moments.

She won in 2021 playing under the flag of the Philippines, the land of her birth. This time she won flying the flag of Japan, the birthplace of her father.

“Winning in 2021, I represented the Philippines. I feel like I was able to give back to my mom,” Saso said. “This year I was able to represent Japan, and I think I was able to give back to my dad. I’m very happy that I was able to do it.

“It’s just a wonderful feeling that I was able to give back to my parents in the same way.”

The 22-year-old Saso shot a 2-under 68 on Sunday — early Monday Japanese time — to win by three shots at the Lancaster Country Club in Lancaster, Pennsylvania.

AP golf: https://apnews.com/hub/golf