Category Archives: Public Humanities

Detecting and Integrating ChatGPT in the College Classroom

Sebastian Williams, PhD

Instructors can use several methods to detect whether a text has been authored by a chat bot such as ChatGPT, including both heuristic approaches and GPTZero. Additionally, instructors can limit the use of ChatGPT and other chat bots by modifying assignment prompts, asking for pre-writing (e.g., outlines) and early drafts, or by requiring more handwritten work.

Ultimately, however, faculty should try to integrate these technologies into the learning process. ChatGPT does not fully eliminate the need for problem solving or navigating complex issues, and, in many cases, it frees up writers to worry less about grammar or style than higher order issues such as developing original claims or expressing an individual position.

Heuristic Methods for Detection

ChatGPT can be detected by many writing instructors based on organization, word choice, and the originality of ideas.

For example, when asked to write a close reading of William Wordsworth’s “The World is Too Much with Us,” ChatGPT authored a relatively formulaic response that moved from the start of the poem to the end (i.e., chronologically) and was unable to produce a coherent argument when asked to organize topically instead. (See Appendix for the sample text created by Chat GPT.)

This reflects another tendency in the current iteration of ChatGPT to organize short essays using a tripartite structure. For example, ChatGPT now tends to write “First, X occurred [. . .] Then, Y happened [. . .] Finally, Z was present.” Though machine-learning programs are able to adapt over time, some writing experts have noted a simple, formulaic organization method (“X, Y, Z”). This may only be true if a user does not modify an original response (e.g., by asking ChatGPT to write more).[1]

Other basic issues indicated that the text was automated: the AI-generated text misquotes the final line of Wordsworth’s poem as “the sea that bares her bosom to the moon; / The winds that will be howling at all hours.” The bot includes two lines (not a single line) from the third quatrain and replaces “This Sea” (with a capital “S”) with “the sea.” Plus, the actual final line of the sonnet is significantly different; these surface-level problems caught my attention even before submitting it to GPTZero.

The chat bot also tends to elide in-text citations (though, again, this may change as the technology adapts). And, perhaps most notably for contemporary writing instructors, the essay reiterates only basic ideas or the most accepted interpretations similar to those on Sparknotes, Cliff Notes, and Wikipedia. I.e., it offers the most accepted reading and the main points easily found elsewhere, and, in terms of argumentation, the close reading is not an original or unique interpretation. The sample essay probably would not score an “A” based on expectations for college-level students to go beyond the commonplace reading, balance multiple interpretations at once, or to “read against the grain.”

Formulaic organization, simple interpretations, and basic errors when identifying the lines of a poem are not definitive proof that cheating has occurred, however. A student (or parent) might easily point out that the issues above are reasonable for a student writer.

GPTZero

GPTZero detects AI-generated text based on “perplexity” and “burstiness.” Perplexity refers to a probabilistic detection of unique words within natural language, including randomness and complexity. It also measures “burstiness,” or what rhetoricians commonly refer to as sentence variation (including random variations between complex, compound, simple, and compound complex sentences). GPTZero detects patterns in word choice and sentence length based on massive databases of stored information. In sum, human writers are often more random than a chat bot, though, obviously ChatGPT is not a static technology – it “learns” over time based on inputs.

GPTZero is currently free to use and generates a descriptive report that not only evaluates a sample text but also explains terminology.

In the sample text, GPTZero detected that the close reading of Wordsworth’s poem had a comparatively low rate of perplexity in several instances (see Figure 1) – an indication that certain parts of the text were not written by a human.

**Fig 1** The perplexity evaluates the unique word choices that writers make. AI-generated text is less perplex, but, as the program notes, it’s also important to note perplexity across sentences (i.e., relative to the individual text more so than general word use).

Repeated use of GPTZero with the same text found the same results:

Perplexity: 16
Perplexity (across sentences): 46.1
The line with the highest perplexity: 151
GPT Zero wrongly concluded that the text was human-generated (see Figure 2)

**Fig 2** The green text indicates that false-negative conclusion of GPTZero in this first attempt.

The last two bullet points are related. The most unique line in the essay came from a direct quote of the poem (it is actually two sentences, but it is missing punctuation). In other words, the AI-generated text included several quotes from a Romantic poet whose work is highly distinct (see Figure 3). When summarizing the results, this led to a false-negative report. The tool was fooled into believing all of the text was written by a human because several parts were in fact written by Wordsworth.

**Fig 3** The most perplex sentences were written by Wordsworth, which, given their frequency in the essay, ultimately led the GPTZero bot to make a false conclusion.

Submitting a Text without Direct Quotes

I submitted a close reading of the poem without direct quotations. This involved deleting quotes while still maintaining the integrity of the syntax in the original essay. The results were promising:

Perplexity: 10
Perplexity (across sentences): 32.8.
The line with the highest perplexity: 58.
GPT Zero correctly concluded that the text was AI-generated.

**Fig 4** When eliminating direct quotations, GPTZero came to the correct conclusion.

This indicates that GPTZero is an effective tool for recognizing when text is AI-generated. However, when using the tool, instructors need to be aware that false-negatives are highly possible in a sample text that quotes other material.

Finally, GPTZero users should acknowledge two important factors. First, GPTZero was “trained” on a previous iteration of ChatGPT and will likely become less effective over time. Second, GPTZero uses a limited, probabilistic method and cannot confirm its results with 100-percent accuracy. Students accused of using ChatGPT could easily point out that GPTZero is subject to errors in several cases.

Other Ways of Addressing AI-Generated Text

Instructors should also recognize that they may need to modify their assignment prompts, ask for pre‑writing or drafts, or require more handwritten work to diminish use of ChatGPT.

For example, ChatGPT is less effective when writing about events, texts, or issues within the last few years (post-2020).[2] So, asking a student to analyze a historical document and then to connect it to current events may impact their ability to use a chat bot in place of original work.

Alternatively, asking students to write assignments by hand or to turn in pre-writing and drafts will likely offset the use of ChatGPT. But students can obviously generate an automated text and work backward from it, either copying it by hand or creating an outline after the fact (which is something many students admit to doing in composition classes anyway).

In short, these are not fool-proof methods.

Conclusion: ChatGPT in the Classroom

Perhaps the best method to address automated text in classrooms is to change the mindset that instructors have when approaching the issue. Rather than viewing ChatGPT as yet another tool for plagiarism, instructors can adopt methods for integrating it into classwork.

For example, in a previous writing course, I asked students to generate poetry using Google Verse, which is a machine-learning chat bot similar to ChatGPT that is designed to write poetry specifically. One assignment required students to generate a poem using AI and then to analyze what is omitted in the process. The assignment recognizes that these technologies are highly effective, yet it shows students the value of individual thought, critical analysis, and creativity.

In a literary studies class, a similar assignment might ask students to read a ChatGPT-generated close reading of Wordsworth’s poetry. Then, students could write about what the chat bot has omitted (I have used a similar assignment with SparkNotes and Wikipedia entries). This acknowledges the value of balancing multiple, often contradictory, interpretations of a literary text while still finding meaning in imaginative writing. Most importantly, it does not necessarily downplay the value of chat bots in organizing information – especially regarding dominant interpretations or common knowledge.

Ultimately, such technology will likely allow students and instructors to create more complex assignments and courses overall. Rather than viewing ChatGPT as a floodgate for new issues, instructors should use technologies to their advantage to eliminate some of the more tedious aspects of assignments and course design.

As a concluding remark, it seems worth mentioning that the text I chose to sample, Wordsworth’s “The World is Too Much with Us,” is itself a meditation on how technology changes the human imagination. But a twenty-first-century interpretation of the poem might recognize that what Wordsworth laments is not unique to the Industrial Revolution, and that the persona is ultimately critiquing how humankind uses and responds to technology – not necessarily the technology itself.

Appendix-ChatGPT-Samples Download

[1] Ian Bogost, “ChatGPT is Dumber than You Think.” The Atlantic, 7 Dec. 2022, https://www.theatlantic.com/technology/archive/2022/12/chatgpt-openai-artificial-intelligence-writing-ethics/672386/. Bogost is a world-renowned digital humanist and rhetorical theorist.

[2] Kevin Roose, “The Brilliance and Weirdness of ChatGPT,” New York Times, 5 Dec. 2022, https://www.nytimes.com/2022/12/05/technology/chatgpt-ai-twitter.html. ChatGPT does not “crawl the web for current events,” and its knowledge is somewhat restricted.

What is the Value of the Humanities?

The humanities refers to that group of fields—literature, history, philosophy, and so forth—that investigate the nature of meaning, or how we make sense of the world around us. Unlike some scientific fields, the humanities are less interested in the mechanics of the physical world and are more focused on questions of quality, value, language, comparison, law, and religion. While an engineer might develop a ramp to make a building more accessible for wheelchair users, a humanist would be more likely to investigate the assumptions of the architect who designed an inaccessible building in the first place.

Though it’s common in the public and among academics to say, “Naturally, there’s an inherent value to the humanities,” few people can offer indisputable reasons for explaining why this is true. To be fair, many of the sciences run up against the same problem when we press the issue—it seems odd to say that number theory has more “real-world” application than professional writing. But in a social climate in which job training is what administrators push, it can be difficult to convince a hiring manager that something like a history or philosophy degree has value that isn’t simply intrinsic, beyond explanation. Philosophy requires a high degree of reading comprehension, analytic skills, and often mathematical knowledge, but you’ll rarely see a job post for a “philosopher” position on LinkedIn.

Helen Small’s 2013 book The Value of the Humanities offers some insights on this question of value. It’s a little different than other books with similar titles, primarily because Small’s approach to the question of value is not a polemic rant—it’s an analysis. There are some common myths or arguments she’s taking a look at to gain some clarity on the issue, many of which have to do with what we mean when we claim something is “valuable.”

Intrinsic Value?

One common argument goes like this: The humanities have intrinsic value; once you ask about use-value, you automatically frame the humanities in a neoliberal, capitalistic frame with which they are at odds.

This is kind of true if you’re a critic of neoliberalism (and anyone familiar with that word is probably already a critic), but it’s also a very limiting answer that doesn’t get at the question of value. A moderate approach to this idea might suggest that the humanities do in fact teach you skills, or that the arts have a functional use-value. The argument that something has an intrinsic value is problematic because it ignores the ways (1) value is embedded in social meaning and (2) the ways humanists can apply their knowledge in broad ways.

The common argument against focusing on use-value is intelligent in that it considers ideological frameworks, but it also tends to shut down conversation without actually addressing the point. Yes, concepts like “utility” are derived from eighteenth-century notions of capital, but the humanities also contribute significantly to the GDP and cultural development as well—whether you want them to or not.

Self-Culture and Individual Happiness

Common argument #2: The humanities contribute to social and individual happiness and well-being; it’s a way of understanding the human holistically.

This argument has its roots largely in nineteenth-century thinking, particularly John Stuart Mills’s theory of utilitarianism. But the problem with this idea, as Small notes, is that the humanities won’t necessarily make you happier; instead, they help you better understand what happiness means at any given moment.

Kathryn Hamilton Warren’s essay on “Self-Culture and the Private Value of the Humanities” (2018) touches on this issue, but from a slightly different, less utilitarian angle. Building on the transcendentalists such as Henry David Thoreau, Warren argues that the humanities should emphasize self-criticism and examination. Studying books won’t necessarily bring joy into your world, but it could give you insights about what you value in your life.

The position of “self-growth” is a bit controversial because it’s not always clear how you should make a living when you’re “growing”—Thoreau famously had his mother do his laundry while at Walden Pond, living in a cabin that was built on land borrowed from Emerson—but it goes back to the first question about how we frame value. As Small suggests, the emphasis on happiness and utility in the humanities is the product of social conditioning—something toward which we should retain a healthy dose of skepticism.

Better Citizens—or Elitism?

Common argument #3: Democracy needs the humanities. “Democracy is good; therefore the humanities are good”—that’s the way this argument goes. But, there’s a small bit that often gets little scrutiny, which is how the humanities (arguably) make us more democratic.

This idea is rooted in the liberal-arts tradition, and it dates to Socrates’ Apology: “I am the gadfly of the Athenian people, given to them by God.” The “gadfly” argument is that the philosopher is a supreme check on the politicians; the problem, though, is that Socrates thinks a little too highly of himself. The major critique of the notion that the humanities are central to democracy is that it tends to be elitist—who, after all, gets to be a humanist and shape our society?

Additionally, for those in post-secondary education, this argument is mostly for primary or secondary school. There’s an access issue here once we move to college demographics: only 12 to 15 percent of people study humanities in higher education, so the argument tends toward stewardship, a deeply paternalistic model. Socrates, after all, thinks he was “given by God” to steer the political life of the Athenians.

This argument also puts politics first and the humanities second—the goal of the humanities is, per this logic, to support the political realm. It’s an argument that skips past the humanities directly and emphasizes the political world without clarifying how the two are related.

Martha Nussbaum has a unique take on this argument, and many others suggest that the key to making the humanities good for democracy is universalizing them, making them accessible to everyone. This avoids the paternalistic “sent by God” model, but we also need to recognize that the humanities have always had a troubling history of racism, imperialism, sexism, ableism, and various other types of “isms.”

What’s the Answer?

We should avoid saying that the humanities are absolute needs, per Small, in the rudimentary sense of the word “needs,” even though some argue “creative expression” is a human right. Suggesting the humanities are an “absolute need” tends to have more symbolic than real significance, especially from a rhetorical perspective. In the same way that the argument of “art for art’s sake” tends to ignore social context, the idea that the humanities are an “absolute necessity” tends to change the meaning of “absolute need” for the assertion to stick.

That said, the value of the humanities can be found in some version of the common arguments above. They all essentially revolve around the idea that we need to change our perspective when it comes to value, and that change in perspective is partly what the humanities offers. We can claim that the humanities are necessary for democracy, as long as we critically investigate that idea. And, the same goes for the idea that the humanities can’t be measured by use-value, as long as we recognize that, in fact, the humanities do have use-value. Self-culture is important, too, if we don’t fool ourselves into thinking happiness is a consequence of humanities scholarship, or as long as we’re aware we need some degree of privilege to be able to use our time for contemplation.

Small, Helen. The Value of the Humanities. Oxford University Press, 2013.

Warren, Kathryn Hamilton. “Self-Culture and the Private Value of the Humanities.” College Literature, vol. 45, no. 4, 2018, pp. 587–595.