Supervised Classifier

This tab shows the output of a calibrated supervised classifier trained to distinguish human-written from AI-generated academic text. The score (P(AI)) is a probability between 0 and 1, where higher values indicate a greater likelihood of AI-generated writing.Research design. The classifier is trained on paired examples: original pre-ChatGPT thesis sections (confirmed human, label = 0) and their LLM-rewritten counterparts (label = 1). Rewrites are produced by multiple language models (to avoid learning one model's fingerprint) using two prompt variants (formal academic rewrite and concise rewrite). The feature vector has 9 dimensions: 7 stylometric signals (burstiness, TTR, sentence length variance, transition density, passive voice rate, first-person rate, hedging density) plus the Binoculars score and GPT-2 perplexity for that section. Both training and scoring operate at the section level (matching the granularity of the training data); the document-level P(AI) shown here is the mean across all scored sections. Cross-validation is grouped by thesis so no thesis appears in both train and test folds. The model is calibrated (logistic regression or Platt-scaled GBM), so P(AI) can be interpreted as an approximate posterior probability. Feature inputs are winsorized at the 1st/99th percentile (fitted on training data) to limit the influence of outliers.Interpretation. Because the training data uses pre-ChatGPT originals as the human baseline, the classifier captures the stylometric shift that occurs when academic text is rewritten by an LLM. A high P(AI) does not prove AI use; it means the thesis's stylometric profile is closer to LLM-rewritten text than to typical pre-ChatGPT student writing. Theses with unusual writing styles (e.g., highly formulaic or non-native English) may score high without any AI involvement.

Distribution

Overlaid density histograms for the pre-ChatGPT (blue) and post-ChatGPT (orange) periods. A rightward shift in the post-ChatGPT distribution would indicate that more recent theses resemble LLM-rewritten text.

Time Trends

Each dot is one thesis. The dashed vertical line marks the first academic year after ChatGPT's launch. The year-level summary smooths individual variation with standard error bars.

Binoculars vs. Supervised

The supervised classifier uses the section-level Binoculars score as one of 9 input features. This scatter shows how the document-level Binoculars score relates to the aggregated P(AI). Strong correlation is expected because both capture text predictability, but the supervised model adds stylometric and perplexity signals that Binoculars alone does not capture.

Section-Level Analysis

The classifier scores each thesis section independently using features extracted from that section's text. This matches the granularity of the training data (section-level pairs) and can reveal targeted AI use in specific parts of a thesis.

Highest P(AI) (Top 20)

Overview

Binoculars measures how predictable text is across two language models. Higher scores indicate more human-like writing; scores below the 0.85 threshold suggest possible AI generation.Perplexity captures how surprising the text is to GPT-2. Higher values indicate less predictable (more human-like) writing; lower values suggest formulaic or AI-generated text.

Stylometric Features

Burstiness measures how clustered vocabulary usage is across a document (human writers tend to cluster topic words in specific sections). TTR (type-token ratio) reflects vocabulary diversity. Sentence length variance captures structural variation; AI-generated text tends toward more uniform sentence lengths. Transition density counts formulaic connectives per 1,000 words ('Furthermore', 'Moreover', 'It is important to note'); AI text reliably overuses these. Passive voice rate is the share of sentences with a passive construction; AI academic writing is systematically more passive. First-person rate counts 'I', 'we', 'my', etc. per 1,000 words; AI text avoids first-person voice, so lower values are more AI-typical. Hedging density tracks formulaic hedging markers per 1,000 words ('it can be argued', 'it appears that'); AI text uses these more mechanically. Section score variance is the standard deviation of the six Binoculars section scores; a large spike in one section alongside normal scores elsewhere may indicate targeted AI use.

Summary Statistics

Descriptive statistics for all scoring variables. The raw table shows unmodified values; the winsorized table shows values after clipping at the 1st and 99th percentiles.

Raw (Unmodified)

Winsorized (1st/99th Percentile)

Winsorization Details

Winsorization clips values below the 1st or above the 99th percentile to reduce the influence of extreme outliers on visualizations. Across all winsorized variables, 272 observations were clipped (out of 1543 theses). Binoculars Score is intentionally excluded from winsorization because its low-end outliers are the primary signal of interest. New stylometric columns are only shown once a rescore and merge have been run.

Z-Score Normalization

Z-score columns (_z suffix): the merged dataset contains 17 standardized columns expressing each score as standard deviations from the pre-ChatGPT baseline mean. A z-score of −2 means the thesis is 2 SDs below the typical pre-ChatGPT level. See the Z-Score Analysis tab for visualizations and cross-metric comparisons.

Time Trends

Each dot is one thesis. The dashed vertical line marks the first academic year after ChatGPT's launch (November 2022). Dotted horizontal lines show the overall mean. A downward shift in perplexity or stylometric variation after the cutoff could suggest increased use of AI-generated text.

Stylometric Features

Burstiness and sentence length variance tend to be lower in AI-generated text (more uniform structure). TTR captures vocabulary diversity; context-dependent and not a standalone AI indicator.

Extended Stylometric Features

Transition density and hedging density are counts per 1,000 words; higher values are more AI-typical. Passive voice rate is the share of sentences with a passive construction; higher is more AI-typical. First-person rate counts first-person pronouns per 1,000 words; lower values are more AI-typical (AI avoids 'I' and 'we'). Section score variance measures how unevenly Binoculars scores are distributed across the six thesis sections.