BERT-based AI-Generated Academic Text Detector

A high-accuracy BERT model for detecting AI-generated academic text with 99.57% accuracy on paragraph-level samples.

Online Demo

🌐 Try the model online: https://followsci.com/ai-detection

Free web interface with real-time detection, no installation or API key required.

Model Details

Model Description

Model Type: BERT-base-uncased fine-tuned for binary text classification
Architecture: BERT-base-uncased (110M parameters)
Task: Binary classification (Human-written vs AI-generated text)
Input: Academic text paragraphs (up to 512 tokens)
Output: Binary label (0 = Human-written, 1 = AI-generated) with confidence scores

Training Information

Training Samples: 1,487,400 paragraph-level samples
Validation Samples: 185,930 paragraph-level samples
Test Samples: 185,930 paragraph-level samples
Total Dataset: 1,859,260 paragraphs
Training Data:
- Human-written: Academic papers from arXiv
- AI-generated: Text generated by various large language models (GPT, Claude, etc.)

Performance

Test Set Results

Metric	Value
Accuracy	99.57%
F1-Score	99.58%
Precision	99.23%
Recall	99.94%
False Positive Rate	0.82%
False Negative Rate	0.06%

Confusion Matrix (Test Set)

	Predicted: Human	Predicted: AI
Actual: Human	89,740 (TN)	740 (FP)
Actual: AI	60 (FN)	95,390 (TP)

Inference Speed: ~20,900 samples/second on RTX 3090 (batch size 64)

Usage

Quick Start

from transformers import BertTokenizer, BertForSequenceClassification
import torch

# Load model and tokenizer
model_name = "followsci/bert-ai-text-detector"
tokenizer = BertTokenizer.from_pretrained(model_name)
model = BertForSequenceClassification.from_pretrained(model_name)
model.eval()

# Detect AI text
text = "Your academic paragraph here..."
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=512)

with torch.no_grad():
    outputs = model(**inputs)
    probs = torch.nn.functional.softmax(outputs.logits, dim=-1)
    ai_prob = probs[0][1].item() * 100
    human_prob = probs[0][0].item() * 100
    
    print(f"AI-generated probability: {ai_prob:.1f}%")
    print(f"Human-written probability: {human_prob:.1f}%")
    
    if ai_prob > 50:
        print("Prediction: AI-generated")
    else:
        print("Prediction: Human-written")

Batch Processing

texts = [
    "First paragraph...",
    "Second paragraph...",
    # ... more texts
]

inputs = tokenizer(
    texts,
    return_tensors="pt",
    truncation=True,
    max_length=512,
    padding=True
)

with torch.no_grad():
    outputs = model(**inputs)
    probs = torch.nn.functional.softmax(outputs.logits, dim=-1)
    
    for i, prob in enumerate(probs):
        ai_prob = prob[1].item() * 100
        print(f"Text {i+1}: AI probability = {ai_prob:.1f}%")

Using with Transformers Pipeline

from transformers import pipeline

classifier = pipeline(
    "text-classification",
    model="followsci/bert-ai-text-detector",
    tokenizer="followsci/bert-ai-text-detector"
)

result = classifier("Your text here...")
print(result)

Training Details

Training Configuration

Base Model: bert-base-uncased
Batch Size: 64
Learning Rate: 5e-5 (with linear warmup)
Warmup Steps: 5,000
Max Sequence Length: 512
Optimizer: AdamW
Epochs: 3
Training Time: ~11 hours (on RTX 3090)

Dataset Distribution

Split	Total Samples	Human (Label 0)	AI (Label 1)
Train	1,487,400	723,780 (48.7%)	763,620 (51.3%)
Validation	185,930	90,470 (48.7%)	95,460 (51.3%)
Test	185,930	90,480 (48.7%)	95,450 (51.3%)

Limitations

Domain Specificity: The model is trained primarily on academic text. Performance may degrade on:
- Casual text or social media content
- Technical documentation
- Creative writing
Binary Classification: The model only distinguishes between "human" and "AI" text, without:
- Identifying which AI model generated the text
- Providing confidence intervals
- Detecting partially AI-assisted text
Paragraph-Level Detection: The model is optimized for paragraph-level samples:
- Performance on sentence-level or full-document level may vary
- Best results achieved with structured academic paragraphs
False Positives: Approximately 0.82% false positive rate means some human-written text may be flagged as AI-generated.

Ethical Considerations

Use Case: This model is intended as a tool for academic integrity and research purposes
Bias: The model may reflect biases present in the training data
Misuse: Should not be used as the sole criterion for academic misconduct decisions
Transparency: Results should be interpreted with context and domain expertise