MALTO — 2nd Place Solution

2nd place on the MALTO Recruitment Hackathon hosted by MALTO and Politecnico di Torino.

Task

Classify text as human-written or identify which AI model generated it across 6 classes:

The main challenge is severe class imbalance (19:1 ratio) with DeepSeek and Grok as the hardest minority classes.

Solution

The solution ensembles a fine-tuned transformer with a classical n-gram model, optimised via Nelder-Mead on out-of-fold predictions.

Pipeline

ModernBERT-base (5-fold CV) ─┬─ Temperature Scaling ─┬─ Nelder-Mead ─── Threshold ─── Submission
                              │                       │  Per-class blend   Nudge
Full-data ModernBERT (7 ep) ──┘                       │
                                                      │
TF-IDF + Calibrated SVC (5-fold CV) ──────────────────┘

Key Techniques

Per-Class OOF Performance

Score Progression

Prediction Analysis

Beyond training metrics, several features helped validate and understand the model's predictions on the test set.

Expected Class Distribution (Sanity Check)

Assuming the test set follows the same class ratios as training, the expected counts in 600 test samples are:

Distribution alignment is a strong signal that the model is well-calibrated. Large deviations (e.g. predicting 50 Grok and only 8 DeepSeek) indicate systematic classifier bias.

Classifier Agreement Analysis

Comparing transformer ensemble vs calibrated SVC across 600 test samples revealed 20 disagreements (96.7% agreement). All disagreements were DeepSeek ↔ Grok confusions — no Human ↔ AI errors were found.

The SVC systematically over-predicts Grok and under-predicts DeepSeek. TF-IDF n-gram models lack semantic depth to distinguish these two models on short, fact-dense texts. When the transformer and SVC disagree on a DeepSeek/Grok call, the transformer is correct.

Hard Class Characteristics

Features Used to Evaluate Disputed Samples

For the 20 transformer–SVC disagreements, each sample was evaluated along four axes:

1. Text length (word count) — very short texts (< 80 words) carry less signal; classification is less reliable
2. Topic / domain — certain topics are associated with specific AI writing styles
3. SVC calibrated confidence — predict_proba from CalibratedClassifierCV; scores below 0.70 indicate low certainty
4. Transformer softmax gap — margin between top-1 and top-2 logits; a narrow gap flags genuinely ambiguous samples

Repository Structure

MALTO/
├── notebooks/
│   └── solution.ipynb              # Full pipeline notebook
├── src/
│   ├── features.py                 # 46-feature stylometric extractor
│   ├── models.py                   # LDAM loss, temperature scaling, ensemble utils
│   └── utils.py                    # Data I/O and submission helpers
├── scripts/
│   └── generate_figures.py         # Competition result visualizations
├── malto_model/
│   ├── ensemble_config.json        # Saved ensemble parameters and label map
│   ├── char_tfidf.pkl              # TF-IDF character n-gram vectorizer
│   ├── word_tfidf.pkl              # TF-IDF word n-gram vectorizer
│   └── svc_model.pkl               # Calibrated LinearSVC
├── docs/
│   └── writeup.md                  # Detailed technical write-up
├── figures/
│   └── competition_results.png     # Score progression + leaderboard chart
├── archive/                        # Previous experiment notebooks and submissions
├── environment.yml                 # Conda environment spec
├── requirements.txt
├── CONTRIBUTING.md
├── LICENSE
└── README.md

Reproducing the Results

On Kaggle (recommended)

1. Upload notebooks/solution.ipynb to a Kaggle notebook
2. Enable GPU T4×2 in Settings → Accelerator
3. Attach the competition dataset
4. Run All Cells (~155 min)

The notebook auto-detects /kaggle/input/ vs local paths.

Load the transformer locally

from transformers import AutoModelForSequenceClassification, AutoTokenizer

model     = AutoModelForSequenceClassification.from_pretrained("malto_model")
tokenizer = AutoTokenizer.from_pretrained("malto_model")

Requirements

torch>=2.0
transformers>=4.40
scikit-learn>=1.4
scipy>=1.12
numpy>=1.24
pandas>=2.0
joblib>=1.3
tqdm>=4.65
matplotlib>=3.8

See environment.yml for the full reproducible conda environment.

License

MIT — see LICENSE.