About this project

What we’re asking

Can a classifier trained on broad, mostly US-centric news titles transfer to UK BBC headlines — and does continued fine-tuning on UK text close the gap?

We compare five systems on the same frozen UK test set: Naive Bayes @ AG, DistilBERT @ AG, Naive Bayes @ UK, DistilBERT UK-only (no AG), and DistilBERT AG→UK continued fine-tuning. Charts live on the classification page.

Hardware: Mac Studio

Training and inference ran locally on Apple Silicon — not a cloud GPU box. DistilBERT is small enough that this machine is more than enough; the practical constraint was getting PyTorch’s MPS backend working cleanly.

MPS notes: we avoid fp16 mixed-precision quirks on MPS, keep batches modest (32 / 16), and set dataloader_pin_memory=False. Stage 1 is typically under 1–2 hours; Stage 2 is minutes; Naive Bayes is seconds.

Model: DistilBERT

Backbone: distilbert-base-uncased with a fresh 4-way sequence-classification head (AutoModelForSequenceClassification).

• ~66M parameters — about 40% of BERT-base, trained by knowledge distillation.
• Uncased WordPiece tokenizer; max sequence length capped at 64 tokens (headlines / short prefixes).
• 4 labels: business, sport, tech, politics.
• Classical baseline: Multinomial Naive Bayes + bag-of-words unigrams (CountVectorizer, min_df=2).

Lower-level architecture

DistilBERT is a compressed Transformer encoder. It keeps the BERT-style stack but drops the next-sentence prediction objective and halves the depth, then trains the student to match a BERT teacher.

Encoder stack

What “fine-tuning” updates

We do full continued fine-tuning: every DistilBERT weight plus the classification head receive gradients. LoRA / adapters were not needed — the model is small, the UK set is small, and full updates finish quickly on MPS.

• Forward: tokenize → pad/truncate to ≤64 → encoder → logits.
• Loss: cross-entropy over the four labels.
• Optimizer path: Hugging Face Trainer (AdamW-style), lr 2e-5, weight decay 0.01, seed 42.
• Selection: best checkpoint by validation macro-F1.

Why DistilBERT (not a generative LLM)

This is single-label topic classification. An encoder with a linear head is the right inductive bias: cheap, stable, and easy to compare against a Naive Bayes bag-of-words baseline. We deliberately do not train a large generative model from scratch.

Fine-tuning techniques

The core method is two-stage domain adaptation (continued fine-tuning), with an ablated UK-only run to separate “UK data helps” from “AG pre-training still helps.”

Not used: LoRA, QLoRA, prefix-tuning, or frozen-encoder linear probes. Full updates are appropriate here; parameter-efficient methods would mostly add complexity without a clear win on this size of model and data.

Data & label mapping

Unified 4-way labels

AG World → BBC politics is an imperfect bridge on purpose — that mismatch is part of what UK fine-tuning has to fix. BBC rows often lack a clean title field; we use title/headline when present, else the first sentence or first ~16 tokens as a headline proxy.

Software pipeline

Scripts are meant to be run in order after source .venv/bin/activate. Prepared CSVs already live in data/; model weights stay gitignored while metrics and charts are committed for GitHub Pages.

1. prepare_data.py — optional regenerate of splits
2. train_nb.py — Naive Bayes @ AG and @ UK
3. train_stage1.py → evaluate.py (before metrics)
4. train_stage2.py → evaluate.py (after metrics)
5. train_uk_only.py → evaluate.py (UK-only metrics)
6. assemble_results.py, embed_similarity.py, novel_words.py

Stack: PyTorch, Hugging Face Transformers / Datasets / Accelerate, scikit-learn, pandas, matplotlib. See requirements.txt and AGENT_PLAN.html for the locked brief.

Evaluation design

• Frozen UK test — never used for training or early stopping.
• Metrics: accuracy, macro-F1, micro-F1, per-class F1, confusion matrices.
• Micro-F1 = accuracy here (single-label multi-class); both reported for lecture-style comparison.
• Headline results: AG→UK fine-tuned reaches ~0.96 accuracy / macro-F1 on UK test; largest class gain vs AG-only is politics. Details on Classification and in RESULTS.md.

Side analyses dig into why transfer is hard: embedding centroid similarity, headline length distributions, and UK words missing from AG train.

Caveats & design risks

• Label bridge World ↔ politics is noisy by construction.
• BBC “headlines” are sometimes truncated article leads, not newsstand titles.
• UK data is small — Stage 2 early-stops on val; over-training would memorize fast.
• AG-only DistilBERT already beats NB @ AG; most of the big jump to ~0.95 comes from UK data itself, with AG→UK still edging UK-only.