What our students taught us about assessment

When we started collecting text for Nigerian Pidgin, the hardest part was not the modelling. It was agreeing on how the language is written down, because the same word shows up in four spellings in a single newspaper.

This post walks through the three decisions that mattered most:

1. Choose a spelling policy and write it down before collecting anything.
2. Keep the original text next to every normalised form.
3. Pay annotators for disagreement, not only for agreement.

None of this is novel, but each decision saved weeks later. If you are building a corpus for a language without a standard orthography, start with the policy document, then the tools.

The corpus itself will be shared once the licensing review is complete.