A rubric is only as reliable as the raters using it. Standardise before every rating session.
Standardisation session (60-90 minutes)
- Read the rubric together; clarify terms ("range", "occasional").
- Benchmarks: listen to 3-4 recorded performances already rated by experienced raters (low, middle, high, borderline). Discuss which descriptor words justify each band.
- Practice: rate 3-4 new recordings independently; compare; discuss differences of a band or more.
- Qualify: each rater rates 4-6 recordings; agreement within one band on at least 90% before rating live.
- Record decisions about borderline features in a rater guide.
During rating
- Rate criteria in the same order every time; take notes on evidence (quotes, error examples).
- Rate the performance, not the person or the previous candidate (avoid contrast effects).
- Do not average criteria in your head; give each criterion its own band.
- For live interviews: one interlocutor and one assessor where possible; the assessor does not speak.
Double rating
- Double-rate a sample (10-20%) or all performances for high stakes.
- Agreement targets: within one band 90%+; exact agreement 60%+; weighted kappa 0.70+.
- Differences of two or more bands go to a third rater; report the two closest scores or the third rater's.
- Track each rater's mean against the group (severity) and over time (drift).
scripts/rater_agreement.py calculates all of these from a two-column score file.
Common rater effects
| Effect | Sign | Fix |
|---|---|---|
| Severity / leniency | One rater's mean consistently higher or lower | Feedback with data; benchmarks |
| Central tendency | Few top or bottom bands | Benchmarks at the extremes |
| Halo | All criteria get the same band | Rate one criterion across all candidates; notes per criterion |
| Accent bias | Lower pronunciation bands for familiar or unfamiliar accents | Focus on intelligibility; varied benchmarks |
| Drift | Ratings change during a long session | Breaks; re-check benchmarks midway |
Converting bands to scores and levels
- Total = sum or weighted sum of criteria; report criteria too.
- Map totals to levels with cut points set on benchmark samples, not by percentage alone.
- Round consistently and state the rule (e.g. totals ending in .5 round down).
academic-english-eap-course-planner
Plan English for Academic Purposes (EAP) courses: pre-sessional, in-sessional, foundation, IELTS preparation and discipline-specific EAP, with syllabus strands (academic reading, lecture listening and note-taking, essay and report writing, paraphrase, summary and synthesis, referencing and plagiarism, seminars and presentations, academic vocabulary, critical thinking), a 10-week pre-sessional outline, exit assessment, IELTS-CEFR alignment and academic style (formality, hedging, cohesion, reporting verbs). Estimates study hours and weeks to close an IELTS gap from overall or per-skill scores (IELTS rounding, minimum per skill, priority skills, feasibility before a deadline), and checks learner texts for contractions, informal words with academic alternatives, unhedged claims, 'you', And/But/So starts, rhetorical questions and missing citations.
actfl-proficiency-curriculum-mapper
Map world language curricula to the ACTFL framework: proficiency levels (Novice, Intermediate, Advanced, Superior, Distinguished with sublevels), functions, contexts, text types and accuracy, proficiency vs performance, the three modes of communication (interpretive, interpersonal, presentational), the 5 Cs World-Readiness Standards, backward-designed thematic units with Integrated Performance Assessments, typical programme targets and FSI time estimates. Looks up approximate ACTFL-CEFR-ILR equivalences with what each level can do, and checks a unit map for missing modes, rehearsed skits mislabelled as interpersonal, translation or word lists as interpretive tasks, and units without an integrated performance assessment.
adult-literacy-esl-lesson-builder
Build lessons for adult ESL learners who are developing literacy (LESLLA learners: pre-literate, non-literate, semi-literate, literate in a non-Roman script): learner profiles and sensitive initial assessment, principles (oral language first, known to new, systematic phonics with adult-relevant text, repetition, classroom skills, respect for experience), a lesson routine (oral warm-up, review, new grapheme, blending, segmenting, decodable text, writing, functional literacy, Language Experience Approach), a phonics sequence for adults, sight words, functional literacy targets (forms, signs, medicine labels) and portfolio assessment. Checks whether a reading text is decodable with the letter-sound patterns and sight words taught so far, catching untaught digraphs, listing non-decodable words with the untaught part, and suggesting the most useful next patterns.
authentic-material-lesson-adapter
Turn authentic material (news articles, podcasts, videos, songs, menus, timetables, adverts, social media posts, forms, infographics) into language lessons by grading the task rather than the text. Selects texts for interest, relevance, exploitability, coverage, cultural load and rights; finds the 4-6 words to pre-teach, words to gloss on the page and words learners can ignore, plus names and cultural references to explain, with reading time by level; and builds a lesson shape (lead-in, pre-teach, gist or scanning, graded detail task, language focus on chunks and register, response) with task ideas per material type and level from A1 to C1. Covers copyright, fair dealing and fair use, Creative Commons and attribution.
can-do-statement-writer
Write and review can-do statements for language courses, units, lessons and learner self-assessment, CEFR or NCSSFL-ACTFL style: action (observable verb) + text or situation + level-specific conditions, learner ('I can') and descriptor ('Can') versions, progressions of one activity from A1 to C1, and coverage of reception, production, interaction and mediation. Includes a verb bank by mode and level and condition and text-type banks. Lints statements for teacher-centred phrasing, knowledge instead of use (learn, know, understand the passive), missing context, two skills in one statement, vague quality words, grammar items as outcomes, overlong statements and missing modes in a set.
cefr-aligned-syllabus-builder
Build language course syllabuses aligned to the CEFR (A1-C2, plus levels, Companion Volume 2020): entry and exit levels, guided learning hours and course length (with an L1-distance adjustment), syllabus type (topic, functional-notional, task-based, skills-based, multi-strand), unit-by-unit can-do outcomes, grammar from a core inventory by level, vocabulary, functions, pronunciation, balance of reception, production, interaction and mediation, recycling and assessment points. Checks a syllabus for units without can-do statements, core grammar for the level not covered, items above the course level, skills missing from most units and units with no vocabulary or functions.