Seven criteria with weights, four levels from 0 to 3, the scoring rule and a worked, fictitious scoring example.
Author
R. Heller
The group project is assessed with the rubric below. The same rubric is used for feedback on the interim report in W6 and on the defence in W8, so you know at every point what is expected. Assessment and participation conditions for a particular run of the course are announced by the course team before the course starts.
Criteria and weights
Table 1: Assessment criteria of the group project. The weights add up to 100%.
Criterion
Weight
Observable anchor for a strong project
Question, data unit and comparison
15%
Purpose, time zero, population and baseline fit the decision.
R analysis, validation and uncertainty
25%
Someone outside the group can reproduce a figure and identify the independent test data set.
Ethical weighing and communication
15%
One affected person, one conflict of goals and the reasoned consequence are concrete.
Regulatory classification
10%
Intended purpose, role, relevant source and open classification are traceable.
Data protection
10%
Data flow, legal or review question, recipients and deletion point are consistent.
Information security
10%
A plausible threat receives an effective control, a test and a responsible role.
Recommendation, defence and revision
15%
The recommendation states conditions, a named counter-position and a verifiable next step.
Levels
Every criterion is scored on the same four levels.
Table 2: The four levels of the rubric.
Level
Meaning
Example for “data protection”
0
Missing. The criterion is not addressed.
No data flow.
1
Named. The relevant point is mentioned in general terms.
“Patient data are pseudonymised.”
2
Evidenced in the case. The point is worked out for this project with its specific facts.
A data flow with recipients, purpose and a deletion point for the gait videos.
3
Evidenced, tested against an alternative and translated into the decision.
As level 2, plus a comparison with keeping only joint angles, and the recommendation to delete raw videos after 30 days.
The split is a proposal; the course team states the conditions that apply before the course starts. Self-assessment and peer feedback serve as feedback only and never enter a grade.
Worked example (fictitious)
A fictitious group receives the following levels. The calculation is done in R so that every step can be checked.
rubric <-data.frame(criterion =c("Question, data unit and comparison", "R analysis, validation and uncertainty","Ethical weighing and communication", "Regulatory classification", "Data protection","Information security", "Recommendation, defence and revision"),weight =c(15, 25, 15, 10, 10, 10, 15), # percentlevel =c(2, 3, 2, 1, 2, 2, 3) # fictitious levels 0-3)rubric$points <- rubric$weight * rubric$level /3project_percent <-sum(rubric$points)exam_correct <-15# fictitious: 15 of 20 questions correctexam_percent <-100* exam_correct /20course_percent <-0.6* exam_percent +0.4* project_percentrubric
#> criterion weight level points
#> 1 Question, data unit and comparison 15 2 10.000000
#> 2 R analysis, validation and uncertainty 25 3 25.000000
#> 3 Ethical weighing and communication 15 2 10.000000
#> 4 Regulatory classification 10 1 3.333333
#> 5 Data protection 10 2 6.666667
#> 6 Information security 10 2 6.666667
#> 7 Recommendation, defence and revision 15 3 15.000000
# Checks: the weights add up to 100 and the example is reproduced exactlystopifnot(sum(rubric$weight) ==100, all(rubric$level %in%0:3),isTRUE(all.equal(project_percent, 230/3)),isTRUE(all.equal(course_percent, 0.6*75+0.4*230/3)))
The fictitious group reaches 76.7% in the project. With 15 of 20 examination questions correct (75%), the proposed weighting gives 75.7% for the course. Use the example to calibrate assessors: two assessors score the same project independently, compare their levels criterion by criterion and agree on the reasons before scoring further projects.
Common mistakes the rubric catches
An AUC without the number of independent test units (criterion 2 stays at level 1).
“Data are anonymised” without a data flow (data protection stays at level 1).
A control without a test (“we use encryption”) stays at level 1 for information security.
A recommendation without a condition for stopping is at most level 2.