Evaluation before prediction
Why a fall-event classifier is not evidence of future fall prediction.
Technical note
Recognising an event in a labelled recording differs from predicting an event before it occurs. The latter requires an explicit prediction horizon, longitudinal outcomes and controls for leakage.
Proposed evaluation discipline
Report observation time and false alerts alongside sensitivity. Separate participants and settings between development and testing. Publish negative findings and uncertain cases.
Evidence status
This is a methodological note, not an experimental discovery.
A concrete source of misleading confidence
If consecutive frames from one recording appear in both training and test sets, a model may benefit from shared room conditions or participant characteristics. A high score in that arrangement may say little about a different person in a different room.
A proposed evaluation checklist
- Define the event and the observation period before selecting a metric.
- Specify what information would have been available at the time of a prediction.
- Keep participants and recording sessions separated where required by the question.
- Report false alerts relative to observation time.
- Inspect failure cases and uncertain outputs, including routine activity.
- Describe the limits of the reference labels and annotation agreement.
Interpreting results responsibly
Sensitivity alone does not describe the burden of alerts. Overall accuracy can hide poor performance on rare events. A complete report should explain the dataset composition, thresholds, confidence intervals and whether the evaluation was chosen before inspecting the held-out results.