Open Access

Evidence of Concussion Signs in National Rugby League Match Play: a Video Review and Validation Study

  • Andrew J. Gardner1, 9Email author,
  • David R. Howell3, 4, 5,
  • Christopher R. Levi1, 2 and
  • Grant L. Iverson6, 7, 8
Sports Medicine - Open20173:29

https://doi.org/10.1186/s40798-017-0097-9

Received: 12 February 2017

Accepted: 11 August 2017

Published: 22 August 2017

Abstract

Background

Many professional sports have introduced sideline video review to help recognise concussions. The reliability and validity of identifying clinical and observable signs of concussion using video analysis has not been extensively explored. This study examined the reliability and validity of clinical signs of concussion using video analysis in the National Rugby League (NRL).

Methods

All 201 professional NRL matches from the 2014 season were reviewed to document six signs of possible concussion (unresponsiveness, slow to get up, clutching/shaking head, gait ataxia, vacant stare, and seizure).

Results

A total of 127,062 tackles were reviewed. Getting up slowly was the most common observable sign (2240 times in the season, 1.8% of all tackles) but only 223 times where it appeared to be a possible concussion (0.2% of all tackles and 10.0% of the times it occurred). Additionally, clutching/shaking the head occurred 361 times (on 212 occasions this sign appeared to be due to a possible concussion), gait ataxia was observed 102 times, a vacant stare was noted 98 times, unresponsiveness 52 times, and a possible seizure 4 times. On 383 occasions, one or more of the observable signs were identified and deemed associated with a possible concussion. There were 175 incidences in which a player appeared to demonstrate two or more concussion signs, and 54 incidences where a player appeared to demonstrate three or more concussion signs. A total of 60 diagnosed concussions occurred, and the concussion interchange rule was activated 167 times. Intra-rater reliability (κ = 0.65–1.00) was moderate to perfect for all six video signs; however, the inter-rater reliability was not as strong (κ = 0.22–0.76). Most of the signs had relatively low sensitivity (0.18–0.75), but high specificity (0.85–1.00).

Conclusions

Using video replay, observable signs of concussion appear to be sensitive to concussion diagnoses when reviewing known injuries among professional rugby league players. When reviewing an entire season, however, certain signs occur very commonly and did not identify concussion. Thus, the implementation of video review in the NRL is challenging, but can provide a useful addition to sideline concussion identification and removal from play decisions.

Keywords

Concussion Video analysis Injury management Return to play

Key Points

  • The addition of video review to the assessment of concussion injury events may help improve consistency in the management of players, as well as assist in diagnostic decision-making in cases where signs may be transient and resolved by the time of the medical assessment.

  • The signs of concussion appear to be quite sensitive to concussion when reviewing known injuries; however, when reviewing an entire season, some signs occur very commonly and usually do not reflect a concussive injury.

  • Most signs of concussion had high specificity but low sensitivity when examining all tackles across a sporting season.

Background

Participation in many full contact and collision sports, such as rugby league, carries with it a risk of concussion [10]. In-game concussion diagnosis, however, remains a highly challenging task for the athletic trainer and sports medicine physician. On-field or sideline clinical assessments can be challenging due to the heterogeneous presentation of an athlete following a head impact, the non-specific nature of many of the clinical signs and symptoms of concussion [29], the absence of a reliable concussion biomarker [40], and the regularity with which some concussion signs emerge and evolve over time [31]. Recognising a potential concussion and removing an athlete from play is understood to be an important intervention for reducing the risk of a worse clinical outcome following injury [31]. However, it is acknowledged that in some instances, concussions may be missed from the sideline [25]. This may occur for a variety of reasons, but commonly the transient early physical signs may resolve before the player can be removed from play and assessed [25].

Some prior studies suggest that worse outcomes following concussion are associated with on-field signs and symptoms, such as loss of consciousness [28], amnesia [8, 28], mental status change for more than 5 min [8], and dizziness [23]. It is important to appreciate that the literature on the association between on-field signs and symptoms is mixed. For example, loss of consciousness has been associated with worse clinical outcomes in some [1, 4, 15, 28, 35, 39], but not in most studies [2, 3, 5, 8, 9, 16, 18, 22, 27, 30, 32, 3638, 42, 43]. The vast majority of studies examining loss of consciousness base this finding on a questionnaire or interview completed with the athlete, not video review of the injury event for confirmation. Similarly, post-traumatic amnesia has been associated with worse clinical outcomes in some [15, 24, 28], but not in most studies [13, 8, 16, 18, 23, 27, 30, 34, 37, 38, 42]. Dizziness has been observed as an on-field symptom associated with a protracted recovery of greater than 21 days (6.34 time more likely) [23], but assessing dizziness is subjective and may or may not manifest as an objective sign (e.g., gait ataxia). Thus, video review may allow for the quantification of objective concussion signs, but not subjective symptoms.

In the sport of rugby league, the concussion incidence rates have been reported to vary widely depending on the level of competition [10]. In one study of three National Rugby League (NRL) clubs, a concussion incidence rate of 14.8 concussions per 1000 player match hours was reported [13], while a rate of 28.3 concussion per 1000 player match hours were reported from one NRL club over a 15-year (1998–2012) period [41].

The use of video footage on the sideline for reviewing a concussion has been introduced in a number of professional sports as a method to improve the recognition of a possible concussion that may be missed by on-field medical personnel [25]. Video studies have now been conducted in a variety of sports such as rugby league [1113], rugby union [21], hockey [7], and Australian rules football [6, 25, 26]. In addition to the introduction of sideline video review, the governing body of the sport in Australia (the National Rugby League) implemented a new the “concussion interchange rule” (CIR). The CIR requires the mandatory removal of any player suspected of having sustained a concussion. The CIR allows a player to be removed from play for 15 min to be assessed by the club medical officer, including completing the Sports Concussion Assessment Tool third edition (SCAT-3). Following the assessment, if the player was not diagnosed with concussion, they are permitted to return to play without using an interchange. The incidence of use of the CIR was 24.0 (95% CI 20.7–27.9) uses of the CIR per 1000 NRL player match hours [12], and 44.9 (95% CI 38.5–52.3) uses of the CIR per 1000 National Youth Competition player match hours [13].

The primary aim of this study was to determine the rate of six objective concussion signs that occurred during the 2014 NRL season, as well as the sensitivity and the specificity of these signs to classify whether or not a diagnosed concussion resulted from the event. The secondary objective of this study was to analyse the intra- and inter-rater reliability of these signs using the video recordings of each incident that activated the CIR.

Methods

Participants

Participants for this study were the entire league of NRL players involved in match play during the 2014 season. Each team is comprised of a squad of approximately 25 players, and there are 16 teams in the league. There are 13 players on the field during the match.

Procedure

This study conducted a video analysis of every game during the 2014 National Rugby League (NRL) season. The full game video analysis for all games during the season was conducted retrospectively, at the conclusion of the season by the lead author. The video review was conducted independent of any club trainer or team medical staff. The video was used to examine all on-field signs. There was no prerequisite for recording the presence of a sign (e.g., contact with the head was not required). The lead researcher obtained information on the use of the CIR (i.e., the player’s name and round in which the CIR was used). Each NRL team physician provided a list of players who had been diagnosed with concussion during the season. The club physician made the final diagnosis of concussion, based on conventional clinical examination techniques. For 162/167 (97%) uses of the CIR [12], and for all diagnosed concussions (n = 60), the Observational Review and Analysis of Concussion (ORAC) form [14] was used to record the game circumstances and details of the event as well as the presence or absence of six concussive signs (clutching or shaking head, slow to get up, gait ataxia, blank/vacant stare, unresponsiveness, and seizure). There were five uses of the CIR where the video of the incident was not located; none of these cases were diagnosed with a concussion. There were 162 (97%) cases where the CIR incident was identified, and 60 (100%) cases were medically diagnosed with a concussion. Two reviewers conducted the video analysis of these cases. To assess intra-rater reliability, one of the reviewers also completed a second review of 162 video clips of uses of the CIR, which included a review of all cases of medically diagnosed concussion.

All diagnosed concussions and 97% of the uses of the CIR were independently reviewed by two raters. The video raters were blinded to the club medical staff’s documentation (i.e., they conducted their retrospective review of the video signs completely independently of the club medical staff). Both raters have experience in the identification of concussion on the sideline and with retrospective video review. The two raters determined whether any of six signs of concussion were present, absent, or indeterminable based on the available footage of the incident for every case. When there was disagreement between the two raters, both raters reviewed and discussed those cases in an effort to reach consensus. In the cases where consensus could not be achieved, ratings from an experienced third rater were used. Consent was obtained from all participants who used the CIR. This study was approved by the University of Newcastle Human Ethics Committee (H-2012-0344).

The ORAC form was created to provide a simple but standardised framework for coding and analysing video footage of the situations and consequences of concussion events in rugby league. The form was developed based on work conducted previously in ice hockey in North America [19], but was adapted to include specific information to rugby league. The form includes concussion signs that have been previously examined in video review studies in rugby league [1113]. The form consists of various sections related to the player and game characteristics (e.g., ball carrier versus tackler, tackle height, type of play, etc.), the anatomical region of contact, the injury location on the field of play, the injured player’s on-field management, and six concussive signs (defined in Table 1).
Table 1

Definition of video signs

Concussion sign

Definition

Clutch or shake head

The player holds his head or face in the palm of his hand or hands, or the player rubs or shakes his head in a manner that appears to demonstrate they are experiencing discomfort (i.e., not just wiping his face).

Slow to return to feet/play

Player took longer than usual to return to his feet (e.g., remained on the ground, got to his knees or to his haunches, and waited momentarily before standing), in the absence of another player holding him down.

Gait ataxia (wobbly legs)

Unable to stand steadily unaided or walk normally. Appears “clumsy”. Upon standing and walking the player has unsteadiness, wobbly legs, balance problems, stumbles or falls over, drags his feet, or cannot walk straight independently.

Blank/vacant stare

The player is not visually focused on doctor/trainer when being spoken to, or assessed and asked to attend, and/or the player appears to be looking off into the distance. The player’s face is expressionless.

Evidence of unresponsiveness

The player’s body goes limp/floppy, the player loses control of muscles, the player does not protect (i.e., brace) himself when falling (e.g., ragdoll-like appearance). The player remains/lies motionless on the ground for a period of time longer than expected, or the player shows a lack of visible responsiveness to verbal stimuli.

Post-impact seizure

Tonic posturing—stiffening of limbs; or clonic movements—involuntary repetitive contraction and relaxation of the muscles, could include jerky movements, or shaking/convulsion of upper and/or lower limbs, or body.

The full-match digital records of all 2014 NRL season matches (n = 210) were reviewed by a single rater (AG) using the Quicktime Multimedia Player V.7.7.5. The season prevalence of the six concussion signs were recorded by this rater. There was an important distinction made between observing a sign and instances where the sign ‘looked like it was a concussion-related’ sign, so there were two tallies made for each sign (i.e., an entire season tally of the sign regardless of the reasons for the player displaying that sign, and a season tally of the sign where the sign appeared to reflect a possible concussion). We operationally defined ‘looked like a concussion’ as all instances where there was an absence of another explanation for the sign. The digital records of events leading to the use of the CIR and medically diagnosed concussion were independently reviewed by two raters, on two separate occasions. The only circumstances that rendered a reported use of the CIR or concussion ineligible for inclusion were (i) any event where the video of the player was obstructed by other players or the match official(s) (n = 0), or (ii) where the incidence was unable to be located following a full-match review (n = 5 uses of the CIR; n = 0 diagnosed concussions). The ORAC form was used to code each application of the CIR and medically diagnosed concussion, in an identical methodology to previous studies in this setting [1113]. See Fig. 1 for details.
Fig. 1

Hierarchy of methods and procedures of video review process

Statistical Analysis

Descriptive analyses are presented as means (standard deviation) for continuous variables; categorical variables are presented as numbers included (n) or percentage of the total. Sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were calculated for each of the six concussion signs using 2 × 2 contingency tables. The number of times a player demonstrated one or more of these signs that looked like a possible concussion sign (n = 383) during match play for the entire season was used as the denominator for calculating sensitivity, specificity, PPV, and NPV.

Intra- and inter-rater reliability analyses using Cohen’s kappa (κ) statistics [17] were used for 162 (97%) cases where the CIR was used and for all of those cases that were medically diagnosed with concussion, to determine consistency among the two raters for each of the six individual signs. Unlike the total percent agreement, Cohen’s kappa considers the proportional agreement that could occur simply by chance. The κ coefficients are calculated by considering the proportion of rater agreement and the expected proportion [17]. Using the interpretations of κ described by McHugh [33], κ agreement was categorised as almost perfect (>.90), strong (.80–.90), moderate (.60–.79), weak (.40–.59), minimal (.21–.39), and none (0–.20). All analyses were performed using IBM SPSS Statistics V.23.0 [20] and used two-sided tests for significance at the 0.05 level, with either 90 or 95% confidence intervals (CIs).

Results

During the 2014 season, the CIR was used on 167 occasions (97% [n = 162] were located and reviewed), and there were 60 medically diagnosed concussions. The incidence rate for concussion was 8.92 (95% CI = 6.96–11.43) per 1000 National Rugby League player match hours, or was approximately one concussion every 3.35 games. There were 127,062 tackles reviewed in the 201 matches, for an average of 632.1 tackles per match (Table 2). There was an important distinction made between observing a sign, and instances where the sign looked like it reflected a concussion. For example, in the 201 games reviewed, slow to get up was observed 2240 times but of those 2240 times on only 223 occasions slow to get up looked like it may have been a concussion-related sign (10.0%). Therefore, on 2017 occasions a player was slow to get up due to reasons other than a possible concussion. For example, the player stayed down to stay out of the way of play and avoid being penalised = 837, appeared to be fatigued = 706 or sustained another injury = 365; play had stopped = 78; or the opposition scored and the player stayed down = 31. The sign clutching or shaking head was observed 361 times and for 212 times the sign was considered possibly concussion-related (58.7%). Examples of when it was not considered concussion-related included when the player sustained a facial (non-nose) injury = 64 or a nose injury = 42, exhibited a disappointed or dismayed reaction to play = 27; wiped or shook off sweat from his head = 15, or appeared to have been poked in the eye = 1. All instances of observing unresponsiveness (n = 52), gait ataxia (n = 102), vacant stare (n = 98), and a post-impact seizure (n = 4) throughout the season were considered to be possibly concussion-related (Table 3).
Table 2

Data summary of video analysis from the 2014 National Rugby League season

Variable

Total

Matches

201

Total number of tackles in the season

127,062

Average number of tackles per match

632.1

Average concussion interchange rule user per match

0.78

Concussion interchange rule (CIR) uses

167

 Returned to play

80

 No return to play

76

 Diagnosed concussions

60

 CIR, no return to play, and no concussion

16

Note: video footage for 5 cases of the use of the CIR could not be located and therefore were not coded for concussion signs

There were 6 CIR cases where return to play data is missing

Table 3

Summary of the rates of concussion signs for the entire season, concussion-related signs, use of the CIR, and medically diagnosed concussions

Concussion sign

Entire season total

Possible concussion sign

Use of the CIR

CIR missing dataa

Medically Diagnosed concussion

Medically Diagnosed concussion missing data

(n = 162)a

(n = 60)

Unresponsiveness

52

52

50

2

24

1

Slow to get up

2240

223

153

0

60

0

Clutching head

361

212

110

1

38

0

Gait ataxia

102

102

87

18

35

11

Vacant stare b

98

98

93

18

45

6

Possible seizure

4

4

3

0

3

0

Note: CIR concussion interchange rule

The “Entire Season Total” refers to every instance during the season that the sign was observed (whether it appeared to be a sign of concussion or not). The “Possible Concussion Sign” refers to the subset of the Entire Season Total in those instances where the sign was observed, and the sign appeared to be a possible concussion

aVideo footage for 5 cases of the use of the CIR could not be located and therefore were not coded for concussion signs

b The frequency of missing data for the sign ‘vacant stare’ for the entire season total was not recorded

Of the 223 instances of a player being slow to get up, 153 players were removed under the CIR (68.6%); return to play data was available for only 156/162 (96.3%) instances of the use of the CIR. Of those 153, 60 were medically diagnosed as having a concussion (39.2%), and all players who were medically diagnosed with concussion demonstrated the sign “slow to get up”. The PPV and NPV for slow to get up were 0.27 and 1.00, respectively. Of the 212 instances of clutching or shaking head, 110 players were removed under the CIR (51.9%) and 38 were diagnosed with a concussion (34.5%). Of those who were diagnosed with concussion, players clutched or shook their heads 63.3% (38/60) of the time. The PPV and NPV for clutching or shaking head were 0.18 and 0.87, respectively. Of the 102 instances of gait ataxia, 87 players were removed under the CIR (85.3%) and 35 were diagnosed with concussion (40.2%). Gait ataxia was seen in 58.3% of players medically diagnosed with concussion. The PPV and NPV for gait ataxia were 0.34 and 0.91, respectively. Of the 98 instances of a blank or vacant stare, 93 players were removed under the CIR (94.9%) and 45 were diagnosed with concussion (48.4%). A blank or vacant stare was observed in 75.0% of players medically diagnosed with concussion. The PPV and NPV for blank or vacant stare were 0.46 and 0.95, respectively. Of the 52 instances in which a player appeared to have brief unresponsiveness following a tackle, 50 players were removed under the CIR (96.2%), of which 24 were diagnosed with a concussion (48%). Unresponsiveness on the field was observed in 40.0% (24/60) of players medically diagnosed with concussion. The PPV and NPV for unresponsiveness were 0.46 and 0.89, respectively. Of the 4 instances in which a player appeared to experience a post-impact seizure, 3 players were removed under the CIR (75.0%). All three players were medically diagnosed with a concussion (100%). There were three (5%) players who were medically diagnosed with concussion that showed signs of seizure-like activity. The PPV and NPV for seizure-like activity were 0.75 and 0.85, respectively (Table 4).
Table 4

Rates, sensitivity, specificity, positive predictive value, and negative predictive value for concussion signs

Concussion sign

Sign present

Medically Dx concussion with sign

Sign absent

Medically Dx concussion without sign

Sign sens

95% CI

Sign spec

95% CI

PPV

95% CI

NPV

95% CI

Clutching or shaking head

212

38

171

22

0.63

0.51–0.75

0.46

0.44–0.48

0.18

0.14–0.21

0.87

0.83–0.91

Slow to get up

223

60

160

0

1.00

0.93–1.00

0.50

0.48–0.50

0.27

0.25–0.27

1.00

0.97–1.00

Gait ataxia

102

35

281

25

0.58

0.46–0.70

0.79

0.77–0.81

0.34

0.27–0.41

0.91

0.86–0.94

Blank or vacant stare

98

45

285

15

0.75

0.63–0.84

0.84

0.81–0.85

0.46

0.39–0.52

0.95

0.92–0.97

Unresponsiveness

52

24

331

36

0.40

0.29–0.50

0.91

0.89–0.93

0.46

0.34–0.58

0.89

0.87–0.91

Possible seizure

4

3

379

57

0.05

0.02–0.07

1.00

0.99–1.00

0.75

0.22–0.99

0.85

0.84–0.85

Note: All calculations were made using the number of times the sign was seen in concussed players, contrasted to the total number of times one or more signs (that appeared to represent a concussion) were seen (n = 383), for the entire season

≥ greater than or equal to, CI confidence interval, Dx diagnosed, n number, NPV negative predictive value, PPV positive predictive value, Sens sensitivity, Spec specificity

One or more video signs of possible concussion were seen during 383 tackles (i.e., only 0.3% of the total number of tackles), most commonly demonstrating the characteristic “slow to get up” (Table 5). Considering five signs simultaneously (unresponsiveness, clutching/shaking head, gait ataxia, slow to get up, or blank/vacant stare), 34 (57%) of players medically diagnosed with concussion had three or more observable signs of concussion. There were a greater total number of signs observed in the concussed group compared to players that used the CIR and were cleared to return to play in the same game (Table 5). For the individual signs, those diagnosed with concussion were more likely to show evidence of unresponsiveness and a vacant stare. There was no difference between the number of observed signs between groups for clutching or shaking the head, or gait ataxia.
Table 5

Summary of video analysis findings for those situations in which the concussion interchange rule was used

Signs of concussion

Medically diagnosed concussion (n = 60)

No diagnosed concussion and no return to playa (n = 16)

Returned to playa (n = 80)

Diagnosed concussion vs. returned to playa

 

Present

Absent

Missing

Present

Absent

Missing

Present

Absent

Missing

   
 

% (n)

% (n)

% (n)

% (n)

% (n)

% (n)

% (n)

% (n)

% (n)

χ2

p

RR (90% CI)

Unresponsiveness

40 (24)

58.3 (35)

1.7 (1)

43.8 (7)

56.3 (9)

0 (0)

23.8 (19)

75 (60)

1.3 (1)

4.35

.037

1.52 (1.06–2.10)

Slow to get up

100 (60)

0 (0)

0 (0)

100 (16)

0 (0)

0 (0)

96.3 (77)

3.7 (3)

0 (0)

Not applicable

Clutching head

63.3 (38)

36.7 (22)

0 (0)

87.5 (14)

12.5 (2)

0 (0)

72.5 (58)

26.3 (21)

1.3 (1)

1.62

.203

0.77 (0.55–1.12)

Gait ataxia

58.3 (35)

23.3 (14)

18.3 (11)

62.5 (10)

37.5 (6)

0 (0)

52.5 (42)

38.8 (31)

8.8 (7)

2.43

.119

1.46 (0.94–2.38)

Vacant stare

75 (45)

15 (9)

10 (6)

62.5 (10)

37.5 (6)

0 (0)

47.5 (38)

37.5 (30)

15 (12)

10.43

.001

2.35 (1.40–4.30)

Possible seizure

5 (3)

95 (57)

0 (0)

0 (0)

100.0 (16)

0 (0)

0 (0)

100 (80)

0 (0)

Not applicable

Total signs observed (n)

205

64

174

Note. Video footage for 5 cases of the use of the CIR could not be located and therefore were not coded for concussion signs; there were 6 CIR cases where return to play data is missing

aReturn to play refers to return to play in the same game; RR risk ratio

The players who were medically diagnosed with a concussion were significantly more likely (1.52 times) to exhibit unresponsiveness and a vacant stare (2.35 times) than players who were removed from play under the CIR, not diagnosed with a concussion, and were returned to play. Clutching the head and gait ataxia were not significantly different signs between those who were and were not medically diagnosed with a concussion (see Table 1).

A sign that was always present in cases of diagnosed concussion was slow to get up (sensitivity = 100%), although it had low specificity (50%). A possible seizure was observed only four times during the season and on three occasions those athletes were medically diagnosed as having a concussion (3/60 diagnosed concussions; sensitivity = 5%, specificity = 100%). A blank or vacant stare had fairly high sensitivity and specificity (75% and 84%, respectively; see Table 4).

The intra-rater reliability of the video analysis of each sign of concussion was moderate to perfect, and perfect agreement was observed in three (clutch or shake head, slow to get up, and post-impact seizure) of the six signs. Moderate inter-rater reliability was observed in three of six of signs: clutch or shake head, slow to get up, and unresponsiveness (Table 6).
Table 6

Intra-rater reliability and inter-rater reliability for the concussion signs

 

Intra-rater (κ) (95% CIs)

McHugh [33]

κ agreement classification

Inter-rater (κ) (95% CIs)

McHugh [33]

κ agreement classification

Absolute disagreement (CIR cases, n = 162)

Clutch of shake head

1.00 (–,–)

Perfect

0.76 (0.60–0.91)

Moderate

43

Slow to get up

1.00 (–,–)

Perfect

0.64 (0.42–0.83)

Moderate

27

Gait ataxia

0.77 (0.63–0.90)

Moderate

0.37 (0.21–0.53)

Minimal

36

Blank/vacant stare

0.65 (0.44–0.84)

Moderate

0.22 (0.05–0.40)

Minimal

71

Unresponsiveness

0.84 (0.70–0.94)

Strong

0.75 (0.56–0.92)

Moderate

22

Post-impact seizure

1.00 (–,–)

Perfect

0.40 (0.00–0.72)

Weak

3

Note. CIs confidence intervals, κ Kappa McHugh, κ agreement was categorised as almost perfect (>.90), strong (.80–.90), moderate (.60–.79), weak (.40–.59), minimal (.21–.39), and none (0–.20)

Discussion

To expand previous video analysis work in collision sports, this study explored the rate of six observable signs of concussion, as well as their sensitivity and specificity during match play of a National Rugby League season. The results of the current study revealed moderate to perfect intra-rater reliability or better for all video signs. For inter-rater reliability, clutching or shaking head, slow to get up, and unresponsiveness had moderate inter-rater reliability, whereas post-impact seizure had weak inter-rater reliability, and gait ataxia and blank or vacant stare had poor inter-rater reliability. These findings are relatively consistent with those reported from a video review study of concussion signs during Australia Football League (AFL) match play, which found that intra-rater reliability was generally better than inter-rater reliability, and blank or vacant stare was the sign with the lowest agreement [25].

Blank or vacant stare was the only sign that had reasonably high sensitivity (75%) and specificity (84%); all other signs did not have both high sensitivity and specificity for a concussion diagnosis. Some of the signs had high sensitivity but low specificity, suggesting that those signs may be a useful marker in flagging a potential concussion, but not helpful in confirming the diagnosis. Therefore, medical personnel working in tandem on the sideline and with individuals reviewing video may be an appropriate strategy to achieve a sensitive and specific approach to concussion diagnosis. In a video review of concussion signs during match play in the Australian Football League (AFL), slow to get up had the highest sensitivity (87%), but low specificity (19%). All other video concussion signs examined in this prior study had high specificity but low sensitivity, such as blank and vacant look [specificity 100%, sensitivity 9%], motor incoordination [specificity 95%, sensitivity 29%], impact seizure [specificity 93%, sensitivity 7%], and rag doll appearance [specificity 91%, sensitivity 16%]). The highest PPV was found with the “blank and vacant look” (100%), and “motor incoordination” (81%) signs [25]. However, in the current study, we found that the highest PPV was for seizure (75%), while slow to get up (100%), blank or vacant stare (95%), and gait ataxia (91%) were the signs with the highest NPV.

The sign with the highest relative risk of diagnosed concussion was a blank or vacant stare (2.35 times more likely; 90% CI = 1.40–4.30), while unresponsiveness was 1.52 times more likely (90% CI = 1.06–2.10). Gait ataxia (1.46 times more likely; 90% CI = 0.94–2.38) and clutching or shaking head (0.77 times more likely; 90% CI = 0.55–1.12) were not statistically different. Blank or vacant stare appears to be a conceptually different sign to the other five. The detection of a blank or vacant stare tends to be through a secondary screening process, in that it is often only observed if, and when, a player is injured and the camera zooms in closely. It is also arguably the most subjective of the six signs and therefore vulnerable to greater variation between raters when classifying this sign. This notion is supported by the weaker inter-rater reliability of this sign not only in the current study but also in previous video review studies [12, 13, 25]. In our study, the low inter-rater reliability was due to one rater seeing the sign more often than the other, recording it as present, and then both raters ultimately agreeing that it was present. Observing a blank or vacant stare was rare in a recently published NHL video review study [7]. We do not know why the sign was so uncommon in that study compared to ours. It might be more difficult to see the sign through a hockey helmet, and it might also be more difficult to see it based on camera angles when players are moving off the ice. Among the other signs, inter-rater reliability was the strongest for clutch or shake head, slow to get up, and unresponsiveness, each with a moderate agreement classification. These signs may be more easily seen during field-side evaluations; their intra-rater reliability was strong to perfect. In contrast, the minimal agreement between raters for gait ataxia or blank/vacant stare warrants caution with their use.

There were several limitations of the current study. In some instances, the available video footage was not sufficiently clear to code all signs (i.e., the view from the available camera angle was obscured, or a close up of the incident was not available). This ‘missing data’ was excluded from the analyses, which might have slightly improved the support for the utility of some of the visible signs. This was particularly true for the blank or vacant stare sign, where missing data on this sign was common. Despite being the most sensitive and specific sign for concussion diagnosis, blank or vacant stare had the worst inter-rater reliability (0.22, with 71 absolute disagreements between raters). For these reasons, if video review within professional sports are implemented, then access to high quality reviews with the capability of multi-angle and slow motion replays to allow for close ups would be optimal [25] and would reduce the likelihood of missing data. Although the video reviewer was blinded to the sideline assessment results and the medical diagnosis of concussion for this study, they were only partially blinded to the use of the CIR. Given that the process for enacting a use of this rule requires the trainer to provide a signal to the sideline, and the official on the sideline identifies the interchange with a green card, the video reviewer was able to identify many instances where the CIR was used. Another study limitation was that there were no concussions that were subsequently (beyond match day) diagnosed by the club medical and reported to the researchers, which is inconsistent with other video review studies that have reported a numbers of cases of post-game diagnosis of concussion [7]. A further limitation of the current study pertains to the generalizability of the current findings to other levels of rugby league. The current study was a post-game review of an adult, male, professional league, and as such the results may not necessarily be generalizable to in-match real-time analysis, or other levels of match play. Our use of an operational definition of exclusion to categorise the observed signs into ‘plausible concussion signs’ versus signs that were more likely attributable to other factors, was also a limitation in terms of its subjectivity and reproducibility for future work in video review of concussion signs. Finally, only one reviewer completed the coding of the entire game, for every game in the season; the inter-rater reliability of that type of coding is unknown.

Conclusions

A conservative approach to sideline concussion management would be to remove a player from play based on any evidence of possible concussion signs. This approach is encouraged as the best management strategy and in the best interest of the welfare of the player [31]. However, there is limited information available on the reliability and validity of identifying the objective signs of concussion when using video analysis, and indeed not all instances of observed concussion signs occur as a result of the player having sustained a concussion. The signs of concussion appear to be quite sensitive to concussion when reviewing known (i.e., medically diagnosed concussions) or suspected (i.e., players who have used the CIR) injury. When reviewing an entire season of match play, specific objective concussion signs such as slowness in getting up and clutching/shaking the head occurred commonly during professional rugby league match play, but did not typically reflect concussion occurrence. For these reasons, video injury surveillance can be difficult to interpret, but may provide a useful adjunct to the clinical assessment of potential concussion. With improved access to video replays, clear definitions and education regarding the observable signs, and improved communication between video observers and sideline medical personnel, the detection of concussion may improve.

Declarations

Acknowledgements

The authors would like to thank Dr Jinho Lee (School of Human Movement and Nutrition Sciences, University of Queensland) for completing the video review of all uses of the CIR.

Funding

Funding was provided by Hunter Medical Research Institute (HMRI) supported by Anne Greaves.

Availability of data and materials

Not applicable

Authors’ contributions

AG conceived the design of the study, reviewed all game video, collected all of the study data, and drafted, revised, and finalised the manuscript. He had final veto on the submission. DH assisted with the design of the study, conducted the statistical analysis, provided expert editorial comment for all drafts of the manuscript, and had final veto on the submission. CL assisted with the design of the study and provided editorial comment to the final draft of the manuscript. GI assisted with the design of the study, assisted with the statistical analysis of the data, provided expert editorial comment for all drafts of the manuscript, and had final veto on the submission. All authors read and approved the final manuscript.

Ethics approval and consent to participate

The study was approved by the University of Newcastle Human Ethics Committee (Ref No. H-2012-0344). Consent to participant was provided by all participants.

Consent for publication

Not applicable.

Competing interests

Andrew Gardner has a clinical practice in neuropsychology involving individuals who have sustained sport-related concussion (including current and former athletes). He has operated as a contracted concussion consultant to the Australian Rugby Union (ARU) from July 2016. He has received travel funding from the Australian Football League (AFL) to present at the Concussion in Football Conference in 2013 and 2017. Previous grant funding includes the NSW Sporting Injuries Committee, the Brain Foundation (Australia), and the Hunter Medical Research Institute (HMRI), supported by Jennie Thomas. He is currently funded through the HMRI, supported by Anne Greaves, and the University of Newcastle’s Priority Research Centre for Stroke and Brain Injury.

David Howell and Christopher Levi have no competing interests to declare.

Grant Iverson has been reimbursed by the government, professional scientific bodies, and commercial organisations for discussing or presenting research relating to mild TBI and sport-related concussion at meetings, scientific conferences, and symposiums. He has a clinical and consulting practice in forensic neuropsychology involving individuals who have sustained mild TBIs (including athletes). He has received research funding from several test publishing companies, including ImPACT Applications, Inc., CNS Vital Signs, and Psychological Assessment Resources (PAR, Inc.). He acknowledges unrestricted philanthropic support from the Mooney-Reed Charitable Foundation and ImPACT Applications, Inc.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Authors’ Affiliations

(1)
Centre for Stroke and Brain Injury, School of Medicine and Public Health, University of Newcastle
(2)
Hunter New England Local Health District Sports Concussion Program, John Hunter Hospital
(3)
The Micheli Center for Sports Injury Prevention
(4)
Division of Sports Medicine, Department of Orthopaedics, Boston Children’s Hospital
(5)
Brain Injury Center, Boston Children’s Hospital
(6)
Center for Health and Rehabilitation, Department of Physical Medicine and Rehabilitation, Harvard Medical School
(7)
Spaulding Rehabilitation Hospital
(8)
MassGeneral Hospital for Children™ Sport Concussion Program, and Home Base, A Red Sox Foundation and Massachusetts General Hospital Program
(9)
Priority Research Centre for Stroke and Brain Injury, Level 5, McAuley Building, Calvary Mater Hospital

References

  1. Asplund CA, McKeag DB, Olsen CH. Sport-related concussion: factors associated with prolonged return to play. Clin J Sport Med. 2004;14:339–43.View ArticlePubMedGoogle Scholar
  2. Bock S, Grim R, Barron TF, Wagenheim A, Hu YE, Hendell M, Deitch J, Deibert E. Factors associated with delayed recovery in athletes with concussion treated at a pediatric neurology concussion clinic. Childs Nerv Syst. 2015;31:2111–6.View ArticlePubMedGoogle Scholar
  3. Brown NJ, Mannix RC, O’Brien MJ, Gostine D, Collins MW, Meehan WP. Effect of cognitive activity level on duration of post-concussion symptoms. Pediatrics. 2014;133:e299–304.View ArticlePubMedPubMed CentralGoogle Scholar
  4. Chermann JF, Klouche S, Savigny A, Lefevre N, Herman S, Bohu Y. Return to rugby after brain concussion: a prospective study in 35 high level rugby players. Asian J Sports Med. 2014;5:e24042.View ArticlePubMedPubMed CentralGoogle Scholar
  5. Collins MW, Iverson GL, Lovell MR, McKeag DB, Norwig J, Maroon J. On-field predictors of neuropsychological and symptom deficit following sports-related concussion. Clin J Sport Med. 2003;13:222–9.View ArticlePubMedGoogle Scholar
  6. Davis G, Makdissi M. Use of video to facilitate sideline concussion diagnosis and management decision-making. J Sci Med Sport 2016; online first.Google Scholar
  7. Echemendia RJ, Bruce JM, Meeuwisse W, Hutchison MG, Comper P, Aubry M. Can visible signs predict concussion diagnosis in the National Hockey League? Br J Sports Med 2017; bjsports-2016-097090.Google Scholar
  8. Ellis MJ, Ritchie LJ, Koltek M, Hosain S, Cordingley D, Chu S, Selci E, Leiter J, Russell K. Psychiatric outcome after pediatric sports-related concussion. J Neurosurg Pediatr. 2015;16:709–18.View ArticlePubMedGoogle Scholar
  9. Erlanger D, Kaushik T, Cantu R, Barth JT, Broshek DK, Freeman JR, Webbe FM. Symptom-based assessment of the severity of a concussion. J Neurosurg. 2003;98:477–84.View ArticlePubMedGoogle Scholar
  10. Gardner A, Iverson GL, Levi CR, Schofield PW, Kay-Lambkin F, Kohler RMN, Stanwell P. A systematic review of concussion in rugby league. Br J Sports Med 2014; bjsports-2013-093102.Google Scholar
  11. Gardner AJ, Iverson GL, Quinn TN, Makdissi M, Levi CR, Shultz SR, Wright DK, Stanwell P. A preliminary video analysis of concussion in the national rugby league. Brain Inj. 2015;29:1182–5.View ArticleGoogle Scholar
  12. Gardner AJ, Iverson GL, Stanwell P, Ellis J, Levi CR. A video analysis of use of the new “concussion interchange rule” in the national rugby league. Int J Sports Med. 2016;Google Scholar
  13. Gardner AJ, RMN K, Levi CR, Iverson GL. Usefulness of video review of possible concussions in National Youth Rugby League. Int J Sports Med. 2016; in pressGoogle Scholar
  14. Gardner AJ, Levi CR, Iverson GL. Observational review and analysis of concussion: a method for conducting a standardised video analysis of concussion in rugby league. Sport Med - Open. 2017;3:26.View ArticleGoogle Scholar
  15. Guskiewicz KM, McCrea M, Marshall SW, Cantu RC, Randolph C, Barr W, Onate JA, Kelly JP. Cumulative effects associated with recurrent concussion in collegiate football players: the NCAA concussion study. JAMA. 2003;290:2549–55.View ArticlePubMedGoogle Scholar
  16. Guskiewicz KM, Ross SE, Marshall SW. Postural stability and neuropsychological deficits after concussion in collegiate athletes. J Athl Train. 2001;36:263–73.PubMedPubMed CentralGoogle Scholar
  17. Hallgren KA. Computing inter-rater reliability for observational data: an overview and tutorial. Tutor Quant Methods Psychol. 2012;8:23–34.View ArticlePubMedPubMed CentralGoogle Scholar
  18. Hinton-Bayre AD, Geffen G. Severity of sports-related concussion and neuropsychological test performance. Neurology. 2002;59:1068–70.View ArticlePubMedGoogle Scholar
  19. Hutchison MG, Comper P, Meeuwisse WH, Echemendia RJ. An observational method to code concussions in the National Hockey League (NHL): the heads-up checklist. Br J Sports Med. 2014;48:125–9.View ArticlePubMedGoogle Scholar
  20. IBM Corp . IBM SPSS statistics for Macintosh, version 22.0. Armonk: IBM Corp.Google Scholar
  21. Kohler R, Makdissi M, McDonald W, Partridge B, Gardner AJ. A preliminary video review of in-game head injury incidents (HII) and use of the head injury assessment (HIA) from the 2015 super rugby season. Br J Sports Med. 2017;51:A78–9.Google Scholar
  22. Lau B, Lovell MR, Collins MW, Pardini J. Neurocognitive and symptom predictors of recovery in high school athletes. Clin J Sport Med. 2009;19:216–21.View ArticlePubMedGoogle Scholar
  23. Lau BC, Kontos AP, Collins MW, Mucha A, Lovell MR. Which on-field signs/symptoms predict protracted recovery from sport-related concussion among high school football players? Am J Sports Med. 2011;39:2311–8.View ArticlePubMedGoogle Scholar
  24. Lovell MR, Collins MW, Iverson GL, Field M, Maroon JC, Cantu R, Podell K, Powell JW, Belza M, Fu FH. Recovery from mild concussion in high school athletes. J Neurosurg. 2003;98:296–301.View ArticlePubMedGoogle Scholar
  25. Makdissi M, Davis G. The reliability and validity of video analysis for the assessment of the clinical signs of concussion in Australian football. J Sci Med Sport 2016; online first.Google Scholar
  26. Makdissi M, Davis G. Using video analysis for concussion surveillance in Australian football. J Sci Med Sport 2016; online first.Google Scholar
  27. Makdissi M, Davis G, Jordan B, Patricios J, Purcell L, Putukian M. Revisiting the modifiers: how should the evaluation and management of acute concussions differ in specific groups? Br J Sports Med. 2013;47:314–20.View ArticlePubMedGoogle Scholar
  28. McCrea M, Guskiewicz K, Randolph C, Barr WB, Hammeke TA, Marshall SW, Powell MR, Woo Ahn K, Wang Y, Kelly JP. Incidence, clinical course, and predictors of prolonged recovery time following sport-related concussion in high school and college athletes. J Int Neuropsychol Soc. 2013;19:22–33.View ArticlePubMedGoogle Scholar
  29. McCrea M, Iverson GL, Echemendia RJ, Makdissi M, Raftery M. Day of injury assessment of sport-related concussion. Br J Sports Med 2013; 47: 272–284 Available from: http://www.ncbi.nlm.nih.gov/pubmed/23479484.
  30. McCrea M, Kelly JP, Randolph C, Cisler R, Berger L. Immediate Neurocognitive effects of concussion. Neurosurgery. 2002;50:1032–40.PubMedGoogle Scholar
  31. McCrory P, Meeuwisse WH, Aubry M, Cantu B, Dvorák J, Echemendia RJ, Engebretsen L, Johnston K, Kutcher JS, Raftery M, Sills A, Benson BW, Davis GA, Ellenbogen RG, Guskiewicz K, Herring SA, Iverson GL, Jordan BD, Kissick J, McCrea M, McIntosh AS, Maddocks D, Makdissi M, Purcell L, Putukian M, Schneider K, Tator CH, Turner M. Consensus statement on concussion in sport: the 4th international conference on concussion in sport held in Zurich, November 2012. Br J Sports Med. 2013;47:250–8.View ArticlePubMedGoogle Scholar
  32. McDevitt J, Tierney RT, Phillips J, Gaughan JP, Torg JS, Krynetskiy E. Association between GRIN2A promoter polymorphism and recovery from concussion. Brain Inj. 2015;29:1674–81.View ArticlePubMedGoogle Scholar
  33. McHugh ML. Interrater reliability: the kappa statistic. Biochem Med. 2012;22:276–82.View ArticleGoogle Scholar
  34. Meehan WP, Mannix RC, Stracciolini A, Elbin RJ, Collins MW. Symptom severity predicts prolonged recovery after sport-related concussion, but age and amnesia do not. J Pediatr. 2013;163:721–5.View ArticlePubMedPubMed CentralGoogle Scholar
  35. Miller JH, Gill C, Kuhn EN, Rocque BG, Menendez JY, O’Neill JA, Agee BS, Brown ST, Crowther M, Davis RD, Ferguson D, Johnston JM. Predictors of delayed recovery following pediatric sports-related concussion: a case-control study. J Neurosurg Pediatr. 2016;17:491–6.View ArticlePubMedGoogle Scholar
  36. Morgan CD, Zuckerman SL, Lee YM, King L, Beaird S, Sills AK, Solomon GS. Predictors of postconcussion syndrome after sports-related concussion in young athletes: a matched case-control study. J Neurosurgery Pediatr. 2015;15:589–98.View ArticleGoogle Scholar
  37. Nelson LD, Guskiewicz KM, Marshall SW, Hammeke T, Barr W, Randolph C, McCrea MA. Multiple self-reported concussions are more prevalent in athletes with ADHD and learning disability. Clin J Sport Med. 2016;26:120–7.View ArticlePubMedGoogle Scholar
  38. Nelson LD, Tarima S, LaRoche AA, Hammeke TA, Barr WB, Guskiewicz K, Randolph C, McCrea MA. Preinjury somatization symptoms contribute to clinical recovery after sports-related concussion. Neurology. 2016;86:1856–63.View ArticlePubMedPubMed CentralGoogle Scholar
  39. Pellman EJ, Viano DC, Casson IR, Arfken C, Powell J. Concussion in professional football: injuries involving 7 or more days out-part 5. Neurosurgery. 2004;55:1100–19.View ArticlePubMedGoogle Scholar
  40. Putukian M, Raftery M, Guskiewicz K, Herring S, Aubry M, Cantu RC, Molloy M. Onfield assessment of concussion in the adult athlete. Br J Sports Med. 2013;47:285–8.View ArticlePubMedGoogle Scholar
  41. Savage J, Hooke C, Orchard J, Parkinson R. The incidence of concussion in a professional Australian rugby league team, 1998–2012. J Sports Med 2013; online first.Google Scholar
  42. Terwilliger VK, Pratson L, Vaughan CG, Gioia GA. Additional post-concussion impact exposure may affect recovery in adolescent athletes. J Neurotrauma. 2016;33:761–5.View ArticlePubMedPubMed CentralGoogle Scholar
  43. Zuckerman SL, Yengo-Kahn AM, Buckley TA, Solomon GS, Sills AK, Kerr ZY. Predictors of postconcussion syndrome in collegiate student-athletes. Neurosurg Focus. 2016;40:E13.View ArticlePubMedGoogle Scholar

Copyright

© The Author(s). 2017