Several professional contact and collision sports have recently introduced the use of sideline video review for club medical staff to help identify and manage concussions. As such, reviewing video footage on the sideline has become increasingly relied upon to assist with improving the identification of possible injury. However, as yet, a standardized method for reviewing such video footage in rugby league has not been published. The aim of this study is to evaluate whether independent raters reliably agreed on the injury characterization when using a standardized observational instrument to record video footage of National Rugby League (NRL) concussions.
Methods
Video footage of 25 concussions were randomly selected from a pool of 80 medically diagnosed concussions from the 2013–2014 NRL seasons. Four raters (two naïve and two expert) independently viewed video footage of 25 NRL concussions and completed the Observational Review and Analysis of Concussion form for the purpose of this inter-rater reliability study. The inter-rater reliability was calculated using Cohen’s kappa (κ) and intra-class correlation (ICC) statistics. The two naïve raters and the two expert raters were compared with one another separately.
Results
A considerable number of components for the naïve and expert raters had almost perfect agreement (κ or ICC value ≥ 0.9), 9 of 22 (41%) components for naïve raters and 21 of 22 (95%) components for expert raters. For the concussion signs, however, the majority of the rating agreement was moderate (κ value 0.6–0.79); both the naïve and expert raters had 4 of 6 (67%) concussion signs with moderate agreement. The most difficult concussion sign to achieve agreement on was blank or vacant stare, which had weak (κ value 0.4–0.59) agreement for both naïve and expert raters.
Conclusions
There appears to be value in expert raters, but less value for naive raters, in using the new Observational Review and Analysis of Concussion (ORAC) Form. The ORAC Form has high inter-rater agreement for most data elements, and it can be used by expert raters evaluating video footage of possible concussion in the NRL.
Keywords
ConcussionVideo analysisInjury management
Key Points
Identifying concussion from the sideline during a match is challenging, but with the use of video out-of-view or fleeting signs may be captured and a player can be removed from play.
Having a reliable form for coding and analysing concussion can be a useful adjunct to the sideline clinical management strategy of the athletic trainer and team physician.
We present the first objective and reliable coding form for rugby league to capture the game situation, the mechanism of injury, and possible signs of concussion.
Background
Rugby league is a high-intensity collision sport [23]. The game is played continuously in two 40-min halves, and game-play involves two teams of 13 on-field players and four interchange players who may be switched in and out of the game. The published incidence rates of concussion in rugby league vary [12]; at the National Rugby League (NRL) level, medically diagnosed concussions in three clubs from the 2013 season revealed an incidence rate of 14.8 concussions per 1000 player match hours [15], while a rate of 28.3 concussion per 1000 player match hours were reported from one NRL club over a 15-year (1998–2012) period [41]. The incidence of use of the concussion interchange rule (CIR) was 24.0 (95% CI 20.7–27.9) uses of the CIR per 1000 NRL player match hours [14] and 44.9 (95% CI 38.5–52.3) uses of the CIR per 1000 National Youth Competition player match hours [15].
One method that has becoming increasingly relied upon to assist with improving the identification of possible concussion has been the review of video footage on the sideline. The use of video for reviewing a concussion may identify signs of injury that may have been blocked from view or otherwise missed by medical staff. A number of professional contact and collision sports have recently introduced the use of sideline video review for club medical staff to help identify and manage concussions [29]. A number of studies of video footage have been conducted in a variety of sports, for example, rugby league [13–15], rugby union [25], and Australian Rules Football [9, 29, 30]. Other sports, such as boxing [37], soccer [1], taekwondo [24], ice hockey [6, 10, 18, 19], and lacrosse [28], have also reported on the use of video footage for understanding the circumstances and mechanisms of injury unique to their sports. A risk prediction model among National Hockey League (NHL) players reported that both visual signs of concussion and information pertaining to mechanisms of injury improved a clinician’s ability to identify athletes who should be removed from play and evaluated [6]. Specifically, the study indicated that suspected loss of consciousness, motor incoordination or balance problems, being in a fight, having an initial hit from another player’s shoulder, and having a secondary hit on the ice were all associated with increased risk of subsequent concussion diagnosis.
Sport-specific coding criteria of concussion for game situational factors and injury mechanisms have been developed for hockey (e.g., the ‘Heads-Up Checklist’ [20]), but these criteria do not generalize to other sports like rugby union or rugby league. Video criteria and coding forms require validation in each individual sport [29, 30]. In a more recent NHL study examining the predictive ability of visual signs of concussion, loss of consciousness, motor incoordination, and blank/vacant look had a positive association with concussion diagnosis, whereas slow to get up and clutching of the head, despite occurring frequently, had low positive predictive values [10].
Several video studies have examined signs of concussion, together with player characteristics, injury characteristics, and match situational factors, in professional rugby league [13–15]. In 2014, video reviews of injury have been implemented in the NRL to help medical staff and promote player health and safety. The aim of this study was to present a standardized observational recording form and to determine whether independent raters agreed on the antecedent events, mechanisms of injury, and concussion signs when using the form to code digital video records of concussions in the NRL.
Methods
Participants
This study was conducted in the national professional rugby league competition in Australia during the 2013 and 2014 seasons. All medically diagnosed concussion events during 2013 and 2014 NRL seasons were available to be included in this study.
Procedure
For this study, 25 medically diagnosed concussions were randomly selected from the 2013 to 2014 NRL seasons’ video library (n = 80). The video library included only excerpts of the incidents for each case, not the entire game. The duration of the game footage recorded for each of the 25 cases selected from the library and used in this study ranged from 138 to 473 s. Four raters (two ‘expert’ and two ‘naïve’) independently reviewed the video footage of the 25 NRL concussion events. The naïve raters were novices of the sport. They had limited to no knowledge or experience with rugby league match play and no experience identifying and managing concussion. The expert raters were defined as individuals with experience in rugby league match play and expertise in concussion management at the professional (NRL) level. Both expert raters had at least one NRL season of experience working on the sideline for an NRL club with the responsibility of identifying and assessing athletes suspected of having sustained a concussion.
The medically diagnosed NRL concussion library was gathered from three teams during the 2013 season (n = 20 concussions) [13], and all teams during the 2014 NRL season (n = 60 concussions). The Concussion in Sport Group consensus definition of concussion was used by all clubs involved in this study [32]. Raters viewed the match digital records of 25 concussions using the Quicktime Multimedia Player V.7.7.5. Each rater completed all components of the form for each of the 25 concussion events. The raters were permitted to view the incident as many times as required and in any playback speed as deemed necessary to complete all categories of the form. All participants provided informed consent. This study was conducted in accordance with the standards of the ethics outlined in the Declaration of Helsinki. Approval of this study was provided by the University of Newcastle Human Ethics Committee.
Instruments
A rating form was created to provide a simple but standardized framework for coding and analysing video footage of the situations and consequences of concussion events in rugby league. The form was developed by a neuropsychologist with extensive experience in the sport and concussion management and was based on work conducted previously in ice hockey in North America, and a similar, but not identical, approach to validating the form was used for the validation of the ‘Heads-Up Checklist’ [20]. The form was adapted to include specific information to rugby league. Concussion signs that have been previously examined in video review studies in rugby league were included [13–15]. The form consists of various sections related to the player and game characteristics (e.g., ball carrier versus tackler, tackle height, type of play, etc.), the anatomical region of contact, the injury location on the field of play, the injured player’s on-field management, and six possible concussion signs (see Fig. 1).
Fig. 1
The Observational Review and Analysis of Concussion (ORAC) Form
Statistical Analysis
The results from the two naïve raters and the expert raters were considered separately. The intra-class correlation (ICC) was used to determine the level of agreement between the two naïve raters and the two expert raters for interval and ratio variables (i.e. ‘number of players in tackle’ and ‘time taken to leave’). Inter-rater reliability analyses using Cohen’s kappa (κ) statistics [17] were used to determine the level of agreement between the two naïve raters and the two expert raters for all other (nominal) variables. Unlike the total percent agreement, Cohen’s kappa considers the proportional agreement that could occur simply by chance. The κ coefficients are calculated by considering the proportion of rater agreement and the expected proportion [17]. Using the interpretations of κ described by McHugh [34], κ agreement was categorized as almost perfect (>.90), strong (.80–.90), moderate (.60–.79), weak (.40–.59), minimal (.21–.39) and none (0–.20). All analyses were performed using IBM SPSS Statistics V.23.0 [21] and used two-sided tests for significance at the 0.05 level, with 95% confidence intervals (CIs).
Results
The inter-rater reliabilities for the various components of the rating form for both the naïve and expert raters are presented in Table 1. For the naïve raters, 6 of 20 (30%) components of the form, and 5 of 6 (83%) concussions signs, had κ values between .60 and .79 (‘moderate’ agreement), while 2 of 2 (100%) interval/ratio variables of the form had very good ICC. According to the interpretations of κ described by McHugh [34], 8 of 22 components were categorized as ‘almost perfect’; 3 components were classified as ‘strong’; 2 components were classified as ‘weak’; 2 components were classified as ‘minimal’; and one of the components (playing position) was not classified. For the concussion signs, the naïve raters had no signs that had ‘almost perfect’ or ‘strong’ agreement; 5 (83%) signs were classified as ‘moderate’ and 1 (17%) sign was classified as ‘weak’.
Table 1
Inter-rater reliability (κ, ICC) for naïve and expert raters for each component of the form
McHugh κ Agreement Classification: almost perfect (>.90), strong (.80–.90), moderate (.60–.79), weak (.40–.59), minimal (.21–.39), and none (0–.20)
CIs confidence intervals, ICC intra-class correlation, κ kappa, N/A not applicable, NR not recorded
For the expert raters, 19 of 20 (95%) components of the form had κ values of between .90–1.00 (‘almost perfect’ agreement) and one (5%) had moderate agreement. The expert raters had perfect ICC for 2 of 2 (100%) interval/ratio variables. For the concussion signs, the expert raters had 1 of 6 (17%) of concussions signs with κ values between .80–.90 (‘strong’ agreement); 4 (67%) of were classified as ‘moderate’; and 1 (17%) was classified as ‘weak’ agreement. No signs classified by either the naïve or expert raters had a ‘minimal’ or ‘none’ level of agreement.
There were nine components that were all rated with ‘almost perfect’ agreement by both the naïve and the expert raters (game time, score, whether the concussed player was a ball carrier or a tackler, the number of players involved in the tackle, whether the offending player was placed on report by a match official, the initial contact, the region of contact, whether the player was removed from play, and how the player left the field). The level of agreement between the expert raters and between the naïve raters was also very consistent for the tackle height and whether the injury occurred as a result of foul play (i.e., the offending player was penalized). The naïve raters had a ‘strong’ level of agreement for these components. The naïve raters had a moderate level of agreement on whether the game was played during the night or day, the tackle number in the set, the anatomical location of the impact, and the location of the field where the concussion took place, whereas all of these components had an ‘almost perfect’ level of agreement between the two expert raters. The expert raters also achieved an ‘almost perfect’ level of agreement on the secondary contact, whether the concussed player had anticipated the impact that caused the injury, and the time taken to leave the field of play. However, the naïve raters only had a ‘minimal’ level of agreement on these components. For type of play, and whether or not the player returned to play, the naïve raters had a ‘weak’ agreement on these components compared to the ‘almost perfect’ agreement by the expert raters. Whether or not there was secondary contact was the most difficult component to agree upon; the expert raters’ level of agreement was ‘moderate’ for this component, and the naive raters’ level of agreement was ‘minimal’ (see Table 1).
Regarding concussion signs, slow to get up had the best level of agreement between expert and naïve raters of all possible concussion signs (strong and moderate agreement, respectively), whereas a blank or vacant stare had the worst agreement (both rater groups had a ‘weak’ level of agreement). Clutch or shake head, gait ataxia (or having wobbly legs), unresponsiveness, and post-impact seizure-like features had moderate agreement for both expert and naïve raters.
Discussion
Rugby League is a full contact collision sport that has high concussion incidence rates [13–15]. The in-game management and decision-making process surrounding concussion is a challenge. Video review is increasingly being used as one method for improving this in-game decision-making process for medical staff, although a standardized approach to the use of such information had not been published. Although there is a large body of research examining on-field markers of concussion and their association with outcome [2–5, 7, 8, 11, 16, 22, 26, 27, 31, 33, 35, 36, 38–40, 42–46], very few of these studies have been focused on possible signs of concussion at the time of injury (versus collected later as part of a questionnaire or interview with the athlete). This study presents a standardized observational form and examines intra-rater and inter-rater agreement on the antecedent events, mechanisms of injury, and concussion signs. Overall, the results of this study suggest that a certain level of knowledge about the game is required to complete the form components accurately. Expert rates achieved an ‘almost perfect’ level of agreement on 21/22 (95%) of components compared to only 9/22 (41%) components for the naïve raters.
In a similar study conducted with the ‘Heads-Up Checklist’ for National Hockey League (NHL) concussions, the naïve raters also had worse agreement across components pertaining to the antecedent events and mechanism of injury compared to the expert raters. Of the 15 components in version 1 of the Heads-Up Checklist, naïve raters 7 (47%) had weak or minimal agreement, compared to only 1 of the 15 (7%) components for the expert raters [20]. For the Heads-Up Checklist, the acceleration of the head (which was not considered a component or review item in our form) was the single component with the worst agreement across naïve and expert raters. Rating secondary contact was also challenging in the hockey study as it was in the current study. The location of the playing surface where the concussion occurred and the time in the game when the concussion occurred were the two components with the strongest agreement by naïve and expert raters for the hockey study [20]. For the current study, the time in the game was rated well. However, the location on the field did not have a high agreement for the naïve raters. The discrepancy between naïve raters for the hockey study compared to this rugby league study may have occurred for at least three reasons. First, we divided the playing surface in our study into 12 different components and the hockey study used fewer zones. Second, the hockey study designated offensive ends and defensive ends, whereas the rugby league study required the raters to record the direction of the play, and some of the disagreement between the naïve raters for the location on the field was due to the indication of the direction of the play. Finally, the hockey study used naïve raters who where more familiar with their sport (i.e., ‘individuals with limited experience who might have played or coached [ice] hockey at a competitive level’), whereas our naïve raters were complete novices, who had limited to no experience even watching the sport as fans and certainly no experience identifying concussions.
In the current study, there were a number of variables that appear to rely on knowledge, understanding, and experience with rugby league match play (i.e., the expert raters outperformed the naïve raters). For example, there were large differences between the coding by expert and naïve raters of variables such as secondary contact and anticipation of impact. There was also a large difference between the coding by expert and naïve raters on whether the player returned to play. This variable required the raters to watch the remainder of a game (following the injury) to determine if the injured athlete subsequently returned to the field of play. Interchanges can occur during play or during a stoppage in play, and they are not always announced on the broadcaster footage. It appears that the naïve raters were not as savvy in identifying the return to play of an interchanged athlete and/or did not identify the athlete as being re-involved in match play following their return to the field of play.
As with our previous video reviews of concussion signs [14, 15], we once again found that determining whether a concussed player had a blank or vacant stare was difficult to agree upon. We had weak agreement between naïve (0.44, 95% CI = 0.15–0.71) and expert (0.50, 95% CI = 0.23–0.76) raters in this study, and our previous work has also revealed difficulty with agreement between raters (i.e., 0.36 (95% CI = 0.29 to 0.43) [14] and 0.62 (95% CI = 0.37 to 0.88) [15]). In a recent Australian Football League (AFL) video review, inter-rater reliability for the blank/vacant stare on first review was reported to be 0.24 (95% CI = 0.04 to 0.41) and minimal improvements were observed on second review [0.26 (95% CI = 0.07 to 0.43)]. The intra-rater reliability in the AFL study was somewhat better for the two raters over the two rating sessions [i.e. 0.63 (95% CI = 0.50 to 0.74) and 0.36 (95% CI = 0.18 to 0.51)]. The concussion sign ‘blank/vacant stare’ was reported to have 9% sensitivity, 100% specificity, 100% positive predictive value and 58% negative predictive value in the sample of AFL concussions [29]. When the quality of the video (including the zoom capacity to see the players face) is limited, attempting to code the presence or absence of a blank or vacant stare from video is challenging [15]. This supports the notion that good-quality video from multiple camera angles are crucial for effective video surveillance of injuries [30]. In the current study, however, this was not a limitation, suggesting that it is also important to have clear definitions, including the inclusion and exclusion criteria for coding concussion signs [30]. In a recent series of video reviews of concussions from the AFL [9, 29, 30], Makdissi and Davis indicated that video review may be an avenue that facilitates the assessment of the mechanism and impact of injury and allows for the identification of brief early signs of concussion [29]. The authors suggest that video analysis may be a useful adjunct to the sideline assessment of possible concussion [29] and that the implementation of a flowchart may improve the timely assessment of concussion [9].
We recently completed a study on the frequency (or base rates) of concussion signs in NRL match play (Gardner et al., under review). That study reviewed every game (n = 201) from the 2014 NRL season, which included 127,062 tackles, and found unresponsiveness occurred 52 times [24 (46%) were diagnosed with a concussion], slow to get up occurred 2240 times [60 (3%) were diagnosed with a concussion], clutching or shaking the head occurred 361 times [38 (11%) were diagnosed with a concussion], gait ataxia occurred 102 times [35 (34%) were diagnosed with a concussion], blank or vacant stare occurred 98 times [45 (46%) were diagnosed with a concussion], and a post-impact posturing or seizure occurred 4 times [3 (75%) were diagnosed with a concussion]. The unresponsiveness sign had 40% sensitivity, 91% specificity, 46% positive predictive value, and 89% negative predictive value. The slow to get up sign had 100% sensitivity, 50% specificity, 27% positive predictive value, and 100% negative predictive value. Clutching or shaking the head had 63% sensitivity, 46% specificity, 18% positive predictive value, and 87% negative predictive value. Gait ataxia had 58% sensitivity, 79% specificity, 34% positive predictive value, and 91% negative predictive value. Blank or vacant stare had 75% sensitivity, 84% specificity, 46% positive predictive value, and 95% negative predictive value. Post-impact seizure had 5% sensitivity, 100% specificity, 75% positive predictive value, and 85% negative predictive value in the 2014 NRL season (Gardner et al., under review).
One of the unusual and unexpected findings of this study was the discrepancy observed between the naïve raters in coding variables that were conceivably thought to be obvious (e.g., game time, score, day/night game). The naïve raters did not always have 100% agreement. Because rugby league is a continuous sport, it is common for the game to continue despite an injury, and therefore, the game clock also does not stop. As such, an injury can occur well before the game and the game clock is stopped. The discrepancies in the ‘time in game’ variable are explained by this issue; one of the naïve reviewers recorded the time correctly (i.e., when the injury occurred), whereas the other naïve rater often recorded the time when the game clock was stopped. In terms of the ‘game score’ variable, it is possible that the naïve raters were unfamiliar with teams, and therefore, errors were made in coding the score of each team. For the ‘day/night game’ variable, there were a number of games that were played during twilight, as well as the footage of some of those cases being zoomed in, and the wide view did not make the day/night difference obvious to the naïve raters who do not watch NRL games.
Video review appears to be a useful adjunct to traditional methods for making in-game decisions pertaining to the identification of potential concussion (and an athlete subsequently being removed from play). However, to better understand and quantify the value of this process, future research should be conducted under time limits and/or during a game to replicate the real-world/practical pressure, neither of which was replicated in this study. Future studies might focus on whether agreement between experts improves under ‘ideal circumstances’ (i.e. as many reviews as required without time limitations) versus ‘real-world circumstances’ (i.e. a quick decision required to identify a possible injury and immediately remove the athlete from play).
The current study has several limitations. Firstly, clubs used their own personnel and methods for identifying possible injuries on the field and diagnosing concussions on the sideline, which presumably makes the final specific criteria for a ‘medically diagnosed concussion’ variable across clubs. The current study does not generalize to the real-world use of in-game video analysis because the study was not conducted under the time pressure associated with in-game decision-making. Further, the sample size is small, and only two naïve and two expert reviewers were used. Whether the current results hold true for more cases and a greater number of raters is unknown.
Conclusions
The present study suggests that determining the presence or absence of a blank or vacant stare is challenging for both naïve and expert raters to rate reliably, but that showing unresponsiveness (i.e. possible LOC), clutching or shaking of the head, a post-impact seizure, or being slow to get up are more reliably rated signs. However, in this study, there was no variability in the clinical outcome measure, as our sample came from a pool of individuals who were all medically diagnosed with a concussion. Therefore, the predictive value of any one component or concussion sign, or a combination of these items, is unknown and may be the focus of future research. Given the variability of in-game decision-making in professional rugby league [13–15], we sought to provide validation of a standardized approach for collecting information surrounding possible concussions to help inform the in-game decision-making process. Although the form was created for all levels of competition, it only had a good level of agreement among experienced raters. Therefore, it might only be useful for those teams or clubs that have experts available to them (i.e., the professional level). For lower levels of competition, the form may have less of a benefit, because the naïve raters had a low level of agreement on many components of the form. It is important to note, however, that the management of suspected concussion at these lower levels should always be conservative. If a concussion is suspected, then the athlete should be removed from play and not returned to play the same day [32]. At the professional level, data collected from this form may allow for a thorough understanding of the situational and contextual factors related to concussion, which may be used to strategize future interventions to reduce the risk of concussion at this level.
Declarations
Acknowledgements
The authors would like to thank Mrs. Kathryn Gurr and Ms. Vanessa Case (University of Newcastle, School of Medicine and Public Health) for completing the video review as the nominated naïve raters and Dr. Jinho Lee (School of Human Movement and Nutrition Sciences, University of Queensland) for completing the review as an expert rater.
Funding
The funding was provided by Hunter Medical Research Institute (HMRI) supported by Anne Greaves.
Financial Disclosure
None.
Authors’ Contributions
AG conceived the design of the study, designed the data collection tool, collected the video footage, liaised with the other raters to complete the data collection and monitored the data collection, managed the study database, conducted the statistical analysis, and drafted, revised, and finalized the manuscript. He had the final veto on the submission. CL assisted with the design of the study and provided the editorial comment to the drafts of the manuscript prepared for the submission. GI assisted with the design of the study, assisted with the statistical analysis of the data, provided expert editorial comment for all the drafts of the manuscript, and had the final veto on the submission. All authors read and approved the final manuscript.
Competing Interests
Andrew Gardner is an early career fellow with the National Health and Medical Research Council (NHMRC) and is supported by the School of Medicine and Public Health, University of Newcastle, and the Priority Research Centre for Stroke and Brain Injury, School of Medicine and Public Health, University of Newcastle. He has a clinical practice in neuropsychology involving individuals who have sustained sport-related concussion (including current and former athletes). He has operated as a contracted concussion consultant to the Australian Rugby Union (ARU) from July 2016. He has received travel funding from the Australian Football League (AFL) to present at the Concussion in Football Conference in 2013 and 2017. The previous grant funding includes the NSW Sporting Injuries Committee, the Brain Foundation (Australia), and the Hunter Medical Research Institute (HMRI), supported by Jennie Thomas. He is currently funded through the HMRI, supported by Anne Greaves, and the University of Newcastle’s Priority Research Centre for Stroke and Brain Injury. Christopher Levi declares that he has no conflict of interest. Grant Iverson has been reimbursed by the government, professional scientific bodies, and commercial organizations for discussing or presenting research relating to mild TBI and sport-related concussion at meetings, scientific conferences, and symposiums. He has a clinical and consulting practice in forensic neuropsychology involving individuals who have sustained mild TBIs (including athletes). He has received research funding from several test publishing companies, including ImPACT Applications, Inc., CNS Vital Signs, and Psychological Assessment Resources (PAR, Inc.). He received past salary support from the Harvard Integrated Program to Protect and Improve the Health of National Football League Players Association Members. He acknowledges unrestricted philanthropic support from the Mooney-Reed Charitable Foundation and ImPACT Applications, Inc.
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
Authors’ Affiliations
(1)
Centre for Stroke and Brain Injury, School of Medicine and Public Health, University of Newcastle
(2)
Hunter New England Local Health District Sports Concussion Program, John Hunter Hospital
(3)
Department of Physical Medicine and Rehabilitation, Harvard Medical School, Spaulding Rehabilitation Hospital
(4)
MassGeneral Hospital for Children™ Sport Concussion Program
(5)
Home Base, A Red Sox Foundation and Massachusetts General Hospital Program
References
Andersen TE, Larsen Ø, Tenga A, Engebretsen L, Bahr R. Football incident analysis: a new video based method to describe injury mechanisms in professional football. Br J Sports Med. 2003;37:226–32.View ArticlePubMedPubMed CentralGoogle Scholar
Asplund CA, McKeag DB, Olsen CH. Sport-related concussion: factors associated with prolonged return to play. Clin J Sport Med. 2004;14:339–43.View ArticlePubMedGoogle Scholar
Baker JG, Leddy JJ, Darling SR, Rieger BP, Mashtare TL, Sharma T, Willer BS. Factors associated with problems for adolescents returning to the classroom after sport-related concussion. Clin Pediatr (Phila). 2015;54:961–8.View ArticleGoogle Scholar
Barlow M, Schlabach D, Peiffer J, Cook C. Differences in change scores and the predictive validity of three commonly used measures following concussion in the middle school and high school aged population. Int J Sports Phys Ther. 2011;6:150–7.PubMedPubMed CentralGoogle Scholar
Berz K, Divine J, Foss KB, Heyl R, Ford KR, Myer GD. Sex-specific differences in the severity of symptoms and recovery rate following sports-related concussion in young athletes. Physician Sport Med. 2013;41:58–63.View ArticleGoogle Scholar
Bruce JM, Echemendia RJ, Meeuwisse W, Hutchison MG, Aubry M, Comper P. Development of a risk prediction model among professional hockey players with visible signs of concussion. Br J Sports Med. 2017; bjsports-2016-097091Google Scholar
Chermann JF, Klouche S, Savigny A, Lefevre N, Herman S, Bohu Y. Return to rugby after brain concussion: a prospective study in 35 high level rugby players. Asian J Sports Med. 2014;5:e24042.View ArticlePubMedPubMed CentralGoogle Scholar
Collins MW, Iverson GL, Lovell MR, McKeag DB, Norwig J, Maroon J. On-field predictors of neuropsychological and symptom deficit following sports-related concussion. Clin J Sport Med. 2003;13:222–9.View ArticlePubMedGoogle Scholar
Davis G, Makdissi M. Use of video to facilitate sideline concussion diagnosis and management decision-making. J Sci Med Sport. 2016; online first.Google Scholar
Echemendia RJ, Bruce JM, Meeuwisse W, Hutchison MG, Comper P, Aubry M. Can visible signs predict concussion diagnosis in the National Hockey League? Br J Sports Med. 2017; bjsports-2016-097090.Google Scholar
Ellis MJ, Ritchie LJ, Koltek M, Hosain S, Cordingley D, Chu S, Selci E, Leiter J, Russell K. Psychiatric outcome after pediatric sports-related concussion. J Neurosurg Pediatr. 2015;16:709–18.View ArticlePubMedGoogle Scholar
Gardner A, Iverson GL, Levi CR, Schofield PW, Kay-Lambkin F, Kohler RMN, Stanwell P. A systematic review of concussion in rugby league. Br J Sports Med. 2015;49:495–8.View ArticlePubMedGoogle Scholar
Gardner AJ, Iverson GL, Quinn TN, Makdissi M, Levi CR, Shultz SR, Wright DK, Stanwell P. A preliminary video analysis of concussion in the National Rugby League. Brain Inj. 2015;29:1182–5.View ArticleGoogle Scholar
Gardner AJ, Iverson GL, Stanwell P, Ellis J, Levi CR. A video analysis of use of the new “Concussion Interchange Rule” in the National Rugby League. Int J Sports Med. 2016;37(4):267–3.Google Scholar
Gardner AJ, Kohler RMN, Levi CR, Iverson GL. Usefulness of video review of possible concussions in National Youth Rugby League. Int J Sports Med. 2016; in press.Google Scholar
Guskiewicz KM, McCrea M, Marshall SW, Cantu RC, Randolph C, Barr W, Onate JA, Kelly JP. Cumulative effects associated with recurrent concussion in collegiate football players: the NCAA concussion study. JAMA. 2003;290:2549–55.View ArticlePubMedGoogle Scholar
Hallgren KA. Computing inter-rater reliability for observational data: an overview and tutorial. Tutor Quant Methods Psychol. 2012;8:23–34.View ArticlePubMedPubMed CentralGoogle Scholar
Hutchison MG, Comper P, Meeuwisse WH, Echemendia RJ. A systematic video analysis of National Hockey League (NHL) concussions, part I: who, when, where and what? Br J Sports Med. 2015;49(8):547–51.Google Scholar
Hutchison MG, Comper P, Meeuwisse WH, Echemendia RJ. A systematic video analysis of National Hockey League (NHL) concussions, part II: how concussions occur in the NHL. Br J Sports Med. 2015;49(8):552–5.Google Scholar
Hutchison MG, Comper P, Meeuwisse WH, Echemendia RJ. An observational method to code concussions in the National Hockey League (NHL): the heads-up checklist. Br J Sports Med. 2014;48:125–9.View ArticlePubMedGoogle Scholar
IBM Corp. IBM SPSS Statistics for Macintosh, Version 22.0. Armonk, NY: IBM Corp. 2013.Google Scholar
Kerr ZY, DeFreese JD, Marshall SW. Current physical and mental health of former collegiate athletes. Orthop J Sport Med. 2014;2:1–9.View ArticleGoogle Scholar
King DA, Hume PA, Milburn PD, Guttenbeil D. Match and training injuries in rugby league: a review of published studies. Sport Med. 2010;40:163–78.View ArticleGoogle Scholar
Koh JO, Watkinson EJ, Yoon Y-J. Video analysis of head blows leading to concussion in competition Taekwondo. Brain Inj. 2004;18:1287–96.View ArticlePubMedGoogle Scholar
Kohler R, Makdissi M, McDonald W, Partridge B, Gardner AJ. A preliminary video review of in-game head injury incidents (HII) and use of the head injury assessment (HIA) from the 2015 super rugby season. Br J Sports Med. 2017;51:A78–A79.Google Scholar
Lau B, Lovell MR, Collins MW, Pardini J. Neurocognitive and symptom predictors of recovery in high school athletes. Clin J Sport Med. 2009;19:216–21.View ArticlePubMedGoogle Scholar
Lau BC, Kontos AP, Collins MW, Mucha A, Lovell MR. Which on-field signs/symptoms predict protracted recovery from sport-related concussion among high school football players? Am J Sports Med. 2011;39:2311–8.View ArticlePubMedGoogle Scholar
Lincoln AE, Caswell SV, Almquist JL, Dunn RE, Hinton RY. Video incident analysis of concussions in boys’ high school lacrosse. Am J Sports Med. 2013;41:756–61.View ArticlePubMedGoogle Scholar
Makdissi M, Davis G. The reliability and validity of video analysis for the assessment of the clinical signs of concussion in Australian football. J Sci Med Sport. 2016; online first.Google Scholar
Makdissi M, Davis G. Using video analysis for concussion surveillance in Australian football. J Sci Med Sport. 2016; online first.Google Scholar
McCrea M, Kelly JP, Randolph C, Cisler R, Berger L. Immediate neurocognitive effects of concussion. Neurosurgery. 2002;50:1032–40.PubMedGoogle Scholar
McCrory P, Meeuwisse WH, Aubry M, Cantu B, Dvorák J, Echemendia RJ, Engebretsen L, Johnston K, Kutcher JS, Raftery M, Sills A, Benson BW, Davis GA, Ellenbogen RG, Guskiewicz K, Herring SA, Iverson GL, Jordan BD, Kissick J, McCrea M, McIntosh AS, Maddocks D, Makdissi M, Purcell L, Putukian M, Schneider K, Tator CH, Turner M. Consensus statement on concussion in sport: the 4th International Conference on Concussion in Sport held in Zurich, November 2012. Br J Sports Med. 2013;47:250–8.View ArticlePubMedGoogle Scholar
McDevitt J, Tierney RT, Phillips J, Gaughan JP, Torg JS, Krynetskiy E. Association between GRIN2A promoter polymorphism and recovery from concussion. Brain Inj. 2015;29:1674–81.View ArticlePubMedGoogle Scholar
Meehan WP, Mannix RC, Stracciolini A, Elbin RJ, Collins MW. Symptom severity predicts prolonged recovery after sport-related concussion, but age and amnesia do not. J Pediatr. 2013;163:721–5.View ArticlePubMedPubMed CentralGoogle Scholar
Meehan WP, Monuteaux MC. Early symptom burden predicts recovery after sport-related concussion. Nerurology. 2014;83:2204–10.View ArticleGoogle Scholar
Miele VJ, Bailes JE. Objectifying when to halt a boxing match: a video analysis of fatalities. Neurosurgery. 2007; 60: 307-15-6.Google Scholar
Moor HM, Eisenhauer RC, Killian KD, Proudfoot N, Henriques AA, Congeni JA, Reneker JC. The relationship between adherence behaviors and recovery time in adolescents after a sports-related concussion: an observational study. Int J Sports Phys Ther. 2015;10:225–33.PubMedPubMed CentralGoogle Scholar
Morgan CD, Zuckerman SL, Lee YM, King L, Beaird S, Sills AK, Solomon GS. Predictors of postconcussion syndrome after sports-related concussion in young athletes: a matched case-control study. J Neurosurgery Pediatr. 2015;15:589–98.View ArticleGoogle Scholar
Nelson LD, Guskiewicz KM, Marshall SW, Hammeke T, Barr W, Randolph C, McCrea MA. Multiple self-reported concussions are more prevalent in athletes with ADHD and learning disability. Clin J Sport Med. 2016;26:120–7.View ArticlePubMedGoogle Scholar
Savage J, Hooke C, Orchard J, Parkinson R. The incidence of concussion in a professional Australian rugby league team, 1998–2012. J Sports Med. 2013; online first.Google Scholar
Vargas G, Rabinowitz A, Meyer J, Arnett PA. Predictors and prevalence of postconcussion depression symptoms in collegiate athletes. J Athl Train. 2015;50:250–5.View ArticlePubMedPubMed CentralGoogle Scholar
Zuckerman SL, Apple RP, Odom MJ, Lee YM, Solomon GS, Sills AK. Effect of sex on symptoms and return to baseline in sport-related concussion. J Neurosurg Pediatr. 2014;13:72–81.View ArticlePubMedGoogle Scholar
Zuckerman SL, Solomon GS, Forbes JA, Haase RF, Sills AK, Lovell MR. Response to acute concussive injury in soccer players: is gender a modifying factor? J Neurosurg Pediatr. 2012;10:504–10.View ArticlePubMedGoogle Scholar
Zuckerman SL, Yengo-Kahn AM, Buckley TA, Solomon GS, Sills AK, Kerr ZY. Predictors of postconcussion syndrome in collegiate student-athletes. Neurosurg Focus. 2016;40:E13.View ArticlePubMedGoogle Scholar
