If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
Communications should be addressed to: Dr. Ahrens; Division of Neurology; Department of Pediatrics; The Ohio State University and Nationwide Children’s Hospital; 700 Children’s Drive; Columbus, OH 43205.
Electroencephalography (EEG) technologists commonly screen continuous EEG. Until now, the inter-rater agreement or sensitivity for important EEG findings has been unknown in this group.
Twenty-nine EEG technologists and three clinical neurophysiologists interpreted 90 five-minute samples of pediatric critical care EEG. Inter-rater agreement was examined with Cohen’s kappa and Fleiss’ kappa for EEG findings. A gold-standard consensus agreement was developed for examining sensitivity and specificity for seizures or discontinuity. Kruskal-Wallis tests with Benjamani-Hochberg corrections for multiple comparisons were utilized to examine associations between correct scoring and certification status and years of experience.
Aggregate agreement was moderate for seizures and fair for EEG background continuity among EEG technologists. Individual agreement for seizures and continuity varied from slight to substantial. For individual EEG technologists, sensitivity for seizures ranged from 44 to 93% and sensitivity for continuity ranged from 81 to 100%. Raters with Certified Long Term Monitoring credentials were more likely to identify seizures correctly.
This is the first study to evaluate inter-rater agreement and interpretation correctness among EEG technologists interpreting pediatric critical care EEG. EEG technologists demonstrated better aggregate agreement for seizure detection than other EEG findings, yet individual results and internal consistency varied widely. These data provide important insight into the common practice of utilizing EEG technologists for screening critical care EEG.
Despite standardized terms, EEG remains primarily a qualitative test with imperfect inter-rater agreement. For example, a large single-center study found aggregated agreement for overall EEG interpretation was moderate (kappa (κ) = 0.44) and paired inter-rater agreement was fair to substantial (κ = 0.29 to 0.62) among experienced readers.
Inter-rater agreement for interpreting critical care EEG (CCEEG) varies among board-certified clinical neurophysiologists. The American Clinical Neurophysiology Society (ACNS) CCEEG terminology demonstrated relatively high inter-rater agreement for most terms within single-center groups
However, no published data exist regarding inter-rater agreement or sensitivity for detecting seizures among EEG technologists. The goal of this study was to assess the inter-rater agreement, sensitivity, and specificity of EEG technologists for non-neonate critical care EEG at a National Association of Epilepsy Centers level 4 pediatric epilepsy center using the 2012 ACNS Critical Care EEG Terminology.
We retrospectively collected 90 five-minute sample pediatric video EEG of children and adolescents ranging from more than 48 weeks postmenstrual age to less than 19 years of age. The studies were recorded between January 1, 2016 and December 31, 2016. The clips were selected to represent a variety of EEG backgrounds, interictal findings and seizure types; they were reviewed using commercially available digital EEG review software. All participants in the study were unaware of any patient information, and the exact samples had not been previously viewed by raters. The primary study goal was to assess Cohen’s kappa for seizures with a significance level of 0.05; thus, a sample size of 90 was selected based on a pretest sample size calculation for a seizure frequency of 30%.
Twenty-nine EEG technologists and three clinical neurophysiologists interpreted each sample. Thirteen technologists were certified by the American Board of Registration of Electroencephalographic and Evoked Potential Technologists (ABRET): nine with Registered Electroencephalographic Technologist (R.EEG.T) certification and four with Certified Long Term Monitoring (CLTM) credentials. Sixteen technologists were not ABRET certified but had completed an associate’s degree and one year of hands-on training in pediatric EEG. All noncertified EEG technologists employed at the time of data collection were in various stages of preparing for certification examinations. Raters were asked to provide the number of years of experience working with EEG. Technologist experience, primarily in pediatric EEG, ranged from one to 39 years (mean: 9, median: 4), and all were actively involved in screening pediatric CCEEG at our institution. Neurophysiologists had completed formal fellowships in clinical neurophysiology, were board-certified in clinical neurophysiology and epilepsy, and had five to 13 (mean: 10, median: 12) years of experience in EEG interpretation.
American Clinical Neurophysiology Society terminology for critical care EEG was utilized.
: EEG background, seizures, seizure onset location, focal slowing, interictal epileptiform discharges, interictal epileptiform discharges location, rhythmic/periodic pattern present, rhythmic/periodic pattern type, and pattern location. All raters completed each data field, and clinical information other than patient age was not provided.
Interobserver agreement was measured in aggregate, with Fleiss’ kappa and pairwise agreement for two primary outcomes (EEG background and seizures) was calculated using Cohen’s kappa. Agreement was classified as slight (0-0.20), fair (0.21-0.40), moderate (0.41-0.60), substantial (0.61-0.80), or almost perfect (0.81-1.00).
One feature of kappa is the dependence on marginal probabilities; a variable with very uneven distribution may have a relatively low kappa even if percent agreement is high. Therefore, frequencies of each variable are also provided, to show which variables have uneven distributions (Table 1). Gwet’s AC1 statistic was considered for use instead of the kappa statistic.
A gold standard was defined by the consensus decision of three board-certified clinical neurophysiologists and was used to calculate sensitivity and specificity for each rater on the two primary outcomes, background continuity, and seizures. Continuity of the EEG background had four response categories, so in order to determine sensitivity and specificity, it was dichotomized in two different ways: continuous versus any other response, and continuous or discontinuous versus burst suppression or suppression. Years of experience was categorized based on quartiles of experience, as follows: one to two years, three to four years, five to 12 years, and more than 12 years. Sensitivity and specificity were compared by certification and years of experience using Kruskal-Wallis tests with Benjamani-Hochberg corrections for multiple comparisons. All analyses were conducted using R for Statistical Computing or SPSS. The Nationwide Children’s Hospital Institutional Review Board deemed this work exempt.
Data and the complete study protocol will be available for qualified individuals after approval of the institutional review board and obtaining Data Use Agreements between the involved institutions.
Of the 90 five-minute samples, 29 (32.2%) contained seizures and the background was continuous in 75 (83.3%), discontinuous in two (2.2%), burst suppression in six (6.7%), and suppression in seven (7.8%). The frequency of other interictal findings, such as focal slowing, sporadic interictal epileptiform discharges, and rhythmic/periodic patterns, is detailed in Table 1 as scored by individuals, although a consensus was not defined for each category.
Overall agreement for several main EEG finding categories was measured with Fleiss’ kappa and is displayed in Table 2. The variables with better overall agreement among EEG technologists were seizures (moderate agreement) and EEG background continuity (fair agreement). Focal slowing and rhythmic or periodic patterns had poor interobserver agreement. In comparing certified technologists to those without certification, Fleiss’ kappa for seizure onset location and interictal epileptiform discharge location differed between groups.
TABLE 2Fleiss’ Kappa for Rater Groups for 90 CCEEG Samples
Pairwise agreement for seizures and EEG background continuity was measured with Cohen’s kappa and is displayed in Fig 1 (noncertified technologists) and Fig 2 (certified technologists). Inter-rater agreement varied widely among EEG technologists and ranged from slight to substantial.
We examined the sensitivity and specificity for correctly detecting seizures or EEG background discontinuity, two of the more clinically important features of CCEEG interpretation. Sensitivity for seizure detection was variable among EEG technologists, ranging from 44% to 93%. Background continuity sensitivity was determined by assessing continuity or any other option (discontinuous, burst suppression, suppression), as this was determined to be a clinically relevant cutoff. Continuity sensitivity was also widely variable among EEG technologists, ranging from 81% to 100%. Overall specificity for seizure or EEG background discontinuity was similar across groups. We then examined the relationships between these findings and EEG technologist certification status or years of experience.
Twenty-nine raters were included in the analysis. Median (interquartile range) years of experience was four (two to 12). Fifty-five percent of the raters had no certification, 14% had CLTM training, and 31% had R.EEG.T training. Years of experience was strongly associated with certification; all but two R.EEG.T raters had more than 12 years of experience, most raters with no certification had one to four years of experience, and all CLTM raters had five to 12 years of experience. There were no significant differences in seizure or continuity sensitivity or specificity by experience (Table 3; Fig 3).
TABLE 3Sensitivity and Specificity for Seizure Detection and EEG Background Continuity by Years of Experience
Raters with CLTM certification had higher seizure sensitivity than R.EEG.T raters (P = 0.08) or those without certification (P = 0.08). Seizure sensitivity was similar among those without certification versus R.EEG.T (P = 0.59). Seizure specificity differed significantly by certification, but was high for all three groups. Specificity was higher among R.EEG.T raters than among those without certification (P = 0.05) and was higher among those with CLTM certification than among those without certification (but P = 0.23 due to greater variability), and there was no difference in seizure specificity between R.EEG.T and CLTM raters (P = 0.94). There were no other significant differences by certification (Table 4).
TABLE 4Sensitivity and Specificity for Seizure Detection and EEG Background Continuity by Certification Status
This is the first study to evaluate EEG technologists’ inter-rater agreement as well as sensitivity and specificity for detecting seizures in pediatric CCEEG. These data are vital, as EEG technologists commonly screen CCEEG. We focused our study on two EEG features of greatest clinical impact for screening CCEEG: seizures and EEG background continuity.
Seizure identification by individuals varied widely. Perhaps the most important finding was sensitivity for detecting seizures, which ranged from 44% to 93%. EEG technologists with CLTM certification were more sensitive than R.EEG.T raters or those without certification. It is unclear if this reflects the process of earning CLTM certification or those who seek CLTM are better EEG interpreters. Greater years of experience did not predict enhanced seizure detection. Experience as an isolated predictor for correctness likely suffers from the heterogeneity of training, exposure, and feedback for EEG technologists.
The aggregate agreement between EEG technologists was moderate (κ = 0.51), including for identifying seizure onset location as focal or generalized (κ = 0.40). In comparison, a previous study demonstrated seizure identification agreement as κ = 0.5 in neurologists and κ = 0.29 in neurology residents with various levels of experience.
Agreement with a paired clinical neurophysiologist ranged from fair to almost perfect (κ = 0.26 to 0.82). Therefore, our data reveal aggregate agreement that was within the published range of neurologists, but individual agreement with a neurophysiologist was widely variable.
Sensitivity for EEG background continuity varied from 81% to 100%, with specificity from 25% to 100%. This reflects better ability to detect when a background is continuous (sensitivity), but worse ability to detect when a background was discontinuous. The fair (κ = 0.37) aggregate agreement among EEG technologists using ACNS CCEEG terminology to identify discontinuity, burst suppression, or suppression of the EEG background was less robust than agreement on seizures. This is lower than previously reported substantial agreement (κ = 0.69 to 0.79) for pediatric CCEEG among clinical neurophysiologists.
A possible contributor to the lower kappa among EEG technologists may be the uneven distribution of observations between categories. However, the scoring categories of clinical neurophysiologists were also unevenly distributed, yet the kappa among this group was substantially higher. Therefore, we believe the lower inter-rater agreement among EEG technologists is an accurate assessment of agreement within this study.
Other EEG findings demonstrated worse aggregate agreement among EEG technologists. As defined by the ACNS CCEEG terminology, findings such as focal slowing, sporadic interictal epileptiform discharges, and rhythmic or periodic patterns had lower kappa than published data from neurophysiologists demonstrating moderate agreement in children
and recent reimbursement changes enacted by the Center for Medicaid and Medicare Services have further incentivized continuous monitoring. Due to increasing demand, many EEG laboratories employ EEG technologists with varying levels of certification. Currently, ABRET certifications are attained through multiple pathways without a standard curriculum. EEG technologists work while collecting EEG cases for certification.
This study has several noteworthy strengths. These are the first published data systematically evaluating interpretation of pediatric CCEEG by EEG technologists. We evaluated clinically actionable EEG variables: sensitivity and specificity for seizures and discontinuity. Additionally, we utilized ACNS-standardized CCEEG terminology for comparison to other groups. Samples of EEG were representative of the various EEG findings in this population.
We believe the combination of noncertified and certified technologists is reflective of current practice.
Our findings are limited due to the single-center data sample. Therefore, specific kappa values may be less generalizable between institutions. Additionally, we utilized short clips of CCEEG, which may limit an interpreter’s ability to differentiate specific patterns.
This study prepresents the first published data on inter-rater agreement for pediatric CCEEG among EEG technologists. Our data suggest CLTM certification may be related to improved reliability of seizure detection. However, neither years of experience nor certification status influenced EEG background interpretation. Implementation of a focused educational curriculum targeting seizure identification and background discontinuity may strengthen technologist performance. Future studies should focus on the generalizability of these findings across centers and interventions for improving EEG technologist pediatric CCEEG interpretation.
The authors thank the EEG technologists and staff at Nationwide Children’s Hospital for their enthusiasm for learning and dedication to the patients.
Statistical Analysis: conducted by Melissa Moore-Clingenpeel, Nationwide Children’s Hospital.
Funding: The project described was supported by Award Number UL1RR025755 from the National Center for Research Resources . The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Center for Research Resources or the National Institutes of Health.