Advertisement

Electroencephalography Technologist Inter-rater Agreement and Interpretation of Pediatric Critical Care Electroencephalography

      Abstract

      Objectives

      Electroencephalography (EEG) technologists commonly screen continuous EEG. Until now, the inter-rater agreement or sensitivity for important EEG findings has been unknown in this group.

      Methods

      Twenty-nine EEG technologists and three clinical neurophysiologists interpreted 90 five-minute samples of pediatric critical care EEG. Inter-rater agreement was examined with Cohen’s kappa and Fleiss’ kappa for EEG findings. A gold-standard consensus agreement was developed for examining sensitivity and specificity for seizures or discontinuity. Kruskal-Wallis tests with Benjamani-Hochberg corrections for multiple comparisons were utilized to examine associations between correct scoring and certification status and years of experience.

      Results

      Aggregate agreement was moderate for seizures and fair for EEG background continuity among EEG technologists. Individual agreement for seizures and continuity varied from slight to substantial. For individual EEG technologists, sensitivity for seizures ranged from 44 to 93% and sensitivity for continuity ranged from 81 to 100%. Raters with Certified Long Term Monitoring credentials were more likely to identify seizures correctly.

      Significance

      This is the first study to evaluate inter-rater agreement and interpretation correctness among EEG technologists interpreting pediatric critical care EEG. EEG technologists demonstrated better aggregate agreement for seizure detection than other EEG findings, yet individual results and internal consistency varied widely. These data provide important insight into the common practice of utilizing EEG technologists for screening critical care EEG.

      Keywords

      Introduction

      Continuous electroencephalography (EEG) monitoring is recommended in the intensive care setting for diverse indications.
      • Herman S.T.
      • Abend N.S.
      • Bleck T.P.
      • et al.
      Consensus statement on continuous EEG in critically ill adults and children, part I: indications.
      Identification of seizures and high-risk EEG background patterns may influence seizure treatment and inform prognosis.
      • Ostendorf A.P.
      • Hartman M.E.
      • Friess S.H.
      Early electroencephalographic findings correlate with neurologic outcome in children following cardiac arrest.
      • Abend N.S.
      • Dlugos D.J.
      • Hahn C.D.
      • et al.
      Use of EEG monitoring and management of non-convulsive seizures in critically ill patients: a survey of neurologists.
      • Brophy G.M.
      • Bell R.
      • Claassen J.
      • et al.
      Guidelines for the evaluation and management of status epilepticus.
      • Herman S.T.
      • Abend N.S.
      • Bleck T.P.
      • et al.
      Consensus statement on continuous EEG in critically ill adults and children, part II: personnel, technical specifications, and clinical practice.
      • Hughes J.R.
      Periodic lateralized epileptiform discharges: do they represent an ictal pattern requiring treatment?.
      Standardized terminology has been developed in order to facilitate consistent interpretation and communication.
      • Hirsch L.J.
      • LaRoche S.M.
      • Gaspard N.
      • et al.
      American Clinical Neurophysiology Society’s standardized critical care EEG terminology: 2012 version.
      Despite standardized terms, EEG remains primarily a qualitative test with imperfect inter-rater agreement. For example, a large single-center study found aggregated agreement for overall EEG interpretation was moderate (kappa (κ) = 0.44) and paired inter-rater agreement was fair to substantial (κ = 0.29 to 0.62) among experienced readers.
      • Grant A.C.
      • Abdel-Baki S.G.
      • Weedon J.
      • et al.
      EEG interpretation reliability and interpreter confidence: a large single-center study.
      More specifically, agreement on seizure detection ranges widely from fair to very good (κ = 0.29 to 0.91).
      • Ronner H.E.
      • Ponten S.C.
      • Stam C.J.
      • et al.
      Inter-observer variability of the EEG diagnosis of seizures in comatose patients.
      ,
      • Gaspard N.
      • Hirsch L.J.
      • LaRoche S.M.
      • et al.
      Interrater agreement for critical care EEG terminology.
      Inter-rater agreement for interpreting critical care EEG (CCEEG) varies among board-certified clinical neurophysiologists. The American Clinical Neurophysiology Society (ACNS) CCEEG terminology demonstrated relatively high inter-rater agreement for most terms within single-center groups
      • Gerber P.A.
      • Chapman K.E.
      • Chung S.S.
      • et al.
      Interobserver agreement in the interpretation of EEG patterns in critically ill adults.
      ,
      • Mani R.
      • Arif H.
      • Hirsch L.J.
      • et al.
      Interrater reliability of ICU EEG research terminology.
      and a multicenter CCEEG research group.
      • Gaspard N.
      • Hirsch L.J.
      • LaRoche S.M.
      • et al.
      Interrater agreement for critical care EEG terminology.
      However, identification of specific CCEEG findings was notably more variable in separate studies of clinical neurophysiologists’ agreement on CCEEG interpretation.
      • Halford J.J.
      • Shiau D.
      • Desrochers J.A.
      • et al.
      Inter-rater agreement on identification of electrographic seizures and periodic discharges in ICU EEG recordings.
      • Abend N.S.
      • Gutierrez-Colina A.
      • Zhao H.
      • et al.
      Interobserver reproducibility of electroencephalogram interpretation in critically ill children.
      • Wusthoff C.J.
      • Sullivan J.
      • Glass H.C.
      • et al.
      Interrater agreement in the interpretation of neonatal electroencephalography in hypoxic-ischemic encephalopathy.
      • Abend N.S.
      • Massey S.L.
      • Fitzgerald M.
      • et al.
      Interrater agreement of EEG interpretation after pediatric cardiac arrest using standardized critical care EEG terminology.
      Electroneurodiagnostic technologists are increasingly called upon to screen continuous CCEEG recorded in the critical care setting.
      • Gavvala J.
      • Abend N.
      • LaRoche S.
      • et al.
      Continuous EEG monitoring: a survey of neurophysiologists and neurointensivists.
      However, no published data exist regarding inter-rater agreement or sensitivity for detecting seizures among EEG technologists. The goal of this study was to assess the inter-rater agreement, sensitivity, and specificity of EEG technologists for non-neonate critical care EEG at a National Association of Epilepsy Centers level 4 pediatric epilepsy center using the 2012 ACNS Critical Care EEG Terminology.
      • Hirsch L.J.
      • LaRoche S.M.
      • Gaspard N.
      • et al.
      American Clinical Neurophysiology Society’s standardized critical care EEG terminology: 2012 version.

      Methods

      Electroencephalography samples

      We retrospectively collected 90 five-minute sample pediatric video EEG of children and adolescents ranging from more than 48 weeks postmenstrual age to less than 19 years of age. The studies were recorded between January 1, 2016 and December 31, 2016. The clips were selected to represent a variety of EEG backgrounds, interictal findings and seizure types; they were reviewed using commercially available digital EEG review software. All participants in the study were unaware of any patient information, and the exact samples had not been previously viewed by raters. The primary study goal was to assess Cohen’s kappa for seizures with a significance level of 0.05; thus, a sample size of 90 was selected based on a pretest sample size calculation for a seizure frequency of 30%.

      Raters

      Twenty-nine EEG technologists and three clinical neurophysiologists interpreted each sample. Thirteen technologists were certified by the American Board of Registration of Electroencephalographic and Evoked Potential Technologists (ABRET): nine with Registered Electroencephalographic Technologist (R.EEG.T) certification and four with Certified Long Term Monitoring (CLTM) credentials. Sixteen technologists were not ABRET certified but had completed an associate’s degree and one year of hands-on training in pediatric EEG. All noncertified EEG technologists employed at the time of data collection were in various stages of preparing for certification examinations. Raters were asked to provide the number of years of experience working with EEG. Technologist experience, primarily in pediatric EEG, ranged from one to 39 years (mean: 9, median: 4), and all were actively involved in screening pediatric CCEEG at our institution. Neurophysiologists had completed formal fellowships in clinical neurophysiology, were board-certified in clinical neurophysiology and epilepsy, and had five to 13 (mean: 10, median: 12) years of experience in EEG interpretation.

      Scoring

      American Clinical Neurophysiology Society terminology for critical care EEG was utilized.
      • Hirsch L.J.
      • LaRoche S.M.
      • Gaspard N.
      • et al.
      American Clinical Neurophysiology Society’s standardized critical care EEG terminology: 2012 version.
      The following variables were scored using a standardized form with multiple choice responses within a REDCap database
      • Harris P.A.
      • Taylor R.
      • Thielke R.
      • et al.
      Research electronic data capture (REDCap)--a metadata-driven methodology and workflow process for providing translational research informatics support.
      : EEG background, seizures, seizure onset location, focal slowing, interictal epileptiform discharges, interictal epileptiform discharges location, rhythmic/periodic pattern present, rhythmic/periodic pattern type, and pattern location. All raters completed each data field, and clinical information other than patient age was not provided.

      Statistics

      Interobserver agreement was measured in aggregate, with Fleiss’ kappa and pairwise agreement for two primary outcomes (EEG background and seizures) was calculated using Cohen’s kappa. Agreement was classified as slight (0-0.20), fair (0.21-0.40), moderate (0.41-0.60), substantial (0.61-0.80), or almost perfect (0.81-1.00).
      • Landis J.R.
      • Koch G.G.
      The measurement of observer agreement for categorical data.
      One feature of kappa is the dependence on marginal probabilities; a variable with very uneven distribution may have a relatively low kappa even if percent agreement is high. Therefore, frequencies of each variable are also provided, to show which variables have uneven distributions (Table 1). Gwet’s AC1 statistic was considered for use instead of the kappa statistic.
      • Gwet K.L.
      Computing inter-rater reliability and its variance in the presence of high agreement.
      However, there are a limited number of publications (beyond those by Gwet himself) supporting the use of Gwet’s AC1,
      • Gaspard N.
      • Hirsch L.J.
      • LaRoche S.M.
      • et al.
      Interrater agreement for critical care EEG terminology.
      and some research suggests that the AC1 statistic is flawed.
      • Cicchetti D.V.
      • Klin A.
      • Volkmar F.R.
      Assessing binary diagnoses of bio-behavioral disorders: the clinical relevance of Cohen’s kappa.
      Therefore, we utilized kappa statistics for analysis to facilitate comparisons across a larger number of studies.
      TABLE 1Variable Interpretation Frequency by Interpreter Category for 90 CCEEG Samples Across 32 Raters
      n%
      EEG background continuity
       Burst suppression24010.5
       Continuous150965.7
       Discontinuous35015.2
       Suppression1978.6
      Seizures
       No201269.9
       Yes86830.1
      Seizure onset location
       Bilateral independent506.8
       Focal or lateralized49767.5
       Generalized15320.8
       Multifocal364.9
      Focal slowing
       Absent215874.9
       Present72225.1
      Interictal epileptiform discharges
       Absent155754.1
       Present132345.9
      Interictal epileptiform discharge location
       Focal or lateralized43142.2
       Generalized21821.3
       Multifocal37336.5
      Rhythmic/periodic pattern
       Absent114253.6
       Present98946.4
      Rhythmic/periodic pattern type
       Rhythmic63466.0
       Periodic34035.4
       Spike and wave39441.0
      Rhythmic/periodic pattern location
       Focal or lateralized42044.9
       Generalized35738.1
       Bilateral independent9810.5
       Multifocal14515.5
      Abbreviations:
      CCEEG = Critical care electroencephalography
      EEG = Electroencephalography
      A gold standard was defined by the consensus decision of three board-certified clinical neurophysiologists and was used to calculate sensitivity and specificity for each rater on the two primary outcomes, background continuity, and seizures. Continuity of the EEG background had four response categories, so in order to determine sensitivity and specificity, it was dichotomized in two different ways: continuous versus any other response, and continuous or discontinuous versus burst suppression or suppression. Years of experience was categorized based on quartiles of experience, as follows: one to two years, three to four years, five to 12 years, and more than 12 years. Sensitivity and specificity were compared by certification and years of experience using Kruskal-Wallis tests with Benjamani-Hochberg corrections for multiple comparisons. All analyses were conducted using R for Statistical Computing or SPSS. The Nationwide Children’s Hospital Institutional Review Board deemed this work exempt.

      Data availability

      Data and the complete study protocol will be available for qualified individuals after approval of the institutional review board and obtaining Data Use Agreements between the involved institutions.

      Results

      Of the 90 five-minute samples, 29 (32.2%) contained seizures and the background was continuous in 75 (83.3%), discontinuous in two (2.2%), burst suppression in six (6.7%), and suppression in seven (7.8%). The frequency of other interictal findings, such as focal slowing, sporadic interictal epileptiform discharges, and rhythmic/periodic patterns, is detailed in Table 1 as scored by individuals, although a consensus was not defined for each category.
      Overall agreement for several main EEG finding categories was measured with Fleiss’ kappa and is displayed in Table 2. The variables with better overall agreement among EEG technologists were seizures (moderate agreement) and EEG background continuity (fair agreement). Focal slowing and rhythmic or periodic patterns had poor interobserver agreement. In comparing certified technologists to those without certification, Fleiss’ kappa for seizure onset location and interictal epileptiform discharge location differed between groups.
      TABLE 2Fleiss’ Kappa for Rater Groups for 90 CCEEG Samples
      VariableEEG Technologists (n = 29)Neurophysiologists (n = 3)
      Kappa95% CI% AgreeKappa95% CI% Agree
      EEG background continuity0.370.36-0.38710.850.76-0.9395
      Seizures0.510.50-0.52790.830.71-0.9593
       Seizure onset location0.400.39-0.41760.770.68-0.8689
      Focal slowing0.140.12-0.15680.300.18-0.4270
      Interictal epileptiform discharges (IEDs)0.250.24-0.27630.520.40-0.6477
       IED location0.180.17-0.19560.480.40-0.5671
      Rhythmic/periodic pattern present0.060.05-0.07380.330.21-0.4567
       Main term 2 - rhythmic0.090.08-0.10700.300.21-0.4067
       Main term 2 - periodic0.050.04-0.06800.330.24-0.4390
       Main term 2 - spike/wave0.130.10-0.14790.270.18-0.3783
       Pattern location - focal or lateralized0.130.12-0.14780.320.23-0.4184
       Pattern location - generalized0.160.15-0.17840.360.27-0.4576
       Pattern location - bilateral independent0.030.02-0.04940.300.20-0.4189
       Pattern location - multifocal0.050.04-0.06900.320.20-0.4396
      VariableCertified Techs (n = 13)Noncertified Techs (n = 16)
      Kappa95% CI% AgreeKappa95% CI% Agree
      EEG background continuity0.460.44-0.47720.300.28-0.3172
      Seizures0.540.52-0.56790.490.47-0.5180
       Seizure onset location0.500.48-0.52760.350.33-0.3678
      Focal slowing0.160.14-0.19700.110.09-0.1366
      Interictal epileptiform discharges0.370.35-0.39690.180.16-0.2059
       Interictal epileptiform discharge location0.340.32-0.36590.080.06-0.0959
      Rhythmic/periodic pattern present?0.110.09-0.13470.030.01-0.0437
       Main term 2 - rhythmic0.140.11-0.16680.060.04-0.0873
       Main term 2 - periodic0.050.03-0.08770.030.01-0.0582
       Main term 2 - spike/wave0.150.13-0.18760.10.08-0.1282
       Pattern location - focal or lateralized0.180.15-0.20740.070.05-0.0983
       Pattern location - generalized0.240.22-0.27810.10.08-0.1187
       Pattern location - bilateral independent0.080.05-0.10930.01−0.02 to 0.02 (P = 0.08)95
       Pattern location - multifocal0.060.03-0.08890.030.01-0.0591
      Abbreviations:
      CCEEG = Critical care electroencephalography
      EEG = Electroencephalography
      Pairwise agreement for seizures and EEG background continuity was measured with Cohen’s kappa and is displayed in Fig 1 (noncertified technologists) and Fig 2 (certified technologists). Inter-rater agreement varied widely among EEG technologists and ranged from slight to substantial.
      Figure thumbnail gr1
      FIGURE 1Pairwise agreement for seizures and continuity. Cohen’s kappa between noncertified technologists and clinical neurophysiologists (A) or certified technologists and neurophysiologists (B). Raters (T = noncertified technologists; C = certified technologists; A = attending neurophysiologists) are arranged by years of experience (“Yrs”; lowest in top left). Certified Long Term Monitoring (CLTM) technicians are shaded in gray. Agreement classified as slight (0-0.20; dark red), fair (0.21-0.40; pink), moderate (0.41-0.60; white), substantial (0.61-0.80; light green), or almost perfect (0.81-1.00; dark green). The color version of this figure is available in the online edition.
      Figure thumbnail gr2
      FIGURE 2EEG technician interpretation by years of experience. Seizure sensitivity (A), seizure specificity (B), EEG continuity sensitivity (C), and EEG continuity specificity (D) across years of experience. Box plots show the minimum, 25th percentile, median, 75th percentile, and maximum values. Violin plots are overlaid and show the distribution of sensitivity and specificity values (turned sideways). EEG, electroencephalography. The color version of this figure is available in the online edition.
      We examined the sensitivity and specificity for correctly detecting seizures or EEG background discontinuity, two of the more clinically important features of CCEEG interpretation. Sensitivity for seizure detection was variable among EEG technologists, ranging from 44% to 93%. Background continuity sensitivity was determined by assessing continuity or any other option (discontinuous, burst suppression, suppression), as this was determined to be a clinically relevant cutoff. Continuity sensitivity was also widely variable among EEG technologists, ranging from 81% to 100%. Overall specificity for seizure or EEG background discontinuity was similar across groups. We then examined the relationships between these findings and EEG technologist certification status or years of experience.
      Twenty-nine raters were included in the analysis. Median (interquartile range) years of experience was four (two to 12). Fifty-five percent of the raters had no certification, 14% had CLTM training, and 31% had R.EEG.T training. Years of experience was strongly associated with certification; all but two R.EEG.T raters had more than 12 years of experience, most raters with no certification had one to four years of experience, and all CLTM raters had five to 12 years of experience. There were no significant differences in seizure or continuity sensitivity or specificity by experience (Table 3; Fig 3).
      TABLE 3Sensitivity and Specificity for Seizure Detection and EEG Background Continuity by Years of Experience
      Variable[All] N = 291-2 yr N = 103-4 yr N = 55-12 yr N = 7>12 yr N = 7P-value
      Certification<0.001
       None16 (55.2%)10 (100%)4 (80.0%)2 (28.6%)0 (0.00%)
       CLTM4 (13.8%)0 (0.00%)0 (0.00%)4 (57.1%)0 (0.00%)
       R.EEG.T9 (31.0%)0 (0.00%)1 (20.0%)1 (14.3%)7 (100%)
      Seizure sensitivity0.68 [0.67, 0.79]0.68 [0.67, 0.80]0.68 [0.67, 0.73]0.79 [0.68, 0.81]0.68 [0.57, 0.75]0.765
      Seizure specificity0.88 [0.86, 0.92]0.86 [0.84, 0.88]0.87 [0.87, 0.92]0.90 [0.86, 0.92]0.90 [0.88, 0.94]0.269
      Continuity sensitivity0.98 [0.95, 0.99]0.96 [0.88, 0.99]0.98 [0.95, 0.98]0.97 [0.94, 0.98]0.98 [0.98, 1.00]0.241
       Continuity sensitivity >0.9520 (69.0%)6 (60.0%)3 (60.0%)4 (57.1%)7 (100%)0.240
      Continuity specificity0.59 [0.43, 0.76]0.75 [0.74, 0.94]0.37 [0.32, 0.76]0.55 [0.46, 0.75]0.50 [0.44, 0.55]0.109
      Abbreviations:
      CLTM = Certified Long Term Monitoring
      EEG = Electroencephalography
      R.EEG.T = Registered Electroencephalographic Technologist
      Figure thumbnail gr3
      FIGURE 3EEG technician interpretation by certification. EEG technician seizure sensitivity (A), seizure specificity (B), EEG continuity sensitivity (C), and EEG continuity specificity (D) across certification. Box plots show the minimum, 25th percentile, median, 75th percentile, and maximum values. Violin plots are overlaid and show the distribution of sensitivity and specificity values (turned sideways). EEG, electroencephalography. The color version of this figure is available in the online edition.
      Raters with CLTM certification had higher seizure sensitivity than R.EEG.T raters (P = 0.08) or those without certification (P = 0.08). Seizure sensitivity was similar among those without certification versus R.EEG.T (P = 0.59). Seizure specificity differed significantly by certification, but was high for all three groups. Specificity was higher among R.EEG.T raters than among those without certification (P = 0.05) and was higher among those with CLTM certification than among those without certification (but P = 0.23 due to greater variability), and there was no difference in seizure specificity between R.EEG.T and CLTM raters (P = 0.94). There were no other significant differences by certification (Table 4).
      TABLE 4Sensitivity and Specificity for Seizure Detection and EEG Background Continuity by Certification Status
      Variable[All] N = 29None N = 16CLTM N = 4R.EEG.T N = 9P-value
      Seizure sensitivity0.68 [0.67, 0.79]0.67 [0.64, 0.74]0.81 [0.80, 0.82]0.71 [0.67, 0.79]0.081
      Seizure specificity0.88 [0.86, 0.92]0.86 [0.84, 0.88]0.91 [0.88, 0.94]0.90 [0.89, 0.92]0.043
      Continuity sensitivity0.98 [0.95, 0.99]0.96 [0.88, 0.98]0.98 [0.97, 0.98]0.98 [0.98, 1.00]0.163
       Continuity sensitivity >0.9520 (69.0%)9 (56.2%)3 (75.0%)8 (88.9%)0.253
      Continuity specificity0.59 [0.43, 0.76]0.74 [0.47, 0.77]0.62 [0.42, 0.82]0.52 [0.44, 0.59]0.623
      Years of experience4.00 [2.00, 12.0]2.00 [1.00, 3.00]7.50 [6.75, 9.00]18.0 [16.0, 19.0]<0.001
      Experience category<0.001
       1-2 yr10 (34.5%)10 (62.5%)0 (0.00%)0 (0.00%)
       3-4 yr5 (17.2%)4 (25.0%)0 (0.00%)1 (11.1%)
       5-12 yr7 (24.1%)2 (12.5%)4 (100%)1 (11.1%)
       >12 yr7 (24.1%)0 (0.00%)0 (0.00%)7 (77.8%)
      Abbreviations:
      CLTM = Certified Long Term Monitoring
      EEG = Electroencephalography
      R.EEG.T = Registered Electroencephalographic Technologist

      Conclusion

      This is the first study to evaluate EEG technologists’ inter-rater agreement as well as sensitivity and specificity for detecting seizures in pediatric CCEEG. These data are vital, as EEG technologists commonly screen CCEEG. We focused our study on two EEG features of greatest clinical impact for screening CCEEG: seizures and EEG background continuity.
      • Herman S.T.
      • Abend N.S.
      • Bleck T.P.
      • et al.
      Consensus statement on continuous EEG in critically ill adults and children, part I: indications.
      ,
      • Ostendorf A.P.
      • Hartman M.E.
      • Friess S.H.
      Early electroencephalographic findings correlate with neurologic outcome in children following cardiac arrest.
      ,
      • Brophy G.M.
      • Bell R.
      • Claassen J.
      • et al.
      Guidelines for the evaluation and management of status epilepticus.
      Seizure identification by individuals varied widely. Perhaps the most important finding was sensitivity for detecting seizures, which ranged from 44% to 93%. EEG technologists with CLTM certification were more sensitive than R.EEG.T raters or those without certification. It is unclear if this reflects the process of earning CLTM certification or those who seek CLTM are better EEG interpreters. Greater years of experience did not predict enhanced seizure detection. Experience as an isolated predictor for correctness likely suffers from the heterogeneity of training, exposure, and feedback for EEG technologists.
      The aggregate agreement between EEG technologists was moderate (κ = 0.51), including for identifying seizure onset location as focal or generalized (κ = 0.40). In comparison, a previous study demonstrated seizure identification agreement as κ = 0.5 in neurologists and κ = 0.29 in neurology residents with various levels of experience.
      • Ronner H.E.
      • Ponten S.C.
      • Stam C.J.
      • et al.
      Inter-observer variability of the EEG diagnosis of seizures in comatose patients.
      Other studies examining neurophysiologists experienced in CCEEG revealed κ ranging from 0.46 to 0.93.
      • Gaspard N.
      • Hirsch L.J.
      • LaRoche S.M.
      • et al.
      Interrater agreement for critical care EEG terminology.
      ,
      • Abend N.S.
      • Gutierrez-Colina A.
      • Zhao H.
      • et al.
      Interobserver reproducibility of electroencephalogram interpretation in critically ill children.
      ,
      • Abend N.S.
      • Massey S.L.
      • Fitzgerald M.
      • et al.
      Interrater agreement of EEG interpretation after pediatric cardiac arrest using standardized critical care EEG terminology.
      Agreement with a paired clinical neurophysiologist ranged from fair to almost perfect (κ = 0.26 to 0.82). Therefore, our data reveal aggregate agreement that was within the published range of neurologists, but individual agreement with a neurophysiologist was widely variable.
      Sensitivity for EEG background continuity varied from 81% to 100%, with specificity from 25% to 100%. This reflects better ability to detect when a background is continuous (sensitivity), but worse ability to detect when a background was discontinuous. The fair (κ = 0.37) aggregate agreement among EEG technologists using ACNS CCEEG terminology to identify discontinuity, burst suppression, or suppression of the EEG background was less robust than agreement on seizures. This is lower than previously reported substantial agreement (κ = 0.69 to 0.79) for pediatric CCEEG among clinical neurophysiologists.
      • Abend N.S.
      • Gutierrez-Colina A.
      • Zhao H.
      • et al.
      Interobserver reproducibility of electroencephalogram interpretation in critically ill children.
      ,
      • Abend N.S.
      • Massey S.L.
      • Fitzgerald M.
      • et al.
      Interrater agreement of EEG interpretation after pediatric cardiac arrest using standardized critical care EEG terminology.
      A possible contributor to the lower kappa among EEG technologists may be the uneven distribution of observations between categories. However, the scoring categories of clinical neurophysiologists were also unevenly distributed, yet the kappa among this group was substantially higher. Therefore, we believe the lower inter-rater agreement among EEG technologists is an accurate assessment of agreement within this study.
      Other EEG findings demonstrated worse aggregate agreement among EEG technologists. As defined by the ACNS CCEEG terminology, findings such as focal slowing, sporadic interictal epileptiform discharges, and rhythmic or periodic patterns had lower kappa than published data from neurophysiologists demonstrating moderate agreement in children
      • Abend N.S.
      • Massey S.L.
      • Fitzgerald M.
      • et al.
      Interrater agreement of EEG interpretation after pediatric cardiac arrest using standardized critical care EEG terminology.
      or almost perfect in adult
      • Gaspard N.
      • Hirsch L.J.
      • LaRoche S.M.
      • et al.
      Interrater agreement for critical care EEG terminology.
      ,
      • Mani R.
      • Arif H.
      • Hirsch L.J.
      • et al.
      Interrater reliability of ICU EEG research terminology.
      CCEEG samples. This may be meaningful if EEG technologists are tasked to prepare reports, as is the case for some centers or remote EEG monitoring services.
      Our study reflects a common “real-world” scenario for EEG technologist CCEEG screening and may be generalizable based on the current credentialing paradigms in use. CCEEG is commonly used,
      • Herman S.T.
      • Abend N.S.
      • Bleck T.P.
      • et al.
      Consensus statement on continuous EEG in critically ill adults and children, part II: personnel, technical specifications, and clinical practice.
      ,
      • Gavvala J.
      • Abend N.
      • LaRoche S.
      • et al.
      Continuous EEG monitoring: a survey of neurophysiologists and neurointensivists.
      and recent reimbursement changes enacted by the Center for Medicaid and Medicare Services have further incentivized continuous monitoring. Due to increasing demand, many EEG laboratories employ EEG technologists with varying levels of certification. Currently, ABRET certifications are attained through multiple pathways without a standard curriculum. EEG technologists work while collecting EEG cases for certification.
      This study has several noteworthy strengths. These are the first published data systematically evaluating interpretation of pediatric CCEEG by EEG technologists. We evaluated clinically actionable EEG variables: sensitivity and specificity for seizures and discontinuity. Additionally, we utilized ACNS-standardized CCEEG terminology for comparison to other groups. Samples of EEG were representative of the various EEG findings in this population.
      • Abend N.S.
      • Arndt D.H.
      • Carpenter J.L.
      • et al.
      Electrographic seizures in pediatric ICU patients: cohort study of risk factors and mortality.
      We believe the combination of noncertified and certified technologists is reflective of current practice.
      Our findings are limited due to the single-center data sample. Therefore, specific kappa values may be less generalizable between institutions. Additionally, we utilized short clips of CCEEG, which may limit an interpreter’s ability to differentiate specific patterns.
      This study prepresents the first published data on inter-rater agreement for pediatric CCEEG among EEG technologists. Our data suggest CLTM certification may be related to improved reliability of seizure detection. However, neither years of experience nor certification status influenced EEG background interpretation. Implementation of a focused educational curriculum targeting seizure identification and background discontinuity may strengthen technologist performance. Future studies should focus on the generalizability of these findings across centers and interventions for improving EEG technologist pediatric CCEEG interpretation.

      Acknowledgments

      The authors thank the EEG technologists and staff at Nationwide Children’s Hospital for their enthusiasm for learning and dedication to the patients.

      References

        • Herman S.T.
        • Abend N.S.
        • Bleck T.P.
        • et al.
        Consensus statement on continuous EEG in critically ill adults and children, part I: indications.
        J Clin Neurophysiol. 2015; 32: 87-95
        • Ostendorf A.P.
        • Hartman M.E.
        • Friess S.H.
        Early electroencephalographic findings correlate with neurologic outcome in children following cardiac arrest.
        Pediatr Crit Care Med. 2016; 17: 667-676
        • Abend N.S.
        • Dlugos D.J.
        • Hahn C.D.
        • et al.
        Use of EEG monitoring and management of non-convulsive seizures in critically ill patients: a survey of neurologists.
        Neurocrit Care. 2010; 12: 382-389
        • Brophy G.M.
        • Bell R.
        • Claassen J.
        • et al.
        Guidelines for the evaluation and management of status epilepticus.
        Neurocrit Care. 2012; 17: 3-23
        • Herman S.T.
        • Abend N.S.
        • Bleck T.P.
        • et al.
        Consensus statement on continuous EEG in critically ill adults and children, part II: personnel, technical specifications, and clinical practice.
        J Clin Neurophysiol. 2015; 32: 96-108
        • Hughes J.R.
        Periodic lateralized epileptiform discharges: do they represent an ictal pattern requiring treatment?.
        Epilepsy Behav. 2010; 18: 162-165
        • Hirsch L.J.
        • LaRoche S.M.
        • Gaspard N.
        • et al.
        American Clinical Neurophysiology Society’s standardized critical care EEG terminology: 2012 version.
        J Clin Neurophysiol. 2013; 30: 1-27
        • Grant A.C.
        • Abdel-Baki S.G.
        • Weedon J.
        • et al.
        EEG interpretation reliability and interpreter confidence: a large single-center study.
        Epilepsy Behav. 2014; 32: 102-107
        • Ronner H.E.
        • Ponten S.C.
        • Stam C.J.
        • et al.
        Inter-observer variability of the EEG diagnosis of seizures in comatose patients.
        Seizure J Br Epilepsy Assoc. 2009; 18: 257-263
        • Gaspard N.
        • Hirsch L.J.
        • LaRoche S.M.
        • et al.
        Interrater agreement for critical care EEG terminology.
        Epilepsia. 2014; 55: 1366-1373
        • Gerber P.A.
        • Chapman K.E.
        • Chung S.S.
        • et al.
        Interobserver agreement in the interpretation of EEG patterns in critically ill adults.
        J Clin Neurophysiol. 2008; 25: 241-249
        • Mani R.
        • Arif H.
        • Hirsch L.J.
        • et al.
        Interrater reliability of ICU EEG research terminology.
        J Clin Neurophysiol. 2012; 29: 203-212
        • Halford J.J.
        • Shiau D.
        • Desrochers J.A.
        • et al.
        Inter-rater agreement on identification of electrographic seizures and periodic discharges in ICU EEG recordings.
        Clin Neurophysiol. 2015; 126: 1661-1669
        • Abend N.S.
        • Gutierrez-Colina A.
        • Zhao H.
        • et al.
        Interobserver reproducibility of electroencephalogram interpretation in critically ill children.
        J Clin Neurophysiol. 2011; 28: 15-19
        • Wusthoff C.J.
        • Sullivan J.
        • Glass H.C.
        • et al.
        Interrater agreement in the interpretation of neonatal electroencephalography in hypoxic-ischemic encephalopathy.
        Epilepsia. 2017; 58: 429-435
        • Abend N.S.
        • Massey S.L.
        • Fitzgerald M.
        • et al.
        Interrater agreement of EEG interpretation after pediatric cardiac arrest using standardized critical care EEG terminology.
        J Clin Neurophysiol. 2017; 34: 534-541
        • Gavvala J.
        • Abend N.
        • LaRoche S.
        • et al.
        Continuous EEG monitoring: a survey of neurophysiologists and neurointensivists.
        Epilepsia. 2014; 55: 1864-1871
        • Harris P.A.
        • Taylor R.
        • Thielke R.
        • et al.
        Research electronic data capture (REDCap)--a metadata-driven methodology and workflow process for providing translational research informatics support.
        J Biomed Inform. 2009; 42: 377-381
        • Landis J.R.
        • Koch G.G.
        The measurement of observer agreement for categorical data.
        Biometrics. 1977; 33: 159-174
        • Gwet K.L.
        Computing inter-rater reliability and its variance in the presence of high agreement.
        Br J Math Stat Psychol. 2008; 61: 29-48
        • Cicchetti D.V.
        • Klin A.
        • Volkmar F.R.
        Assessing binary diagnoses of bio-behavioral disorders: the clinical relevance of Cohen’s kappa.
        J Nerv Ment Dis. 2017; 205: 58-65
        • Abend N.S.
        • Arndt D.H.
        • Carpenter J.L.
        • et al.
        Electrographic seizures in pediatric ICU patients: cohort study of risk factors and mortality.
        Neurology. 2013; 81: 383-391