The aim of the study was to determine the preliminary psychometric performance of the NFAS to diagnose OPD. The objectives were to determine the sensitivity, specificity and accuracy of the NFAS in comparison to the MBSS and to verify inter-rater reliability.

A comparative within-subject design (Meline, 2010) was used to investigate the psychometric properties of the NFAS by comparing the NFAS and MBSS results.

Neonates admitted to a 29-bed neonatal intensive care unit (NICU) at a tertiary academic hospital in Gauteng Province, South Africa, were purposively selected. Mothers were verbally informed of the study and through a brochure in English, Setswana or Afrikaans, the most prominent languages spoken in the city where the study was conducted. Written or verbal (in case of illiterate participants) informed consent was obtained from all mothers. Twenty neonates were selected. The participant inclusion criteria were that the neonate should have a high-risk status such as prematurity, low birth weight (LBW), exposure to HIV or another risk factor (e.g. craniofacial anomaly), predisposing the neonate to feeding and swallowing difficulties; be an in-patient in the NICU; be medically stable for assessment as determined by the treating physician; be within the age range of > 32 weeks gestational age to 4 months corrected age post-term at time of assessment. Neonates younger than 32 weeks gestational age are expected to display feeding and sucking difficulties as a result of immaturity and are typically not fed orally; they were not included. Participant characteristics are presented in Table 1.

According to Table 1 the participants were born at a mean premature gestational age of 35.15 weeks (SD = 3.066). The mean birth weight of the participants was low, 2.17 kg (SD = 0.845) and the mean length of stay in the NICU was 6 days. Additionally, the sample consisted of slightly more female participants (60%). Other risk factors contributing to feeding difficulties were HIV exposure in utero or during delivery (30%, n = 6), respiratory distress syndrome (55%, n = 11) and hyperbilirubinaemia (55%, n = 11). Prematurity (80%, n = 16) and LBW (85%, n = 17) were the most significant known risk factors for OPD (Pados et al., 2016).

The newly developed feeding scale (NFAS) and an MBSS data collection form (based on Arvedson & Brodsky, 2002; Hall, 2001; Swigert, 2010) were used. The MBSS form indicated the stages of swallowing (oral, pharyngeal and oesophageal stages), the presence or absence of any form of dysphagia, and penetration or aspiration in the pharyngeal stage. In addition, a parent interview schedule included pre-, peri- and postnatal information and a description of the feeding problem according to the parents (based on Arvedson & Brodsky, 2002; Hall, 2001; Swigert, 2010). Medical records were used for additional information.

The development and content of the NFAS were discussed in a previous study (Viviers et al., 2016). The item selection in the sections of the NFAS was based on theoretical constructs related to neonatal and early infant feeding and the clinical assessment of feeding skills. The instrument relies on physiological observations of the infant during feeding, how infant state is influenced by feeding and how feeding may subsequently disrupt a regulated state in the infant with feeding difficulties and an associated display of stress cues (Figure 1).

The MBSS was performed using a fluoroscope (SYSCO 19” version Multi DiagnostEleva FD screening machine from Philips, Amsterdam, Netherlands) with DVD recording capabilities.

Clearance was obtained from the research ethics committee at the university and the medical ethics committee at the tertiary academic hospital where the study was conducted. The mothers of the participants were interviewed, medical files were reviewed, a clinical feeding assessment was conducted using the NFAS and a MBSS was performed. The MBSS was conducted within 7 days of the clinical assessment. The interviews, medical file review and clinical feeding assessments were conducted by the first author, a qualified SLT, and three graduate students in speech-language pathology. All data collectors were trained. Training was provided in a 6-hour session on the content, administration and scoring of the NFAS. After the training session each trainee was expected to accumulate four practice assessments before data collection was initiated. Inter-rater reliability data was obtained for two of the four data collectors (excluding the first author) on 10 infants (50% of sample). Two senior SLTs working at the hospital conducted the MBSS while blinded to the infants’ feeding history and diagnostic outcome of the clinical assessments.

Because feeding is an integrated process, with infant responses in the different sections occurring simultaneously, the order in which sections of the NFAS are completed may vary. A breastfeeding session was observed, or the mother was asked to prepare the bottle feed (expressed breast milk or formula) or supplemented breastfeeding with tube feeding if the infant was not fully breastfed. The complete data collection procedures for the NFAS are presented in Appendix A. Scoring instructions for each section are indicated on the instrument. A binary yes or no system is used. The outcome of each section is a yes or no conclusion regarding the possible presence of OPD. Each section score is transferred to the last page of the instrument, where the overall diagnostic outcome of the assessment is calculated. When a score of three or more yes responses is obtained, the assessment outcome indicates that OPD is likely to be present. At least one of the three yes responses required for reaching the final diagnosis of OPD must either be obtained in Section E or F (Viviers et al., 2016).

During the MBSS a solution of barium sulphate was reconstituted by mixing the powder with the mother’s expressed breast milk or recommended formula. During fluoroscopy the pulsed mode with appropriate collimation was used to limit radiation exposure (Hernanz-Schulman, Goske, Bercha & Strauss, 2011; Scott, Fujii, Behrman & Dillon, 2014). A NUK (Bonn, Germany) MedicPro First Choice^TM 120 ml infant bottle with a MedicPro^TM disposable thermoplastic elastomer (TPE) Teat Size 1 was used. Participants were positioned with appropriate supported seating in a Tumble Forms 2 Feeder Seat^TM, Warrenville, United States.

Frequency distributions were calculated for the NFAS data. Criterion validity was determined by calculating sensitivity (%) and specificity (%) scores based on the comparative data sets. Sensitivity determines the probability of the presence of OPD, whereas specificity reveals the probability that OPD will truly be absent when using the NFAS (Dawson & Trapp, 2004). Positive predictive value (PPV) and negative predictive value (NPV) indicate whether the NFAS predicted the true positive and true negative diagnoses correctly (Dawson & Trapp, 2004). The higher the percentage score derived for PPV and NPV calculations, the better and more valid the predictive ability of the instrument (Dawson & Trapp, 2004). Cohen’s kappa with accompanying asymptotic standard error (ASE) was used to investigate the inter-rater reliability coefficient, together with p-bar calculations for the results obtained by two independent raters. The interpretation of the inter-rater reliability calculations (kappa) according to Dawson and Trapp (2004) and Landis and Koch (1977) are provided in Table 2. A kappa value of greater than 0.41 was considered a minimal reliability criterion (Dawson & Trapp, 2004). Accuracy of agreement between the NFAS and the MBSS diagnosis of OPD was also investigated.

The NFAS was administered on a sample of 20 participants to determine preliminary psychometric properties. The clinical assessment results were compared to the MBSS results to determine which participants presented with true OPD. Table 3 presents the data obtained from the NFAS assessment.

According to Table 3 nine infants (45%) presented with OPD on the NFAS. Positive identification of OPD could be explained by the participant characteristics (see Table 1) and the previously stated associated risk factors in the sample. As per scoring guidelines, the nine participants obtained a minimum score of three yes responses in the five sections, with one of the yes responses in either Section E or F of the NFAS. In Sections C (stress cues during feeding) and F (clinical feeding and swallowing evaluation) the highest number of indicators were observed in those neonates diagnosed with OPD. Some of the neonates were not attached to heart rate and respiratory monitors. Therefore certain items could not be scored in Sections A and B (physiological status and state of alertness), resulting in low scores in the combined section. As a result of the low scores in the physiological status and state of alertness sections, the contributions of these sections to diagnose OPD should be investigated further in a larger sample. The NFAS results were then compared to the MBSS results to determine validity.

Criterion validity determined the extent to which the NFAS agreed with the gold standard (MBSS) measuring the same variable. Measures to determine criterion validity included the predictive ability, sensitivity, specificity and accuracy of the instrument. The comparative results are presented in Table 4.

When comparing the MBSS and NFAS outcomes in Table 4, all six neonates who presented with OPD were correctly identified with the NFAS; however, three were incorrectly identified, resulting in a false positive rate of 21.4%. This comparison revealed the NFAS presented with a sensitivity of 100% when identifying OPD in neonates. The specificity of 78.6% reflects the probability of the NFAS to determine that a neonate does not present with dysphagia.

The PPV and NPV were calculated using the data in Table 4. The PPV was 100% (6 / 6 × 100) and the NPV was 78.6% (11 / 14 × 100). The higher the PPV and NPV (closer to 100%), the better the new assessment scale is at diagnosing OPD when compared to the gold standard (Parikh, Mathai, Parikh, Chandra Sekhar & Thomas, 2008). Based on the PPV and NPV scores the NFAS showed adequate predictive ability to determine when OPD would be present or absent. It was concluded that among those participants who had OPD the predictive ability of dysphagia being present was 100% and among those participants who did not have OPD the predictive ability of not having dysphagia was 78.6%.

The overall accuracy was calculated using the specificity and sensitivity data. The accuracy of agreement on diagnosis of OPD between the NFAS and MBSS was 85% (11 + 6/20 × 100). The closer the accuracy score is to 100%, the better agreement there is between the newly developed instrument and the gold standard (Dawson & Trapp, 2004).

The NFAS therefore presented with good preliminary sensitivity (100%) (Dawson & Trapp, 2004). Specificity was also considered to be good (Dawson & Trapp, 2004) at 78.6%. An assessment tool with a high specificity, sensitivity, PPV, NPV and accuracy is considered valuable in clinical practice (Lalkhen & McCluskey, 2008). The participants not diagnosed with OPD on MBSS presented with oesophageal dysphagia or normal swallowing ability.

Apart from the six participants diagnosed with OPD on the NFAS and the MBSS, the MBSS revealed additional results as expected. Based on MBSS results, 40% (n= 8) of the participants presented with oesophageal dysphagia, 10% (n = 2) had OPD co-occurring with oesophageal dysphagia and four participants had normal swallowing. Two of the six neonates diagnosed with OPD on the NFAS presented with this co-occurrence.

Inter-rater reliability for all the sections and diagnostic outcome of the NFAS between two independent raters were determined using half of the sample (n = 10). A kappa value of greater than 0.410 was considered a minimal reliability criterion and a p-bar value of 0.50 (Dawson & Trapp, 2004). The inter-rater reliability calculations of each section of the instrument are presented in Table 5.

The inter-rater reliability for two of the five sections of the instrument demonstrated substantial agreement beyond chance. In the combined Sections A and B as well as for Section D, the assessment criteria were clear (0.90–1.00 p-bar), therefore rendering the kappa calculation obsolete for these sections. For Section C the results indicated only slight agreement, which may be due to the variability of infant state during the feeding process. Thus the variability inherent to infant state may have increased the difficulty to evaluate this section objectively. The two raters agreed on the instrument outcome in 90% (n = 9) of the cases. The agreement on diagnostic outcome between the two raters was considered substantial beyond chance with an ASE of 0.241 (Dawson & Trapp, 2004).

The high sensitivity and specificity of the NFAS provide evidence of the ability of the scale to accurately diagnose the presence of OPD and in turn to also recognise the absence of OPD, rendering very few false positives. There appears to be limited information on the sensitivity and specificity properties of comparable assessments for oral motor difficulties in neonates and infants, such as the EFS, NOMAS and SOMA (Da Costa, Van Den Engel-Hoek & Bos, 2008). The diagnostic accuracy (85%) of the NFAS and its good predictive ability (Dawson & Trapp, 2004) in clinical use showed that the scale is capable of measuring what it intends to measure.

As expected of a direct instrumental observational procedure, the MBSS gave additional diagnoses. Different types of dysphagia exist in neonates, depending on the stage of swallowing that is affected (Pados et al., 2016). Different types of dysphagia can also co-occur. The MBSS diagnosed oesophageal dysphagia and clearly showed the co-occurrence of the two types of dysphagia, OPD and oesophageal dysphagia. Because the focus of SLTs is on assessment and intervention of OPD, preliminary results indicate that the NFAS could serve this purpose. All participants who truly presented with OPD were identified. When relying on clinical assessments only in contexts where MBSS is not available, the three false positive OPD results may not be viewed as disadvantageous. Further research is required to determine whether subsequent assessments on the same neonate using the NFAS may show different results.