Background: We recently demonstrated that quality of spirometry in primary care could markedly improve with remote offline support from specialized professionals. It is hypothesized that implementation of automatic online assessment of quality of spirometry using information and communication technologies may significantly enhance the potential for extensive deployment of a high quality spirometry program in integrated care settings.
Objective: The objective of the study was to elaborate and validate a Clinical Decision Support System (CDSS) for automatic online quality assessment of spirometry.
Methods: The CDSS was done through a three step process including: (1) identification of optimal sampling frequency; (2) iterations to build-up an initial version using the 24 standard spirometry curves recommended by the American Thoracic Society; and (3) iterations to refine the CDSS using 270 curves from 90 patients. In each of these steps the results were checked against one expert. Finally, 778 spirometry curves from 291 patients were analyzed for validation purposes.
Results: The CDSS generated appropriate online classification and certification in 685/778 (88.1%) of spirometry testing, with 96% sensitivity and 95% specificity.
Conclusions: Consequently, only 93/778 (11.9%) of spirometry testing required offline remote classification by an expert, indicating a potential positive role of the CDSS in the deployment of a high quality spirometry program in an integrated care setting.
High Quality Spirometry Testing
High quality spirometry testing across health care levels is pivotal for proper management of patients with prevalent chronic respiratory disorders, namely asthma and chronic obstructive pulmonary disease (COPD) .
We have recently reported the effectiveness of a Web-based application for remote offline expert support to enhance the quality of spirometry in primary care. High quality testing improved in a sustainable manner with the remote support . A relevant difference was observed between the intervention group, 2419/3383 (71.50%) high quality spirometry, and the control group, 713/1198 (59.52%) high quality spirometry, throughout the 12 month follow-up period (P<.001). Similar figures have been obtained in pharmacy offices, as part of a COPD case finding program [ ].
In the Basque Country (Spain), the ongoing regional deployment of the Web-based offline support program from specialists to primary care will cover the entire population, 2.2 million inhabitants, by the end of 2014 [, ]. Interestingly, their results [ ] are similar to those reported in the initial randomized controlled trial [ ] described above.
Ideally, extensive deployment of a high quality spirometry program in the community should offer accessibility to standardized raw spirometric data through a technological architecture providing interoperability across health care tiers. To this end, a Clinical Document Architecture for spirometry using Health Level Seven v3 standards was recently made available by the Catalan Health Department , such that spirometric testing will be available at the regional level. In this scenario, automatic assessment of quality of spirometry testing should enhance the efficiency of the program. Unfortunately, current applications for online assessment of quality of spirometry misclassify the tests, as compared with examinations done by expert professionals [ ].
Clinical Decision Support System
We hypothesize that elaboration and validation of a clinical decision support system (CDSS) for online automatic assessment and certification of quality of spirometry in primary care may represent a pivotal step toward regional adoption of the high quality spirometry program with an integrated care approach.
The current study is part of the refinement of the ongoing deployment of the high quality spirometry program in Catalonia , an European region of 7.5 million inhabitants.
Building-Up the Clinical Decision Support System
shows the initial step in the process for elaboration of the CDSS was the identification of the optimal sampling frequency to achieve the highest sensitivity and specificity in the analysis of the spirometric curves. To this end, a systematic examination of a large range of sampling frequencies, from 6.25 Hz to 100 Hz, was done during the first iterative process.
The process was done using the 24 standard flow-volume and volume-time curves from the pulmonary waveform generator recommended by the American Thoracic Society/European Respiratory Society (ATS/ERS) . This set of 24 standard curves cover the entire spectrum of clinical abnormalities, as well as common spirometric artifacts. They are used as a reference material for calibration purposes and, in general, to facilitate comparisons among lung function laboratories.
The construction of an initial version of the CDSS was carried out using the 24 standard spirometry curves [, ] following an iterative process, as displayed in . In each step, the results generated by the CDSS were compared with the criteria of one expert in the field of lung function testing (FB), and the iterative process was maintained until sensitivity and specificity of the results generated by the CDSS showed 24/24 (100.0%) agreement with the expert.
The CDSS combines the different aspects assessed on the spirometry curve in one score with three different categories: (1) grade 0, rejected due to unacceptable morphology of the spirometry curve; (2) grade 1, acceptable for further classification according to; or (3) grade 2, undefined characteristics of the spirometry (see for examples of the three categories in Figure 1S). The two first categories, grades 0 and 1, allow proper online automatic classification of spirometry testing as well as the generation of a certified spirometry curve to be potentially shared across health care tiers; whereas grade 2 requires offline expert assessment.
|Aa||3 acceptable maneuvers, and best 2 matched with differences in FVCband/or FEV1<150 ml|
|B||3 acceptable maneuvers, and best 2 matched with differences in FVCband/or FEV1c<200 ml|
|C||2 acceptable maneuvers, and best 2 matched with differences in FVC and/or FEV1c<250 ml|
|D||1 acceptable maneuver|
|F||No acceptable maneuvers|
aHigh quality spirometries, A and B scores, correspond to A, 3 acceptable maneuvers with differences in FVC and/or FEV1<150 ml; and B, 3 acceptable maneuvers with differences in FVC and/or FEV1<200 ml; C, to high variability among maneuvers; D, only one acceptable maneuver; and F no acceptable maneuver.
bFVC = forced vital capacity
cFEV1= forced expiratory volume in the first second
The Characteristics and the Algorithm
The CDSS systematically assessed 27 different characteristics of each spirometry curve, as displayed in. There were four out of the 27 characteristics that were extracted from the international recommendations for standardization of the test, jointly reported by the ATS and the ERS [ ]; whereas the remaining 23 were introduced during the current research. Each of these 27 features had a well defined specific algorithm for calculations. The mathematical description of a feature constituted the so-called metric. It is of note that a given feature may require more than one metric. The quantitative values of a given metric were denominated thresholds that were used for quality assessment. It is also of note that some metrics may have primary and secondary thresholds. The initial parameters of the automatic algorithm for online assessment of quality of spirometry were refined through successive iterations until the final version of the CDSS was obtained ( ). As indicated above, the performance of each of the successive versions of the CDSS was compared with the results provided by the expert. A refined version of the CDSS was achieved using 270 curves from 90 patients from [ ].
|Forced spirometry curve||Criteriai|
|BEVa trad||Back extrapolation >0.15 L or < 5% of FVCg|
|EOTVb trad||End of test criteria, volume < 0.025 L in time ≥1 s|
|Texc||Time of end FVCg(Tex>6 s)|
|EOTVb new (5 criteria)||a) EOTVb< 0.025 L or Texc>6 s;|
b) If Texc>6 s EOTVb<0.025 L in time 0.5 s;
c) If Texc>6 s, EOTVb< 0.1 L;
d) EOTVb(Texc) < 0.025 L; and
e) EOTVb< 0.025 * Tex/6 L
|Peak_Valley_Single||High local maximum (peak) and minimum (valley) in FVecurve|
|Peak_Valley_Combined||High local maximum (peak) and minimum (valley) in FVecurve close to FEV1h|
|VTd end||Irregularity or oscillation at the end of FTmcurve|
|FVe_slope_single||Variation of FVeslope or high FVeslope|
|FVe_slope_combined||Variation of FVeslope and high FVeslope|
|FVeSlope_Test_Combo||Irregularity and variation of FVeslope or high FVeslope|
|FVeSlope_Test_Combo_Area Under Rectj||Irregularity or variation of FVeslope and high FVeslope|
|FVeSlope_Test_Combo4||Irregularity and variation of FVeslope and high FVeslope|
|Diff_singlek||Irregular concavity-convexity before the PEFfvalue in FVecurve|
|Diff_combinedl||Irregular slope and irregular concavity-convexity before the PEFfvalue in FVecurve|
|PEFf TimeUp||Time to archive PEFf< 130 milliseconds|
|PEFf TimeDown||Time to archive PEFf> 0.25 milliseconds|
|PEFf Cut||PEFfis not a peak in FVecurve (is plane), volume (Fn=PEFf) > 15 % FVCg|
|PEFf Cut2 FEV1h||PEFfis not a peak in FVecurve (is plane), volume (Fn=PEFf) > 17.5 % FEV1h|
|PEFf DoublePeak||PEFfbimodal in FVecurve|
|PEFf Slow||Volume to archive PEFf< 20% FVCg|
aBEV = back extrapolation
bEOTV = end of test criteria, volume
cTex = Time to end FVC
dVT = volume/time curve
eFV = flow/volume curve
fPEF = peak expiratory flow
gFVC = forced vital capacity
hFEV1= forced expiratory volume in the first second
iThe list includes the classical parameters used by ATS/ERS guidelines .
jRect = rectum
kDiff single= irregular concavity-convexity before the PEFf value in flow volumen curve concavity or convexity exists if the extracted signal metric
lDiff_combined = irregular slope and irregular concavity-convexity before the peak expiratory flow value in flow volume curve
mFT = flow/time curve
Clinical Decision Support System Validation
The refined version of the CDSS was compared with a database of 778 curves from 291 patients from one of the primary care centers in Barcelona. The spirometry testing was done using a spirometer (Sibel 120, SIBELMED, Barcelona Spain). Again, the score generated by the CDSS was compared with the one obtained from the same expert evaluator.
The use of the two patient databases, for refinement and validation purposes, was approved by the Ethical Committee of the Hospital Clínic i Provincial de Barcelona.
The ATS database  contains volume (V) values of each curve, from which flow (F) values were obtained by discrete differentiation (equation 1, ). The two patient’s databases contained F values, from which V values were obtained by discrete integration (equation 2, ). The sample period is Δt=0.01s, so the sample frequency is 100 Hz. Sensitivity and specificity of the CDSS were calculated for all curves classified as grades 0 or 1 using equations 3 and 4 in .
The Sampling Frequency
The sampling frequency that provided the highest sensitivity and specificity for the analysis carried out with the 24 standard spirometry curves recommended by the ATS  was 100 Hz ( and , Table 2S), this frequency is widely used in commercial spirometers, and it is reasonable from the electronic transferability point of view. This result was confirmed in the 270 curves from 90 subjects [ ].
Both sensitivity and specificity of the CDSS were initially calculated with the 24 standard spirometry curves recommended by the ATS  using only grade 0 and grade 1 curves. The results were as follows, grade 0, n=15; grade 1, n=6; grade 2, n=3 with 24/24 (100.0%) sensitivity and 24/24 (100.0%) specificity. Up to five complete versions of the CDSS were generated in the two iterative processes indicated in , until a final version of the CDSS was ready for validation.
Grading the Curves
The validation study using 778 curves from 291 patients showed the following distribution of spirometry curves, 419/778 maneuvers (53.8%) were appropriately classified as bad curves (grade 0); 266/778 maneuvers (34.2%) were appropriately classified as good curves (grade 1); and only 93/778 maneuvers (11.9%) needed an offline review by a lung function expert to assess quality of the test (grade 2; see). Sensitivity and specificity calculations for grade 0 and grade 1 curves were 96.1 and 94.9%, respectively.
The Current Research
The current research has generated and validated a CDSS that shows the ability to classify a reasonable percentage of spirometry curves, 685/778 (88.1%) as either acceptable (grade 1) or bad maneuvers (grade 0). Only 93/778 (11.9%) of the curves were classified as undefined (grade 2) and were candidates for offline remote validation by an expert. Moreover, we observed that both sensitivity and specificity of the CDSS were very high. Consequently, the results seem to indicate that a vast majority of spirometry testing carried out by nonspecialized professionals in primary care can be reliably assessed online, and the high quality spirometry program partly based on remote automatic evaluation of the testing could be considered ready for regional scalability. Obviously, further steps toward extensive deployment of the program must be planned with caution. A proper monitoring of the potential for generalization of the current results and the need for further refinements of the current CDSS should be taken into account.
The results of the current research overcome some of the limitations of the existing computer-based algorithms generating automatic feedback, as reported in [, ]. It is acknowledged, however, that automatic feedback based on enhanced algorithms like the one proposed by the current research may be effective only if they are part of a comprehensive program for high quality forced spirometry.
In the new scenario, as indicated by the business process management notation (BPMN) diagram (, Figure 2S), acceptable maneuvers (grade 1) will be automatically addressed to the algorithm indicated in that classifies and certifies spirometry testing prior to its recording into the local (electronic health record) and regional repositories. In contrast, those maneuvers classified as bad curves (grade 0) will generate an online specific error message to the professional, indicating the need to perform additional testing until quality acceptance is reached. As indicated, we estimate that approximately 12% of the curves will not be properly classified (grade 2), and they will need an offline remote supervision by an expert professional. In this case, the spirometry testing of a given patient may need to be rescheduled.
Previous reports have indicated the potential of telemedicine to enhance both quality and diagnostic potential of spirometry testing carried out by nonexpert professionals [- ], but the quality control in those studies was based on offline analyses by expert professionals carried out in a time consuming manner [ - ]. Likewise, the need for an external, likely centralized, quality control program [ , - ] is well established. The results of the current study refine previous achievements [ ] and open the way to explore extensive and efficient adoption of this type of high quality spirometry programs.
We acknowledge that high quality spirometry programs combine several different dimensions, namely: (1) professional coaching [, ]; (2) remote support [ ]; (3) interoperability of testing across health care levels [ ]; (4) standards for procurement of equipment [ , ]; and (5) support to interpretation of testing [ , ]. The current study provides pivotal results to efficiently address issues associated to remote support of spirometry testing. But, a proper integration of all the above elements needs to be considered in the process of shaping a successful high quality spirometry program for scalability at regional level.
Limitations of the Study
We acknowledge two principal limitations of the study. First, we included only one expert observed (FB). The CDSS should be reassessed in the future with the inclusion of at least 3 different experts. Moreover, the current study evaluates the CDSS in an isolated manner. But, further assessment of the whole clinical process as defined in the BPMN (see, Figure 2S) should be done before specific plans for scalability are undertaken.
To our knowledge, the current study constitutes the first successful attempt to validate an automatic CDSS for large scale online assessment of quality of spirometry testing. The incorporation of the CDSS into the Web-based application for remote assistance to primary care professionals  may facilitate sustainable high quality spirometry generating a significant added value in an integrated care scenario.
The results indicate a high potential of the CDSS for discrimination between good and poor quality results of spirometry testing, but they require further independent validation before specific plans for implementation are materialized.
The authors thank Jordi Giner of Hospital de la Santa Creu i Sant Pau, in Barcelona for providing the validation database. This project was supported by Inforegió (AGAUR) 2008; NEXES (Supporting Healthier and Independent Living for Chronic Patients and Elderly, CIP-ICT-PSP-2007-225025); FIS PI09/90634. Servicios Innovadores de Atencion Integrada para Pacientes Crónicos - PITES- ISCIII 2010-12; EC-FP7 Programme, Synergy-COPD, GA nº 270086; TAMESIS (TEC2011-22746, Spanish Government) CIBER of Bioengineering, Biomaterials and Nanomedicine; Research Fellowship Grant FPU AP2009-0858 from the Spanish Government; and, Catalan Master Plan of Respiratory Diseases (PDMAR).
Conflicts of Interest
Multimedia Appendix 1
The algorithm for computing maneuver acceptability, using the 27 set of criteria.PDF File (Adobe PDF File), 6KB
Multimedia Appendix 2
Three examples with curves classified as Grade 0, 1 and 2. The Business Process Model Notation (BPMN) diagram displays the use of the CDSS for quality control in primary care within a coordinated care scenario. The results of the protocol undertaken to identify the optimal sampling frequency during the first iterative process are shown here.PDF File (Adobe PDF File), 113KB
Multimedia Appendix 3
For each FS curve, the results generated by the CDSS are compared with those provided by the expert professional. It is of note, that only the expiratory portion of the FS manouevres was taken into account for analysis.PDF File (Adobe PDF File), 27KB
- Celli BR. The importance of spirometry in COPD and asthma: Effect on approach to management. Chest 2000 Feb;117(2 Suppl):15S-19S. [Medline]
- Burgos F, Disdier C, de Santamaria EL, Galdiz B, Roger N, Rivera ML, e-Spir@p Group. Telemedicine enhances quality of forced spirometry in primary care. Eur Respir J 2012 Jun;39(6):1313-1318 [FREE Full text] [CrossRef] [Medline]
- Castillo D, Burgos F, Giner J, Estrada M, Soriano JB, Flor X. Arch Bronconeumologia. 2011. Cribado de EPOC: nuevas herramientas y circuitos sanitarios. Resultados preliminares del Farmaepoc II. (FARMAEPOC) URL: http://www.researchgate.net/publication/266081643 [accessed 2014-10-03] [WebCite Cache]
- Eustat - population and housing census in Euskadi. 2011. URL: http://www.eustat.es/estadisticas/tema_159/opt_0/ti_Poblacion/temas.html [accessed 2014-09-11] [WebCite Cache]
- Marina Malanda N. Agencia de Evaluación de Tecnologías Sanitarias del País Vasco. Informes de evaluación de tecnologías sanitarias: OSTEBA Nº; 2012. URL: http://www.msssi.gob.es/organizacion/sns/planCalidadSNS/ [accessed 2014-09-25] [WebCite Cache]
- Marina Malanda N, López de Santa María E, Gutiérrez A, Bayón J, Garcia L, Gáldiz J. Telemedicine spirometry training and quality assurance program in primary care centers of a public health system. Telemed J E Health 2014 Apr;20(4):388-392. [CrossRef] [Medline]
- Salas T, Rubies C, Gallego C, Muñoz P, Burgos F, Escarrabill J. Technical requirements of spirometers in the strategy for guaranteeing the access to quality spirometry. Arch Bronconeumol 2011 Sep;47(9):466-469 [FREE Full text] [CrossRef] [Medline]
- Tresserras R, en nombre del Grupo de Trabajo de Planes Directores. Planning driven by health priorities. Master planning criteria. Med Clin (Barc) 2008 Dec;131 Suppl 4:42-46. [Medline]
- ATS. Am Rev Respir Dis. 1979 May. ATS statement--snowbird workshop on standardization of spirometry URL: http://www.unboundmedicine.com/medline/citation/453705/ATS_statement__Snowbird_workshop_on_standardization_of_spirometry_ [accessed 2014-09-25] [WebCite Cache]
- Am Rev Respir Dis. 1987 Nov. Standardization of spirometry: A summary of recommendations from the American Thoracic Society: The 1987 update URL: http://annals.org/article.aspx?articleid=701103 [accessed 2014-09-25] [WebCite Cache]
- Miller MR, Hankinson J, Brusasco V, Burgos F, Casaburi R, Coates A, ATS/ERS Task Force. Standardisation of spirometry. Eur Respir J 2005 Aug;26(2):319-338 [FREE Full text] [CrossRef] [Medline]
- Müller-Brandes C, Krämer U, Gappa M, Seitner-Sorge G, Hüls A, von Berg A, et al. LUNOKID: Can numerical American Thoracic Society/European Respiratory Society quality criteria replace visual inspection of spirometry? Eur Respir J 2014 May;43(5):1347-1356. [CrossRef] [Medline]
- Masa JF, González M, Pereira R, Mota M, Riesco JA, Corral J, et al. Validity of spirometry performed online. Eur Respir J 2011 Apr;37(4):911-918 [FREE Full text] [CrossRef] [Medline]
- Bellia V, Pistelli R, Catalano F, Antonelli-Incalzi R, Grassi V, Melillo G, et al. Quality control of spirometry in the elderly. The SA.R.A. study. Salute respiration nell'Anziano = Respiratory health in the elderly. Am J Respir Crit Care Med 2000 Apr;161(4 Pt 1):1094-1100. [CrossRef] [Medline]
- Spirometry monitoring technology - SPIROLA. Centers for Disease Control and Prevention. NIOSH - Spirometry in occupational medicine URL: http://www.cdc.gov/niosh/topics/spirometry/spirola-software.html [accessed 2014-10-02] [WebCite Cache]
- Montes de Oca M, Tálamo C, Perez-Padilla R, Jardim JR, Muiño A, Lopez MV, PLATINO Team. Chronic obstructive pulmonary disease and body mass index in five Latin America cities: The platino study. Respir Med 2008 May;102(5):642-650 [FREE Full text] [CrossRef] [Medline]
- Pérez-Padilla R, Vázquez-García J, Márquez M, Menezes AM, PLATINO Group. Spirometry quality-control strategies in a multinational study of the prevalence of chronic obstructive pulmonary disease. Respir Care 2008 Aug;53(8):1019-1026 [FREE Full text] [Medline]
- Janssens W, Liu Y, Liu D, Kesten S, Tashkin DP, Celli BR, et al. Quality and reproducibility of spirometry in COPD patients in a randomized trial (UPLIFT®). Respir Med 2013 Sep;107(9):1409-1416. [CrossRef] [Medline]
- Enright PL. How to make sure your spirometry tests are of good quality. Respir Care 2003 Aug;48(8):773-776 [FREE Full text] [Medline]
- Enright PL, Skloot GS, Cox-Ganser JM, Udasin IG, Herbert R. Quality of spirometry performed by 13,599 participants in the World Trade Center Worker and Volunteer Medical Screening Program. Respir Care 2010 Mar;55(3):303-309 [FREE Full text] [Medline]
- Steenbruggen I, Mitchell S, Severin T, Palange P, Cooper BG, Spirometry HERMES Task Force. Harmonising spirometry education with HERMES: Training a new generation of qualified spirometry practitioners across Europe. Eur Respir J 2011 Mar;37(3):479-481 [FREE Full text] [CrossRef] [Medline]
- Escarrabill J, Roger N, Burgos N, Giner J, Molins A, Tresserras R. Diseño de un programa de formación básico para conseguir espirometrías de calidad. Educ. méd 2012 Jun;15(2):103-107. [CrossRef]
- Miller MR, Crapo R, Hankinson J, Brusasco V, Burgos F, Casaburi R, ATS/ERS Task Force. General considerations for lung function testing. Eur Respir J 2005 Jul;26(1):153-161 [FREE Full text] [CrossRef] [Medline]
- Pellegrino R, Viegi G, Brusasco V, Crapo RO, Burgos F, Casaburi R, et al. Interpretative strategies for lung function tests. Eur Respir J 2005 Nov;26(5):948-968 [FREE Full text] [CrossRef] [Medline]
- Pellegrino R, Brusasco V, Viegi G, Crapo RO, Burgos F, Casaburi R, et al. Definition of COPD: Based on evidence or opinion? Eur Respir J 2008 Mar;31(3):681-682 [FREE Full text] [CrossRef] [Medline]
|ATS: American Thoracic Society|
|BPMN: business process management notation|
|CDSS: clinical decision support system|
|COPD: chronic obstructive pulmonary disease|
|ERS: European Respiratory Society|
Edited by G Eysenbach; submitted 06.04.14; peer-reviewed by P Enright, M Stanbrook; comments to author 16.06.14; revised version received 27.07.14; accepted 21.08.14; published 21.10.14Copyright
©Felip Burgos, Umberto Melia, Montserrat Vallverdú, Filip Velickovski, Magí Lluch-Ariet, Pere Caminal, Josep Roca. Originally published in JMIR Medical Informatics (http://medinform.jmir.org), 21.10.2014.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Medical Informatics, is properly cited. The complete bibliographic information, a link to the original publication on http://medinform.jmir.org/, as well as this copyright and license information must be included.