F. Rundo, A. Genovese, R. Leotta, F. Scotti, V. Piuri, S. Battiato, Advanced 3D deep non-local embedded system for self-augmented X-ray-based COVID-19 assessment, Proc. of the 2021 IEEE/CVF Int. Conf. on Computer Vision Workshops (ICCVW 2021), pp. 423-432, Montreal, QC, Canada, October 2021, ISSN 978-1-6654-0191-3.
Medical imaging
X-Ray-based COVID-19 Assessment
The X-Ray-based COVID-19 Assessment project investigates artificial-intelligence methods for assisting the analysis of chest radiographs in patients with suspected COVID-19. Chest X-ray imaging is inexpensive, widely available, relatively fast, and less invasive than computed tomography, making it potentially suitable for large-scale screening and point-of-care assessment. The system is intended as clinical decision support rather than a replacement for laboratory testing or specialist diagnosis.
A central challenge is the limited availability of labelled COVID-19 radiographs, particularly during the early phases of the pandemic. Conventional deep-learning models may lose accuracy when trained on small datasets, while standard augmentation methods process transformed copies independently and may fail to exploit the relationships among multiple augmented representations of the same image.
The proposed pipeline begins by segmenting the lung region from the input chest X-ray. A fully convolutional network based on a modified ResNet-50 backbone and recurrent criss-cross attention generates a lung mask, which is applied to suppress unrelated anatomical structures and focus subsequent analysis on the pulmonary region.
Criss-cross attention collects contextual information along horizontal and vertical directions, while recurrent application of the attention mechanism improves the representation of long-range spatial relationships. This allows the segmentation stage to identify the lung lobes while preserving contextual dependencies across the radiograph.
The segmented image is then processed by an original self-augmentation module called the reinforcement-learning spatio-temporal feature generator. Instead of applying random transformations independently, the method converts the two-dimensional lung image into a three-dimensional volume containing multiple related feature maps.
The generator is based on nonlinear cellular neural networks and creates 32 discriminative feature configurations for each segmented chest X-ray. Reinforcement learning and a genetic-style optimisation procedure adapt the internal templates according to the loss of the downstream classifier, retaining feature configurations that improve discrimination.
The resulting volume of interest has a size of 32 × 256 × 256 and is classified by a custom 3D Non-Local DenseNet. The architecture combines densely connected three-dimensional convolutional blocks, separable convolutions, transition layers, and embedded Gaussian non-local blocks.
The non-local attention layers model relationships among distant positions in the feature volume. This enables the classifier to integrate local pulmonary patterns with more widely distributed image evidence that may contribute to distinguishing COVID-19 pneumonia from normal radiographs and other viral pneumonias.
The system performs a three-class classification task: normal chest X-ray, COVID-19-induced pneumonia, and non-COVID viral pneumonia. This distinction is clinically relevant because it aims not only to identify possible COVID-19 involvement, but also to separate it from other viral conditions that may produce similar radiographic appearances.
The method was evaluated using the public COVID-19 Radiography Database. The updated dataset included 3,616 COVID-19-positive images, 10,192 normal images, and 1,345 viral-pneumonia images. A balanced subset was also used for comparison with several established deep-learning architectures.
On the balanced three-class experiment, the proposed model achieved 98.82% accuracy, 97.67% precision, 98.82% sensitivity, a 98.25% F1 score, and 98.82% specificity. These results were higher overall than those of the evaluated pretrained architectures with and without conventional image augmentation.
On the larger updated dataset, the system achieved 98.05% accuracy, 97.54% precision, 96.59% sensitivity, a 97.06% F1 score, and 98.78% specificity. It outperformed the compared DenseNet-201 configuration in accuracy, sensitivity, and F1 score, while obtaining the same specificity.
The study also compared the model’s decision regions with radiologists’ observations. The reported analysis indicated that the pulmonary regions influencing the classification were consistent with clinically relevant areas, supporting the interpretability of the attention-based architecture.
A further contribution is the implementation of the pipeline as an embedded point-of-care prototype. The system uses two connected platforms: an NVIDIA Jetson TX2 for segmentation and classification, and an STA1295 Accordo5 platform for feature augmentation and graphical rendering.
The prototype required approximately ten seconds for segmentation and about twenty seconds for classification and rendering. Although not instantaneous, the implementation demonstrated that the complete pipeline could be transferred from workstation-based experimentation to compact embedded hardware suitable for point-of-care environments.
Overall, the project integrates medical-image segmentation, attention mechanisms, reinforcement learning, cellular neural networks, 3D deep learning, and embedded processing. Its principal contribution is a self-augmented and attention-driven framework capable of extracting richer information from limited chest X-ray datasets while supporting accurate three-class pulmonary assessment.
The reported results remain experimental and require broader clinical validation before routine medical use. Future work identified in the study includes large-scale clinical evaluation and extension from disease detection toward assessment of patient prognosis.
A central challenge is the limited availability of labelled COVID-19 radiographs, particularly during the early phases of the pandemic. Conventional deep-learning models may lose accuracy when trained on small datasets, while standard augmentation methods process transformed copies independently and may fail to exploit the relationships among multiple augmented representations of the same image.
The proposed pipeline begins by segmenting the lung region from the input chest X-ray. A fully convolutional network based on a modified ResNet-50 backbone and recurrent criss-cross attention generates a lung mask, which is applied to suppress unrelated anatomical structures and focus subsequent analysis on the pulmonary region.
Criss-cross attention collects contextual information along horizontal and vertical directions, while recurrent application of the attention mechanism improves the representation of long-range spatial relationships. This allows the segmentation stage to identify the lung lobes while preserving contextual dependencies across the radiograph.
The segmented image is then processed by an original self-augmentation module called the reinforcement-learning spatio-temporal feature generator. Instead of applying random transformations independently, the method converts the two-dimensional lung image into a three-dimensional volume containing multiple related feature maps.
The generator is based on nonlinear cellular neural networks and creates 32 discriminative feature configurations for each segmented chest X-ray. Reinforcement learning and a genetic-style optimisation procedure adapt the internal templates according to the loss of the downstream classifier, retaining feature configurations that improve discrimination.
The resulting volume of interest has a size of 32 × 256 × 256 and is classified by a custom 3D Non-Local DenseNet. The architecture combines densely connected three-dimensional convolutional blocks, separable convolutions, transition layers, and embedded Gaussian non-local blocks.
The non-local attention layers model relationships among distant positions in the feature volume. This enables the classifier to integrate local pulmonary patterns with more widely distributed image evidence that may contribute to distinguishing COVID-19 pneumonia from normal radiographs and other viral pneumonias.
The system performs a three-class classification task: normal chest X-ray, COVID-19-induced pneumonia, and non-COVID viral pneumonia. This distinction is clinically relevant because it aims not only to identify possible COVID-19 involvement, but also to separate it from other viral conditions that may produce similar radiographic appearances.
The method was evaluated using the public COVID-19 Radiography Database. The updated dataset included 3,616 COVID-19-positive images, 10,192 normal images, and 1,345 viral-pneumonia images. A balanced subset was also used for comparison with several established deep-learning architectures.
On the balanced three-class experiment, the proposed model achieved 98.82% accuracy, 97.67% precision, 98.82% sensitivity, a 98.25% F1 score, and 98.82% specificity. These results were higher overall than those of the evaluated pretrained architectures with and without conventional image augmentation.
On the larger updated dataset, the system achieved 98.05% accuracy, 97.54% precision, 96.59% sensitivity, a 97.06% F1 score, and 98.78% specificity. It outperformed the compared DenseNet-201 configuration in accuracy, sensitivity, and F1 score, while obtaining the same specificity.
The study also compared the model’s decision regions with radiologists’ observations. The reported analysis indicated that the pulmonary regions influencing the classification were consistent with clinically relevant areas, supporting the interpretability of the attention-based architecture.
A further contribution is the implementation of the pipeline as an embedded point-of-care prototype. The system uses two connected platforms: an NVIDIA Jetson TX2 for segmentation and classification, and an STA1295 Accordo5 platform for feature augmentation and graphical rendering.
The prototype required approximately ten seconds for segmentation and about twenty seconds for classification and rendering. Although not instantaneous, the implementation demonstrated that the complete pipeline could be transferred from workstation-based experimentation to compact embedded hardware suitable for point-of-care environments.
Overall, the project integrates medical-image segmentation, attention mechanisms, reinforcement learning, cellular neural networks, 3D deep learning, and embedded processing. Its principal contribution is a self-augmented and attention-driven framework capable of extracting richer information from limited chest X-ray datasets while supporting accurate three-class pulmonary assessment.
The reported results remain experimental and require broader clinical validation before routine medical use. Future work identified in the study includes large-scale clinical evaluation and extension from disease detection toward assessment of patient prognosis.