LAUSR.org creates dashboard-style pages of related content for over 1.5 million academic articles. Sign Up to like articles & get recommendations!

MFormer: Taming Masked Transformer for Unsupervised Spectral Reconstruction

Photo by tsawwunna24 from unsplash

Spectral reconstruction (SR) aims to recover the hyperspectral images (HSIs) from the corresponding RGB images directly. Most SR studies based on supervised learning require massive data annotations to achieve superior… Click to show full abstract

Spectral reconstruction (SR) aims to recover the hyperspectral images (HSIs) from the corresponding RGB images directly. Most SR studies based on supervised learning require massive data annotations to achieve superior reconstruction performance, which is limited by complicated imaging techniques and laborious annotation calibration in practice. Thus, unsupervised strategies attract the attention of the community, however, existing unsupervised SR works still face a fatal bottleneck from low accuracy. Besides, traditional convolutional neural network (CNN)-based models are good at capturing local features but experience difficulty in global features. To ameliorate these drawbacks, we propose an unsupervised SR architecture with strong constraints, especially constructing a novel Masked Transformer (MFormer) to excavate latent hyperspectral characteristics to restore realistic HSIs further. Concretely, a dual spectralwise multihead self-attention (DSSA) mechanism embedded in transformer is proposed to firmly associate multihead and channel dimensions and then capture the spectral representation in the implicit solution spaces. Furthermore, a plug-and-play mask-guided band augment (MBA) module is presented to extract and further enhance the bandwise correlation and continuity to boost the robustness of the model. Innovatively, a customized loss based on the intrinsic mapping from HSIs to RGB images and the inherent spectral structural similarity is designed to restrain spectral distortion. Extensive experimental results on three benchmarks verify that our MFormer achieves superior performance over other state-of-the-art (SOTA) supervised and unsupervised methods under a no-label training process equally.

Keywords: spectral reconstruction; masked transformer; reconstruction; mformer taming; transformer

Journal Title: IEEE Transactions on Geoscience and Remote Sensing
Year Published: 2023

Link to full text (if available)


Share on Social Media:                               Sign Up to like & get
recommendations!

Related content

More Information              News              Social Media              Video              Recommended



                Click one of the above tabs to view related content.