LAUSR

As flowchart images become diverse and complex, existing flowchart recognition methods no longer achieve satisfactory recognition accuracy, particularly for images that contain rarely used symbols and texture backgrounds. Existing deep-learning-based object detectors and line segment detectors are promising in recognizing symbols and connecting edges separately. However, using two separate detectors for symbol and edge detection will inevitably cause unnecessary training and inference costs. Moreover, the insufficient volume and diversity of available dataset further limit the overall recognition accuracy. To address these issues, this paper proposes an end-to-end multi-task network FR-DETR (Flowchart Recognition DETection TRansformer) and a new dataset for precise and robust flowchart recognition. FR-DETR comprises a CNN backbone and a shared multi-scale Transformer structure with two prediction heads for symbol detection and edge detection respectively. The multi-scale Transformer encodes and decodes feature maps with different resolutions to jointly detect symbols and edges in a coarse-to-fine refinement process. The coarse stage uses features with low resolution and suggests candidate regions that contain potential targets for the fine stage to produce accurate predictions using features with high resolution. At each stage, every task detects targets using shared features and its respective prediction head. A new dataset is constructed to provide more symbol types and complex backgrounds for network training and evaluation. It contains more than 1000 machine-generated flowchart images, 25K+ symbol instances with nine categories, and 20K+ line segments. The experiments show that FR-DETR achieves an overall precision and recall of 94.0% and 93.1% on the proposed dataset, and 98.7% and 98.1% on the CLEF-IP dataset, respectively, which all outperform the prior methods.