On November 25th, 2022, in response to national and university epidemic prevention policies, Associate Professor Han Yuexing’s research group organized the annual team dinner and welcome party for new students in Room 402 of the Computer Science Building. The attendees included Professor Han Yuexing, Professor Zhang Rui, and Professor Chen Qiaochuan. During the event, Professor Han introduced the new and existing members of the group to Professor Zhang, and encouraged everyone to actively organize sports activities alongside their academic research. This would not only promote physical fitness but also enhance the group’s atmosphere. Professor Chen shared some interesting stories from their time as students, bringing back nostalgic memories. Professor Zhang shared about her happy family life, adding a personal touch to the gathering. During the dinner, students from each grade chose their favorite songs to share and perform, including popular old songs and currently trending hits. The atmosphere was filled with laughter and joy. The dinner concluded on a high note with everyone enjoying themselves.

Wei Huishan, who graduated with a bachelor’s degree from Anhui University of Science and Technology, joined the School of Computer Engineering and Science at Shanghai University in 2019 to pursue a research-oriented master’s degree. Under the guidance of Professor Han Yuexing, her main research focus is material image segmentation. Over the course of three years under Professor Han’s supervision, she has accomplished the following research projects:

1. We proposed a material image segmentation method based on graph convolution and deep learning to address the challenges of small sample sizes and complex textures in material images. This method incorporates residual connections and multi-scale fusion modules to enrich the information in feature maps. It also utilizes a dual attention mechanism based on graph convolution to enhance the focus on crucial features. Additionally, convolution layers in the deconvolutional part are added to improve the network’s non-linear expressive power.

2. To address the issue of feature loss when using excessive convolutional layers on small sample datasets, we designed a graph attention module based on skip connections using UNet as the backbone network. This method combines the ideas of convolutional neural networks and connects graph convolution and graph attention layers. It aims to fuse multi-dimensional node features from a graph structure perspective. By deepening the network while reducing the loss of pixel-level and spatial information, the goal is to improve the segmentation performance of the network.

3. We implemented the cross-domain application of graph convolution techniques in semantic segmentation tasks. We proposed a graph encoder and graph decoder that transform feature maps into a graph structure. This transformation allows the feature maps generated during the convolution process to be converted into a graph structure with a corresponding number of nodes. This approach facilitates the application of graph convolutional neural networks in semantic segmentation tasks.

After graduating, Wei Huishan joined the ZeroBeam Software Branch of SAIC Group. The three years of graduate studies have broadened her horizons and exposed her to the fascinating field of computer vision. She has witnessed the extensive applications of deep learning, especially in image processing, and has had the opportunity to meet many inspiring mentors and friends. Wei Huishan aspires to continue progressing and making the most of her journey in the future.

Essay: Research on Material Image Segmentation based on Graph Convolutional Neural Networks

Our team recently published a paper titled “Center-Environment Feature Models for Material Image Segmentation based on Machine Learning” in the international journal “Scientific Reports” (IF: 4.996). The School of Computer Engineering and Science at Shanghai University is listed as the first affiliation, with Associate Professor Han Yuexing as the first author and corresponding author. This work also received strong support from Professor Chen Qiaochuan, Professor Wang Bing, and Professor Liu Yi. Additionally, a significant contribution was made by Li Ruiqi and Yang Shen.

The performance of materials depends not only on their composition but also on their microstructure under various processing conditions. So far, the analysis of complex microstructural images has heavily relied on human expertise, lacking automated and quantitative characterization methods. Machine learning provides an emerging and essential tool for intelligently identifying various complex material phases. In this work, we propose a “Center-Environment Segmentation” (CES) feature model for image segmentation, which leverages machine learning techniques to segment images using annotated environmental features and domain knowledge. The CES model introduces neighborhood information as features for a given pixel, reflecting the relationships between the pixel of interest and its surrounding environment. An iterative ensemble machine learning approach is then employed to train and refine the image segmentation model. The CES model has been successfully applied to segment images of seven different materials with complex textures, including steel, wood, and others. In the study of steel image segmentation, the CES method demonstrates superior overall performance in determining boundary contours compared to many traditional methods. This work demonstrates that the iterative incorporation of domain knowledge and environmental features can enhance the accuracy of segmenting various complex material microstructure images.

Li Ruiqi

Yang Shen

Essay: Center-environment feature models for materials image segmentation based on machine learning

Zhang Hongkun, who graduated with a bachelor’s degree from Inner Mongolia Agricultural University, joined the School of Computer Engineering and Science at Shanghai University in 2019 to pursue a research-oriented master’s degree. After starting his studies, he joined the research group led by Professor Han Yuexing to focus on research in material image processing. Under the careful guidance of Professor Han, Zhang Hongkun has accomplished the following research projects over the course of three years:

1. Faced with complex material images with uneven distribution and overlapping structures, a research study was conducted on feature information processing methods based on complex network theory. This method utilizes the community structure in complex networks to describe different organizations within the material. By incorporating R and T thresholds during the network construction process to accelerate the dynamic evolution of network topology, an RT-modularity measure is proposed to evaluate the network’s topology and facilitate image processing. The effectiveness of this method is validated through segmentation experiments conducted on images of ceramics and steel.

2. In the face of complex texture images with multiple features, a feature information processing method based on traditional image processing techniques is proposed. This method takes advantage of the characteristics of material images and designs corresponding algorithms for feature information processing. It utilizes the extracted shape features to accelerate the processing of texture features, significantly reducing the processing time. The effectiveness of this method is demonstrated through experiments conducted on thin film patterns with irregular shapes and complex textures.

3. To further enhance processing speed and reduce computational resource consumption, a feature information processing method based on deep learning is designed and proposed. This method applies lightweight neural network models to texture authentication research. By introducing coordinated attention mechanisms and designing loss functions, the algorithm ensures accurate recognition rates. Compared to other studies that utilize deep learning for texture authentication, this method not only reduces resource consumption but also validates its effectiveness on a large-scale material image database with complex textures.

After graduation, Zhang Hongkun joined AMD Corporation to engage in software development-related work. Throughout his three-year graduate career at Shanghai University, Zhang Hongkun demonstrated a strong commitment to learning, continuously enhancing his professional knowledge and research presentation skills. He displayed enthusiasm in building connections with mentors and friends who provided valuable guidance and support. We hope that Zhang Hongkun will stay true to his original aspirations, remember his mission, and bravely overcome challenges while forging ahead on his future path.

Essay: Research on Feature Information Processing Methods for Complex Material Images

Yang Shen, who graduated with a bachelor’s degree from Anhui University of Traditional Chinese Medicine, pursued a master’s degree in Computer Application Technology as part of the 2019 cohort. Under the guidance of Professor Han Yuexing, Yang Shen’s research primarily focused on material image segmentation. After three years of hard work, Yang Shen proposed a segmentation method for the segmentation and recognition of complex textures in material microstructures. This method made contributions in addressing challenges such as small sample sizes, imbalanced data distributions, and complex textures in material images. It played a role in the construction of material gene databases. After graduation, Yang Shen will be joining ZTE Corporation to work in wireless product development. In addition, Yang Shen has a passion for reading and writing outside of academics and actively participates in extracurricular activities, where they have had the opportunity to meet interesting and talented teachers and classmates. The three years of graduate life have been like climbing a mountain, experiencing tears, laughter, and exhaustion, but ultimately reaching the summit and witnessing a unique scenery of their own. We hope that in the future, Yang Shen maintains their enthusiasm and fearlessly moves forward towards new horizons！

During the graduate period, Yang Shen worked on the following projects:

To address the challenges of small sample sizes, imbalanced data distributions, and complex textures in material images, a combination of various deep learning techniques was employed to automate the segmentation of microstructures in material images. This approach aimed to provide a data foundation for the construction of material gene databases.

1. To address the issue of small sample sizes in material image segmentation, a method combining deep learning and superpixels is proposed. This method takes advantage of the high similarity of pixels within the same material region and utilizes a superpixel algorithm to obtain rectangular blocks, effectively addressing the problem of small sample sizes in material images. An improved version of DenseNet is introduced, which incorporates a feature enhancement module to preserve texture features while removing interference from redundant features. Additionally, a designed transition layer upsampling method is employed to effectively restore the information of feature maps. By leveraging the characteristics of superpixels and the enhanced DenseNet, the proposed method achieves accurate and robust segmentation of material images, even when faced with limited sample sizes. The preservation of texture features and the effective recovery of feature map information contribute to the overall segmentation performance, enabling better analysis and understanding of material microstructures.

2. To tackle the issue of imbalanced data distributions in material image segmentation, two types of loss functions are used and improved: (a) Precision Focal Loss for Classification: In addressing the imbalanced data distribution in classification tasks, an improvement is made by proposing the Precision Focal Loss. This loss function replaces confidence with precision, providing a more accurate reflection of the difficulty of sample classification. The difficulty information is then fed back to the network to optimize the training process. By emphasizing hard-to-classify samples, the Precision Focal Loss helps the network focus on learning from challenging instances, leading to improved performance in imbalanced datasets. (b) CE-Dice Loss for Segmentation: For addressing the imbalanced data distribution in segmentation tasks, the CE-Dice Loss is introduced based on two components: the cross-entropy (CE) loss and the Dice loss. By combining these two loss functions, the CE-Dice Loss provides a smoother training process and optimization of the segmentation results. The cross-entropy loss helps with accurate pixel-wise classification, while the Dice loss evaluates the overlap between the predicted and ground truth segmentation masks, encouraging better boundary localization and segmentation accuracy. The utilization of these improved loss functions aids in mitigating the effects of imbalanced data distributions in both classification and segmentation tasks within material image analysis. By incorporating the specific difficulties and characteristics of these tasks, the proposed loss functions contribute to more effective and accurate training, resulting in better performance and more reliable segmentation outcomes.

3. The precise segmentation of material images with complex textures has been achieved through the following approaches: In Chapter 3, an improved version of DenseNet is proposed. This modified DenseNet architecture aims to preserve crucial texture features while removing redundant features that can interfere with the identification of rectangular blocks. By incorporating a feature enhancement module, the network can effectively retain important texture information and improve the accuracy of segmentation. The modified DenseNet architecture provides a robust framework for handling material images with complex textures. In Chapter 4, an improved Fully Convolutional Network (FCN) is introduced to achieve accurate segmentation of material images with similar textures. The improved FCN incorporates several modules to enhance segmentation performance. A cascaded feature fusion module combines high-level and low-level semantic features to capture multi-scale information effectively. A multi-scale learning module is employed to delve into fine-grained details and global contextual information. An attention mechanism module is used to focus on important feature maps and optimize resource allocation. These three modules complement each other and work together to improve the accuracy and robustness of texture-based material image segmentation. By implementing these advancements in DenseNet and FCN architectures, the research has successfully achieved precise segmentation of material images with complex textures. These improvements address the challenges posed by intricate textures, redundant features, and the need for multi-scale information, leading to more accurate and reliable segmentation results in material image analysis.

Essay: Research on segmentation and recognition methods for complex texture in material microstructures

Our team has recently published a paper titled “UNet for Material Image Segmentation” in the Chinese core journal “Journal of Computer Applications Research.” The School of Computer Engineering and Science at Shanghai University is listed as the primary affiliation, with Wei Huishan as the first author and Associate Professor Han Yuexing as the corresponding author. This work was also guided by Professor Chen Qiaochuan and other members of the research group.

The microscopic structure of material images often exhibits diverse shapes, complex textures, and blurred boundaries. These characteristics have posed limitations on the development of deep learning methods in the field of material image processing. In this study, we propose Graph-UNet, which combines the UNet architecture with graph convolutional neural networks to address the challenge of automatic segmentation of small-sample material images. We transfer the ideas of multidimensional feature fusion and skip connections from convolutional neural networks to graph convolutional neural networks, enabling an effective combination of graph convolutions and graph attention. Additionally, we establish a versatile module to facilitate the conversion between feature maps and graph structures. Comparative and ablation experiments were conducted on material image datasets, demonstrating that Graph-UNet outperforms many state-of-the-art methods in terms of segmentation results. It accurately identifies various material structures, thereby advancing the exploration of the relationship between material structure and performance.

Essay: Graph-UNet applied in material image segmentation

Our team has recently published a paper titled “Lightweight Network-Based Algorithm for Anti-Counterfeiting Label Detection” in the core journal “Journal of Shanghai University (Natural Science Edition).” The School of Computer Engineering and Science at Shanghai University is listed as the primary affiliation, with master’s student Zhang Hongkun as the first author and Professor Han Yuexing as the corresponding author. This work has also received tremendous support from Professor Chen Qiaochuan and Professor Wu Jinbo.

In recent years, the economic losses caused by counterfeit and pirated products have been increasing, and counterfeit techniques continue to advance. As a result, the issue of anti-counterfeiting detection has attracted widespread attention from researchers. To address the challenges of high computational complexity, resource consumption, and lengthy detection time in existing anti-counterfeiting detection methods, this work proposes a lightweight network-based model for anti-counterfeiting label recognition and detection. The model utilizes lightweight convolutional neural networks (CNN) for shape and texture recognition. To enhance the model’s learning capability in shape recognition tasks, the pooling layer size is reduced. For texture classification tasks, a coordinate attention (CA) module is employed to enhance the model’s information retrieval from individual feature maps. The model’s ability to distinguish between genuine and counterfeit samples is reinforced through the design of a loss function. Finally, the prediction results are obtained by selecting the maximum value from the feature vectors. Experimental results demonstrate that the proposed method achieves an overall accuracy rate of 95.67% in label recognition and detection. Moreover, the detection time is significantly improved compared to traditional methods.

Our team has recently published a paper titled “Facial Expression Recognition in Facial Occlusion Scenarios: A Path Selection Multinetwork” in the international journal “Displays” with an impact factor of 2.167. The School of Computer Engineering and Science at Shanghai University is listed as the primary affiliation, with Ruan Liheng as the first author and Associate Professor Han Yuexing as the second author and corresponding author.

In today’s ongoing pandemic situation, wearing masks has become a common practice during outings, which obstructs the nose and mouth regions of the face. Additionally, common facial occlusion scenarios include wearing sunglasses, hats, or objects casting shadows on the face. Facial occlusion poses challenges for facial expression recognition. In this paper, based on three common facial occlusion scenarios, namely upper-face occlusion, lower-face occlusion, and eye occlusion, we propose a path selection-based multi-network structure. The method consists of two parts: the first part involves a multi-network structure, where the original dataset is divided into three subsets based on labels, referred to as sub-datasets. Each sub-dataset inherits some labels from the original dataset and is used to train three corresponding sub-networks. The second part is a path selection-based multi-network integration method, where images from each sub-dataset, treated as the same label, are combined into a new dataset for training an initial network. The final prediction is obtained by selecting one of the sub-networks based on the prediction output of the initial network. In this study, the Fer2013, Jaffe, KDEF, and RAF-DB datasets, commonly used for facial expression recognition, are merged into a larger dataset to increase the training sample size and simulate occlusion scenarios. Experimental results demonstrate that our method effectively recognizes facial expressions under occlusion and is applicable to various facial occlusion scenarios, enabling more accurate and reliable facial expression recognition in a wide range of real-world applications.

Essay: https://doi.org/10.1016/j.displa.2022.102245

Project: https://github.com/han-yuexing/A-Path-Selection-Multinetwork

Our team recently published a paper titled “A Method for Recognizing Morphology Features of Hot Barrier Coatings Based on Digital Image Processing Techniques” in the core journal “Journal of Shanghai University (Natural Science Edition)”. The School of Computer Engineering and Science at Shanghai University is listed as the primary affiliation, with Master’s student Liu Yuhong as the first author, and Professor Han Yuexing as the corresponding author. This work also received strong support from Professor Zeng Yi and Ms. Wang Yuyan.

In response to the drawbacks of manual detection of morphology features in hot barrier coatings, such as complexity and high error rates, we propose a method for automatically recognizing morphology features of hot barrier coatings and calculating their parameters using machine vision. We have achieved automatic contour extraction based on mathematical morphology and calculated the spreading morphology parameters. The method utilizes the maximum interclass variance method to determine the threshold for binary segmentation. Mean filtering and morphological operations are applied to denoise the images while preserving the connectivity of individual layers. Contour extraction is performed to obtain the edge information of each layer, and then the solidity parameter of each layer is calculated based on the extracted contours. Additionally, we have further developed an automatic identification and length calculation method for cracks in hot barrier coatings using a traversal search approach. First, the layers in the image are identified and removed, and the fractures in the cracks are repaired using a closing operation. The crack skeleton is obtained through image thinning. Then, each individual crack is traversed and its length is calculated. The results demonstrate that the proposed method effectively detects layer contours and identifies cracks with good resistance to noise interference. It accurately calculates morphology feature parameters and plays a significant role in studying the deposition behavior of thermal spray droplets on substrate surfaces.

Essay: https://www.journal.shu.edu.cn/CN/10.12066/j.issn.1007-2861.2371

Our team recently published a paper titled “Recognition and Segmentation of Complex Texture Images Based on Superpixel Algorithm and Deep Learning” in the international journal “Computational Materials Science” (IF: 3.3000). The School of Computer Engineering and Science at Shanghai University is listed as the primary affiliation, with Associate Professor Han Yuexing as the first author and Professor Chen Qiaochuan as the corresponding author within our research group. Furthermore, a significant contribution was made by Yang Shen, who played a crucial role in this research.

Material images often lack a sufficient number of training samples, which hinders the application of machine learning and deep learning techniques in material image analysis. In this study, we leverage an important characteristic of material images, namely the high similarity among pixels of the same phase, and propose a method for recognizing and segmenting the microstructure of material images based on superpixel algorithms and deep learning. The method consists of three steps: 1. Rectangular block extraction: Initially, we employ a classic superpixel algorithm, such as SLIC, to obtain different numbers of superpixels. Then, we extract the largest inscribed rectangular block within each superpixel. 2. Rectangular block recognition: We input these rectangular blocks into a convolutional neural network (CNN), with DenseNet chosen and enhanced as the backbone network for recognition. Additionally, considering the issues of non-uniform phase distribution and difficulty in distinguishing certain phases in material images, we select and improve the Focal loss to adapt to material images. 3. Pixel-level prediction and segmentation: Finally, we predict the class labels for each pixel in the entire image. After training, we slide a window of size l*l (l is an odd number) with a stride of 1 over the n*n-sized image, obtaining n*n rectangular blocks. The model predicts the class labels for these rectangular blocks, representing the class labels of the pixels in the middle of each block. By connecting pixels of the same class label, we achieve the recognition and segmentation of the microstructure in material images.

Essay：https://doi.org/10.1016/j.commatsci.2022.111398

Essay：https://www.sciencedirect.com/science/article/pii/S0927025622001690?dgcid=coauthor

Our team has recently published a paper titled “Automatic detection of Kikuchi bands based on Radon transform and PPHT” in the international journal “Journal of Microscopy” (IF: 1.758, indexed in the 4th quartile of Chinese Academy of Sciences). The School of Computer Engineering and Science at Shanghai University is listed as the first affiliation, with Associate Professor Han Yuexing as the first author and corresponding author. Li Ruiqi, Zeng Yi, and Liu Mengyang have contributed significantly to this work.

The structure and orientation information of crystals can be obtained by analyzing Electron Backscatter Diffraction (EBSD) patterns, which are acquired using EBSD devices. The reliability and accuracy of the obtained information depend on the precise localization of the EBSD pattern bands and intersection points. In this study, we propose a method for automatically obtaining the positions of EBSD pattern (Kikuchi band) and intersection points. The method utilizes the Radon transform and cumulative probability Hough transform to detect the lines and line segments representing the edges of Kikuchi bands. Subsequently, the Kikuchi bands can be approximated by fitting hyperbolic curves to the endpoints of the line segments. These results provide a quantitative description of the Kikuchi bands’ information. Experimental results demonstrate that the proposed method is robust and capable of detecting more accurate Kikuchi bands and intersection points. By automating the localization of EBSD pattern bands and intersection points, our method enhances the reliability and accuracy of the obtained crystal structure and orientation information. It offers a robust and effective approach for analyzing EBSD patterns and extracting valuable data for crystallographic studies.

Essay: http://dx.doi.org/10.1111/jmi.13079