Deconstructing historical astronomy images using machine learning

Summary

Machine learning model to detect figures in scientific publications.

Key words

Machine Learning, Figure Detection, Astronomy, Python, PyTorch

Goals

The primary goal of this project is to train a machine learning algorithm that can detect figures in scans of astronomical publications. Key point here is to ensure that the found bounding box around a figure includes the axis labels. To achieve this, a new loss function called "Boundary Enhance Loss" developed by Yu et al (2023) specifically designed for figure detection in scientific publications is re-implemented and the results claimed by the authors shall be verified

Initial situation

Jill Naiman (customer) has developed a complex processing pipeline that tries to detect and interpret figures in old scans of astronomical publications. However, the machine learning algorithm used to detect figures has difficulty with including the axis labels of figures inside the detected bounding box, making an interpretation of these figures nearly impossible in future processing steps.

Even with a high Intersection over Union (IoU) a predicted bounding box can still miss the labels due to them being located at the edges of the figures

Results

The result of the project is a re-implementation of the mentioned Boundary Enhance Loss function by Yu et al (2023) and trained models on two different datasets, comparing metrics when trained with the Boundary Enhance Loss function and metrics when trained with the Cross Entropy Loss function.

We trained on 2 different datasets, creating 2 models per dataset (one using the Cross Entropy Loss function and one using the Boundary Enhance Loss function) to compare their performance.

Model	Recall	FPR	FNR	TNR	IoU	F1	Precision
1	0.9330	0.0116	0.0670	0.9884	0.8339	0.8572	0.8714
2	0.9560	0.0138	0.0440	0.9862	0.8420	0.8645	0.8653
3	0.8116	0.0761	0.1884	0.9239	0.5210	0.6754	0.6028
4	0.9645	0.0948	0.0355	0.9052	0.5702	0.7193	0.5835

Table showing the metrics from all the trained models. Models 1 & 2 are trained using the Deepfigures dataset and models 3 & 4 are trained using the ReadingTimeMachine dataset. The models 1 & 3 were trained with the Cross Entropy Loss function and models 2 & 4 with the Boundary Enhance Loss function.

These metrics clearly show an improvement when trained with the Boundary Enhance Loss function.

This is how a prediction of one of the trained models looks like. Green pixels indicate the pixel was labeled as "Figure" in the ground truth and red pixels mark predicted "Figure" pixels by the trained model

Project key figures

• Project start and end date: 12th February 2024 - 16th August 2024
• Project time budget: 180 hours
• Team size: 1

Client

Jill Naiman (University of Illinois)
https://ischool.illinois.edu/people/jill-naiman

Project Team

Tino Heuberger

Contact

Prof. Dr. Arzu Çöltekin arzu.coltekin@fhnw.ch
Dr. Marianne Santaholma marianne.santaholma@fhnw.ch

<< zurück