Secure Semi-Automated Health Data Extraction with OCR and NLP

A Brazilian project leveraging OCR and LLM for health data extraction, improving workflow and data processing efficiency.

Initial situation

Hospitals and healthcare institutions, such as the National Institute of Cardiology (INC) and the Federal University of Rio de Janeiro (UFRJ), manage extensive volumes of patient data, typically exceeding 1 TB per patient. This data includes complex, unstructured formats such as echocardiograms, MRI scans, clinical reports, and data from other databases. However, the full potential of this valuable information can be further optimised by using the latest technologies and modern AI-driven solutions, for example by automating data collection, rather than relying on manual processes.

Problem statement / Project goal

Manual extraction of unstructured medical data is time-consuming, error-prone, and restricts clinical efficiency. This project utilizes cutting-edge AI technologies, specifically Optical Character Recognition (OCR) and Large Language Models (LLM), to automate the extraction and structuring of multilingual medical data. The resulting solution delivers reliable clinical insights, supports informed decision-making, and ultimately improves patient care outcomes.

Solution developed and its benefits

The developed solution features a streamlined document processing workflow with multiple stages. The system combines custom OCR preprocessing algorithms with a local and secure LLM to extract structured data from clinical reports, while incorporating human verification to ensure high accuracy. Key benefits include:

• Advanced image preprocessing pipeline with AWS Textract achieving high scores in OCR accuracy and outperforming open-source alternatives
• Specialized medical prompting with Llama 3.1 LLM to extract standardized cardiac measurement variables while maintaining data privacy
• Interactive web frontend with human-in-the-loop verification that achieves the balance between automation efficiency and clinical accuracy
• Secure document processing with isolated systems hosted on AWS servers and local LLM execution to protect sensitive patient information

Key terms

• OCR Technologies AWS Textract, Docling, and PyTesseract
• NLP Llama 3.1 8B and Llama 3.2-vision for local and secure LLM processing
• Human-in-the-loop Semi-automated expert verification workflow
• Multilingual Support for Brazilian Portuguese and English medical terminology
• Data Privacy Secure local data handling
• Technology Stack Python, Flask, JavaScript

Customer

The Federal University of Rio de Janeiro (UFRJ) and National Institute of Cardiology (INC)

Team