nw  

Projekttitel

Summary

We tried improving a BERTweet classifier's performance on an imbalanced dataset using Large Language Models (LLMs) for weak labeling and data augmentation. The results showed our approach wasn't optimal. The unpredictable behavior of LLMs poses challenges. However, LLMs hold vast potential. To tap into this potential, we advise refining test methodologies, balancing data, and exploring larger models.

Schlüsselbegriffe

Natural Language Processing, Text Classification, Few-Shot Learning, Large Language Models, Weak Labeling, Data Augmentation

Context

In the current digital age, the rapid spread of false information, especially on platforms like Twitter, holds the power to greatly impact societies. In Switzerland, Twitter is an important space for political and scientific circles. Twitter data was analyzed in a previous project by the I4DS. The work involved identifying state-linked information campaigns and building a machine-learning model for tweet classification. The tweets were labeled in an annotation process where three raters annotated 5'000 tweets.

Project Goals

This project seeks to refine and advance the tweet classification methodologies previously presented in a report by the I4DS. The core challenge addressed is the inadequate performance of the multi-label classification model on underrepresented classes. Four primary tasks guide the project's trajectory:

  1. Data preparation and understanding.
  2. Reproducing results from the original report.
  3. Implementing Few-Shot Learning (FSL) techniques, encompassing weak labeling using LLMs and data augmentation to tackle the underrepresentation of certain classes.
      4. Try to improve the F1-Scores of the reproduced model by integrating weakly labeled data into its training set or by using data augmentation.
      Results

      We tested several LLMs using handcrafted prompts and weakly labeled 2'000 tweets with them. Due to the "binary" nature of the prompts (see image below depicting an example prompt with an LLM) and time constraints, we weakly labeled the 6 lowest performing classes. Some of the weakly labeled classes improved in performance while the other classes declined in performance due to the added noise. We also highlight that our approach is not optimal and can be easily improved. Our tests with LLMs show the sensitivity of models to small word changes in prompts. LLMs are prone to not understanding complex instructions and tend to "hallucinate" pointless information. Future research might consider exploring varied LLM parameters, harnessing their inherent ability to generate detailed explanations, or a different data sampling strategy.

      weak_labeling
      Projektdaten

      Projektdauer: 20.02.2023 - 18.08.2023

      Aufwand: 360h

      Teamgrösse: 1

      Auftraggeber

      armasuisse S+T

      Cyber Defence Campus

      Feuerwerkerstrasse 39, 3602 Thun

      www.armasuisse.ch

      Projektteam

      Bruno Baptista Kreiner, BSc Data Science

      Kontakt

      Daniel Perrouchoud, daniel.perruchoud@fhnw.ch

      Fernando Benites, fernando.benites@fhnw.ch

      << zurück