Content area
Large Language Models (LLMs) have rapidly advanced the field of Natural Language Processing and become powerful tools for generating and evaluating scientific text. Although LLMs have demonstrated promising as evaluators for certain text generation tasks, there is still a gap until they are used as reliable text evaluators for general purposes. In this thesis project, I attempted to fill this gap by examining the discernibility of LLMs from human-written and LLM-generated scientific news. This research demonstrated that although it was relatively straightforward for humans to discern scientific news written by humans from scientific news generated by GPT-3.5 using basic prompts, it is challenging for most state-of-the-art LLMs without instruction-tuning. To unlock the potential evaluation capability of LLMs on this task, we propose guided-few-shot (GFS), an instruction-tuning method that significantly improves the discernibility of LLMs to human-written and LLM-generated scientific news. To evaluate our method, we built a new dataset, SANews, containing about 362 triplets of scientific news text, LLM-generated news text, and the corresponding scientific paper abstract on which the news articles were based. This work is the first step for further understanding the feasibility of using LLMs as an automated scientific news quality evaluator.
