In this study, we explore the performance of transformers in predicting RNA secondary structures using both synthetic and biological datasets. We also discuss the generalisation capabilities of transformer models for RNA secondary structure prediction, focusing on both classification and translation tasks. Our analysis revealed that even with a straightforward synthetic dataset of fixed length (12 nucleotides), transformers struggled to generalise across various structural patterns, such as hairpin, bulge, and interior loops.
Our experiments indicate that the primary challenge stems from a large number of sequences mapping to a limited set of structural patterns, a pattern that differs from that observed in translation tasks in natural language processing. Additionally, both biological and synthetic datasets showed that underrepresented structures performed poorly relative to those with more associated sequences. We argue that one major issue in the generalisation failure of DL models is the imbalance between the number of RNA sequences and RNA structures, which results in a lack of evolutionary information. This lack of evolutionary information often renders DL methods incapable of capturing the intrinsic properties of RNA secondary structures.