Title: Improving Large-Scale Foundation Models Via Attention-Aware Techniques

Date: Thursday, August 1st, 2024

Time: 2:00 pm - 3:00 pm ET

Location: Virtually via Zoom (https://gatech.zoom.us/j/9960405372?pwd=bzhIbVdWRkxweW9naUh0aUt4ci9WZz09)

PhD Student:

Zhongzhi Yu, School of Computer Science, Georgia Institute of Technology

Committee Members:

Dr. Yingyan (Celine) Lin (Advisor) – School of Computer Science, Georgia Institute of Technology

Dr. Chao Zhang – School of Computational Science and Engineering, Georgia Institute of Technology

Dr. Judy Hoffman – School of Interactive Computing, Georgia Institute of Technology

Dr. Pavlo Molchanov – Nvidia Corporation

Abstract:

Foundation models, which are a series of large-scale transformer models, have shown impressive performance across a diverse range of applications, from natural language processing to computer vision. The key enabler behind their success is the attention module, which controls how these models extract relationships among input tokens. However, despite the importance of the attention module, our understanding of its role during the inference and fine-tuning stages remains limited, leading to challenges such as potentially sub-optimal model performance and a lack of interpretability.

My thesis research focuses on understanding the potentially suboptimal attention distributions generated by foundation models and developing attention-aware techniques to improve their performance. The primary insight from my research is that certain high-attention tokens can negatively affect foundation model performance during both fine-tuning and inference. Building on this insight, my research presents state-of-the-art solutions to enhance the performance of foundation models, including an attention-aware data augmentation technique that enhances the data efficiency of the fine-tuning process and an attention calibration technique that improves inference accuracy.