3

Theory of Scaling Laws for In-Context Regression - Depth, Width, Context and Time
We study in-context learning (ICL) of linear regression in a deep linear self-attention model, characterizing how performance depends …
Blake Bordelon, Mary Letey, Cengiz Pehlevan
PDF Cite DOI
Pretrain-Test Task Alignment Governs Generalization in In-Context Learning
In-context learning (ICL) is a central capability of Transformer models, but the structures in data that enable its emergence and …
Mary Letey, Jacob A Zavatone-Veth, Yue M. Lu, Cengiz Pehlevan
PDF Cite DOI