Transformer networks

1 Introductions

So many.

TODO: rank in terms of lay-person-friendliness.

• “Attention”, “Transformers”, in Neural Network “Large Language Models” I quite like Cosma’s characteristically idiosyncratic way of learning and re-deriving things. I found his contrarian statistician’s take on the transformer architecture to be enlightening.

• [2502.17814] An Overview of Large Language Models for Statisticians

• 3b1b: But what is a GPT? Visual intro to transformers

• Compact precise definition of a transformer function – foreXiv

• Understanding einsum for Deep learning: implement a transformer with multi-head self-attention from scratch | AI Summer

• Jay Alammar’s Illustrated Transformer series is good.

  • How GPT3 Works - Visualizations and Animations
  • The Illustrated BERT, ELMo, and co. (How NLP Cracked Transfer Learning)
  • The Illustrated GPT-2 (Visualising Transformer Language Models)

• Lilian Weng, Large Transformer Model Inference Optimization

• Lilian Weng, The Transformer Family Version 2.0

• nostalgebraist, An exciting new paper on neural language models

• 1A - Scaled Dot Product Attention explained

• John Thickstun, The Transformer Model in Equations

• Large language models, explained with a minimum of math and jargon

• [1hr Talk] Intro to Large Language Models

• Jeremy Howard on X: Here, in full directly on Twitter, is “A Hackers’ Guide to Language Models”. This 90 minute tutorial is designed to be the one place I point coders at when they ask “hey, tell me everything I need to know about LLMs!” It covers both OpenAI models and open source ones in depth.

• Large language models, explained with a minimum of math and jargon

• A good paper read is Yannic Kilcher’s.

• Xavier Amatriain, Transformer models: an introduction and catalog — 2023 Edition

• Noam Shazeer’s Shape Suffixes post implicitly makes the case that transformers are simply a confusing way of smashing tensors together.

2 Power of

Transformers are pretty good at weird stuff, e.g. automata — see Unveiling Transformers with LEGO (Y. Zhang et al. 2022).

How about Bayesian inference? (Müller et al. 2022)

Can they be an engine of intelligence? Controversial — see the Stochastic Parrots paper (Bender et al. 2021), and the entire internet commentariat from November 2022 onwards.

What do they do in society? Who knows, but see AI democratization and AI economics.

3 As set functions

Transformers are neural set functions (!).

4 As recurrent state

See Transformers and RNNs

5 For forecasting of non-linguistic material

See Transformers and numerical time series.

6 Practicalities

For you and me, see AI democratization.

7 Embedding vector databases

See embedding vector databases.

8 Incoming

• Transformers.js

• LMQL: Programming Large Language Models: “LMQL is a programming language for language model interaction.” (Beurer-Kellner, Fischer, and Vechev 2022)

  LMQL generalises natural language prompting, making it more expressive while remaining accessible. For this, LMQL builds on top of Python, allowing users to express natural language prompts that also contain code. The resulting queries can be directly executed on language models like OpenAI’s GPT models. Fixed answer templates and intermediate instructions allow the user to steer the LLM’s reasoning process.

• How does in-context learning work? A framework for understanding the differences from traditional supervised learning

  TL;DR — In-context learning is a mysterious emergent behaviour in large language models (LMs) where the LM performs a task just by conditioning on input-output examples, without optimising any parameters. In this post, we provide a Bayesian inference framework for understanding in-context learning as “locating” latent concepts the LM has acquired from pretraining data. This suggests that all components of the prompt (inputs, outputs, formatting, and the input-output mapping) can provide information for inferring the latent concept. We connect this framework to empirical evidence where in-context learning still works when provided training examples with random outputs. While output randomisation cripples traditional supervised learning algorithms, it only removes one source of information for Bayesian inference (the input-output mapping).

• What Are the Different Approaches for Detecting Content Generated by LLMs Such As ChatGPT? And How Do They Work and Differ?

• Large Language Models as General Pattern Machines

  We observe that pre-trained large language models (LLMs) are capable of autoregressively completing complex token sequences—from arbitrary ones procedurally generated by probabilistic context-free grammars (PCFG), to more rich spatial patterns found in the Abstract Reasoning Corpus (ARC), a general AI benchmark, prompted in the style of ASCII art. Surprisingly, pattern completion proficiency can be partially retained even when the sequences are expressed using tokens randomly sampled from the vocabulary. These results suggest that without any additional training, LLMs can serve as general sequence modellers, driven by in-context learning. In this work, we investigate how these zero-shot capabilities may be applied to problems in robotics—from extrapolating sequences of numbers that represent states over time to complete simple motions, to least-to-most prompting of reward-conditioned trajectories that can discover and represent closed-loop policies (e.g., a stabilising controller for CartPole). While difficult to deploy today for real systems due to latency, context size limitations, and compute costs, the approach of using LLMs to drive low-level control may provide an exciting glimpse into how the patterns among words could be transferred to actions.

• karpathy/nanoGPT: The simplest, fastest repository for training/finetuning medium-sized GPTs.

9 References