arXiv.org / 2024

Attention Is All You Need But You Don't Need All Of It For Inference of Large Language Models

Georgy Tyukin, G. Dovonon, Jean Kaddour, Pasquale Minervini

Foundation ModelsGenerative AILarge Language ModelsPopular and Landmark Papers

The inference demand for LLMs has skyrocketed in recent months, and serving models with low latencies remains challenging due to the quadratic input length complexity of the attention layers. In this work, we investigate the effect of dropping MLP and attention layers at inference time on the performance of Llama-v2 models. We find that dropping dreeper attention layers only marginally decreases performance but leads to the best speedups alongside dropping entire layers. For example, removing 33\% of attention layers in a 13B Llama2 model results in a 1.8\% drop in average performance over the OpenLLM benchmark. We also observe that skipping layers except the latter layers reduces performances for more layers skipped, except for skipping the attention layers.

2 citations0 influential

Full paper

Read the original paper

Source page

A direct open-access PDF is not available in the database yet. Use the source page or learning resources below to open the complete paper from the publisher or index.

Learning resources

Google Scholar referencesGoogle Scholar Papers with Code searchPapers with Code Semantic Scholar paper pageSemantic Scholar YouTube explanationsYouTube

Reading state

Discuss in ChatGPT

Uses your own ChatGPT account. The paper context is copied into a tutor prompt before ChatGPT opens.

Preview prompt

You are my AI/ML research paper instructor. I want to deeply understand the paper below.

First, teach it in layers:
1. One-paragraph intuition.
2. Problem statement and why it mattered.
3. Key method, architecture, or algorithm.
4. Important equations or mechanisms, explained intuitively.
5. Experiments and evidence.
6. Limitations, assumptions, and failure modes.
7. How this paper influenced later AI/ML/Deep Learning/GenAI work.
8. A 30-minute study plan with checkpoints.
9. Quiz me with 5 questions and wait for my answers.

When something is not available in the attached context, say what is missing and infer carefully.

### Paper attached as context
Title: Attention Is All You Need But You Don't Need All Of It For Inference of Large Language Models
Authors: Georgy Tyukin, G. Dovonon, Jean Kaddour, Pasquale Minervini
Year: 2024
Venue: arXiv.org
Categories: Foundation Models, Generative AI, Large Language Models, Popular and Landmark Papers
Citations: 2
Paper URL: https://www.semanticscholar.org/paper/081e0d061ba6efeb7e46507a5e3dc88375c20837
Open PDF: Not available

Abstract:
The inference demand for LLMs has skyrocketed in recent months, and serving models with low latencies remains challenging due to the quadratic input length complexity of the attention layers. In this work, we investigate the effect of dropping MLP and attention layers at inference time on the performance of Llama-v2 models. We find that dropping dreeper attention layers only marginally decreases performance but leads to the best speedups alongside dropping entire layers. For example, removing 33\% of attention layers in a 13B Llama2 model results in a 1.8\% drop in average performance over the OpenLLM benchmark. We also observe that skipping layers except the latter layers reduces performances for more layers skipped, except for skipping the attention layers.

Learning resources:
- Google Scholar: Google Scholar references (https://scholar.google.com/scholar?q=Attention%20Is%20All%20You%20Need%20But%20You%20Don't%20Need%20All%20Of%20It%20For%20Inference%20of%20Large%20Language%20Models)
- Papers with Code: Papers with Code search (https://paperswithcode.com/search?q=Attention%20Is%20All%20You%20Need%20But%20You%20Don't%20Need%20All%20Of%20It%20For%20Inference%20of%20Large%20Language%20Models)
- Semantic Scholar: Semantic Scholar paper page (https://www.semanticscholar.org/paper/081e0d061ba6efeb7e46507a5e3dc88375c20837)
- YouTube: YouTube explanations (https://www.youtube.com/results?search_query=Attention%20Is%20All%20You%20Need%20But%20You%20Don't%20Need%20All%20Of%20It%20For%20Inference%20of%20Large%20Language%20Models+paper+explained)