Compression Intelligence Benchmark

LLM Compression

Introduction

The following introduction comes from the abstract of Compression Represents Intelligence Linearly:

>There is a belief that learning to compress well will lead to intelligence. Recently, language modeling has been shown to be equivalent to compression, which offers a compelling rationale for the success of large language models (LLMs): the development of more advanced language models is essentially enhancing compression which facilitates intelligence. ...our findings suggest that compression efficiency, as an unsupervised metric derived from raw text corpora, serves as a reliable evaluation measure that is linearly associated with the model capabilities. We open-source our compression datasets as well as our data collection pipelines to facilitate future researchers to assess compression properly.

Official Links

• Paper: Compression Represents Intelligence Linearly
• GitHub Repository: llm-compression-intelligence

Overview and Usage

Dataset

The dataset, which consists of three external corpora, can be downloaded using the following python script:

Note: Refer to the original repository for more details on data collection and design.

Inference

The inference stage (SWCELossInferencer) consists of the following key steps:

1. For each candidate model, we obtain the encodings of each sample of the dataset using its tokenizer.
2. Concatenate the encodings of all samples into a single array and construct a PyTorch Dataset. Each item of __getitem__ is a chunk of the array based on a sliding window. To reproduce results from the original paper, set block_size=1900 and stride=512.
3. For each batch, calculate the cross entropy loss based on model logits and targets. The losses within each batch is reduced to a single loss by summation.
4. Output the losses and total_chr_num to BPCEvaluator for evaluation.

Evaluation

BPCEvaluator: Using the total loss for each batch and the total number of characters in the original dataset from the inference stage, calculate the Bits per Character (BPC) metric for each model:

$$ BPC = \frac{TotalCrossEntropyLoss}{TotalCharacterNumber*log(2)} $$

Summarization

Config Files

1. Dataset config: configs/datasets/llm-compression.py
2. Evaluation config: examples/eval_llm_compression.py

Evaluation Results

FAQ

I am getting this warning during inference, should I truncate long samples to max_seq_len to avoid further errors?

>A: This warning comes from the tokenizer indicating that the input sequence length exceeds the model's input length, but it does not affect the operation of the tokenizer. For loss calculation, as long as we set a block_size of the sliding window less than max_seq_len, we can safely ignore this warning.

Reference