GitHub热门项目: GPTFast

GitHub项目：GPTFast 仓库地址：https://github.com/MDK8888/GPTFast Stars：683 | 作者：MDK8888 项目描述：Accelerate your Hugging Face Transformers 7.6-9x. Native to Hugging Face and PyTorch. ================================================== README 内容： # GPTFast Accelerate your Hugging Face Transformers 7.6-9x with GPTFast! # Background [GPTFast](https://github.com/pytorch-labs/gpt-fast) was originally a set of techniques developed by the PyTorch Team to accelerate the inference speed of Llama-2-7b. This pip package generalizes those techniques to all Hugging Face models. # Demo GPTFast Inference Time|Eager Inference Time --|-- | # Roadmap - ⟳ 0.7.x (xx/xx/xx): Medusa, Speculative Sampling, Eagle - ⟳ 0.6.x (xx/xx/xx): BitNet and 1-bit quantization, AWQ, QoQ, GGUF, HQQ - ⟳ 0.5.x (xx/xx/xx): PagedAttention (vLLM) + FlashAttention integration - ⟳ 0.4.x (xx/xx/xx): Tensor parallelism + GPU distributed inference - ✅ 0.3.x (06/20/24): GPTQ int4 quantization and optimized int4 matmul kernels enabled for all HF models (**9x inference acceleration**) - ✅ 0.2.x (04/02/24): static key-value cache enabled for all HF models (**8.5x inference acceleration**) - ✅ 0.1.x (02/22/24): torch.compile, int8 quantization, speculative decoding (**7x inference acceleration**) # Getting Started ## WARNING: The below documentation is now deprecated with version 0.3.0. New docs will be up soon! ## * Make sure that your python version >= 3.10, and you are on a cuda enabled device. * Make a virtual environment on your machine and activate it. ```bash $python3 -m venv VENV_NAME source VENV_NAME/bin/activate #./VENV_NAME/scripts/activate if you are on Windows ``` * Call the following: ```pip install gptfast``` * Copy the following code into a python file: ```python import os import torch from transformers import AutoTokenizer from GPTFast.Core import gpt_fast from GPTFast.Helpers import timed torch._dynamo.reset() os.environ["TOKENIZERS_PARALLELISM"] = "false" device = "cuda" if torch.cuda.is_available() else "cpu" def argmax_variation(self, probabilities:torch.Tensor, temperature:float = 1, k:int = 5): # Apply temperature scaling device = probabilities.device scaled_probabilities = probabilities / temperature # Ensure k is within a valid range k = min(k, probabilities.size(-1)) # Get the indices of the top-k scaled probabilities along the specified dimension top_k_indices = torch.topk(scaled_probabilities, k, dim=-1).indices # Generate random indices for sampling random_indices = torch.randint(0, k, (1,) * probabilities.dim()).to(device) # Use gathered indices to get the final sampl