Release GPTQ Quantization, Low-level API · huggingface/peft
github.com
GPTQ Integration Now, you can finetune GPTQ quantized models using PEFT. Here are some examples of how to use PEFT with a GPTQ model: colab notebook and finetuning script. GPTQ Integration by @Sun...

With release 0.5.0, PEFT now officially supports fine tuning quantized GPTQ models! This is a pretty big deal as it allows you to download a much smaller model for fine tuning!