llama : add custom RoPE (#2054) · ggerganov/llama.cpp@6e7cca4
github.com
* Implement customizable RoPE The original RoPE has pre-defined parameters theta_i = 10000^(−2(i−1)/d), for i in [1, 2, ..., d/2] Our customizable RoPE, ggml_rope_custom_inplace, uses t...
  • h3ndrik@feddit.deEnglish
    7·
    2 years ago

    nice. a bigger context size was my dream for many weeks. seems to work pretty well.