I am using a code-completion model for (will be open sourced very soon).
Qwen2.5-coder 1.5b though tends to repeat what has already been written, or change it slightly. (See the video)
Is this intentional? I am passing the prefix and suffix correctly to ollama, so it knows where it currently is. I’m also trimming the amount of lines it can see, so the time-to-first-token isn’t too long.
Do you have a recommendation for a better code model, better suited for this?
Your assumption is very good, but I am sure that the completion is setup correctly. Sometimes it does fill in correctly, like suggesting variable types and adding comments to functions.
So sometimes completion works fieny but other times it doesn’t. I use the ollama REST API for the completion, so the token handling isn’t on my side.
I hope someone else chimes in and can offer some advice. You could have a look at the ollama log / debug output and see if the
<|fim_prefix|>,<|fim_suffix|>and<|fim_middle|>tokens are at the correct spots when fed into the LLM. (as per https://github.com/QwenLM/Qwen2.5-Coder?tab=readme-ov-file#3-file-level-code-completion-fill-in-the-middle ) Other than that, I don’t have a clue. You could also try a different model. But I guess there is something wrong somewhere. I mean coding sometimes is repetetive. But it shouldn’t do it like that.