Researchers have found that large language models (LLMs) tend to parrot buggy code when tasked with completing flawed snippets.
That is to say, when shown a snippet of shoddy code and asked to fill in the blanks, AI models are just as likely to repeat the mistake as to fix it.
As a software developer I’ve never used AI to write code, but several of my friends use it daily and they say it really helps them in their jobs. To explain this to non-programmers, they don’t tell it “Write some code” and then watch TV while it does their job. Coding involves a lot of very routine busy work that’s little more than typing. AI can generate approximately what they want, which they then edit, and according to them this helps them work a lot faster.
A hammer is a useful tool, even though can’t build a building by itself and is really shitty as a drill. I look at AI the same way.
We have a handful of Python tools that we require to adhere to PEP8 formatting, and have Jenkins pipeline jobs to validate it and block merge requests if any of the code isn’t properly formatted. I haven’t personally tried it yet, but I wonder if these AI’s might be good for fixing up this sort of formatting lint.
Exactly. I have a coworker use it effectively.
Personally, I’ve been around the block so it’s usually faster for me to just do the busy work myself. I have lots of tricks for manipulating text quickly (I’m quite proficient with vim), so it’s not a big deal to automate turning JSON into a serializer class or copy and modify a function a bunch of times to build out a bunch of controllers or something. What takes others on my team 30 min I can sometimes get done in 5 through the power of regex or macros.
But at the end of the day, it doesn’t really matter what tools you use because you’re not being paid for your typing speed or ability to do mundane work quickly, you’re being paid to design and support complex software.
That’s right. You watch it type it out and right where it gets to the important part you realize that’s not what you meant at all, so you hit the stop button. Then you modify the prompt and repeat that one more time. That’s when you realize there are so many things it’s not even considering which gives you the satisfaction that your job is still secure. Then you write a more focused prompt for one aspect of them problem and take whatever good enough bullshit it spewed as a starting point for you to do the manual work. Rinse and repeat.