How Good Is AI at Coding, Really?

Study finds that while AI can be great, it also struggles due to training limitations

In collaboration with IEEE Xplore, this piece is a part of our unique IEEE Journal Watch series.

It's a full circle moment: after decades of programming code for AI models, programmers are now using AI to write code. But what is the difference between a human coder and an AI code generator?

A research assessing the functionality, complexity, and security of the code generated by OpenAI's ChatGPT was published in the IEEE Transactions on Software Engineering in June.

 

According to the results, ChatGPT's success rate in producing functional code varies greatly, ranging from as low as 0.66 percent to as high as 89 percent, depending on the task's complexity, the programming language, and a variety of other factors.

 

The investigation also reveals significant security issues with AI-generated code, even though in some circumstances the AI generator could produce better code than humans.

 

Yutian Tang, a University of Glasgow lecturer, participated in the research. He points out that there may be some benefits to using AI-based code generation in terms of increasing productivity and automating software development processes, but it's critical to comprehend the advantages and disadvantages of these models.

 

 

“By undertaking a complete analysis, we will discover cappotential troubles and obstacles that get up withinside the ChatGPT-primarily based totally code era... [and] enhance era techniques,” Tang explains.

 

To discover those obstacles in extra detail, his crew sought to check GPT-3.5`s cappotential to deal with 728 coding troubles from the LeetCode checking out platform in 5 programming languages: C, C++, Java, JavaScript, and Python.