As part of testing, Coinbase crypto exchange employees independently analyzed 20 smart contracts of various tokens for risks, and then used ChatGPT for the same purpose.

In 12 cases out of 20, the conclusions of the neural network about the level of risks coincided with the conclusions of the experts.

“Of the eight cases of incorrect risk assessment, ChatGPT assigned a low risk rating to high-risk tokens in five cases. This is the worst possible mistake – underestimating the risk of an asset can be much more dangerous than overestimating the level of risk, ”company representatives say.

Analysts noted the benefits of ChatGPT, but said that it is too early to leave manual testing of smart contracts. According to Coinbase experts, a neural network does not perform well when it does not have enough contextual data to perform analysis. In addition, ChatGPT pays too much attention to comments in the code of smart contracts, while the logic of the program itself is more important.

“While ChatGPT shows potential for rapid risk assessment of smart contracts, the findings of the neural network are too uncertain to be integrated into Coinbase’s asset security assessment processes. We expect that with the right engineering, the accuracy of the tool can be improved, but it is impossible to fully rely on it, ”conclude representatives of the exchange.

The popularity of the ChatGPT neural network attracted the attention of scammers. Recently, the developers of the Harvest Keeper project, which “used AI and neural networks to optimize trading,” escaped with over $1 million in user assets.