A very nice paper was published by Maria del Rio-Chanona, Nadzeya Laurentsyeva, and Johannes Wachs in July 2023, titled Are Large Language Models a Threat to Digital Public Goods? Evidence from Activity on Stack Overflow.
The empirical methodology of the paper looks sound to me, and its general point is indicated by its title. Namely, as ChatGPT and other LLMs trained on discussions of programming languages, including programming language code itself, increases in use, that tends to displace use of Stack Overflow to seek analogous answers.
del Rio-Chanona and colleagues address a very reasonable point. The main concern they evince is that decreased use of sites where human programmers share code and insights will lead to decreased source material for future models. As new programming languages, paradigms, libraries, etc. develop, the corpora to use for training LLMs may dry up.
In their paper, they analyze sites in other natural languages that are less used to train popular models, and thereby show less of this effect. As well, less popular languages—with less representation in the training corpora, show less of this displacement.
In my mind, a concern that is at least as important is the ways in which robot assistance may break up the natural community-building function of code sharing sites like Stack Overflow. When I—or other developers—utilize sites like that, certainly it is true that a large, even primary, motivation is in seeking answer to specific programming challenges we face on a particular day. But a wonderful and positive side effect of that effort is that we often form bonds with the human beings whom we come to recognize as providing answers, or who appreciate the answers we provide them.
Clearly, interacting with an AI—regardless of how good or bad its code solutions and algorithm descriptions might be—loses all opportunity to form such passing, but rewarding, connections… connections which often lead to future, longer-term, collaboration on minimally related projects. The internet, social media, pandemic, and other factors, have already led to more social isolation than anyone can wish for. It would be a shame for generative AI coding assistants to exacerbate that negative trend.