Skip to main content
improved phrasing
Source Link
cottontail
  • 1.3k
  • 3
  • 7
  • 14

Attribution, where art thou?

A vast majority of Stack Overflow users want their posts read and be useful to other users. If Stack Overflow itself has an assistant feature at the top that supposedly searches through posts but never show where it got its answer or just gives irrelevant posts that puts off the searcher, Stack Overflow will lose both askers and answerers IMO.

If AI Assist on Stack Overflow is not grounded in Stack Overflow posts or does not surface Stack Overflow posts or has to rely on the underlying LLM (that used Stack Overflow posts in pre-training) to generate a response that rephrases Stack Overflow answers without attribution, what is the point?


I'll give 2 examplesan example.

Case 1: No reference

If it is truly searching through Stack Overflow posts, then it should include links to the actual Stack Overflow posts it used, but it doesn't. The question below: "how to make a function a member of an instance of a class in Python" was already asked and answered on Stack Overflow 14 years ago. In fact, if you look at AI Assist's answer, the code is identical to the accepted answer there. OpenAI's model has definitely seen this Q/A. So shouldn't AI Assist show this answer as a supporting document? Apparently not.

no attribution

Case 2: Irrelevant reference

If I explicitly ask for attributionan explicit reference is asked (note that the question is otherwise identical to above), I get a supporting post that is not aboutgiven but it doesn't answer the question I asked. Also the answerAI Assist's generation doesn't seem to reference the retrieved post either because what the answer saysgenerated solution is different from the one in the linked post.

wrong post

Attribution, where art thou?

A vast majority of Stack Overflow users want their posts read and be useful to other users. If Stack Overflow itself has an assistant feature at the top that supposedly searches through posts but never show where it got its answer or just gives irrelevant posts that puts off the searcher, Stack Overflow will lose both askers and answerers IMO.


I'll give 2 examples.

Case 1: No reference

If it is truly searching through Stack Overflow posts, then it should include links to the actual Stack Overflow posts it used, but it doesn't.

no attribution

Case 2: Irrelevant reference

If I explicitly ask for attribution, I get a supporting post that is not about the question I asked. Also the answer doesn't seem to reference the retrieved post either because what the answer says is different from the linked post.

wrong post

Attribution, where art thou?

A vast majority of Stack Overflow users want their posts read and be useful to other users. If Stack Overflow itself has an assistant feature at the top that supposedly searches through posts but never show where it got its answer or just gives irrelevant posts that puts off the searcher, Stack Overflow will lose both askers and answerers IMO.

If AI Assist on Stack Overflow is not grounded in Stack Overflow posts or does not surface Stack Overflow posts or has to rely on the underlying LLM (that used Stack Overflow posts in pre-training) to generate a response that rephrases Stack Overflow answers without attribution, what is the point?


I'll give an example.

Case 1: No reference

If it is truly searching through Stack Overflow posts, then it should include links to the actual Stack Overflow posts it used, but it doesn't. The question below: "how to make a function a member of an instance of a class in Python" was already asked and answered on Stack Overflow 14 years ago. In fact, if you look at AI Assist's answer, the code is identical to the accepted answer there. OpenAI's model has definitely seen this Q/A. So shouldn't AI Assist show this answer as a supporting document? Apparently not.

no attribution

Case 2: Irrelevant reference

If an explicit reference is asked (note that the question is otherwise identical to above), a supporting post is given but it doesn't answer the question. Also the AI Assist's generation doesn't seem to reference the retrieved post either because the generated solution is different from the one in the linked post.

wrong post

Source Link
cottontail
  • 1.3k
  • 3
  • 7
  • 14

Attribution, where art thou?

A vast majority of Stack Overflow users want their posts read and be useful to other users. If Stack Overflow itself has an assistant feature at the top that supposedly searches through posts but never show where it got its answer or just gives irrelevant posts that puts off the searcher, Stack Overflow will lose both askers and answerers IMO.


I'll give 2 examples.

Case 1: No reference

If it is truly searching through Stack Overflow posts, then it should include links to the actual Stack Overflow posts it used, but it doesn't.

no attribution

Case 2: Irrelevant reference

If I explicitly ask for attribution, I get a supporting post that is not about the question I asked. Also the answer doesn't seem to reference the retrieved post either because what the answer says is different from the linked post.

wrong post