Add AI disclosure to PR form - #1634
Conversation
Also, tiny wording change for "Expected Usage" section comment. It's tricky to comment out a section; much easier to delete a section. So, now it says to delete it (if it doesn't apply) instead of to comment it out.
Sevans711
added
documentation
| <!-- If you did not use AI, please write "AI Usage: N/A" and remove these checklist items or mark as [N/A]. | ||
| Otherwise, please briefly specify all tools used and how they were used. Include model if known. Examples: | ||
| "AI Usage: Claude (Fable 5) made all code edits, but I came up with the ideas and design for this feature myself." | ||
| "AI Usage: discussion with Gemini and ChatGPT (5.5 Instant) to help find and understand the cause of this bug." |
There was a problem hiding this comment.
Do we care if a developer discussed an issue with an AI if they wound up writing the code themselves?
There was a problem hiding this comment.
Personally, yes, but I'm also curious to hear what others think about this.
My understanding is that using AI influences our thoughts, behaviors, work, etc, even if we do not utilize its suggestions directly. E.g., this study (with faster-to-read press release) finds that being exposed to inline autocomplete suggestions from a biased AI affects people's views, even for people who are informed about the AI's bias ahead of time.
I would guess there are probably similar studies about how your problem-solving gets affected by discussing something with an AI, though I haven't taken a close look at any papers like that yet so I could easily be wrong.
There was a problem hiding this comment.
I would say no, from my experience. Having a back-and-forth with Claude is a very different workflow than just vibing it. And anyway, trying to control that would get us into the position of having to be the thought police.
There was a problem hiding this comment.
I agree with @Sevans711, and this distinction may matter less with time, but I think it's still good to know for now. Without knowing much about user or the model, results can be pretty unpredictable. Like Chatgpt with no access to the whole file, being prompted by a new user may have wildly different credibility than a core user/developer using Claude Code which has access to a variety of inputs, like git, other files, dependencies, and more.
I could also see myself ticking "yes" even if I wrote the code, because I'll use models as "proofreading" after writing code, and it can either expand on or validate my findings. Similar to spell check in writing products. In this case it may be nice to know that the user at least went through some form of proofreading/testing before submitting.
There was a problem hiding this comment.
I guess I just don't see the ROI here. Can we imagine a situation where someone uses AI to think through a problem, writes the code themselves, takes responsibility for the code, and then... what? I can't imagine anything seriously negative happening in that situation.
There was a problem hiding this comment.
I see @Sevans711 point about influencing thinking and it would be nice to have stats on that but this feels intrusive and would be discouraging. Maybe just ask "if you used AI, what model did you use?" and we'll leave "use" undefined. Definitely want the checkboxes for responsibility.
There was a problem hiding this comment.
I don't think the main purpose here is to control the ways people use AI, at least, the AI Usage examples aren't trying to do that. That feels more like a job for an AI Usage Policy (which would also be good but a separate topic).
A situation I can imagine is something like: "I used AI to discuss the problem and to help me while drafting this PR. But, all the AI Usage examples in the comments were describing how it was used to edit the code. I didn't edit code with it, so I didn't actually, really use AI, then, right? All I did was discuss and take the ideas it gave me and write them up? I think I probably don't really need to worry about checking these boxes either, then. I'll just put N/A." And then later, we review and ask follow up questions, and get replies like "oh, well Claude said this is the way to go, so I think it has to be right." or "I didn't really understand that, I was just following what ChatGPT said."
In terms of ROI, I think the possible returns are high (saving reviewers' time, and reviewing PRs can take a while) though it's hard to guess their likelihood (not sure how often a situation like the one above would come up). Meanwhile, the cost feels pretty low. I would guess that it takes adds at most a minute of extra time to think about and type out a sentence like "yes I discussed this with Claude".
There was a problem hiding this comment.
this feels intrusive and would be discouraging. Maybe just ask "if you used AI, what model did you use?" and we'll leave "use" undefined. Definitely want the checkboxes for responsibility.
(Woops, didn't see your response before I sent my reply above.)
If it would be discouraging that's definitely an extra cost that I didn't consider above. Can you clarify a bit more on why it would be discouraging? Examples, maybe as simple as "taking the extra time to type something out would get annoying", or maybe more rooted in people's perceptions about AI use, like "I don't want people to think I rely on AI too much", or "I don't want my contributions to be devalued because I used AI"?
Not trying to place any judgements, just trying to understand the reasoning a bit better, if possible!
There was a problem hiding this comment.
Every little bit of friction is discouraging to some extent. People's time and effort is limited, and we should be mindful of imposing too much bureaucracy where it isn't needed. A lot of projects succeed just because they're more organized and easier to work on than the alternatives.
I guess I'm way more comfortable with letting some things be a black box. I don't see this as any different from using Google or StackOverflow.
There was a problem hiding this comment.
Discouraging (to new contributors) because you have to write more and possibly a lot more if you really want an accounting of every tool and LLM used and how it was used. I just want the LLM and maybe the harness.
|
(There are two approving reviews but my suggestion would be to wait to merge this for a little while, maybe 1 week(?), to give everyone linked here a chance to take a look and hash things out in the discussion above.) |
4 participants
Fixes part of #1630 but does not fully close it.
Overview
Adds a simple AI Disclosure section to PR form. Aiming for a relatively lightweight section here, in accordance with original issue description (#1630) and discussion (#1613).
This PR does not fully close the issue because it does not attempt to create an AI Usage Policy; that should be scoped to a different PR.
Also, unrelated to original issue, made a tiny rephrase for comment in "Expected Usage" section: now it says to delete the section (if it doesn't apply) instead of commenting it out. Reason for this change: if you're not super familiar with markdown syntax, it's easier to just delete something, rather than figuring out how to comment it out, and both outcomes are functionally the same here.
PR Checklist
General