We all have repositories with open issues. I'd suggest that you open up github, click the first issue you have open, and try to close it with ChatGPT assistance. For me, this revealed a gap between its spectacular skill at programming puzzles and its total incompetence at closing issues- it can do dynamic programming challenges instantly but can't modify (a function that loads a file from disk, or downloads that file if it isn't already present) to (the same function but it correctly handles a failed download)
I did exactly this. I'm doing some frontend work where I only have basic knowledge and there was a small feature that was a nice to have. When I originally wrote my own solution, it didn't work. So I took my failed solution and out it into Chat GPT and went back and fourth a few times. It also didn't provide solutions that worked after trying them. So I started asking about alternative solutions and eventually that sparked a new idea I could ask about. That ended up working.
Probably a 5 min conversation with a frontend dev would have gotten me that answer faster, but that isn't something I have access to at the moment. And it was far enough down my priority list that it probably would have stayed unimplemented for a long time. With the back and forth with ChatGPT I got there in maybe 30 minutes of playing around. So a direct solution - for sure not. But better than my alternatives, I think so.
One way I think about it, AI can learn when there are many high quality examples. There are tons of blog articles on small topics but none really like "here's 1000 lines of sparsely documented code, good luck". The ability to navigate that is a secondary skill and can't be achieved with even genius parroting.
Until the parrot becomes even more clever, perhaps...
Fix a function to gracefully handle an old version of git
This last one is illustrative, as its proposed solution looks elegant, correct, and well explained, but totally fails.
QuadmasterXLII:
This function crashes if an old version of git is installed that doesn't have the :(exclude) feature. Can you fix it?
```
def initialize(run_name=None, output_root="results/"):
global output_dir_impl
global initialized
if initialized:
raise Exception("footsteps can only be initialized once")
initialized = True
if not run_name:
if "FOOTSTEPS_NAME" in os.environ:
run_name = os.environ["FOOTSTEPS_NAME"]
else:
print("Input name of experiment:")
run_name = input()
output_dir_impl = os.path.join(output_root, run_name) + "/"
suffix = 0
while os.path.exists(output_dir_impl):
suffix += 1
output_dir_impl = os.path.join(output_root, run_name) + "-" + str(suffix) + "/"
os.makedirs(output_dir_impl)
print("Saving results to " + output_dir_impl)
with open(output_dir_impl + "info.txt", "w") as f:
f.write("Command:\n")
f.write(" ".join(sys.argv) + "\n")
f.write("System:\n")
f.write(subprocess.check_output(["hostname"]).decode())
f.write("Python:\n")
f.write(shutil.which("python") + "\n")
f.write("Git Hash:\n")
f.write(
subprocess.check_output(["git", "describe", "--always"]).strip().decode()
+ "\n"
)
f.write("Uncommitted changes:\n")
f.write(
subprocess.check_output(
["git", "diff", "HEAD", "--", ".", ":(exclude)*.ipynb"]
).decode()
)
f.write("Current working dir:\n")
f.write(os.getcwd() + "\n")
try:
shutil.copy(sys.argv[0], output_dir_impl + os.path.basename(sys.argv[0]))
except:
pass
```
ChatGPT:
This function checks whether the :(exclude) feature is available in the installed version of git by calling git diff with :(exclude).ipynb. If the feature is not available, git will return an error and the function will crash.
To fix this, we can add a check to see if git supports the :(exclude) feature before calling git diff with it. We can do this by calling git help diff and checking the output for a mention of :(exclude). If the feature is not supported, we can call git diff without the :(exclude) feature.
Because this code has 7 or 8 unit tests but none of them verify that notebooks are actually excluded from the diff, the gpt code which is wrong and so always calls the fallback _passes the unit test suite_. Brace yourselves for a new era of buggy code