The hypothesis in the description is wrong. A generated artifact goes through the following steps:

1. Multipart/form data with metadata (generator arguments + trailing “file”) to deployments service.

2. Deployments parses metadata form and uploads the file section to s3

   1. The s3 path is the same as the final destination path for the artifact

3. Deployments generates 2 signed URLs for retrieving and deleting the object and ships these as parameters to the workflows service.

4. The workflows service queues the job and eventually triggers create-artifact-worker

5. create-artifact-worker downloads the file to local storage (/tmp) and immediately removes the file from s3.

6. create-artifact-worker invokes the single-file-artifact-generator script with the supplied parameters as arguments

7. The resulting artifact is sent back to deployments using the regular artifact upload endpoint.

If this issue has only been observed for generated artifacts, the only potential issue I can see is the data race for the object path. If the file is really small the time between the pre-signed DELETE call and the following PUT with the actual artifact can be very tight potentially deleting the new object. To mitigate this issue, I created this PR: which adds a suffix to the raw file to distinguish the two.

If this is not the case, I’m afraid that the bug is not unique to generated artifact, but a bug in the upload endpoint although I have a hard time seeing how this could be. The endpoint only creates the object in the database if the file was successfully uploaded and all the sanity checks passes.