Need a better/simpler way to update indexes inside Lab SDK #1602
Description
Followup from this PR:
Background:
In this case we were trying to solve the problem of coming up with a file system lock inside Lab SDK so there aren't race conditions on updating the jobs.json index. Things like filelock won't work because of remote backends, and the solution we used in this PR isn't actually atomic on S3.
Alternatively, instead of trying to solve the lock problem, we were thinking that we could change how we build the index. For example, we could make an index a directory and just create/move empty files around and use a directory listing instead of reason a JSON file. Or something else?