Expand identifier set

[wasade]

Replaces #3498

This pull request introduces sample_idx, prep_sample_idx and artifact_sample_idx values. These are Qiita-wide unique and provide an integer identifier for use within study, prep and artifact respectively.

[coveralls]

coverage: 92.179% (+0.01%) from 92.168%
when pulling b2e9fa9 on wasade:expand_identifier_set
into 74ad1ba on qiita-spots:dev.

[antgonza]

Thank you @wasade, this looks great. I completely understand why the code creates the unique ids on demand but my concern is that this might create an extra layer of complexity vs. adding to all from the start. In specific, my concern is that the order of the unique ids has no actual relationship on artifact/prep/study-id, which might be OK. Thus, just confirming that this is desirable before merging. What do you think?

[wasade]

Local order within a study, or prep, or artifact, is the primary goal, which we largely get, but which we cannot guarantee at all levels as samples can be added at anytime.

Global order is impossible as samples, preps and artifacts can be added at anytime.

Local order provides considerable benefit to the downstream use motiving the implementation. There is no need for semi-global order e.g. all artifacts within a prep, or all preps within a study, to be contiguous. Similarly, there is no need for global order.

As to the broader question: why don't we generate all IDs for extant data? We don't need the majority of the IDs for the downstream use. I took the current approach as it seemed like the path of least resistance with least modifications. It would take, I'm pretty sure, quite a bit more code to retroactively add all the IDs, as well as to ensure the entry points which create studies, preps, artifacts etc are generating these all correctly upfront when new things are added. Even if we did that, we wouldn't have global contiguity or semi-global contiguity for the reasons noted; for the downstream use, the approaches are qualitatively the same.

[antgonza]

Thank you for the explanation. I think it makes sense. FWIW, I think is always good to remember that "private/public" data in qiita shouldn't be removed but we have had a few occasions where we needed to allow this; I mention this because if global order is important, it might be a few/rarely a few occasions where this could happen (a gap between two ids). Anyway, merging now.

[wasade]

Thanks, ya, gaps are not an issue