Current:
Latest Morph.io CSV dataset from 27-06-2020 in 'status' column has only items with status:
Example data table (27-06-2020): https://docs.google.com/spreadsheets/d/1wY_RHSyR-Pc9H_tA0No95bED6KVGsW_h60Bhmcpc2Js/edit#gid=142384647
Expected:
For comparison CSV dataset from 15-09-2019 in 'status' column had items with these statuses:
Example data table (15-09-2019): https://docs.google.com/spreadsheets/d/1dKDJqRBNNItWSgUtGxIvfyXh8VaWrCsnT5nNwMhFD9I/edit#gid=1368599000
I think something happened here because every time we scrape from official source, we need to compare to our last available Morph.io data and store correct status:
- updated --> only if something was updated in official source data in comparison with last data available on Morph.io
- created --> only for new entries detected in official source data in comparison with last data available on Morph.io
- removed --> only for non-existing entries in the official source data in comparison with last data available on Morph.io (for every new scraping all removed items should be retained indefinitely - i.e. add removed items from Morph.io and keep them in Morph.io dataset always)
- unchanged - when entry was not changed (no change) in the official source data in comparison with last data available on Morph.io (in old definition this was specified as 'nan' status, but I think 'unchanged' would be more appropriate)
Requirements document: https://docs.google.com/document/d/1m3K_JsLjjo1QafiqDJa9mDE5Lk_ntsSWOzx9OQZrvgc/edit#
Current:
Latest Morph.io CSV dataset from 27-06-2020 in 'status' column has only items with status:
Example data table (27-06-2020): https://docs.google.com/spreadsheets/d/1wY_RHSyR-Pc9H_tA0No95bED6KVGsW_h60Bhmcpc2Js/edit#gid=142384647
Expected:
For comparison CSV dataset from 15-09-2019 in 'status' column had items with these statuses:
Example data table (15-09-2019): https://docs.google.com/spreadsheets/d/1dKDJqRBNNItWSgUtGxIvfyXh8VaWrCsnT5nNwMhFD9I/edit#gid=1368599000
I think something happened here because every time we scrape from official source, we need to compare to our last available Morph.io data and store correct status:
Requirements document: https://docs.google.com/document/d/1m3K_JsLjjo1QafiqDJa9mDE5Lk_ntsSWOzx9OQZrvgc/edit#