Skip to content

Missing some of 'status' enumerations #7

Description

@schlos

Current:
Latest Morph.io CSV dataset from 27-06-2020 in 'status' column has only items with status:

  • updated

Example data table (27-06-2020): https://docs.google.com/spreadsheets/d/1wY_RHSyR-Pc9H_tA0No95bED6KVGsW_h60Bhmcpc2Js/edit#gid=142384647

Expected:
For comparison CSV dataset from 15-09-2019 in 'status' column had items with these statuses:

  • updated
  • created
  • removed

Example data table (15-09-2019): https://docs.google.com/spreadsheets/d/1dKDJqRBNNItWSgUtGxIvfyXh8VaWrCsnT5nNwMhFD9I/edit#gid=1368599000

I think something happened here because every time we scrape from official source, we need to compare to our last available Morph.io data and store correct status:

  • updated --> only if something was updated in official source data in comparison with last data available on Morph.io
  • created --> only for new entries detected in official source data in comparison with last data available on Morph.io
  • removed --> only for non-existing entries in the official source data in comparison with last data available on Morph.io (for every new scraping all removed items should be retained indefinitely - i.e. add removed items from Morph.io and keep them in Morph.io dataset always)
  • unchanged - when entry was not changed (no change) in the official source data in comparison with last data available on Morph.io (in old definition this was specified as 'nan' status, but I think 'unchanged' would be more appropriate)

Requirements document: https://docs.google.com/document/d/1m3K_JsLjjo1QafiqDJa9mDE5Lk_ntsSWOzx9OQZrvgc/edit#

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions