Refactor TI Providers to create generic context providers

[Ian Hellen (ianhelle)]

This is a discussion to widen the scope of issue #263.

FlorianBracq is proposing the following model:
ContextProvider/ContextLookup generic classes in the msticpy.context folder.
TILookup/TIProviders would be subclasses of these, specifically scoped for TI-related entities.

All of the generic functionality such as pre-processing, results, etc. would move to context folder.

We do have legacy naming to contend with. E.g.

• contextprovider.lookup_item -> tiprovider.lookup_ioc

Ian Hellen (@ianhelle) proposed keeping the inherited name and adding compatibility attributes

class TIProviderX(ContextProvider):

    def lookup_item(observable=None, obs_type=None, .... **kwargs):
           """
          Return TI results from blah...

          observable: Optional[str]
              description....
          ioc: Optional[str]
               alias of observable.
          ....
          """
          obs = observable or kwargs.pop("ioc", None)
          ....
    
    lookup_ioc = lookup_item

[Ian Hellen (ianhelle)]

Florian has proposed branch here https://github.com/FlorianBracq/msticpy/tree/context_provider

[FlorianBracq]

I did something a tad different:

• Define a generic class Provider
• Have children classes inheriting from Provider
  • TIProvider => base class for all TI Providers
  • ContextProvider => base class for all ContextProviders
  • HTTPLookupProvider => base class for Providers relying on HTTP to retrieve data
• Define HttpTIProvider inheriting from TIProvider and HTTPLookupProvider => base class for API-based TI Providers
• Define HttpContextProvider inheriting from TIProvider and HTTPLookupProvider => base class for API-based Context Providers

that way if we have other Providers (more than the 2 currently identified), we can replicate the logic. However, looking Ian Hellen (@ianhelle)'s comment that could indeed be over-engineering....

[Ian Hellen (ianhelle)]

I think we can adapt this to working for HTTPLookupProvider. I think the concept would (should) work in the same way.
In any case, I won't try to PR this work until the generic provider work is done - obviously we'd want to be able to have providers of any type use this declarative format.

[Ian Hellen (ianhelle)]

BTW I'll try to review properly next week. Hackathon work has kept me pinned down all week.

[Ian Hellen (ianhelle)]

A couple of other things (notes to self) that I'd like to do in this work:

• Have lookup_ioc (or lookup_observable) actually call lookup_iocs (wrapping the single ioc in an iterable) - that way it will benefit from the async lookup across providers. To do this I'll implement an internal _lookup_single_observable and have lookup_iocs call that. Then lookup_ioc will be free to call lookup_iocs([ioc])
• Implement an lru_cache bypass for lookup_ioc - taking a kwarg of cache=False. This would call the raw _lookup_single_observable. If cache=True (default) then it calls a wrapper _lookup_cached_single_observable. This might be needed if you wanted to lookup immediate changes (more likely for ServiceNow, etc. than TI).
• function to manually load TI provider (by name or from definition)
• implement providers (or maybe just TILookup) as a singleton - so multiple "instances" retain the same cache and configuration (e.g. enable/disable providers).
• it would also be good to implement multiple instances of the same provider - e.g. having > 1 config entry that uses the same class but with different params/API keys.

[Ian Hellen (ianhelle)]

This is implemented in release 2.2
Some of the items from my last comment are being done in 2.3