produce_characteristic_explorer for non-English-language Corpus

[cedoard]

Hi,

I was trying to use produce_characteristic_explorer for a corpus in Italian language. You wrote this in your tutorial:

Corpus characteristicness is the difference in dense term ranks between the words in all of the documents in the study and a general English-language frequency list.

Is it possible to specify a non-English-language frequency list in produce_characteristic_explorer method?

Thank you so much!
Edoardo

[JasonKessler]

Hi Edoardo,

This is still a bit experimental and inelegant (like of Scattertext), but you can pass a subclass of CharacteristicScorer into the characteristic_scorer parameter of produce_scattertext_explorer (or any similar function).

You can see an example of this in https://github.com/JasonKessler/scattertext/blob/5af28c8860d718feb9da1d24c3cb698a946b9c70/scattertext/characteristic/DenseRankCharacteristicness.py

In short, your subclass should implement a get_scores(self, corpus) method which should return a tuple consisting of:

• a float \in [0,1] which indicates the median value of the following series (this is currently unused)
• a pd.Series which is indexed on each term in the corpus (e.g., found in corpus.get_terms()) and has values which correspond to characteristic scores

Hope this helps.

[cedoard]

Thank for fast reply and kudos for Scattertext, I love it! I guess it's a bit too demanding for my coding skills but I'll give it a try.

[antonrytting]

I have the same need (for Polish) and so if there is any update on this from either of you (Edoardo or Jason), or more details or example code snippets of how to accomplish this (creating a get_scores() method for a non-English corpus), I would appreciate it.

Failing that, a little more detail on how the English background frequency list is being used is helpful. I'm not familiar with the pkgutils library or the get_data() function, but it appears that data/count_1w.txt is simply a list of English words and their frequencies in a very large corpus (perhaps the one provided by Peter Norvig on this webpage, as I believe I read in another post?). If an equivalent frequency list of Polish words or lemmas is provided as an argument of BackgroundFrequencies.get_background_frequency_df(frequency_path) in lieu of data/count_1w.txt, would that be sufficient to get the get_background_frequency_df() function to work properly?

Is that change sufficient to implement a get_scores(self, corpus) method or get produce_characteristic_explorer working for a non-English language like Polish, or are there other changes that have to be made?

[JasonKessler]

Thanks for your question Anton.

I've updated the Readme and some of the API to make it easier to use custom background frequency lists.

Please take a look at this new section of the Readme: https://github.com/JasonKessler/scattertext#using-custom-background-word-frequences