Add comprehensive README and fix UTF-8 encoding in results export - #1
Open
ahmedmest81-ctrl wants to merge 7 commits into
Open
Add comprehensive README and fix UTF-8 encoding in results export#1ahmedmest81-ctrl wants to merge 7 commits into
ahmedmest81-ctrl wants to merge 7 commits into
Conversation
…README - Add requirements.txt with gensim, numpy, scipy, scikit-learn, matplotlib - Fix TypeError in save_results(): open() does not accept ensure_ascii; use encoding='utf-8' instead - Rewrite README with step-by-step setup, model download instructions, and usage table https://claude.ai/code/session_019RwDEQvGb32Sy8e6SWd1dY
Fixes in q1_root_cluster_density.py: - Fix SyntaxError: backslash inside f-string (line 317) — extract label to variable - Fix crash when only 1 family qualifies: guard cross-family sampling with len < 2 check - Fix np.bool_/numpy float stored in result dict: cast significant to bool(), p_value to float() (np.bool_ breaks json.dump with the default encoder) Add test_q1.py: 36 unit tests covering cosine_similarity, get_available_words, compute_pairwise_similarities, compute_root_cluster_density, save_results, and lexicon sanity https://claude.ai/code/session_019RwDEQvGb32Sy8e6SWd1dY
- run_analysis.py: runs full Q1 analysis on all word families using synthetic embedding models; generates 4 plots + JSON results to q1_report/ (swap SyntheticModel for real KeyedVectors once AraVec / GloVe downloaded) - generate_report.py: standalone illustrative report with Markdown summary - Fix float32 not JSON serializable: cast cosine_similarity return to float() - Update .gitignore: exclude venv/ and q1_report/ output directory https://claude.ai/code/session_019RwDEQvGb32Sy8e6SWd1dY
Summarises Measurement 1 of the AL-MIR'ĀH framework: - Full methodology, results tables, and per-root/per-stem statistics - Comparative analysis (Arabic gap 0.1882 vs English 0.0137, ratio 13.75×) - Cohen's d discussion (Arabic 3.325 vs English 0.240) - Limitations and next-steps for production model runs https://claude.ai/code/session_019RwDEQvGb32Sy8e6SWd1dY
…base - Reorganize into almiraah/ layout: layer0/, q1_experiment/, framework/ - Add strip_diacritics() fix for AraVec undiacritised vocab lookup - Apply diacritic stripping to all Arabic family lookups (arabic=True flag) - Add per-root vocabulary coverage % to stats and output JSON - Add run_q1_real.py: production runner for Twitter/full AraVec + word2vec - Update run_analysis.py output path to q1_experiment/results/synthetic/ - Update .gitignore: exclude *.bin, *.model (too large for git) - English word2vec-google-news-300 downloaded and cached in /root/gensim-data/ - Placeholder results/twitter/ slot ready for Twitter AraVec run https://claude.ai/code/session_019RwDEQvGb32Sy8e6SWd1dY
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds complete documentation for the AL-MIR'ĀH Q1 research framework and fixes a UTF-8 encoding issue in the results export functionality.
Key Changes
encoding='utf-8'insave_results()to ensure proper handling of Arabic text and special characters in JSON outputImplementation Details
The encoding fix in
q1_root_cluster_density.pyline 481 ensures that Arabic diacritics and special characters (like ʾ in "AL-MIR'ĀH") are correctly written to the JSON results file, while maintaining the existingensure_ascii=Falseparameter for proper Unicode representation.https://claude.ai/code/session_019RwDEQvGb32Sy8e6SWd1dY