[WP]: Static Site Generator Migration to MkDocs

[shorodilov]

Assigned To: project-administrator (Claude Code)
Assigned By: project-manager (Claude Chat)

---

Objective

Migrate the project's static site generator from Sphinx to MkDocs with Material theme by converting content directly from reStructuredText to Markdown, establishing multilingual support for English and Ukrainian, and setting up automated deployment.

---

Backstory

The project currently uses Sphinx with reStructuredText, which creates contributor friction and is overkill for the project's needs. ADR-002 approves migration to MkDocs with Material theme to improve author experience (Markdown), reduce complexity, and maintain professional documentation quality. This migration converts content directly to the new format and location in one operation, preserving the original src/ directory as a fallback until the new SSG is fully validated.

---

Branch Instructions

Create and work on feature branch:

git fetch origin
git branch feature/wp-mkdocs-migration origin/main
git checkout feature/wp-mkdocs-migration
git push --set-upstream origin feature/wp-mkdocs-migration

All work happens on feature/wp-mkdocs-migration branch. Do NOT merge to main until verification is complete.

---

Definition of Done

• All content converted directly from src/*.rst to content/en/*.md with integrity preserved
• Original src/ directory preserved as fallback (not deleted)
• MkDocs installed and configured with Material theme
• mkdocs-static-i18n plugin configured for English/Ukrainian localization
• Ukrainian content structure (content/uk/) established with file-based i18n
• All documentation features working (code highlighting, admonitions, internal links, search)
• Site builds successfully with mkdocs build
• GitHub Actions workflow configured for automated deployment to GitHub Pages
• Site deployed successfully to GitHub Pages and fully validated
• Project documentation updated (README, contributor guides) for Markdown/MkDocs workflows
• src/ directory removed only after full validation of MkDocs site
• All changes committed with comprehensive commit message
• No content loss or corruption during conversion

---

Context

Reference Documents:

• ADR-002: Static Site Generator Replacement (docs/ADR-002-ssg-replacement.md)
• ADR-003: Repository File Structure (docs/ADR-003-repo-file-structure.md)
• MkDocs documentation: https://www.mkdocs.org/
• Material for MkDocs: https://squidfunk.github.io/mkdocs-material/
• mkdocs-static-i18n plugin: https://github.com/ultrabug/mkdocs-static-i18n

Key Context:

• Content currently in src/ as .rst files
• Legacy Russian content being organized by WP-244A (separate, independent work)
• Ukrainian translations exist as gettext .po files in src/_locales/uk/
• MkDocs will use file-based i18n approach (separate content/uk/ directory)
• Deployment target is GitHub Pages (existing setup)
• src/ preserved as fallback until new SSG fully validated

Migration Strategy:

• Direct conversion: src/file.rst → converter → content/en/file.md
• No intermediate moves: Files converted directly to target location
• Preserve fallback: Keep src/ intact until MkDocs proven working
• Clean up only after validation: Remove src/ as final step

Decision from ADR-002:

• Chosen SSG: MkDocs with Material theme
• Format: Markdown
• Localization: mkdocs-static-i18n with file-based approach
• Deployment: GitHub Actions + GitHub Pages

Current Tech Stack:

• Static site generator: Sphinx
• Content format: reStructuredText (.rst)
• Build tool: make + Sphinx
• Deployment: GitHub Pages (current)

Migration Objectives:

1. Convert all .rst to .md directly in target location (content/en/)
2. Replace Sphinx with MkDocs + Material theme
3. Migrate Ukrainian translations from gettext to file-based i18n
4. Configure Material theme for professional appearance
5. Set up automated deployment pipeline
6. Update all project documentation
7. Clean up Sphinx artifacts only after full validation

---

Deliverables

1. Content Format Conversion

Convert directly from source to target:

src/file.rst  →  [converter]  →  content/en/file.md
src/subdir/file.rst  →  [converter]  →  content/en/subdir/file.md

Conversion requirements:

• Preserve content structure and meaning
• Convert Sphinx directives to MkDocs/Material equivalents:
  • .. code-block:: → ```language
  • .. note:: → !!! note
  • .. warning:: → !!! warning
  • Internal links: :doc: → [text](file.md)
  • Images: .. image:: → ![alt](path)
• Preserve code examples, formatting, emphasis
• Maintain directory structure within content/en/
• Verify no content loss or corruption

DO NOT:

• Delete or modify files in src/
• Move files from src/ (convert to new location instead)
• Remove src/ directory (keep as fallback)

Tools/approaches you may use:

• pandoc for bulk conversion (with manual review)
• Manual conversion for complex/edge cases
• Regex/scripting for systematic transformations
• Diff checking to verify accuracy

2. MkDocs Configuration

Create mkdocs.yml at repository root:

site_name: Python for Web Developers
site_url: https://openroost.github.io/pymastery-vp/
repo_url: https://github.com/OpenRoost/pymastery-vp
repo_name: OpenRoost/pymastery-vp

docs_dir: content

theme:
  name: material
  language: en
  features:
    - navigation.tabs
    - navigation.sections
    - navigation.top
    - search.suggest
    - search.highlight
    - content.code.copy
  palette:
    - scheme: default
      primary: indigo
      accent: indigo
      toggle:
        icon: material/brightness-7
        name: Switch to dark mode
    - scheme: slate
      primary: indigo
      accent: indigo
      toggle:
        icon: material/brightness-4
        name: Switch to light mode

plugins:
  - search
  - i18n:
      docs_structure: folder
      languages:
        - locale: en
          name: English
          build: true
          default: true
        - locale: uk
          name: Українська
          build: true

markdown_extensions:
  - admonition
  - codehilite
  - pymdownx.highlight
  - pymdownx.superfences
  - pymdownx.inlinehilite
  - toc:
      permalink: true
  - def_list
  - attr_list
  - md_in_html

extra:
  social:
    - icon: fontawesome/brands/github
      link: https://github.com/OpenRoost/pymastery-vp

copyright: Copyright © 2024 OpenRoost

Adjust as needed based on actual content structure and requirements.

3. Python Dependencies

Create requirements.txt for MkDocs:

mkdocs>=1.5.0
mkdocs-material>=9.5.0
mkdocs-static-i18n>=1.2.0
pymdown-extensions>=10.0

Or use pyproject.toml if project uses modern Python packaging.

Note: Keep Sphinx dependencies temporarily during transition phase. Remove only after full validation.

4. Ukrainian Localization Setup

Create content/uk/ directory structure:

• Mirror content/en/ structure
• Translate Markdown files from English to Ukrainian
• Use existing .po files in src/_locales/uk/ as translation reference
• File-based approach: each English file gets Ukrainian equivalent

Translation strategy options:

1. Manual translation (time-consuming but highest quality)
2. Use existing .po translations as reference (extract msgstr values)
3. Machine translation + manual review (faster, requires QA)

Scope flexibility:

If full Ukrainian translation is too time-intensive for this work package, you may:

• Create stub structure with placeholder files
• Document translation as future work
• Or focus on a subset of content for initial launch

Consult with Project Owner if translation scope needs adjustment.

5. GitHub Actions Deployment

Create .github/workflows/deploy-docs.yml:

name: Deploy Documentation

on:
  push:
    branches:
      - main
  workflow_dispatch:

permissions:
  contents: read
  pages: write
  id-token: write

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      
      - name: Install dependencies
        run: |
          pip install -r requirements.txt
      
      - name: Build documentation
        run: mkdocs build
      
      - name: Upload artifact
        uses: actions/upload-pages-artifact@v3
        with:
          path: site
      
      - name: Deploy to GitHub Pages
        uses: actions/deploy-pages@v4

Adjust based on actual repository settings and deployment requirements.

6. Sphinx Artifact Cleanup (Only After Full Validation)

IMPORTANT: Do this LAST, only after MkDocs site is fully validated and approved.

Remove the following Sphinx-specific files/directories:

• src/ (entire directory - kept as fallback until now)
• conf.py (Sphinx configuration)
• Makefile (if Sphinx-specific)
• make.bat (if exists)
• _build/ (build output directory)

Use git rm so history is preserved.

Verify removal:

# No Sphinx references should remain
grep -r "sphinx" . --exclude-dir=.git

7. Documentation Updates

Update README.rst (or convert to README.md if preferred):

• Replace Sphinx build instructions with MkDocs instructions
• Update "Local Development" section:

  # Install dependencies
  pip install -r requirements.txt
  
  # Local preview
  mkdocs serve
  
  # Build site
  mkdocs build

• Update contributor guidance for Markdown authoring
• Remove reStructuredText references
• Update deployment information

Create/update CONTRIBUTING.md:

• Document Markdown authoring guidelines
• Explain MkDocs admonition syntax
• Provide code block examples
• Document localization workflow (how to add Ukrainian translations)
• Reference Material for MkDocs documentation for advanced features

Update .ai/config.yaml:

• Replace Sphinx references with MkDocs
• Update build tool information
• Update content format (Markdown)
• Update content location (content/en/)

Update any other documentation referencing Sphinx or .rst format.

8. Git Commits

Recommended commit strategy (multiple commits for clarity):

Commit 1: Add MkDocs infrastructure

feat(docs): add MkDocs with Material theme configuration

Part of #243 (ADR-002 SSG Replacement)

Add MkDocs configuration, dependencies, and GitHub Actions workflow.
Original Sphinx setup preserved as fallback during transition.

Changes:
- Add mkdocs.yml with Material theme configuration
- Add requirements.txt with MkDocs dependencies
- Add .github/workflows/deploy-docs.yml for automated deployment
- Configure mkdocs-static-i18n for en/uk localization

Sphinx infrastructure remains active during transition phase.

Refs: #243, ADR-002

Commit 2: Convert content to Markdown

feat(docs): convert course content from reST to Markdown

Part of #243 (ADR-002 SSG Replacement)

Convert all course content directly from src/*.rst to content/en/*.md
format. Original src/ preserved as fallback until full validation.

Changes:
- Convert all .rst files to .md in content/en/
- Migrate Sphinx directives to Material for MkDocs syntax
- Preserve content structure and directory organization
- Verify content integrity (no information loss)

Original src/ directory preserved for rollback capability.

Refs: #243, ADR-002

Commit 3: Add Ukrainian localization structure

feat(docs): establish Ukrainian localization structure

Part of #243 (ADR-002 SSG Replacement)

Create content/uk/ structure for file-based i18n translations.
[Adjust message based on whether full or stub translation done]

Changes:
- Create content/uk/ mirroring content/en/ structure
- [If full: Translate content using src/_locales/uk/ as reference]
- [If stub: Add placeholder files, document as future work]
- Configure mkdocs-static-i18n plugin

Refs: #243, ADR-002

Commit 4: Update documentation

docs: update project documentation for MkDocs workflow

Part of #243 (ADR-002 SSG Replacement)

Update all project documentation to reflect Markdown authoring
and MkDocs build processes.

Changes:
- Update README with MkDocs build instructions
- Update/create CONTRIBUTING.md for Markdown authoring
- Update .ai/config.yaml with new tooling information
- Remove reStructuredText references

Refs: #243, ADR-002

Commit 5: Clean up Sphinx artifacts (ONLY AFTER FULL VALIDATION)

chore: remove Sphinx infrastructure after MkDocs validation

Part of #243 (ADR-002 SSG Replacement)

Remove Sphinx-specific files after successful MkDocs migration
validation. MkDocs site confirmed working on GitHub Pages.

BREAKING CHANGE: Sphinx build system removed

Changes:
- Remove src/ directory (content migrated to content/en/)
- Remove conf.py (Sphinx configuration)
- Remove Makefile and make.bat (Sphinx build scripts)
- Remove _build/ directory
- Remove Sphinx from requirements

MkDocs is now the sole documentation system.

Refs: #243, ADR-002

---

Constraints

CRITICAL - Must Follow:

1. Content Integrity: No content loss or meaning changes during .rst → .md conversion. Verify accuracy.

2. Preserve Fallback: Do NOT delete src/ directory until MkDocs site is fully validated and approved by Project Owner.

3. Direct Conversion: Convert files directly to target location. Do NOT move .rst files to content/en/ then convert in place.

4. Branch Protection: ALL work on feature/wp-mkdocs-migration branch. Do NOT merge to main without approval.

5. Build Success: Site MUST build successfully with mkdocs build before completion.

6. Feature Parity: All current documentation features must work in MkDocs (code highlighting, admonitions, cross-references, search).

7. Deployment Verification: Verify site deploys successfully to GitHub Pages and works correctly before cleanup.

8. Russian Content Unchanged: content/ru/ (if it exists from WP-244A) stays untouched.

9. Asset Preservation: /assets/ directory and contents unchanged.

10. Configuration Accuracy: mkdocs.yml must reference correct paths and languages.

11. Staged Cleanup: Remove Sphinx artifacts ONLY as final step after full validation.

Explicitly DO NOT:

• Delete src/ directory before MkDocs is fully validated
• Move .rst files then convert (use direct conversion instead)
• Modify course content text (only convert format)
• Change Russian legacy content in content/ru/ (if present)
• Modify /assets/ directory
• Make content improvements or fixes (format conversion only)
• Remove git history for any files
• Merge to main before full validation

Scope Flexibility:

If Ukrainian translation proves too time-intensive, you may:

• Create stub content/uk/ structure with placeholder files
• Document translation as future work
• Implement infrastructure for i18n without full content translation

Consult Project Owner if scope needs adjustment.

---

Verification Steps

Self-QC checklist before considering work complete:

1. Conversion Verification

# Verify all content converted to content/en/
find content/en/ -name "*.md" | wc -l
# Should match count of .rst files in src/

# Verify src/ still exists (fallback preserved)
ls -la src/
# Should show original content unchanged

# Spot-check conversions for accuracy
# Pick 3-5 files and manually verify:
# - Code blocks render correctly
# - Admonitions converted properly
# - Internal links work
# - Images display correctly
# - Formatting preserved

2. Build Verification

# Clean build from scratch
mkdocs build --clean

# Build should complete without errors
# Verify output in site/ directory
ls -la site/

# Check for expected structure
ls site/en/
ls site/uk/  # If Ukrainian content exists

3. Local Preview

# Start development server
mkdocs serve

# Visit http://127.0.0.1:8000/
# Verify:
# - Site loads without errors
# - Navigation works (tabs, sections)
# - Search functionality works
# - Dark/light mode toggle works
# - Code highlighting renders correctly
# - Admonitions display properly
# - Internal links navigate correctly
# - Language switcher works (en/uk)
# - All pages accessible and readable

4. Configuration Verification

# Verify mkdocs.yml references correct paths
cat mkdocs.yml | grep -i "docs_dir"
# Should show: docs_dir: content

# Verify requirements.txt has MkDocs dependencies
cat requirements.txt | grep -i "mkdocs"
# Should show mkdocs, mkdocs-material, mkdocs-static-i18n

# Verify GitHub Actions workflow exists
cat .github/workflows/deploy-docs.yml

5. Deployment Verification

# After pushing to GitHub, verify:
# - Workflow runs successfully (check Actions tab)
# - Artifact uploaded
# - Deployment to GitHub Pages succeeds
# - Site accessible at https://openroost.github.io/pymastery-vp/

# Visit the live site and verify:
# - All pages load correctly
# - Navigation works
# - Search works
# - Language switcher works
# - No broken links
# - Images display

6. Documentation Verification

# Verify README updated
cat README.rst  # or README.md
# Should mention MkDocs, not Sphinx

# Verify CONTRIBUTING.md exists and mentions Markdown
cat CONTRIBUTING.md | grep -i "markdown"

# Verify .ai/config.yaml updated
cat .ai/config.yaml | grep -i "mkdocs"

7. Content Integrity Verification

# Compare file counts
src_count=$(find src/ -name "*.rst" | wc -l)
md_count=$(find content/en/ -name "*.md" | wc -l)
echo "Source: $src_count, Converted: $md_count"
# Numbers should match

# Spot-check critical files:
# - Introduction/overview
# - Complex lesson with code/images
# - Exercise or project
# Verify content matches original meaning

8. Ukrainian Localization Verification

# Verify uk/ structure exists
ls -la content/uk/

# If full translation done, verify files mirror en/ structure
diff -r --brief content/en/ content/uk/ | head
# Should show differences in content (translations), not structure

# If stub structure only, verify placeholders exist
# and are documented as future work

9. Fallback Preservation Verification

# Verify src/ directory still exists and is unchanged
test -d src/ && echo "src/ preserved (good)"
ls -la src/

# Verify Sphinx still works (optional, as sanity check)
# make html  # Should still work if Sphinx config exists

10. Final Pre-Cleanup Checklist

Complete this checklist BEFORE removing src/ and Sphinx artifacts:

• MkDocs builds without errors (mkdocs build)
• Local preview works perfectly (mkdocs serve)
• GitHub Actions deployment successful
• Live site on GitHub Pages works correctly
• All navigation, search, and features tested
• No broken links or images
• Content integrity verified (spot-checks passed)
• Project Owner has reviewed and approved live site
• Confirmed ready to remove Sphinx infrastructure

11. Cleanup Verification (After Final Commit)

# Verify src/ removed
test -d src/ && echo "ERROR: src/ still exists" || echo "src/ removed (good)"

# Verify Sphinx artifacts removed
test -f conf.py && echo "ERROR: conf.py still exists" || echo "conf.py removed (good)"
test -f Makefile && echo "ERROR: Makefile still exists" || echo "Makefile removed (good)"
test -d _build && echo "ERROR: _build/ still exists" || echo "_build/ removed (good)"

# Verify no Sphinx references remain
grep -r "sphinx" . --exclude-dir=.git --exclude-dir=site | grep -v "# historical" || echo "Clean"

12. Complete Final Checklist

• All content converted: src/*.rst → content/en/*.md
• MkDocs installed with Material theme
• mkdocs.yml configured correctly
• requirements.txt has MkDocs dependencies
• mkdocs-static-i18n configured for en/uk
• content/uk/ structure exists (full or stub)
• Site builds successfully (mkdocs build)
• Local preview works (mkdocs serve)
• GitHub Actions workflow created and tested
• Site deploys to GitHub Pages successfully
• Live site fully validated by Project Owner
• Project documentation updated (README, CONTRIBUTING, .ai/config.yaml)
• Sphinx artifacts removed ONLY after validation
• No content loss verified
• Multiple clear commit messages created
• All work on feature/wp-mkdocs-migration branch

---

Additional Notes

Content Conversion Strategy

Recommended approach:

1. Automated conversion with Pandoc:

   # Create target directory
   mkdir -p content/en
   
   # Convert all .rst to .md directly to target
   find src/ -name "*.rst" -type f | while read file; do
     # Calculate relative path
     relpath="${file#src/}"
     target="content/en/${relpath%.rst}.md"
     targetdir=$(dirname "$target")
     
     # Create target directory if needed
     mkdir -p "$targetdir"
     
     # Convert
     pandoc -f rst -t markdown -o "$target" "$file"
   done

2. Manual review and cleanup:

   • Check code block formatting
   • Verify admonitions converted correctly
   • Fix internal links
   • Test image rendering

3. Systematic verification:

   • Build site after each major conversion batch
   • Check for broken links
   • Verify navigation structure

Ukrainian Translation Options

Option A: Full translation (high effort, high value)

• Translate all content/en/ files to content/uk/
• Use src/_locales/uk/*.po files as reference
• Time-consuming but complete solution

Option B: Stub structure (low effort, deferred value)

• Create content/uk/ directory structure
• Add placeholder files or minimal translations
• Document translation as future work

Option C: Partial translation (medium effort, incremental value)

• Translate critical pages (index, intro, first few lessons)
• Stub remaining content
• Gradual completion over time

Recommend discussing with Project Owner if full translation is not feasible within this work package timeline.

Phased Approach (Recommended)

Phase 1: Infrastructure + Conversion

• Add MkDocs configuration
• Convert content to content/en/
• Keep src/ as fallback
• Test builds locally

Phase 2: Localization

• Create content/uk/ structure
• Add translations (full or stub)
• Test language switching

Phase 3: Deployment

• Set up GitHub Actions
• Deploy to GitHub Pages
• Validate live site

Phase 4: Documentation

• Update README, CONTRIBUTING
• Update .ai/config.yaml
• Test contributor workflow

Phase 5: Cleanup (ONLY AFTER APPROVAL)

• Remove src/
• Remove Sphinx files
• Clean up old dependencies

Migration Testing Checklist

Before finalizing, test these scenarios:

• Fresh clone builds successfully
• All code examples syntax-highlight correctly
• All admonitions render with proper styling
• Navigation hierarchy makes sense
• Search finds content accurately
• Dark mode displays properly
• Mobile responsive layout works
• Language switcher toggles between en/uk
• GitHub Pages deployment succeeds
• No 404 errors on internal links
• All images load correctly
• Cross-references work
• Table of contents generates correctly

Common Conversion Pitfalls

Watch out for:

1. Code blocks: Sphinx uses .. code-block:: python vs. Markdown ```python
2. Admonitions: Different syntax between Sphinx and Material
3. Internal links: :doc: refs need conversion to [text](file.md)
4. Images: Path references may need adjustment
5. Tables: reStructuredText tables vs. Markdown tables
6. Nested lists: Indentation differences
7. Custom directives: May need manual handling if they don't map cleanly

Expected Challenges

1. Conversion accuracy: .rst → .md may need manual fixes
2. Ukrainian translation scope: Full translation is significant work
3. Link verification: All internal references must be updated
4. GitHub Actions configuration: May need iteration for permissions/deployment
5. Material theme customization: Balancing features vs. simplicity
6. Parallel operation: If WP-244A running simultaneously, coordinate on content/ directory

Success Indicators

• Site builds without errors
• All content renders correctly in browser
• Navigation is intuitive and complete
• Search functionality works well
• Deployment pipeline is automated and reliable
• Documentation is clear for contributors
• Markdown authoring reduces contributor friction
• Professional appearance maintained or improved
• src/ preserved until final approval, then cleanly removed

Rollback Strategy

If critical issues emerge:

Before cleanup (src/ still exists):

• Simply abandon content/en/ and mkdocs.yml
• Continue using Sphinx from src/
• No data loss, easy rollback

After cleanup (src/ removed):

• Use git history to restore src/ and Sphinx files
• Revert commits as needed
• Git preserves all history

Post-Completion

After verification and approval:

• Feature branch feature/wp-mkdocs-migration merged to main
• Sphinx era concluded, MkDocs era begins
• Contributors use Markdown for all future content
• Ukrainian translation can be completed incrementally
• ADR-002 implementation is complete
• Repository ready for modern documentation workflow

---

End of Work Package