[RFE] Migration Complexity Classification for Konveyor Rules

[tsanders-rh]

RFE: Migration Complexity Classification for Konveyor Rules

Summary

Add a standardized migration complexity classification system to Konveyor rules using a five-level scale (TRIVIAL, LOW, MEDIUM, HIGH, EXPERT) to improve AI automation success rates, enable better migration planning, and provide segmented reporting of AI performance.

Problem Statement

Currently, AI-assisted migration success is reported as a single aggregate metric (e.g., "50% pass rate"), which obscures the reality that different migration types have vastly different success rates:

• Namespace changes (javax.* → jakarta.*): 95%+ AI success
• Security reconfigurations: 30% AI success

Result: We hide AI's true value and set incorrect expectations.

Before Classification

Overall Pass Rate: 50% (116/234 failures)

❌ Conclusion: "AI isn't ready for production migrations"

After Classification

TRIVIAL: 95% pass rate (19/20)   ← Automate these!
LOW:     82% pass rate (41/50)   ← Automate with review
MEDIUM:  64% pass rate (96/150)  ← Good acceleration
HIGH:    45% pass rate (8/18)    ← Useful scaffolding
EXPERT:  20% pass rate (1/5)     ← AI generates checklist

✅ Conclusion: "AI excels at 60% of migrations, assists with the rest"

Proposed Solution

Add an optional migration_complexity field to the Konveyor rule schema:

rules:
  - id: "javax-to-jakarta-00001"
    description: "Replace javax.* imports with jakarta.*"
    migration_complexity: trivial  # NEW FIELD
    category: mandatory

Five Complexity Levels

Level	Description	Expected AI Success	Use Case
TRIVIAL	Mechanical find/replace	95%+	Full automation
LOW	Straightforward API equivalents	80%+	Automation + light review
MEDIUM	Requires context understanding	60%+	AI acceleration
HIGH	Architectural changes	30-50%	AI scaffolding
EXPERT	Custom/critical code	<30%	AI checklist only

Benefits

For Migration Planners

• Estimate effort based on complexity distribution
• Identify which migrations can be automated vs. requiring expert review
• Create realistic timelines and resource allocation

For AI Tool Developers

• Route TRIVIAL/LOW migrations to automated pipelines
• Flag HIGH/EXPERT migrations for human review
• Apply complexity-appropriate strategies and prompts

For Stakeholders

• Understand where AI provides value with segmented metrics
• Justify investment in AI migration tooling with data
• Demonstrate clear ROI for different migration types

Implementation Approach

1. Automated Classification Tool

   • Analyzes rules based on keywords, imports, annotations, and patterns
   • Generates classifications with confidence scores
   • Allows manual override for domain-specific knowledge

2. Hybrid Approach for Custom Rules

   • Auto-classifies custom rules with same algorithm
   • Defaults to MEDIUM complexity on low confidence
   • Supports explicit override by rule authors

3. Solution Server Feedback Loop (Self-Improving System)

   • Track user acceptance/rejection/modification of AI-generated fixes
   • Automatically refine complexity ratings based on real-world usage patterns
   • Example: Rule classified as LOW with only 30% acceptance → reclassified to MEDIUM
   • Continuous learning ensures classifications stay accurate over time
   • Provides data-driven validation of initial heuristic classifications

4. Backward Compatible

   • Optional field - existing rules remain valid
   • Tools that don't support complexity simply ignore it
   • Gradual adoption across projects

5. Phased Rollout

   • Phase 1: Schema addition and classification tooling (2-3 weeks)
   • Phase 2: Segmented reporting integration (1-2 weeks)
   • Phase 3: Complexity-aware workflows (2-3 weeks, optional)

Proof of Concept

✅ Already demonstrated:

• 2,680+ rules classified with automated tool
• Evaluation framework with segmented reporting
• Clear patterns showing AI excels at lower complexity levels

Resources:

• GitHub: https://github.com/tsanders-rh/konveyor-iq
• Live Sample Report: https://tsanders-rh.github.io/konveyor-iq/
• Slide Deck Presentation: https://docs.google.com/presentation/d/1Yz5Hecc0zdqAS24-sN02VBMcclOU_6GTcuUUa0kXKzU/edit?usp=sharing
• Full Enhancement Proposal: (will be submitted as PR)

Use Cases

Use Case 1: Migration Planning

Project has 500 rules to address:
- 200 TRIVIAL (40%) → Automate (2 days)
- 150 LOW (30%)     → Automate + review (5 days)
- 100 MEDIUM (20%)  → AI assist (10 days)
- 40 HIGH (8%)      → Expert review (8 days)
- 10 EXPERT (2%)    → Manual migration (5 days)

Total estimate: 30 days vs. "unknown complexity"

Use Case 2: AI Automation Routing

if rule.migration_complexity in [TRIVIAL, LOW]:
    apply_ai_fix_automatically(rule)
elif rule.migration_complexity == MEDIUM:
    suggestion = ai_suggest_fix(rule)
    queue_for_developer_review(suggestion)
else:  # HIGH or EXPERT
    checklist = ai_generate_migration_checklist(rule)
    assign_to_expert(checklist)

Use Case 3: Transparent Reporting

Instead of "AI failed on 50% of migrations," report:

• "AI automated 95% of TRIVIAL migrations (40% of total work)"
• "AI accelerated 64% of MEDIUM migrations (20% of total work)"
• "AI provided scaffolding for HIGH/EXPERT migrations (10% of total work)"

Questions for Community Discussion

1. Schema Integration: Should migration_complexity be part of the official Konveyor rule schema, or evaluation-only metadata?

2. Classification Ownership: Should rules be auto-classified with manual override, or require explicit classification by rule authors?

3. Integration Points: How should this integrate with:

   • Analyzer rule metadata?
   • KAI automation workflows?
   • Hub rule catalog display?

4. Scope: Are there other complexity factors to consider (multi-file migrations, version-specific complexity, etc.)?

5. Feedback Loop: Should the Solution Server track user acceptance/rejection to refine complexity ratings over time?

Next Steps

If this RFE receives community support, I will:

1. ✅ Submit formal enhancement proposal as PR (following template)
2. Collaborate with community on schema design
3. Develop classification tooling for community review
4. Pilot on core rulesets (javax, Spring, Quarkus migrations)
5. Integrate with existing Konveyor reporting and tooling

Related Work

• KAI Enhancement: AI-assisted code migration
• Solution Server: Tracks user acceptance of AI suggestions
• Analysis Insights: Rule-based migration analysis

Contact

• GitHub: @tsanders-rh
• Enhancement Proposal: Will submit PR to /enhancements/migration-complexity-classification/

---

Note: This RFE is based on real-world evaluation of AI migration success rates across 2,680+ rules. The complexity classification system has been demonstrated in the konveyor-iq proof-of-concept framework.

[konveyor-ci-bot]

This issue is currently awaiting triage.
If contributors determine this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.
The triage/accepted label can be added by org members.

[tsanders-rh]

Update: Alignment with AppCAT Metadata Enhancement

I've updated the proposal to align with the optional metadata pattern from the AppCAT enhancement (#254). Both enhancements share the same principles:

Shared Approach

• Optional fields - migration_complexity is optional, just like AppCAT's message, domain, and metadata.confidence fields
• Backward compatible - Existing rules remain valid; tools that don't support the field ignore it
• Single source of truth - Konveyor remains the upstream authority
• Gradual adoption - Ecosystem tools can adopt at their own pace

Complementary Goals

• AppCAT (Adopt Azure Migrate application and code assessment for Java (formerly AppCAT) ruleset enhancements in Konveyor #254): Adds metadata to improve rule precision and reduce false positives
• This proposal: Adds complexity classification to improve AI automation success rates

Both enhancements enrich Konveyor's metadata without fragmenting the schema.

Resolution: Open Question #1

Based on the AppCAT pattern, I've resolved the open question about whether to add migration_complexity to the schema:

Decision: Add to schema as optional field (following #254's approach)

Key Updates to Proposal

• ✅ Added explicit alignment section referencing Adopt Azure Migrate application and code assessment for Java (formerly AppCAT) ruleset enhancements in Konveyor #254
• ✅ Clarified that confidence scores stay in tooling (not schema) - different from AppCAT's metadata.confidence which indicates detection reliability
• ✅ Strengthened backward compatibility guarantees
• ✅ Enhanced schema examples showing rules with/without complexity field

Happy to discuss how these two enhancements can work together to improve Konveyor's migration capabilities!