Skip to content

feat(x86_64/mm): support memory protection keys #2229

Description

@fslongjin

Is your feature request related to a problem?

DragonOS defines the following x86_64 Memory Protection Keys syscalls, but they are not implemented yet:

  • pkey_mprotect(2)
  • pkey_alloc(2)
  • pkey_free(2)

Linux applications and runtimes that use Memory Protection Keys currently receive ENOSYS and cannot use this memory isolation mechanism.

DragonOS has some PKEY-related code scaffolding, but it does not currently provide effective hardware-backed protection. The syscalls must not report success unless the requested protection is actually enforced.

Describe the solution you would like

Add Linux 6.6-compatible Memory Protection Keys support on x86_64, including:

  • allocating and freeing protection keys;
  • applying keys to memory ranges with pkey_mprotect;
  • controlling access through PKRU;
  • preserving the correct state across threads, process creation, exec, and task switches;
  • Linux-compatible SIGSEGV information for protection-key violations;
  • Linux-compatible behavior on CPUs without PKEY support;
  • compatibility with normal mprotect behavior.

The initial scope is x86_64. RISC-V and LoongArch should continue returning ENOSYS until architecture-specific support is implemented.

Roadmap

  • Add x86_64 PKEY capability detection and enablement.
  • Add address-space and per-thread protection-key state management.
  • Apply protection keys to memory mappings and permission changes.
  • Implement pkey_alloc, pkey_free, and pkey_mprotect.
  • Cover fork/clone, exec, task switching, and signal behavior.
  • Add DragonOS dunitest coverage and run the portable Linux PKEY selftests.
  • Test on CPUs with and without PKEY support.

Acceptance criteria

  • All three syscalls have Linux 6.6-compatible behavior on supported x86_64 CPUs.
  • Protection keys enforce actual userspace read/write restrictions rather than only updating software state.
  • Invalid arguments, key exhaustion, repeated frees, and unsupported hardware match Linux error behavior.
  • State remains correct across multithreading, fork/clone, exec, task switching, and signals.
  • Protection-key violations report the correct SIGSEGV, SEGV_PKUERR, and key information.
  • Normal mprotect and applications that do not use PKEY do not regress.
  • The implementation does not use successful no-op handlers, application-specific checks, or software checks presented as hardware protection.

Alternatives considered

Registering no-op syscall handlers, or treating pkey_mprotect as ordinary mprotect, would give applications a false security guarantee and is therefore not acceptable.

Additional context

  • Linux 6.6 reference: mm/mprotect.c
  • Linux x86 references: arch/x86/include/asm/pkeys.h, arch/x86/mm/pkeys.c
  • Linux selftests: tools/testing/selftests/mm/protection_keys.c

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions