Make :stopdoc:/:startdoc:/:enddoc: lexically scoped in the Ruby parser (#1763)

Part of #1591. This PR only changes the region directives
(`:stopdoc:`/`:startdoc:`/`:enddoc:`). `:nodoc:` semantics are unchanged
and will be addressed separately.

## Problem

These directives are currently implemented as mutations of shared
`CodeObject` state
(`document_self`/`document_children`/`done_documenting`). Since classes
are open and shared across files in the store, a `:stopdoc:` without a
matching `:startdoc:` leaks into later reopenings of the class within
the same file. The actual behavior is a product of mutation order, and
neither the documentation nor the implementation defines what these
directives mean.

## New semantics

The parser now tracks the document control state
(`:startdoc:`/`:stopdoc:`/`:enddoc:`) as a lexical region:

- The state is inherited when entering a class/module scope and
automatically restored at its `end`. A `:stopdoc:` can no longer leak
out of the scope it was written in.
- `:startdoc:` inside a nested scope re-enables documentation for that
scope only.
- `:enddoc:` disables documentation for the rest of the current scope
and cannot be cancelled within it, even by `:startdoc:`. It no longer
marks the container object as `done_documenting` globally.
- Classes/modules defined inside a suppressed region are still created
as namespaces (marked as ignored) and their bodies are visited, so an
inner `:startdoc:` works. They become documentable when they receive
documentable contents outside the region, possibly from another file.
Namespaces that never receive contents are removed from the store on
`Store#complete` (`Context#remove_from_documentation?` now also covers
ignored contexts, which is what `CodeObject#ignore` was documented to do
in GH #55).
- Document control scopes are the syntactic units that change the
definition context (`class`/`module` bodies, `class << self`, and
definition-creating blocks). `if`/`while`/`begin` do not create scopes,
consistent with Ruby's own scoping rules.

## Bug fixes

On master, these directives mutate the state of whatever object they
happen
to be applied to, so their effect escapes the scope they were written
in.
Some examples of what this PR fixes:

**A `:stopdoc:` inside `class << self` no longer leaks out of the
singleton
scope:**

```ruby
class Net::HTTPResponse
  class << self
    # :stopdoc:
    def internal; end
  end

  # master: NOT documented / this PR: documented
  attr_reader :http_version
end
```

This is exactly what happened to [the Ruby 4.0
documentation](https://docs.ruby-lang.org/en/4.0/Net/HTTPResponse.html)

**A `:stopdoc:` no longer leaks into later reopenings of the class.**
Note
that on master the two ways of reopening `A::B` below do not even agree
with
each other: `f` is documented but `g` is not, because the leaked state
only
affects the nested-class form. The behavior was an artifact of the
implementation, not a specification:

```ruby
class A
  # :stopdoc:
  def internal; end
end

class A::B
  # master: documented / this PR: documented
  def f; end
end

class A
  class B
    # master: NOT documented / this PR: documented
    def g; end
  end
end
```

## Compatibility notes

- `:nodoc:` is untouched. However, `:startdoc:` additionally calls
`start_doc` on the current container, because `module Net #:nodoc:`
followed by `:stopdoc:`/`:startdoc:` regions is a common pattern that
expects `:startdoc:` to make the container documentable again. This is a
transitional measure until `:nodoc:` gets lexical semantics, at which
point the pattern itself becomes unnecessary. Containers created inside
a suppressed region are excluded, so a bare `:startdoc:` cannot revive
an empty hidden class.
- A comment linked to a `class << X` line is now processed in the
enclosing scope. Nothing consumes such a comment as documentation, and
processing it inside the singleton scope would make a `:startdoc:`
written there expire at the singleton's `end` (this pattern appears in
net/http).
- Region directives in modifier position (e.g. `class Foo # :stopdoc:`
at the end of a line) are no longer applied. There are no occurrences of
this form in ruby/ruby, rails, or the bundled gems.

## Validation

Compared generated HTML for ruby/ruby `lib` and six Rails frameworks
between master and this branch. The sets of generated files are
identical, and every remaining difference is the directives working as
their authors intended:

- Module aliases written inside `:stopdoc:` regions (`Ripper =
Prism::Translation::Ripper` in prism's shim.rb,
`HashWithIndifferentAccess = ...` in activesupport) are no longer
documented. This also removes a duplicated "Ripper" entry from the
navigation and stops bare "Ripper" mentions (which refer to the stdlib
Ripper) from being cross-referenced to `Prism::Translation::Ripper`.
Note: the old accidental link on `HashWithIndifferentAccess` in
Hash.html was arguably useful; writing the full name in the comment
restores it.
- `Net::HTTPResponse` regains its `include Net::HTTPHeader`. The old
implementation lost it because a `:stopdoc:` inside `class << self`
leaked out of the singleton scope; net/http.rb contains a "this is to
fix bug in RDoc" workaround comment for this exact class of leak.
- `ActionDispatch::Journey` (`# :nodoc:`) no longer appears in the
navigation. Its page is still generated because it has documented
children.
- Namespaces fully contained in suppressed regions (e.g.
`Net::HTTPBadResponse` region in net/http.rb, `Timeout::Request`) no
longer produce empty pages.

## Follow-ups (out of scope here)

- Lexical/shallow semantics for `:nodoc:` (the hard part of #1591); the
`start_doc` compatibility shim and the `received_nodoc` guard in
promotion can be removed then.
- Warnings for degenerate usages (redundant `:stopdoc:`, directives
after `:enddoc:`, standalone `:nodoc:` in the middle of a class body).
- A visibility trace output (`file:line: stopdoc applied to X`) as
proposed in
#1591 (comment); the
parser now has the lexical information to make this cheap.
- The C parser still uses the old CodeObject-state mechanism and is
intentionally untouched.

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

Author: tompng

lib/rdoc/code_object/context.rb

@@ -1066,8 +1066,10 @@ def record_location(top_level)
   # * All classes and modules have <tt>#remove_from_documentation? == true</tt>
 
   def remove_from_documentation?
+    # Contexts that are still ignored here were created inside a :stopdoc:
+    # region and never received documentable contents afterwards
     @remove_from_documentation ||=
-      @received_nodoc &&
+      (@received_nodoc || @ignored) &&
       !any_content(false) &&
       @includes.all? { |i| !i.module.is_a?(String) && i.module.remove_from_documentation? } &&
       classes_and_modules.all? { |cm| cm.remove_from_documentation? }

lib/rdoc/parser/ruby.rb

@@ -152,6 +152,53 @@ def initialize(top_level, content, options, stats)
     @visibility = :public
     @singleton = false
     @in_proc_block = false
+    @doc_state = :startdoc
+  end
+
+  # Applies document control directives (:startdoc:, :stopdoc: and :enddoc:)
+  # to the current lexical scope. The state is restored when the enclosing
+  # class/module scope is closed.
+
+  def apply_document_control_directive(directives)
+    directives.each do |directive, (_param, line)|
+      case directive
+      when 'startdoc', 'stopdoc'
+        # :enddoc: cannot be cancelled within the scope, even by :startdoc:
+        if @doc_state == :enddoc
+          @options.warn "#{@top_level.relative_name}:#{line}: :startdoc: is ignored after :enddoc:" if directive == 'startdoc'
+          next
+        end
+        @doc_state = directive.to_sym
+        if directive == 'startdoc' && !@container.ignored?
+          # Compatibility: `module Net #:nodoc:` followed by :stopdoc:/:startdoc:
+          # regions is a common pattern that expects :startdoc: to make the
+          # container documentable again. Containers ignored here were created
+          # in a suppressed region and need documentable contents to revive.
+          @container.start_doc
+          @container.force_documentation = true
+        end
+      when 'enddoc'
+        @doc_state = :enddoc
+      end
+    end
+  end
+
+  # Returns true if code objects at the current position should not be
+  # documented, that is, inside a :stopdoc: or :enddoc: region.
+
+  def document_suppressed?
+    @track_visibility && @doc_state != :startdoc
+  end
+
+  # Makes a container that was created inside a :stopdoc:/:enddoc: region
+  # (thus ignored) documentable again when it receives documentable contents
+  # outside the region, possibly from another file.
+
+  def mark_container_documentable(container)
+    return if container.received_nodoc || !container.ignored?
+    record_location(container)
+    container.start_doc
+    mark_container_documentable(container.parent) if container.parent.is_a?(RDoc::ClassModule)
   end
 
   # Suppress `extend` and `include` within block
@@ -172,6 +219,7 @@ def with_container(container, singleton: false)
     old_visibility = @visibility
     old_singleton = @singleton
     old_in_proc_block = @in_proc_block
+    old_doc_state = @doc_state
     @visibility = :public
     @container = container
     @singleton = singleton
@@ -183,6 +231,7 @@ def with_container(container, singleton: false)
     @visibility = old_visibility
     @singleton = old_singleton
     @in_proc_block = old_in_proc_block
+    @doc_state = old_doc_state
     @module_nesting.pop
   end
 
@@ -307,6 +356,7 @@ def prepare_comments(comments)
   # Signature section in the comment
 
   def parse_comment_tomdoc(container, comment, line_no, start_line)
+    return if document_suppressed?
     return unless signature = RDoc::TomDoc.signature(comment)
 
     name, = signature.split %r%[ \(]%, 2
@@ -353,6 +403,7 @@ def call_node_name_arguments(call_node) # :nodoc:
   # Handles meta method comments
 
   def handle_meta_method_comment(comment, directives, node)
+    apply_document_control_directive(directives)
     handle_code_object_directives(@container, directives)
     is_call_node = node.is_a?(Prism::CallNode)
     singleton_method = false
@@ -375,13 +426,16 @@ def handle_meta_method_comment(comment, directives, node)
       end
     end
 
+    return if document_suppressed?
+
     if attributes
       attributes.each do |attr|
         a = RDoc::Attr.new(attr, rw, comment, singleton: @singleton)
         a.store = @store
         a.line = line_no
         record_location(a)
         @container.add_attribute(a)
+        mark_container_documentable(@container)
         a.visibility = visibility
       end
     elsif line_no || node
@@ -427,6 +481,7 @@ def handle_standalone_consecutive_comment_directive(comment, directives, start_w
     elsif normal_comment_treat_as_ghost_method_for_now?(directives, line_no) && start_line != @first_non_meta_comment_start_line
       handle_meta_method_comment(comment, directives, nil)
     else
+      apply_document_control_directive(directives)
       handle_code_object_directives(@container, directives)
     end
   end
@@ -510,6 +565,9 @@ def change_method_visibility(names, visibility, singleton: @singleton)
     new_methods = []
     @container.methods_matching(names, singleton) do |m|
       if m.parent != @container
+        # A copy of an ancestor's method must not be documented
+        # in a :stopdoc:/:enddoc: region
+        next if document_suppressed?
         m = m.dup
         record_location(m)
         new_methods << m
@@ -532,6 +590,10 @@ def change_method_visibility(names, visibility, singleton: @singleton)
 
   def change_method_to_module_function(names)
     @container.set_visibility_for(names, :private, false)
+    # In a :stopdoc:/:enddoc: region, the visibility of instance methods still
+    # changes but the singleton method copies must not be documented
+    return if document_suppressed?
+
     new_methods = []
     @container.methods_matching(names) do |m|
       s_m = m.dup
@@ -552,6 +614,9 @@ def change_method_to_module_function(names)
 
   def handle_code_object_directives(code_object, directives) # :nodoc:
     directives.each do |directive, (param)|
+      # startdoc/stopdoc/enddoc are handled by apply_document_control_directive.
+      # They control the lexical scope of the parser, not the code object.
+      next if directive == 'startdoc' || directive == 'stopdoc' || directive == 'enddoc'
       @preprocess.handle_directive('', directive, param, code_object)
     end
   end
@@ -560,14 +625,18 @@ def handle_code_object_directives(code_object, directives) # :nodoc:
 
   def add_alias_method(old_name, new_name, line_no)
     comment, directives = consecutive_comment(line_no)
+    apply_document_control_directive(directives) if directives
     handle_code_object_directives(@container, directives) if directives
+    return if document_suppressed?
+
     visibility = @container.find_method(old_name, @singleton)&.visibility || :public
     a = RDoc::Alias.new(old_name, new_name, comment, singleton: @singleton)
     handle_modifier_directive(a, line_no)
     a.store = @store
     a.line = line_no
     record_location(a)
     if should_document?(a)
+      mark_container_documentable(@container)
       @container.add_alias(a)
       @container.find_method(new_name, @singleton)&.visibility = visibility
     end
@@ -577,7 +646,9 @@ def add_alias_method(old_name, new_name, line_no)
 
   def add_attributes(names, rw, line_no)
     comment, directives, type_signature_lines = consecutive_comment(line_no)
+    apply_document_control_directive(directives) if directives
     handle_code_object_directives(@container, directives) if directives
+    return if document_suppressed?
     return unless @container.document_children
 
     names.each do |symbol|
@@ -587,7 +658,10 @@ def add_attributes(names, rw, line_no)
       a.type_signature_lines = type_signature_lines
       record_location(a)
       handle_modifier_directive(a, line_no)
-      @container.add_attribute(a) if should_document?(a)
+      if should_document?(a)
+        @container.add_attribute(a)
+        mark_container_documentable(@container)
+      end
       a.visibility = visibility # should set after adding to container
     end
   end
@@ -596,7 +670,11 @@ def add_attributes(names, rw, line_no)
 
   def add_includes_extends(names, rdoc_class, line_no) # :nodoc:
     comment, directives = consecutive_comment(line_no)
+    apply_document_control_directive(directives) if directives
     handle_code_object_directives(@container, directives) if directives
+    return if document_suppressed?
+
+    mark_container_documentable(@container)
     names.each do |name|
       resolved_name = resolve_constant_path(name)
       ie = @container.add(rdoc_class, resolved_name || name, '')
@@ -622,9 +700,12 @@ def add_extends(names, line_no) # :nodoc:
   # Adds a method defined by `def` syntax
 
   def add_method(method_name, receiver_name:, receiver_fallback_type:, visibility:, singleton:, params:, calls_super:, block_params:, tokens:, start_line:, args_end_line:, end_line:)
-    receiver = receiver_name ? find_or_create_lexical_module_path(receiver_name, receiver_fallback_type) : @container
     comment, directives, type_signature_lines = consecutive_comment(start_line)
+    apply_document_control_directive(directives) if directives
     handle_code_object_directives(@container, directives) if directives
+    # Resolve receiver after applying directives so that a namespace created
+    # here is marked as ignored when the comment starts a :stopdoc: region
+    receiver = receiver_name ? find_or_create_lexical_module_path(receiver_name, receiver_fallback_type) : @container
 
     internal_add_method(
       method_name,
@@ -650,8 +731,11 @@ def add_method(method_name, receiver_name:, receiver_fallback_type:, visibility:
     modifier_comment_lines&.each do |line|
       handle_modifier_directive(meth, line)
     end
+    return if document_suppressed?
     return unless should_document?(meth)
 
+    mark_container_documentable(container)
+
     if directives && (call_seq, = directives['call-seq'])
       meth.call_seq = call_seq.lines.map(&:chomp).reject(&:empty?).join("\n") if call_seq
     end
@@ -691,12 +775,18 @@ def add_method(method_name, receiver_name:, receiver_fallback_type:, visibility:
   def find_or_create_lexical_module_path(module_name, create_mode)
     root_name, *path, name = module_name.split('::')
     add_module = ->(mod, name, mode) {
-      case mode
-      when :class
-        mod.add_class(RDoc::NormalClass, name, 'Object').tap { |m| m.store = @store }
-      when :module
-        mod.add_module(RDoc::NormalModule, name).tap { |m| m.store = @store }
-      end
+      created =
+        case mode
+        when :class
+          mod.add_class(RDoc::NormalClass, name, 'Object').tap { |m| m.store = @store }
+        when :module
+          mod.add_module(RDoc::NormalModule, name).tap { |m| m.store = @store }
+        end
+      # add_class/add_module may return an existing object created by another
+      # file (in_files is not empty then), which must not be ignored here.
+      # Documentable again when reopened or receiving contents outside the region.
+      created.ignore if document_suppressed? && created.in_files.empty?
+      created
     }
     if root_name.empty?
       mod = @top_level
@@ -755,17 +845,22 @@ def find_or_create_lexical_constant_owner_name(constant_path)
 
   def add_constant(constant_name, rhs_name, start_line, end_line, alias_path: nil)
     comment, directives = consecutive_comment(start_line)
+    apply_document_control_directive(directives) if directives
     handle_code_object_directives(@container, directives) if directives
+    return if document_suppressed?
+
     owner, name = find_or_create_lexical_constant_owner_name(constant_name)
     return unless owner
 
     constant = RDoc::Constant.new(name, rhs_name, comment)
     constant.store = @store
     constant.line = start_line
     constant.is_alias_for_path = alias_path
-    record_location(constant)
     handle_modifier_directive(constant, start_line)
     handle_modifier_directive(constant, end_line)
+    # A constant marked :nodoc: must not make an ignored owner documentable
+    mark_container_documentable(owner) if constant.document_self && owner.is_a?(RDoc::ClassModule)
+    record_location(constant)
     owner.add_constant(constant)
     return unless alias_path
     mod =
@@ -787,6 +882,7 @@ def add_constant(constant_name, rhs_name, start_line, end_line, alias_path: nil)
 
   def add_module_or_class(module_name, start_line, end_line, is_class: false, superclass_name: nil, superclass_expr: nil)
     comment, directives = consecutive_comment(start_line)
+    apply_document_control_directive(directives) if directives
     handle_code_object_directives(@container, directives) if directives
     return unless @container.document_children
 
@@ -803,7 +899,13 @@ def add_module_or_class(module_name, start_line, end_line, is_class: false, supe
         superclass_full_path = superclass_full_path.sub(/^::/, '')
       end
       # add_class should be done after resolving superclass
-      mod = owner.classes_hash[name] || owner.add_class(RDoc::NormalClass, name, superclass_name || superclass_expr || '::Object')
+      mod = owner.classes_hash[name]
+      unless mod
+        # add_class may return an existing class created by another file
+        # (in_files is not empty then), which must not be ignored here
+        mod = owner.add_class(RDoc::NormalClass, name, superclass_name || superclass_expr || '::Object')
+        mod.ignore if document_suppressed? && mod.in_files.empty?
+      end
       if superclass_name
         if superclass
           mod.superclass = superclass
@@ -812,15 +914,33 @@ def add_module_or_class(module_name, start_line, end_line, is_class: false, supe
         end
       end
     else
-      mod = owner.modules_hash[name] || owner.add_module(RDoc::NormalModule, name)
+      mod = owner.modules_hash[name]
+      unless mod
+        mod = owner.add_module(RDoc::NormalModule, name)
+        mod.ignore if document_suppressed? && mod.in_files.empty?
+      end
     end
 
     mod.store = @store
     mod.line = start_line
-    record_location(mod)
     handle_modifier_directive(mod, start_line)
     handle_modifier_directive(mod, end_line)
-    mod.add_comment(comment, @top_level) if comment
+    unless document_suppressed?
+      # In a :stopdoc:/:enddoc: region, the container is still created as a
+      # namespace but is not recorded to this file nor documented.
+      # The body is also visited: an inner :startdoc: re-enables documentation
+      # in a :stopdoc: region (not in an :enddoc: region), and nested
+      # namespaces need to be created for later promotion from other files
+      if mod.ignored?
+        # Promotes the owner chain too, unless mod received :nodoc:
+        mark_container_documentable(mod)
+      else
+        # A class/module marked :nodoc: must not make an ignored owner documentable
+        mark_container_documentable(owner) if mod.document_self && owner.is_a?(RDoc::ClassModule)
+        record_location(mod)
+      end
+      mod.add_comment(comment, @top_level) if comment
+    end
     mod
   end
 
@@ -975,7 +1095,9 @@ def visit_class_node(node)
     end
 
     def visit_singleton_class_node(node)
-      @scanner.process_comments_until(node.location.start_line - 1)
+      # A comment linked to the `class << ...` line (e.g. a document control
+      # directive) belongs to the enclosing scope, not to the singleton scope
+      @scanner.process_comments_until(node.location.start_line)
 
       if @scanner.has_modifier_nodoc?(node.location.start_line)
         # Skip visiting inside the singleton class. Also skips creation of node.expression as a module
@@ -990,6 +1112,7 @@ def visit_singleton_class_node(node)
       when Prism::ConstantWriteNode
         # Accept `class << (NameErrorCheckers = Object.new)` as a module which is not actually a module
         mod = @scanner.container.add_module(RDoc::NormalModule, expression.name.to_s)
+        mod.ignore if @scanner.document_suppressed? && mod.in_files.empty?
       when Prism::ConstantPathNode, Prism::ConstantReadNode
         expression_name = constant_path_string(expression)
         # If a constant_path does not exist, RDoc creates a module
@@ -1150,6 +1273,7 @@ def constant_path_string(node)
     end
 
     def _visit_call_require(call_node)
+      return if @scanner.document_suppressed?
       return unless call_node.arguments&.arguments&.size == 1
       arg = call_node.arguments.arguments.first
       return unless arg.is_a?(Prism::StringNode)

test/rdoc/generator/darkfish_test.rb

@@ -355,7 +355,8 @@ def test_setup
 
     assert_equal %w[Ignored Klass Klass::A Object],
                  [@ignored, @klass, @klass_alias, @object].map(&:full_name)
-    assert_equal [@ignored, @klass, @klass_alias, @object],
+    # @ignored is removed from the store on Store#complete
+    assert_equal [@klass, @klass_alias, @object],
                  @g.classes.sort_by { |klass| klass.full_name }
     assert_equal [@top_level],                           @g.files
     assert_equal [@meth, @meth, @meth_bang, @meth_bang, @meth_with_html_tag_yield, @meth_with_html_tag_yield], @g.methods