C parser creates a self-referencing superclass when rb_define_class_under reuses a C variable

[skatkov]

Related to #1776 because both ultimately create a self-referencing RDoc superclass. This is an independent C-parser reproducer with one non-symlinked file; source-path deduplication does not fix it.

Summary

RDoc's C parser creates a self-referencing superclass and crashes with
SystemStackError: stack level too deep when a C extension reuses a variable
as both the superclass argument and the assignment target of
rb_define_class_under.

Confirmed against both kgio 1.1.0 and 2.7.0, but the problem can be reproduced
with one C file.

Minimal reproduction

Create example.c:

#include <ruby.h>

void Init_example(void)
{
  VALUE mExample = rb_define_module("Example");
  VALUE cSocket = rb_const_get(rb_cObject, rb_intern("Socket"));
  cSocket = rb_define_class_under(mExample, "Socket", cSocket);
}

Run:

$ rdoc --version
8.0.0

$ rdoc --format=ri example.c
Parsing sources...
100% [ 1/ 1]  example.c
uh-oh! RDoc had a problem:
stack level too deep

run with --debug for full backtrace

This results in:

$ rdoc --debug --format=ri --force-output --output=doc example.c
Parsing sources...
100% [ 1/ 1]  example.c
stack level too deep
/path/to/rdoc/lib/rdoc/code_object/class_module.rb:780:in 'RDoc::ClassModule#superclass'
	/path/to/rdoc/lib/rdoc/code_object/normal_class.rb:13:in 'RDoc::NormalClass#ancestors'
	/path/to/rdoc/lib/rdoc/code_object/normal_class.rb:18:in 'RDoc::NormalClass#ancestors'
	... same NormalClass#ancestors frame repeated 10,073 more times ...
	/path/to/rdoc/lib/rdoc/store.rb:522:in 'block in RDoc::Store#complete'
	/path/to/rdoc/lib/rdoc/store.rb:522:in 'Array#each'
	/path/to/rdoc/lib/rdoc/store.rb:522:in 'RDoc::Store#complete'
	/path/to/rdoc/lib/rdoc/rdoc.rb:529:in 'RDoc::RDoc#document'
	/path/to/rdoc/exe/rdoc:20:in '<main>'

This is reproduced with rdoc 8.0, RDoc 7.2.0, master. Not an issue for YARD.

Expected behavior

RDoc should generate documentation for:

Example::Socket < Socket

It should not crash.

Why the C code is valid

Before rb_define_class_under is called, cSocket points to the top-level
::Socket class. The right-hand side is evaluated before the assignment, so
the new Example::Socket class correctly receives ::Socket as its
superclass. The returned class is then assigned back to cSocket.

This is the pattern used by kgio, for example:

cSocket = rb_const_get(rb_cObject, rb_intern("Socket"));
cSocket = rb_define_class_under(mKgio, "Socket", cSocket);

Probable cause

RDoc::Parser::C#handle_class_module does not resolve the preceding
rb_const_get, so it initially records the superclass as the C variable name
"cSocket".

After creating Example::Socket, the parser registers:

cSocket -> Example::Socket

RDoc::Store#resolve_c_superclasses later resolves the superclass variable
"cSocket" through that mapping and assigns Example::Socket as its own
superclass.

RDoc::NormalClass#ancestors then recursively calls
superclass.ancestors without encountering a terminating superclass.

At minimum, resolve_c_superclasses should avoid assigning a class as its own
superclass. Ideally, the C parser should recognize the preceding
rb_const_get and preserve Socket as the superclass before replacing the
variable mapping.