Use remove_const after specs which set constants

[MaxLap]

So I'm making a tool which runs specs many times, and specs that set consts seems to often not cleanup after themselves. This makes the repeat have annoying warnings about already defined consts.

Those constants existing between specs also seems like something that could create an interaction between them.

So I added the remove_const calls that make sense to avoid all of these warnings. Good news, no tests relied on that!

[eregon]

This makes the repeat have annoying warnings about already defined consts.

What about just ignoring those warnings for that use case?
For example by running specs with $VERBOSE = nil/RUBYOPT=-W0?

Another way would be to define Warning.warn, MSpec used to do that but it can cause problems: ruby/mspec@1d8cf64.
So I don't want to reintroduce that, or at least not by default.

I have 2 concerns with this change:

• It seems very likely to regress so we would need a CI check for this.
• It looks quite messy with these ~130 added remove_const. There are already some remove_const (68) but IIRC those are necessary to run specs with repeat > 1 (some specs would fail otherwise).

[eregon]

Redefining constants seems rather harmless in the context of MSpec, when it's not autoload constants.
OTOH it's true it's global state that "leaks" between repeated runs of the same specs (different spec examples should use different constants so there is no concern without --repeat). But then the spec on the first run does pass without previous state, so it feels rather harmless again.

[MaxLap]

For many cases, you are right that just ignore warnings would probably have been fine.

For some cases, such as when doing include and extend, then with enough repetition of a new module, you get non-linear timings and increasing memory usage. For these, ignoring the warnings is not possible for my use case.

I understand your concerns, and I had them and wondered if you would accept the changes as I was doing them. The way I see it:

• Regression are not a problem for the current normal usage of this tool. This is just some work that was done, which make things cleaner and more reusable by others. I don't think a CI check is needed for this, at least not yet. My goal is not to add more work on maintainers. There will be regression, and someone that needs the remove_const to be right will clean them up next time it's needed, but at least he won't have to repeat the cleanups that I did.
• It's messy when you look only at those 271 lines of code. I feel that the remove_const actually make the individual test clearer as that they highlight what global mutation where actually done. Especially for some of those subtle cases that are changing the singleton_class.

My question to you is, do you think the cleanups should be in ensure or not? If the test fails before the constant is defined, then the ensure will also fail. Not sure how the error message would look then. At the same time, the ensure highlight that it's not part of the test, but that may also be what make it feel noisy when you focus on it.

[eregon]

Looking at this again, I think it would be OK to merge but we would need some kind of check otherwise it will just regress and there is no much point adding it.

[MaxLap]

You're right.

An idea could be that before the test suite runs, create a Set of all the constants (only their full names).

And after the suite is done, create that Set again and if they are different, fail the specs and mention the differences.

If needed, there could be a list of allowed changes for some special cases.

[eregon]

Actually that already exists: https://github.com/ruby/spec/blob/master/.mspec.constants
but it's only for top-level constants, since nested constants are considered to not "leak".
We'd still want to allow ...Specs and ...Specs::Foo (e.g. defined in fixtures files) so we can't detect that after the full suite, we'd need to detect it after every spec (or maybe once per file?), which seems pretty expensive, even if opt-in.

IOW I currently don't see a way to test this which is not a lot of overhead/complexity.

I'm OK to merge this though given it's small, work already done, and seems good to cleanup anyway, could you rebase?

[MaxLap]

Glad to hear that for the merge! Rebase is done.

I'm curious about my idea, I'll still take a stab at implementing my version. Seems like a fun little side quest :)

[MaxLap]

Well, I see now that lots of tests are defining the classes/modules directly in the it when they aren't actually testing constant-related things. That's too bad.

Ideally, they would either be defined outside of the it, or would cleanup after themselves... But those are old tests that no one will waste time changing.

Otherwise, I would have had a pretty interesting solution to efficiently detect the not cleaned up constants. I'm writing it below as reference if I or someone else ever want to do something similar.

Adding this to the end of spec_helper.rb:

class ConstantNameSetBuilder
  def initialize
    @seen_names = Set.new
    @seen_object_ids = Set.new
    @current_ancestors_names = []
    @object_id_method = Object.method(:object_id).unbind
  end

  def build
    @seen_names << "Gem"
    @seen_names << "Module"
    @seen_names << "Object"
    @seen_object_ids << Gem.object_id
    @seen_object_ids << Module.object_id
    @seen_object_ids << Object.object_id

    visit(Object)

    @seen_names
  end

  protected def visit(constant)
    constant.constants.sort.each do |relative_constant_name|
      @current_ancestors_names << relative_constant_name
      @seen_names << @current_ancestors_names.join("::")

      next if constant.autoload?(relative_constant_name)

      sub_constant = constant.const_get(relative_constant_name)
      # This accepts modules and classes
      if ::Module === sub_constant
        sub_constant_object_id = @object_id_method.bind(sub_constant).call

        next if @seen_object_ids.include?(sub_constant_object_id)

        @seen_object_ids << sub_constant_object_id
        visit(sub_constant)
      end
    ensure
      @current_ancestors_names.pop
    end
  end
end

module ConstantsChecker
  def self.finish
    constant_names = ConstantNameSetBuilder.new.build
    binding.irb
  end
  MSpec.register :finish, self
end

This is not a finished solution, but the core of it. At the end of the run, opens a binding.irb where constant_names is a set of all constant names.

The trick now is to add this:

class ContextState
  def it(...)
    # Do nothing
  end
end

By running with this no-op it, you quickly get all of the constants defined out of it blocks. Save the result to a file:

File.write("names_no_execs.txt", constant_names.to_a.sort.join("\n"))

Now, remove the no-op it. Run the specs again. In the console, read the names when it was not run:

before = File.read("names_no_execs.txt").split("\n")

Can now simply do constant_names - before to get the un-cleared names. Some will be from libraries, but a whitelist should take care of that.

[eregon]

Well, I see now that lots of tests are defining the classes/modules directly in the it when they aren't actually testing constant-related things. That's too bad.

Do you have an example? I thought that would be pretty rare.

Another thought that came up is there is Module#const_added since I think 3.3. Maybe that's an easy way to find all constants added by a given it fairly efficiently.

[MaxLap]

To be clear, I was talking about simple class definitions.

Here is a regex to search. \b(?<!\.)class [A-Z]. Ignore the fixture files results, most results are in specs files are defined in the it.

Here is an example: https://github.com/ruby/spec/blob/master/core/kernel/public_send_spec.rb#L7

They won't create problems from re-running, but they create noise for a system that is dedicated to ensuring that tests don't add new constants without cleaning after themself.

Another file with lots of such cases: https://github.com/ruby/spec/blob/master/language/class_spec.rb

For the #const_added idea. That might work too, but I don't know how to apply it globally and it is vulnerable to the same noise as my technique (the classes in it). By the way, the code I posted above adds less than 10 seconds of overhead.