Test flakes when running tests in parallel

[msullivan]

If I install pytest-xdist and run tests with pytest -n8, sometimes (half the time?) the test models_imported_inside_init_file_one_to_one_field fails with a failure like:

Traceback (most recent call last):
  File "/home/msullivan/src/django-stubs/env/lib/python3.6/site-packages/_pytest/runner.py", line 226, in from_call
    result = func()
  File "/home/msullivan/src/django-stubs/env/lib/python3.6/site-packages/_pytest/runner.py", line 198, in <lambda>
    lambda: ihook(item=item, **kwds), when=when, reraise=reraise
  File "/home/msullivan/src/django-stubs/env/lib/python3.6/site-packages/pluggy/hooks.py", line 289, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/home/msullivan/src/django-stubs/env/lib/python3.6/site-packages/pluggy/manager.py", line 68, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/home/msullivan/src/django-stubs/env/lib/python3.6/site-packages/pluggy/manager.py", line 62, in <lambda>
    firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
  File "/home/msullivan/src/django-stubs/env/lib/python3.6/site-packages/pluggy/callers.py", line 208, in _multicall
    return outcome.get_result()
  File "/home/msullivan/src/django-stubs/env/lib/python3.6/site-packages/pluggy/callers.py", line 80, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/home/msullivan/src/django-stubs/env/lib/python3.6/site-packages/pluggy/callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "/home/msullivan/src/django-stubs/env/lib/python3.6/site-packages/_pytest/runner.py", line 123, in pytest_runtest_call
    item.runtest()
  File "/home/msullivan/src/django-stubs/env/lib/python3.6/site-packages/pytest_mypy/item.py", line 139, in runtest
    return_code = typecheck_with_mypy(mypy_cmd_options)
  File "/home/msullivan/src/django-stubs/env/lib/python3.6/site-packages/pytest_mypy/item.py", line 76, in typecheck_with_mypy
    flush_errors=flush_errors, fscache=fscache)
  File "/home/msullivan/src/django-stubs/env/lib/python3.6/site-packages/mypy/build.py", line 162, in build
    result = _build(sources, options, alt_lib_path, flush_errors, fscache)
  File "/home/msullivan/src/django-stubs/env/lib/python3.6/site-packages/mypy/build.py", line 217, in _build
    graph = dispatch(sources, manager)
  File "/home/msullivan/src/django-stubs/env/lib/python3.6/site-packages/mypy/build.py", line 2360, in dispatch
    process_graph(graph, manager)
  File "/home/msullivan/src/django-stubs/env/lib/python3.6/site-packages/mypy/build.py", line 2653, in process_graph
    process_fresh_modules(graph, prev_scc, manager)
  File "/home/msullivan/src/django-stubs/env/lib/python3.6/site-packages/mypy/build.py", line 2731, in process_fresh_modules
    graph[id].fix_cross_refs()
  File "/home/msullivan/src/django-stubs/env/lib/python3.6/site-packages/mypy/build.py", line 1714, in fix_cross_refs
    self.options.use_fine_grained_cache)
  File "/home/msullivan/src/django-stubs/env/lib/python3.6/site-packages/mypy/fixup.py", line 25, in fixup_module
    node_fixer.visit_symbol_table(tree.names)
  File "/home/msullivan/src/django-stubs/env/lib/python3.6/site-packages/mypy/fixup.py", line 77, in visit_symbol_table
    self.quick_and_dirty)
  File "/home/msullivan/src/django-stubs/env/lib/python3.6/site-packages/mypy/fixup.py", line 263, in lookup_qualified_stnode
    return lookup_fully_qualified(name, modules, raise_on_missing=not quick_and_dirty)
  File "/home/msullivan/src/django-stubs/env/lib/python3.6/site-packages/mypy/lookup.py", line 47, in lookup_fully_qualified
    assert key in names, "Cannot find %s for %s" % (key, name)
AssertionError: Cannot find app for myapp.models.app.App

[higherorderfunctor]

I am also seeing this issue when running with the plugin. I cannot reproduce with the plugin disabled. It is very intermittent, however, when hooked to a linter, it triggers quite often. I resorted to turning off incremental mode, but this quite slow while linting. The first manual run after clearing the cache is always successful. Walking lookup.py:lookup_fully_qualified, the modules dict appears to be lacking some of the project's sub-modules causing the walk to fail.

Below is some output after clearing the cache and running manually to rule out concurrency issues with the linter. After stepping into one of my root modules, it fails to find the models sub-module.

[mypy]
plugins = mypy_django_plugin.main
python_version = 3.7
check_untyped_defs = True
disallow_any_decorated = True
disallow_any_generics = True
disallow_any_unimported = True
disallow_incomplete_defs = True
disallow_subclassing_any = True
disallow_untyped_defs = True
follow_imports = silent
ignore_missing_imports = True
show_error_context = True
warn_redundant_casts = True
warn_return_any = True
warn_unused_ignores = True
disallow_untyped_decorators = True

#incremental=False

[mypy_django_plugin]
django_settings = config.settings.development

Traceback (most recent call last):
  File "/home/higherorderfunctor/workspace/myproject/ENV/bin/mypy", line 24, in <module>
    sys.exit(console_entry())
  File "/home/higherorderfunctor/workspace/myproject/ENV/lib/python3.7/site-packages/mypy/__main__.py", line 7, in console_entry
    main(None)
  File "/home/higherorderfunctor/workspace/myproject/ENV/lib/python3.7/site-packages/mypy/main.py", line 84, in main
    res = build.build(sources, options, None, flush_errors, fscache)
  File "/home/higherorderfunctor/workspace/myproject/ENV/lib/python3.7/site-packages/mypy/build.py", line 163, in build
    result = _build(sources, options, alt_lib_path, flush_errors, fscache)
  File "/home/higherorderfunctor/workspace/myproject/ENV/lib/python3.7/site-packages/mypy/build.py", line 218, in _build
    graph = dispatch(sources, manager)
  File "/home/higherorderfunctor/workspace/myproject/ENV/lib/python3.7/site-packages/mypy/build.py", line 2461, in dispatch
    process_graph(graph, manager)
  File "/home/higherorderfunctor/workspace/myproject/ENV/lib/python3.7/site-packages/mypy/build.py", line 2754, in process_graph
    process_fresh_modules(graph, prev_scc, manager)
  File "/home/higherorderfunctor/workspace/myproject/ENV/lib/python3.7/site-packages/mypy/build.py", line 2832, in process_fresh_modules
    graph[id].fix_cross_refs()
  File "/home/higherorderfunctor/workspace/myproject/ENV/lib/python3.7/site-packages/mypy/build.py", line 1803, in fix_cross_refs
    self.options.use_fine_grained_cache)
  File "/home/higherorderfunctor/workspace/myproject/ENV/lib/python3.7/site-packages/mypy/fixup.py", line 25, in fixup_module
    node_fixer.visit_symbol_table(tree.names)
  File "/home/higherorderfunctor/workspace/myproject/ENV/lib/python3.7/site-packages/mypy/fixup.py", line 88, in visit_symbol_table
    self.visit_type_info(value.node)
  File "/home/higherorderfunctor/workspace/myproject/ENV/lib/python3.7/site-packages/mypy/fixup.py", line 45, in visit_type_info
    self.visit_symbol_table(info.names)
  File "/home/higherorderfunctor/workspace/myproject/ENV/lib/python3.7/site-packages/mypy/fixup.py", line 90, in visit_symbol_table
    value.node.accept(self)
  File "/home/higherorderfunctor/workspace/myproject/ENV/lib/python3.7/site-packages/mypy/nodes.py", line 832, in accept
    return visitor.visit_var(self)
  File "/home/higherorderfunctor/workspace/myproject/ENV/lib/python3.7/site-packages/mypy/fixup.py", line 133, in visit_var
    v.type.accept(self.type_fixer)
  File "/home/higherorderfunctor/workspace/myproject/ENV/lib/python3.7/site-packages/mypy/types.py", line 625, in accept
    return visitor.visit_instance(self)
  File "/home/higherorderfunctor/workspace/myproject/ENV/lib/python3.7/site-packages/mypy/fixup.py", line 157, in visit_instance
    a.accept(self)
  File "/home/higherorderfunctor/workspace/myproject/ENV/lib/python3.7/site-packages/mypy/types.py", line 625, in accept
    return visitor.visit_instance(self)
  File "/home/higherorderfunctor/workspace/myproject/ENV/lib/python3.7/site-packages/mypy/fixup.py", line 150, in visit_instance
    inst.type = lookup_qualified_typeinfo(self.modules, type_ref, self.allow_missing)
  File "/home/higherorderfunctor/workspace/myproject/ENV/lib/python3.7/site-packages/mypy/fixup.py", line 240, in lookup_qualified_typeinfo
    node = lookup_qualified(modules, name, allow_missing)
  File "/home/higherorderfunctor/workspace/myproject/ENV/lib/python3.7/site-packages/mypy/fixup.py", line 254, in lookup_qualified
    stnode = lookup_qualified_stnode(modules, name, allow_missing)
  File "/home/higherorderfunctor/workspace/myproject/ENV/lib/python3.7/site-packages/mypy/fixup.py", line 263, in lookup_qualified_stnode
    return lookup_fully_qualified(name, modules, raise_on_missing=not allow_missing)
  File "/home/higherorderfunctor/workspace/myproject/ENV/lib/python3.7/site-packages/mypy/lookup.py", line 47, in lookup_fully_qualified
    assert key in names, "Cannot find %s for %s" % (key, name)
AssertionError: Cannot find models for ipam.models.block.Block
> /home/higherorderfunctor/workspace/myproject/ENV/lib/python3.7/site-packages/mypy/lookup.py(47)lookup_fully_qualified()
-> assert key in names, "Cannot find %s for %s" % (key, name)
(Pdb) import pprint
(Pdb) pprint.PrettyPrinter(indent=4).pprint(sorted(modules.keys()))
[   'Crypto',
    'Crypto.Cipher',
    'Crypto.Cipher.Blowfish',
    'Crypto.Cipher.blockalgo',
    '__future__',
    '_ast',
    '_importlib_modulespec',
    '_markupbase',
    'abc',
    'argparse',
    'ast',
    'attr',
    'attr.converters',
    'attr.exceptions',
    'attr.filters',
    'attr.validators',
    'builtins',
    'codecs',
    'collections',
    'collections.abc',
    'contextlib',
    'copy',
    'core',
    'core.admin',
    'core.exceptions',
    'core.models.readonly',
    'core.routers',
    'core.tests.utils',
    'core.utils',
    'core.utils.datatypes',
    'core.utils.datatypes.either',
    'core.utils.datatypes.lift',
    'core.utils.modeltools',
    'dataclasses',
    'datetime',
    'decimal',
    'django',
    'django.apps',
    'django.apps.config',
    'django.apps.registry',
    'django.conf',
    'django.conf.global_settings',
    'django.conf.urls',
    'django.contrib',
    'django.contrib.admin.decorators',
    'django.contrib.admin.models',
    'django.contrib.admin.views',
    'django.contrib.auth',
    'django.contrib.auth.backends',
    'django.contrib.auth.base_user',
    'django.contrib.auth.models',
    'django.contrib.auth.signals',
    'django.contrib.auth.tokens',
    'django.contrib.auth.validators',
    'django.contrib.contenttypes',
    'django.contrib.contenttypes.fields',
    'django.contrib.contenttypes.models',
    'django.contrib.messages.constants',
    'django.contrib.messages.storage.base',
    'django.contrib.postgres',
    'django.contrib.postgres.fields.array',
    'django.contrib.postgres.fields.citext',
    'django.contrib.postgres.fields.hstore',
    'django.contrib.postgres.fields.jsonb',
    'django.contrib.postgres.fields.mixins',
    'django.contrib.postgres.fields.ranges',
    'django.contrib.sessions',
    'django.contrib.sessions.backends',
    'django.contrib.sessions.backends.base',
    'django.core',
    'django.core.cache',
    'django.core.cache.backends',
    'django.core.cache.backends.base',
    'django.core.checks',
    'django.core.checks.messages',
    'django.core.checks.registry',
    'django.core.exceptions',
    'django.core.files',
    'django.core.files.base',
    'django.core.files.images',
    'django.core.files.storage',
    'django.core.files.temp',
    'django.core.files.uploadedfile',
    'django.core.files.uploadhandler',
    'django.core.files.utils',
    'django.core.handlers',
    'django.core.handlers.base',
    'django.core.handlers.wsgi',
    'django.core.management',
    'django.core.management.base',
    'django.core.management.color',
    'django.core.paginator',
    'django.core.serializers',
    'django.core.serializers.base',
    'django.core.serializers.json',
    'django.core.serializers.python',
    'django.core.servers',
    'django.core.servers.basehttp',
    'django.core.validators',
    'django.core.wsgi',
    'django.db',
    'django.db.backends',
    'django.db.backends.base',
    'django.db.backends.base.base',
    'django.db.backends.base.client',
    'django.db.backends.base.creation',
    'django.db.backends.base.features',
    'django.db.backends.base.introspection',
    'django.db.backends.base.schema',
    'django.db.backends.base.validation',
    'django.db.backends.ddl_references',
    'django.db.backends.sqlite3',
    'django.db.backends.sqlite3.base',
    'django.db.backends.utils',
    'django.db.migrations',
    'django.db.migrations.migration',
    'django.db.migrations.operations',
    'django.db.migrations.operations.base',
    'django.db.migrations.operations.fields',
    'django.db.migrations.operations.models',
    'django.db.migrations.operations.special',
    'django.db.migrations.state',
    'django.db.models',
    'django.db.models.aggregates',
    'django.db.models.base',
    'django.db.models.deletion',
    'django.db.models.expressions',
    'django.db.models.fields',
    'django.db.models.fields.files',
    'django.db.models.fields.mixins',
    'django.db.models.fields.proxy',
    'django.db.models.fields.related',
    'django.db.models.fields.related_descriptors',
    'django.db.models.fields.related_lookups',
    'django.db.models.fields.reverse_related',
    'django.db.models.functions.comparison',
    'django.db.models.functions.datetime',
    'django.db.models.functions.text',
    'django.db.models.functions.window',
    'django.db.models.indexes',
    'django.db.models.lookups',
    'django.db.models.manager',
    'django.db.models.options',
    'django.db.models.query',
    'django.db.models.query_utils',
    'django.db.models.signals',
    'django.db.models.sql',
    'django.db.models.sql.compiler',
    'django.db.models.sql.datastructures',
    'django.db.models.sql.query',
    'django.db.models.sql.subqueries',
    'django.db.models.sql.where',
    'django.db.transaction',
    'django.db.utils',
    'django.dispatch',
    'django.dispatch.dispatcher',
    'django.forms',
    'django.forms.boundfield',
    'django.forms.fields',
    'django.forms.forms',
    'django.forms.formsets',
    'django.forms.models',
    'django.forms.renderers',
    'django.forms.utils',
    'django.forms.widgets',
    'django.http',
    'django.http.cookie',
    'django.http.request',
    'django.http.response',
    'django.middleware',
    'django.template',
    'django.template.backends',
    'django.template.backends.base',
    'django.template.base',
    'django.template.context',
    'django.template.defaultfilters',
    'django.template.defaulttags',
    'django.template.engine',
    'django.template.exceptions',
    'django.template.library',
    'django.template.loader_tags',
    'django.template.loaders',
    'django.template.loaders.base',
    'django.template.response',
    'django.template.smartif',
    'django.template.utils',
    'django.test',
    'django.test.client',
    'django.test.runner',
    'django.test.testcases',
    'django.test.utils',
    'django.urls',
    'django.urls.base',
    'django.urls.conf',
    'django.urls.converters',
    'django.urls.exceptions',
    'django.urls.resolvers',
    'django.urls.utils',
    'django.utils',
    'django.utils.datastructures',
    'django.utils.dateparse',
    'django.utils.datetime_safe',
    'django.utils.deprecation',
    'django.utils.encoding',
    'django.utils.functional',
    'django.utils.html',
    'django.utils.module_loading',
    'django.utils.safestring',
    'django.utils.six',
    'django.utils.text',
    'django.utils.timezone',
    'django.utils.translation.trans_real',
    'django.utils.tree',
    'django.utils.version',
    'django.views.decorators',
    'django.views.decorators.http',
    'django.views.generic.base',
    'email',
    'email.charset',
    'email.contentmanager',
    'email.errors',
    'email.header',
    'email.message',
    'email.policy',
    'enum',
    'functools',
    'gettext',
    'html',
    'html.parser',
    'http',
    'http.cookies',
    'http.server',
    'import_export',
    'import_export.formats',
    'import_export.formats.base_formats',
    'import_export.instance_loaders',
    'import_export.results',
    'import_export.signals',
    'import_export.tmp_storages',
    'import_export.widgets',
    'importlib',
    'importlib.abc',
    'io',
    'ipam',
    'ipam.admin',
    'ipam.admin.models',
    'ipam.admin.models.netelement_snmp_admin',
    'ipam.admin.models.snmp_admin',
    'ipam.admin.resources',
    'ipam.apiv1',
    'ipam.apiv1.serializers',
    'ipam.apiv1.urls',
    'ipam.apiv1.views',
    'ipam.apiv1.views.netelement',
    'ipam.migrations',
    'ipam.models.rir_organization',
    'ipam.tests',
    'ipam.tests.base',
    'ipam.tests.models',
    'ipam.tests.models.base',
    'ipam.tests.models.test_admin',
    'ipam.tests.models.test_agent',
    'ipam.tests.models.test_container',
    'ipam.tests.models.test_container_closure',
    'ipam.tests.models.test_netelement',
    'ipam.utils',
    'itertools',
    'json',
    'json.decoder',
    'json.encoder',
    'logging',
    'mmap',
    'model_utils.choices',
    'model_utils.fields',
    'model_utils.managers',
    'model_utils.tracker',
    'mypy_extensions',
    'numbers',
    'operator',
    'os',
    'os.path',
    'posix',
    'pprint',
    're',
    'rest_framework',
    'rest_framework.schemas.generators',
    'rest_framework.settings',
    'rest_framework.utils',
    'rest_framework.utils.field_mapping',
    'rest_framework.utils.model_meta',
    'simple_history.exceptions',
    'simple_history.manager',
    'simple_history.signals',
    'socket',
    'socketserver',
    'sqlite3',
    'sqlite3.dbapi2',
    'string',
    'sys',
    'tempfile',
    'threading',
    'time',
    'traceback',
    'types',
    'typing',
    'typing_extensions',
    'unittest',
    'unittest.mock',
    'uuid',
    'warnings',
    'wsgiref',
    'wsgiref.handlers',
    'wsgiref.headers',
    'wsgiref.simple_server',
    'wsgiref.types',
    'wsgiref.util']
(Pdb) key
'models'
(Pdb) pprint.PrettyPrinter(indent=4).pprint(sorted(names))
[   '__doc__',
    '__file__',
    '__name__',
    '__package__',
    'default_app_config',
    'utils']
(Pdb) print(modules.get(head))
MypyFile:1(
  /home/higherorderfunctor/workspace/myproject/ipam/__init__.py)
(Pdb) print(modules.get(head).names)
SymbolTable(
  default_app_config : Gdef/Var (ipam.default_app_config) : builtins.str
  utils : Gdef/MypyFile (ipam.utils))