Improving the OpenSSH variant analysis. (#560)

* Improving the OpenSSH variant analysis.

* Working on variant analysis.

* Working on variant analysis.

* Re-bootstrap. Update GAP, VAST, and PASTA.

* Updating dependencies

* bump deps, rebootstrap

README.md

@@ -4,7 +4,7 @@ Multiplier provides precise and comprehensive code understanding capabilities.
 It does so by saving build artifacts into a database, and then making them
 persistently accessible using a C++ or Python API.
 
-Multiplier emphasizes the ability to unique identify *all* entities in a build
+Multiplier emphasizes the ability to uniquely identify *all* entities in a build
 process, including individual tokens, AST nodes, and intermediate
 representations. With Multiplier, an analyst can identify code patterns of
 interest over one of the representations, and then accurately relay results back
@@ -21,6 +21,9 @@ We like to say that with its APIs, *you can get everywhere from anywhere*.
   * [Getting and building the code](docs/BUILD.md)
   * [Installing a pre-built release](docs/INSTALLING.md)
   * [How to index a codebase](docs/INDEXING.md)
+* Writeups
+  * [regreSSHion OpenSSH variant analysis](docs/openssh-variant-analysis.md)
+  * [PHP variant analysis](docs/php-variant-analysis.md)
 * Included tools
   * [Find function calls inside macro argument lists](docs/mx-find-calls-in-macro-expansions.md)
   * [Find possible divergent representations](docs/mx-find-divergent-candidates.md)
@@ -29,6 +32,7 @@ We like to say that with its APIs, *you can get everywhere from anywhere*.
   * [Find "sketchy" casts flowing to function arguments and to return sites](docs/mx-find-sketchy-casts.md)
   * [Extract an entity, e.g. a function, and all of its dependencies into a file](docs/mx-harness.md)
   * [Highlight a specific entity within its surrounding code](docs/mx-highlight-entity.md)
+  * [Highlight all references to an entity](docs/mx-highlight-references.md)
   * [Print a call graph](docs/mx-print-call-graph.md)
   * [Print the reference graph](docs/mx-print-reference-graph.md)
   * [Print a graph relating source code, macros, parsed tokens, and AST nodes](docs/mx-print-token-graph.md)
@@ -42,9 +46,6 @@ We like to say that with its APIs, *you can get everywhere from anywhere*.
   * [List all indexed structures/unions/classes/enums](docs/mx-list-structures.md)
   * [List all indexed variables](docs/mx-list-variables.md)
   * [Search the code with regular expressions](docs/mx-regex-query.md)
-* Writeups
-  * [regreSSHion OpenSSH variant analysis](docs/openssh-variant-analysis.md)
-  * [PHP variant analysis](docs/php-variant-analysis.md)
 
 # License
 

bin/Bootstrap/PASTA.cpp

@@ -793,7 +793,7 @@ static std::set<std::pair<std::string, std::string>> kMethodBlackList{
 
 static const char *SchemaIntType(pasta::Type type) {
   auto t = type;
-  if (auto bt = pasta::BuiltinType::From(type.UnqualifiedType())) {
+  if (auto bt = pasta::BuiltinType::From(type.UnqualifiedDesugaredType())) {
     switch (bt->BuiltinKind()) {
       case pasta::BuiltinTypeKind::kBoolean: return "Bool";
       case pasta::BuiltinTypeKind::kCharacterS: return "Int8";  // `char`.
@@ -3328,6 +3328,13 @@ MethodListPtr CodeGenerator::RunOnClass(
       }
     } else if (snake_name == "calculate_inheritance_model") {
       snake_name = "inheritance_model";
+
+    } else if (snake_name == "function_scope_index") {
+      snake_name = "index";
+
+    // For parameter packs / substituted parameters.
+    } else if (snake_name == "function_scope_depth") {
+      snake_name = "depth";
     }
 
     if (snake_name.ends_with("_type_info")) {
@@ -3372,9 +3379,9 @@ MethodListPtr CodeGenerator::RunOnClass(
 
     std::string camel_name = SnakeCaseToCamelCase(snake_name);
 
-    auto return_type = method.ReturnType().UnqualifiedType();
+    auto return_type = method.ReturnType().UnqualifiedDesugaredType();
     if (auto return_type_ref = pasta::ReferenceType::From(return_type)) {
-      return_type = return_type_ref->PointeeType().UnqualifiedType();
+      return_type = return_type_ref->PointeeType().UnqualifiedDesugaredType();
     }
 
     if (auto record = return_type.AsRecordDeclaration()) {
@@ -3795,6 +3802,9 @@ MethodListPtr CodeGenerator::RunOnClass(
 
       serialize_cpp_os
           << "  b." << setter_name << "(e." << method_name << "());\n";
+
+    } else {
+      std::cerr << "\tMissing " << method_name << '\n';
     }
   }
 

bin/Bootstrap/PythonBindings.py

@@ -400,6 +400,14 @@ class UserToken;
 """
 
 
+STATIC_GENERATOR_SPEC_CALL = """
+            using Gen = ::mx::StaticGenerator<{cxx_generator_params}>;
+            Gen::FuncType *func = &T::{method_name};
+            Gen generator(func{args_comma}{args});
+            return ::mx::generator_to_python(std::move(generator), &Gen::generate);
+"""
+
+
 METHOD_SPEC_CHECK_ARGCOUNT_END = "          }\n"
 
 
@@ -1102,13 +1110,15 @@ def rename_method(self, class_schema: ClassSchema,
 
 ARGUMENT_RENAMES = {
   "from": "from_",
+  "global": "global_",
 }
 
 class BasicRenamer(Renamer):
   METHOD_RENAMES = {
     "from": "FROM",
     "in": "IN",
     "as": "AS",
+    "global": "global_",
   }
 
   def rename_method(self, class_schema: ClassSchema,
@@ -1151,10 +1161,25 @@ def _wrap_method_impl(class_schema: ClassSchema, schema: MethodSchema,
     else:
       cxx_args.append(METHOD_SPEC_CALL_ARG.format(i))
 
-  out.append(METHOD_SPEC_CALL.format(
+  # NOTE(pag): We don't really need the special generator form to turn static
+  #            methods returning generators into instance methods returning
+  #            generators. This turned out to be a side-quest during bug-hunting
+  #            related to a generator lifetime issue, but it was really a simple
+  #            error of a `std::move` in a loop.
+  call_form = METHOD_SPEC_CALL
+  param_types = []
+  if False and is_static and isinstance(schema.return_type, GapGeneratorSchema):
+    call_form = STATIC_GENERATOR_SPEC_CALL
+    param_types = [schema.return_type.element_type.cxx_value_name]
+    for i, arg in enumerate(schema.parameters):
+      param_types.append(arg.element_type.cxx_name)
+
+  out.append(call_form.format(
       this=is_static and "T::" or "obj->",
       method_name=schema.name,
-      args=", ".join(cxx_args)))
+      args_comma=len(cxx_args) and ", " or "",
+      args=", ".join(cxx_args),
+      cxx_generator_params=", ".join(param_types)))
 
   out.append(METHOD_SPEC_CHECK_ARGCOUNT_END)
 

bin/Index/BuildPendingFragment.cpp

@@ -312,9 +312,15 @@ bool PendingFragment::TryAdd(const pasta::Decl &entity) {
 }
 
 bool PendingFragment::TryAdd(const pasta::Stmt &entity) {
+
+  // If we're generating MLIR, then we need the entity IDs of statements to
+  // stay around until code generation time, as we only generate MLIR after
+  // persisting all fragments.
+  auto &entity_ids = em.generate_source_ir ? em.entity_ids : em.token_tree_ids;
+
   return DoTryAdd(
       entity,
-      em.generate_source_ir ? em.entity_ids : em.token_tree_ids,
+      entity_ids,
       [&, this] (mx::EntityOffset offset) {
         mx::StmtId id;
         id.fragment_id = fragment_index;