Merge 4-0-123 to backup_sync #129

They may fail throwing an unrecoverable exception in case of unresolved includes etc. Also, some minor clean-up.

For: https://shiftleftinc.atlassian.net/jira/software/c/projects/SEN/issues/SEN-2777

For: https://shiftleftinc.atlassian.net/browse/SEN-2838

This PR handles: * Move parser tests from the `deprecated` frontend to the new `ruby` frontend. * Fixed parser issues with HashLiterals * Fixed parser issues with one-liner class definitions * Fixed parser issues with arguments in functions

Brings in latest babel and typescript to astgen.

* [ruby] Arrow Lambda Parameter Fix Tests various lambdas and fixes `self` parameter name in methods.

partly to minify the flatgraph diff

* minor work on reachingDef * fmt --------- Co-authored-by: Michael Pollmeier <[email protected]>

This astgen version skipps giant, unparsable files with EMSCRIPTEN code now by default. For: https://shiftleftinc.atlassian.net/browse/SEN-2797

…joernio#4719) For: https://shiftleftinc.atlassian.net/browse/SEN-2777

Also handle fullnames with generics correctly when stubbing types For: https://shiftleftinc.atlassian.net/browse/SEN-2840

This PR adds a few more parser tests based on tests in the `querying/` folder for Ruby.

Inheritance via `<` in Ruby can be arbitrary extensions which warrant post-processing analysis, so this removes any attempt to resolve the type at AST creation to allow for a post-processing pass to handle this instead.

Does not exist anymore.

As the `<body>` call is synthetic and meant to be immediately deterministic, so there is no reason it should be re-determined.

* upgrade cpg and adapt * use released cpg

…joernio#4729) ``` [warn] -- [E092] Pattern Match Unchecked Warning: /home/mp/Projects/shiftleft/joern.1/joern-cli/frontends/pysrc2cpg/src/test/scala/io/joern/pysrc2cpg/PySrc2CpgFixture.scala:74:61 [warn] 74 | path.resultPairs().collect { case (firstElement: String, secondElement: Option[Integer]) => [warn] | ^ [warn] |the type test for Option[Integer] cannot be checked at runtime because its type arguments can't be determined from Option[Int] [warn] | [warn] | longer explanation available when compiling with `-explain` } ```

Earlier, if we pass the directory path which contains multiple go modules. Processing was done with all the `.go` files mapped to single `go.mod` file. With this change, we have segregated the processing by first isolating all the files mapped to respective `go.mod`. This will also make sure to cleanup the memory footprint after every module is being processed. However, this will increase the processing when used with download dependency as it will process all the `go.mod` files for identifying and processing used dependencies.

This PR changes the modelling of Singleton methods on objects. Ex: ```ruby class Animal; end animal = Animal.new class << animal def bark "woof" end end ``` Creates a lambda method for `animal.bark`, and assigns `animal.bark = methodRef(bark)` Resolves joernio#4721

- we had one stackoverflow in fullname - evaluation.getOverload may return null

…oernio#4733) * Fix always-crashing array initializer type resolution * Fix exception when trying to cast ArrayType to ClassOrInterfaceType * Fix formatting

…g merge (joernio#4737) * minify the flatgraph diff by bringing various things in before the big merge * fmt * revert accidental change

For new frontends with potentially malformed ASTs, type recovery must safely handle instances where members don't have AST parents.

The parser emits calls to `super` as different from simple calls, this PR handles them.

Also: no more empty method fullnames

* Safely handles the case when `super` call has a `null` argument from the parser * Shadows keywords when they are used as keys in association keys for named arguments in calls * Handles singleton methods in implicit returns

This PR fixes a bug where method members were not correctly linked to surrounding methods' bound type decls. Additionally, this handles `return` statements without any proceeding expression. Resolves joernio#4732

The initial implementation of the edge creation and captured local node was wrongly added to the `Ast` object instead of the diff graph. This PR rectifies this.

Again for: ShiftLeftSecurity/codescience#7730

Due to Ruby method references being called via a `.call()` method, the type ref that should be emitted, is one that contains the `.call` method, but is not necessarily the same type as the one bound to the actual lambda. This makes that change, however, data-flow no longer works in the open-source data-flow tracker as this is not supported.

* [ruby] Remodelled methods defined on singleton objects to no longer be lambdas * [ruby] Review comments

* Renamed `:program` to `<main>` * Replaced `:` method separator to `.` * Removed `<global>` from full names of types and methods

In the case of a type or method re-definition, the full name is ensured to be unique by a set that tracks all full-names for that compilation unit, and a counter. Resolves joernio#4742

This PR remodels `yield` calls as explicit invocations of the implicit or explicit block parameter of the surrounding method. Resolves joernio#4760

Getters and setters were modelled as their CPG operations in isolation, and not using the ordinary AST creator hooks. This PR lowers the getters and setters as their `RubyNode` equivalents, and hands off the AST creation to `astForMethodDeclaration`.

* [php2cpg] Support array/list unpacking in assignment * [php2cpg] Rename method and fix some tests * [php2cpg] code clean and improved test * [php2cpg] improved test

) * IF Cfg creation for if-statements with empty `then` block. * Update joern-cli/frontends/c2cpg/src/test/scala/io/joern/c2cpg/passes/cfg/CfgCreationPassTests.scala Co-authored-by: maltek <[email protected]> --------- Co-authored-by: maltek <[email protected]>

- Changed test constructs used to test the CFG creation to return List instead of Set. This avoids deduplication which is required for correct tests because the number of edges between two nodes matters. - The above change unveiled a problem with the for-statement CFG: Duplicate edge between loop condition and body

* [ruby] Moved precedence for keywords down, added special handling on local identifiers to check for keywords and member access * [ruby] Finished test for member call with reserved keyword * [ruby] remove debug print from RubyNodeCreator * [ruby] Fixed edge case for reserved keywords * [ruby] Added more checks for keyword handling, added test cases for :: syntax and having a base with the same name as a reserved keyword * [ruby] Added antlr debug flag to ruby config * [ruby] Added edge case for keyword handling

* [php2cpg] Support array/list unpacking in assignment * [php2cpg] Rename method and fix some tests * [php2cpg] code clean and improved test * [php2cpg] improved test * [php2cpg] lowering the init part of foreach statement

…modifier (joernio#4768) Fixes these org.eclipse.cdt.core.dom.ast.ExpansionOverlapsBoundaryException for `.getSyntax` calls.

* [ruby] Added file content and offset to method and type decls * [ruby] Added test case for file content * [ruby] Remove empty curlys

…ernio#4780) * [php2cpg] Support array/list unpacking in assignment * [php2cpg] Rename method and fix some tests * [php2cpg] code clean and improved test * [php2cpg] improved test * [php2cpg] lowering the init part of foreach statement * Add a new operator to mark the creation point of an array. improved array unpacking. * Use `array()` call instead of emptyArray operator

With Ruby attribute assignments being calls, this can lead to costly data-flow tracking and these setters are rarely overridden with custom logic. This PR simplifies this model by representing the attribute assignment from a setter call to a direct field assignment. A similar adjustment is done to member getters. Misc: Simplified a parser bug workaround and linked it to an issue.

…arents) (joernio#4778)

* Added a script that parses and creates a sensible string from a slice that can be tested against. * Fixed a slicing bug where slicing direction was in the opposite direction of neighbour retrieval and ended up ignoring certain nodes. * Added entry in the `test-scripts` job to run the slice script, parse the slice, and assert the expected flow with `grep`. Resolves joernio#4783

There was a case that would mistakenly prepend `@` on a member that start with a capitalized first letter.

on stage: remove module-info.class from dependency jars - a hacky workaround for a scala3 compiler bug: scala/scala3#20421 Fixes joernio#4625

Methods were wrong for that.

`.getArrayModifiers` may contain null values for empty array init

This may throw (FileSystemLoopException, or any other reason why this dir may not be readable). We use Files.walkFileTree now. The only solution where a safe continue mechanism can be implemented. Fixes: https://shiftleftinc.atlassian.net/browse/SEN-2976

- implicit this param for CPP functions - identifiers that are actual member accesses are now transformed, e.g., this->varname if varname is a member and we are in the corresponding context

1. Added the support in the CfgCreator. A THROW control structure now breaks control flow. 2. Adjust c2cpg to generate such a control structure instead of a CALL node.

…#4808)

In this PR joernio#4807 the CFG for `throw` statements was changes in the way that they have no outgoing CFG edges. Since this might break assumptions of some CFG consuming code, we now create outgoing CFG edges from `throw` statements to the method exit nodes.

…ecl (joernio#4817)

* [ruby] Initial commit for new Ast Printer, basic methods implemented * [ruby] Initial AstPrinter finished * [ruby] String parser tests working on new AstPrinter * [ruby] ArrayParserTests, SingleAssignmentTests, BeginExpressionTests moved over to new AstPrinter for parsing tests * [ruby] BooleanParserTests moved to new parser test * [ruby] CaseConditionParserTests moved * [ruby] ControlStructureParserTests moved * [ruby] FieldAccess, HashLiteral, IndexAccess and InvocationWithoutParentheses parser tests moved * [ruby] InvocationWithParentheses moved * [ruby] MethodDef and ProcDef parser tests moved * [ruby] Range, regex, require parser tests moved * [ruby] Return and ternary parser tests moved * [ruby] UnlessStmt parser tests moved * [ruby] Ensure statement parser test moved * [ruby] DoBlock parser tests moved * [ruby] RescueClause parser tests moved * [ruby] MultipleAssignment moved * [ruby] ClassDecl moved * [ruby] Module parser tests moved * [ruby] Working on interpolations * [ruby] Cleanup * [ruby] Cleanup for PR * [ruby] removed newMatch flag

* [ruby] Added handling for String interpolated array literals * [ruby] Added type check in tests * cleanup * [ruby] Fixed parser tests with new parser test framework * [ruby] removed if check on element content

* [ruby] handling added for command literal. Modelled as exec call * [ruby] fixed failing parser test * [ruby] Removed print * [ruby] Added expanded command literal to new parser tests

Using the old SerializedCpg API hat no effect anymore.

* [ruby2cpg] Fix ImplicitRequirePass. - Fix method lookup via AST edges. The approach of looking up module methods via is not possible at the execution time of the pass because AST is not yet linked. I replaced this buy a fullname based regex lookup. We should likely just change the execution time to after AST linking. - The other changes are just for better readability and debugability. * Address review remarks. * Fix field access lookup. We are only interested on those field accesses which operator on "self" since those describe the exported/imported objects.

* [ruby] Fixed bracket assignments * [ruby] Add parser test for BracketAssignment

* [ruby] Added filter to remove file with empty contents on AstCreator side * [ruby] lowered log to info, changed expression * formatting

Symbols/identifiers used as based class where not taken into account for the implicit import creation.

* [ruby] Fix assignment parser test * [ruby] Move assigment parser test to correct subset

…oernio#4832)

* [ruby] Fixed regex parser tests * [ruby] Fixed last parser tests * [ruby] Use pattern matching and string interpolation for building initMethod span * fmt * remove comment

Although the parser rule for picking up pseudo-variables seems fine, it may sometimes choose to represent it as a local variable. This fixes this and adds the related tests.

* [ruby] `self` Param `REF` Edge Fixed issue where no `REF` edges between self params and their identifiers. * Fixed the rest of the cases where `self` would have no refs * Updated test expectation

As Ruby may consider `<<` as either shifting left (if LHS is a number) or as an append (if LHS is an array), one cannot statically approximate the operation. We thus consider this as a call named `<<`.

The implicit require calls that are generated after the AST creation had an older/out-dated format and lead to identifiers not being linked via REF edges. Changed require to `require_relative`

* [ruby] Fixed incorrect class/module name, added NamespaceDeclaration node type for classes/modules defined straight in a namespace * [ruby] Added Namespace block to diffgraph, updated class test * [ruby] update module test * revert changes * [ruby] Added file as parent node of namespace block generated in class/module decl in namespace * [ruby] review changes * [ruby] Remove unused fucntions * [ruby] removed unused import

* [ruby] Constructor Lowering Fix * Fixed constructor lowering structure as per joernio#4822 * Tested and fixed issue in parenthesis-less `new` member calls (could use a better long-term fix in the parser) * Testing and handling an expression base of constructor * Fixed some imports * Using `code(node)`

…4844)

* [python] Include Pipfile[.lock] * scalafmt

`self` parameters within the scope of methods reverts back to initial surrounding type information, and keeps `ANY` otherwise, e.g. `<main>` methods.

* Reverted implicit requires to use `require` instead of `require_relative` * Removed requirement for `ImplicitRequirePass` to need `programSummary` argument * Moved import processing to post-processing

Following the other frontends, migrates the non-deprecated post-processing passes to `x2cpg`

* [ruby] initial commit * [ruby] Added test for locals * [ruby] cleanup * [ruby] removed check for methodFullName on scope * [ruby] removed check for methodFullName on scope * [ruby] rename function

* [ruby] Changed methodFullName on callNode for Initialize to DynamicUnknown instead of Any * [ruby] test fixes * [ruby] fixed failing type recovery tests * remove println

…eResolver` (joernio#4855)

joernio#4856)

Fixed an issue where classes defined in namespaces had different fullNames for the class and singleton instance defined in the namespace. Also resolved an issue where methods defined in class defined in a namespace was not being defined under the new `TypeDecl`

* Modified `raise` calls to be control structures of type `THROW` * If a literal argument is given, this is then explicitly represented as a `StandardError.new`

…xpression` (joernio#4860) * [ruby] Updated rescue expressions to use TRY/CATCH/ELSE/FINALLY controlStructureNodes * [ruby] fixed failing control structure test

joernio#4861)

* Handles various access modifiers private/public/protected * Also considers file-level methods are private and initialize methods cannot be anything other than private * Accounts for nested types where their modifiers are separate from surrounding type

* [ruby] Lowered ||= and &&= assignment operators to required if statements * [ruby] remove empty params in scaladocs

Some calls such as `select` may be commonly shadowed by auto-loaded calls, thus, on an example basis, we will continue to comment out similar calls.

* [ruby] Added operatorExpression to commandArgument * [ruby] removed operatorExpression, added only ternaryExpression to command

For chained calls such as `a().b()`, where the base/receiver of `b` involves a call, to avoid invoking the call in both the base and receiver, we now assign a temporary variable to the first invocation and refer to it on the second invocation.

* [php2cpg] Improve performance. This php parser we use seems to have some warmup phase in which it has very poor performance which makes it very expensive to feed it single files like we so far did. This change invokes the php parser with groups of 20 files which on my machine give ~40% performance increase for the total frontend runtime on large projects. Bigger groups of files strangely only show marginal performance increases despite the fact that profiling the frontend indicates that there is still a lot of time wasted by the large amount of individual parser invocations. For now we keep it like this because for bigger groups we would run into argument length limitations which are especially a pain on systems like Windows. Sadly the php parser does not provide means to specify the to be parsed files other than as a list on the command line. So there would need to be some change there first before we could increase the groups size in a meaningful way.

…oernio#4871) * [php2cpg] Fixed the init method of static/const members of class * clean code

The current `foreach` implementation is based on an older approach that doesn't capture the semantics quite as well. This PR implements the newer `forAst` construct, which encourages de-sugaring and representing what is semantically happening underneath the `foreach` statement. Resolves joernio#4674 (comment)

* ghidra 11.1.2 * drop unused private class which doesn't compile any longer after an api change * avoid `+` which breaks some scripts

* [ruby] Fixed single-assignments without basic variables * [ruby] review comments * [ruby] fix failing test

joernio#4884) * [ruby] Add handling for ArrayLiteral used directly as argument in call * [ruby] Moved parser rule to indexingArgumentList, added ANTLR debugging flag to RubyParserTEst * rename ANTLR rule

* [ruby] Add handling for ArrayLiteral used directly as argument in call * [ruby] Moved parser rule to indexingArgumentList, added ANTLR debugging flag to RubyParserTEst * rename ANTLR rule * [ruby] Updated ANTLR rule for array elements to include decimal literals * [ruby] removed comment

* [ruby] update parameterList rule * [ruby] Add fixed parser test

…4879) * Allows a user of the pass to add additional type identifiers to consider for implicit imports (preference to internal types, motivation is that internal types are generally loaded later which works with zeitwerk's newest-first logic) * Handles qualified type names to be detected and imported * Added tests and better handling for constructors * Deduplicates require calls * Add core gems to `DependencyPass`

* [ruby] fixed naked splat on LHS * [ruby] fixed multiple RHS splat args * [ruby] Added test for middle splat arg

…ferent types of arguments (joernio#4890)

…tatements (joernio#4891)

* [ruby] Temp Var Call `code` Fix Fixed bug where splitting on `.` alone does not create a reasonable code property if parameters are not accounted for. * Improved code property construction

* Added `GroupedParameter` type and handling for new type * Fixed ANTLR grammar rules for different combinations of block parameters * Updated relevant parser tests

* Reduced redundant match done by `AstForTypesCreator->handleDefaultConstructor` * Refactored control structure handling into `AstForControlStructuresCreator` * Re-named `RubyNode` to `RubyExpression` as most-if-not-all Ruby constructs can be evaluated to some value, even statements (assignments, control structures, method definitions, etc. all return a value), and added `RubyStatement` to try to subtly define constructs that stand on their own more often that not -> this will be more fleshed out in Pt. 2 * Noted areas where more clean-up and refactoring is required, but want to keep this diff small-ish

* [semanticcpg] remove `runPass` * further remove "unnecessary" variable declarations

* Handles implicit returns of multi-assignments by returning an array of the LHS assignment targets. * For implicit returns of multi-assignments created as desugaring of splatted parameters, returns `nil` as per what is evaluated in the Ruby interpreter.

@main

…oernio#4900) as discussed in joernio#4896, we currently fail hard when the user passes additional frontend arguments, which we forward to XTypeRecoveryConfig.parse With this case we still print a warning, but at least don't error. Potentially better would be to filter out everything _but_ the expected arguments in this case... simple test case: foo.sc: ``` @main def main() = { importCode.python("tests/code/pythonsrc/simple.py", args=List("--exclude-regex", "test*")) } ``` ``` sbt stage ./joern --script foo.sc ```

Added handling for named arguments when using hash rocket syntax with "symbol" keys.

* All `require`-like call names are now a set under `ImportsPass` * These calls are now static calls with only the arguments as children

* [ruby] Fixed parser rules for lambda expressions to allow parameters without parentheses * [ruby] update parser rule

…oernio#4907) * [ruby] Added parser rules and AST lowering for beginless and endless range expressions * [ruby] changed ordering in indexing argument list

such loops are bugs that should be fixed. But they happen occasionally, and right now they lead to OOM and infinite loops. I'd prefer it if we handled that a bit more gracefully. The cost of the set in `.dedup` should be pretty low.

joernio#4908) * [ruby] add methodInvocationWithoutParentheses to RETURN argumentList * [ruby] Added check for arguments of jsonArg

Only static calls with import related names 'require', 'load" and so on are linked to import nodes. Before that we ended up with import nodes and links for calls like `Marshal.load(...)`.

Fixed an instance where something like `foo.bar` would represent `foo` as a `self.foo` field access instead of a `foo()` call, given that no `foo` variable is present.

* update cpg/flatgraph * empty commit to trigger GH actions

…4919)

…#4921)

If an annotation type full name cannot be resolved we now use `<unresolvedNamespace>.ClassName` instead of `ANY`.

In case of a field access after a map/array access, the following field access had invalid argument indicies because it triggered the code path for dynamic receiver calls. This is now fixed.

* Fixed issue where access modifiers in expression positions would be unhandled * Handling identifiers on LHS of associations as symbols

…nio#4920) * Refactor `Semantics` -> `FullNameSemantics` * Refactor `FlowSemantic`/`FlowNode`/`FlowPath` out of the FullNameSemantics parser * Rename `Parser` -> `FullNameSemanticsParser` * Introduce `Semantics` trait with `initialize` and `forMethod`

)

* [kotlin2cpg] Fix "nacked" call representation. Fix representation of "nacked" calls like `someFunc(someArg)`. These so far have always being represented as statically dispatched which is obviously not correct. * Handle static dispatch with instance argument case.

as brought up in joernio#4936 it can lead to confusing situations if type parameters are shadowed, so for the sake of readability and disambiguity alone we should enable this compiler warning IMO. That being said, I'd like to stress that it's not something fundamentally complicated, afaics. Simple example copied from joernio#4936: ```scala class Example[NodeType <: Object](node: NodeType) extends AnyVal{ def foo[NodeType](x: List[NodeType]): List[NodeType] = x } ``` These two type parameters in your Example class are not related at all. They happen to have the same name, which only means that within `def foo` we cannot reference the `NodeType` type from the `class Example`, but apart from that there's no connection at all between them. Semantically it's similar to something like this on the value level: the two `bar` variables are not related at all, they just happen to have the same name, and therefor we cannot reference the class-level `bar` variable from within `def baz`. ```scala class Foo { val bar = 42 def baz: Unit = { val bar = "123" () } } ```

* Added ANTLR profiling with the `--antlrProfile` frontend argument. * If enabled, will gather parser metrics and log them right next to the file that was parsed, as well as print any syntax ambiguities.

Removes the old frontend and related CLI arguments. Hard deadline 20 Sept 2024.

Profiled `ArrayTests` to detect ambiguity and decisions with high lookaheads and modified test fixture to print profiler logs if enabled. This led to converting certain array rules to use more specific rules and fall back to more general rules less often. Some small improvements on `railsgoat` measured with `time` command on `joern-parse`: ``` // With ambiguity 75.58s user 1.98s system 356% cpu 21.762 total 73.56s user 2.61s system 492% cpu 15.452 total 66.52s user 2.01s system 387% cpu 17.667 total // With reduced ambiguity 65.42s user 1.94s system 443% cpu 15.189 total 74.58s user 2.01s system 557% cpu 13.744 total 74.39s user 1.75s system 560% cpu 13.595 total ```

…and detect project and configuration names automatically (joernio#4934) Fix android project detection and automatic name detection for gradle dependency fetcher

There were some unused parser rules adding unnecessary complexity to the parser

Added `simpleCommandArgumentList` to allow multiple call args for `methodInvocationWithoutParentheses`

* fix assignmentModulo * fix assignmentXor * fix assignmentOr * fix assignmentAnd * fix assignmentExponentiation * fix assignmentShiftLeft * fix assignment{Logical,Arithmetic}ShiftRight * scalafmt

…wSemantics (joernio#4953)

…#4954) These synthetic METHOD nodes did not have an offset at all. For: https://shiftleftinc.atlassian.net/browse/SEN-3250

…o#4955) As soon as `FullNameSemantics` started logging duplicate semantics (in joernio#4954), it was observed that `SemanticTestCpg` was passing a fresh Semantics instance to `EngineContext` instead of passing the one used by `OssDataFlow`. By coincidence, however, they were the same. Nevertheless, this patch guarantees they are the same.

) * [ruby] Fixed Argument Order on `ArgumentListContextHelper` Use `line` and `column` information to order arguments correctly. * Removed instances where sorting was done after the fact * Fixed tests that validated incorrect behaviour

Using the CDT API (getContainingFilename) now. For: joernio#4924

* [ruby] ANTLR Profiler Summary * Introduced global profiling to summarize rule and parse performance across the project * Added a shutdown hook to dump a summary of the profiler rules in a file `antlr_summary.log` at the root of the project with this information * Check parent exists before dumping. May not be there during test cases * Added note around shutdown hook

fixes joernio#4976

Sometimes AST trees would be thrown away, specifically when long chained expressions would be compressed into temporary cache variables. This only happens with expressions, and closures typically write directly to the diff graph. This PR minimizes what is written directly to the diff graph from function creation, and extends `x2cpg.Ast` to also include `CAPTURE` edges. This also introduces `closureToRefs` map to track which `Block` nodes already have live method ASTs in the diff graph.

…rnio#4983) This reverts commit 3c27bf6.

…ernio#4945)

* [ruby] Changed handling of ForEach loops * [ruby] fixed double _astIn reference to identifier * [ruby] fix failing tests

…" (joernio#4985) * Revert "Revert "[ruby] Ignore "Throwaway" AST Structures (joernio#4982)" (joernio#4983)" This reverts commit 464480d. * [ruby] Re-implemented "Ignore "Throwaway" AST Structures (joernio#4982)" This correctly prevents re-use of nodes that are already being used elsewhere by ensuring deep copies.

`RubyExpression` nodes don't inherently consider the `span` in the calculation of its hash, so when put into any hashed context, nodes that only rely on `span` alone will collide in these contexts.

… goastgen (joernio#4989)

… dir (joernio#4956) The logic to guess the base dir of the installation is quite fiddly but works for our use cases for astgen. PhpParser implemented something similar, but not quite - and it failed for buildbot. On buildbot the installation path for php2cpg is `/worker/sptestV2-php2cpg/build/x2cpg-internal/php2cpg/target/universal/stage` which (prior to this PR) leads to an invalid derived php-parser name and the following error: ``` 2024-09-25 09:30:08.623 ERROR Invalid path for PhpParserBin: /worker/sptestV2-/php2cpg/bin/php-parser/php-parser.php ```

…nse header (joernio#4995) Turns out the HTTPServer expects an explicit "Connection: close" header which requests the connection to be closed after the transaction ends. Otherwise, it would wait for a 10sec timeout for the next thread to become available. In case we only allow for one thread (jimple2cpg) that would mean additional waiting which renders the whole server approach useless. This in now fixes as we immediately close the connection after the frontend is finished.

This happened for parameter and variable declarations in parentheses.

…ex for named arguments (joernio#5000)

* upgrade deps * trigger ci

…ed arguments (joernio#5003) * [dataflowengineoss] Fix PassThroughMapping criteria for named arguments * sort result for testing purposes

joernio#4998) DescriptorRendererImpl asserts that the options are locked upon init. This means that the type rendering will fail anywhere with assertions enabled (`-ea`) as the options weren't being locked after setup.

…nio#5005)

* ExternalCommand: more information in error conditions * report exit code if it's non-zero * pass on original error (if any) rather than disregarding it * log.warn stderr output (if any) * add tests * compiler warning fix * exit code `2` on linux, `1` on mac... * fix for mac * error msg is different on windows

Works with Java Process / ProcessBuilder now. No more scala.sys.process. ----------- Co-authored-by: Michael Pollmeier <[email protected]>

fixes the regression from the ExternalCommand refactor, but I couldn't leave the the env var handling as it was either. Fixing that by using a proper library for parsing CLI arguments

* [kotlin2cpg] Improve fullnames. Improve and cleanup the fullname calculation for methods and callsites. The most noteable changes here are: 1. The lambda function naming schema changed from `package.<fileName>.<lambda>counter:signature` to `package.scope.<lambda>counter:signature`. This give a better indication which lambda function is represented also gets rid of exception of using the filename in the method fullnames. 2. Fixed lambda function signatures. 3. The fullname schema of extension methods changed from `extendedClassFullName.extensionMethodName:signatureWithoutThisParameter` to `package.extensionMethodName:signatureWithThisParameter` The fact that the this parameter is part of the signature is an exception in our java fullnames but necessary in order to distinguish otherwise identical extension methods. 4. We avoid the recalculation of the binding context with every file processed. This is now done once at the begining. 5. Calls to `::class` are now represented as calls to `<operator>.class`. * Fix extension method arg and param indicies. The instance arg/param now have index 1 as they are not the receiver on which the call is dispatches. It is a static call after all. * Implement first sane version of lambda binding nodes. Additionally fixed the inheritance information for lambda type decl nodes. * Refactor NameRender. Split some methods into BindingContextUtils and renamed those methods. * fmt * Incorporate review comments. - No more special case for handling `kotlin.Function`. - Removed some unused code. - Stop using currentTimeMillis. * Remove typeInfoProvider.expressionType and replace it via nameRenderer. This results in expresssion types now being renderer in the new fassion. * Translate kotlin.Array type full name to java representation. * Removed multiple further type full name API methods from TypeInfoProvider. * Remove typeFullName for KtDestructuringDeclarationEntry from TypeInfoProvider. * Remove visibility and modality API from TypeInfoProvider. * Remove containingTypeDeclFullName. * Remove unnecessary code. * Remove inheritanceTypes API from TypeInfoProvider. * Remove destructuringEntries API from TypeInfoProvider. * Removed KtTypeAlias related TypeInfoProvider APIs. * Remove typeFullName API for binary expressions from TypeInfoProvider. * Remove typeFullName for annotations from TypeInfoProvider. * Remove unnecessary Option types from BindingContextUtils APIs. * Rework lambda implicit parameter handling. - Removed implicitParameterName and hasApplyOrAlsoScopeFunctionParent APIs from TypeInfoProvider. - Removed wrong parameter deconstruction in lambda. New implemention is missing. - Some lambda to builtin `apply` and `also` functions did not get return statements generated. That is now fixed. * Remove typeFullName and isCompanionObject APIs from TypeInfoProvider. * Remove typeFullName API for KtTypeReference from TypeInfoProvider. * Remove containingDeclType API from TypeInfoProvider. * Remove typeFullName and referenceTargetTypeFullName APIs from TypeInfoProvider. Also refactored assignmentAstForDestructuringEntry. The right hand side base is now created outside of this function allowing for other constructs than pure identifiers. The pure identifier were e.g. wrong in case of class member references. * Remove propertyType and typeFromImports API from TypeInfoProvider. * Remove TypeRenderer. * Improve comment. * Fmt. * Fix for `ScopeFunctionsTests` * Adjusted test expectations for more precise/sound outcomes * Left note about the lambda flows * Add better fallback handling methodFullName of calls. In case of incomplete type information calls with overloads can get ambiguous. We now use the ambiguous function descriptors to get as much of a methodFullName correct as possible. * Fix test fullname. --------- Co-authored-by: David Baker Effendi <[email protected]>

This bypasses tty buffering and read timeouts.

* [dataflowengineoss] add Operators.modulo semantics * sort results for deterministic outcome

…#5031)

* [pysrc2cpg] refactor <module>, ANY and __init__ constants * Replace Constants/Defines, cf. feedback suggestion

Merge 4-0-123 to backup_sync #129

Are you sure you want to change the base?

Merge 4-0-123 to backup_sync #129

Commits on Jul 1, 2024

Commits on Jul 2, 2024

Commits on Jul 3, 2024

Commits on Jul 4, 2024

Commits on Jul 5, 2024

Commits on Jul 8, 2024

Commits on Jul 9, 2024

Commits on Jul 10, 2024

Commits on Jul 11, 2024

Commits on Jul 12, 2024

Commits on Jul 15, 2024

Commits on Jul 16, 2024

Commits on Jul 17, 2024

Commits on Jul 18, 2024

Commits on Jul 19, 2024

Commits on Jul 22, 2024

Commits on Jul 24, 2024

Commits on Jul 25, 2024

Commits on Jul 29, 2024

Commits on Jul 30, 2024

Commits on Jul 31, 2024

Commits on Aug 1, 2024

Commits on Aug 2, 2024

Commits on Aug 5, 2024

Commits on Aug 6, 2024

Commits on Aug 7, 2024

Commits on Aug 12, 2024

Commits on Aug 13, 2024

Commits on Aug 14, 2024

Commits on Aug 15, 2024

Commits on Aug 16, 2024

Commits on Aug 19, 2024

Commits on Aug 20, 2024

Commits on Aug 21, 2024

Commits on Aug 22, 2024

Commits on Aug 23, 2024

Commits on Aug 26, 2024

Commits on Aug 27, 2024

Commits on Aug 28, 2024

Commits on Aug 29, 2024

Commits on Aug 30, 2024

Commits on Sep 2, 2024

Commits on Sep 3, 2024

Commits on Sep 4, 2024

Commits on Sep 5, 2024

Commits on Sep 6, 2024

Commits on Sep 9, 2024

Commits on Sep 10, 2024

Commits on Sep 11, 2024

Commits on Sep 12, 2024

Commits on Sep 13, 2024

Commits on Sep 14, 2024

Commits on Sep 16, 2024

Commits on Sep 17, 2024

Commits on Sep 18, 2024

Commits on Sep 19, 2024

Commits on Sep 20, 2024

Commits on Sep 23, 2024

Commits on Sep 24, 2024

Commits on Sep 25, 2024

Commits on Sep 26, 2024

Commits on Sep 27, 2024

Commits on Sep 29, 2024

Commits on Sep 30, 2024

Commits on Oct 1, 2024

Commits on Oct 2, 2024

Commits on Oct 3, 2024

Commits on Oct 7, 2024

Commits on Oct 9, 2024

Commits on Oct 10, 2024

Commits on Oct 11, 2024

Commits on Oct 14, 2024

Commits on Oct 17, 2024

Commits on Oct 18, 2024

Commits on Oct 21, 2024

Commits on Oct 22, 2024

Commits on Oct 23, 2024