|
| 1 | +# Python Plugin |
| 2 | + |
| 3 | +## Parsing Python projects |
| 4 | +Python projects can be parsed by using the `CodeCompass_parser` executable. |
| 5 | +See its usage [in a seperate document](/doc/usage.md). |
| 6 | + |
| 7 | +## Python specific parser flags |
| 8 | + |
| 9 | +### Python dependencies |
| 10 | +Large Python projects usually have multiple Python package dependencies. |
| 11 | +Although a given project can be parsed without installing any of its dependencies, it is strongly recommended |
| 12 | +that the required modules are installed in order to achieve a complete parsing. |
| 13 | +To install a project's dependencies, create a [Python virtual environment](https://docs.python.org/3/library/venv.html) |
| 14 | +and install the necessary packages. |
| 15 | +When parsing a project, specify the virtual environment path so the parser can successfully resolve the dependencies: |
| 16 | +``` |
| 17 | +--venvpath <path to virtual environment> |
| 18 | +``` |
| 19 | + |
| 20 | +### Type hints |
| 21 | +The parser can try to determine Python type hints for variables, expressions and functions. |
| 22 | +It can work out type hints such as `Iterable[int]` or `Union[int, str]`. |
| 23 | +However, this process can be extremely slow, especially for functions, thus it is disabled by default. |
| 24 | +It can be enabled using the `--type-hint` flag. |
| 25 | + |
| 26 | +### Python submodules |
| 27 | +Large Python projects can have internal submodules and the parser tries to locate them automatically. |
| 28 | +Specifically, it looks for `__init__.py` files and considers those folders modules. |
| 29 | +This process is called submodule discovery and can be disabled using the `--disable-submodule-discovery` flag. |
| 30 | + |
| 31 | +You can also add submodules manually by adding those specific paths to the parser's syspath: |
| 32 | +``` |
| 33 | +--syspath <path> |
| 34 | +``` |
| 35 | +For more information, see the [Python syspath docs](https://docs.python.org/3/library/sys.html#sys.path). |
| 36 | + |
| 37 | +### File references |
| 38 | +By default, the parser works out references by looking for definitions only - if nodes share the same definition |
| 39 | +they are considered references. |
| 40 | +However, this method sometimes misses a few references (e.g. local variables in a function). |
| 41 | +To extend search for references in a file context, apply the `--file-refs` flag. |
| 42 | +Note that using this option can potentially extend the total parsing time. |
| 43 | + |
| 44 | +## Examples of parsing Python projects |
| 45 | + |
| 46 | +### Flask |
| 47 | +We downloaded [flask 3.1.0](https://github.com/pallets/flask/releases/tag/3.1.0) source code to `~/parsing/flask/`. |
| 48 | +The first step is to create a Python virtual environment and install flask's dependencies. |
| 49 | +Create a Python virtual environment and activate it: |
| 50 | +```bash |
| 51 | +cd ~/parsing/flask/ |
| 52 | +python3 -m venv venv |
| 53 | +source venv/bin/activate |
| 54 | +``` |
| 55 | +Next, we install the required dependencies listed in `pyproject.toml`. |
| 56 | +```bash |
| 57 | +pip install . |
| 58 | +``` |
| 59 | +Further dependencies include development packages listed in `requirements/dev.txt`. |
| 60 | +These can be also installed using `pip`. |
| 61 | +```bash |
| 62 | +pip install -r requirements/dev.txt |
| 63 | +``` |
| 64 | +Finally, we can run `CodeCompass_parser`. |
| 65 | +```bash |
| 66 | +CodeCompass_parser \ |
| 67 | + -n flask \ |
| 68 | + -i ~/parsing/flask/ \ |
| 69 | + -w ~/parsing/workdir/ \ |
| 70 | + -d "pgsql:host=localhost;port=5432;user=compass;password=pass;database=flask" \ |
| 71 | + -f \ |
| 72 | + --venvpath ~/parsing/flask/venv/ \ |
| 73 | + --label src=~/parsing/flask/ |
| 74 | +``` |
| 75 | + |
| 76 | +### CodeChecker |
| 77 | +We downloaded [CodeChecker 6.24.4](https://github.com/Ericsson/codechecker/releases/tag/v6.24.4) source code to `~/parsing/codechecker`. |
| 78 | +CodeChecker has an automated way of creating a Python virtual environment and installing dependencies - by running the `venv` target of a Makefile: |
| 79 | +```bash |
| 80 | +cd ~/parsing/codechecker/ |
| 81 | +make venv |
| 82 | +``` |
| 83 | +Next, we can run `CodeCompass_parser`. |
| 84 | +```bash |
| 85 | +CodeCompass_parser \ |
| 86 | + -n codechecker \ |
| 87 | + -i ~/parsing/codechecker/ \ |
| 88 | + -w ~/parsing/workdir/ \ |
| 89 | + -d "pgsql:host=localhost;port=5432;user=compass;password=pass;database=codechecker" \ |
| 90 | + -f \ |
| 91 | + --venvpath ~/parsing/codechecker/venv/ \ |
| 92 | + --label src=~/parsing/codechecker/ |
| 93 | +``` |
| 94 | + |
| 95 | +## Troubleshooting |
| 96 | +A few errors can occur during the parsing process, these are highlighted in color red. |
| 97 | +The stack trace is hidden by default, and can be shown using the `--stack-trace` flag. |
| 98 | + |
| 99 | +### Failed to use virtual environment |
| 100 | +This error can appear if one specifies the `--venvpath` option during parsing. |
| 101 | +The parser tried to use the specified virtual environment path, however it failed. |
| 102 | + |
| 103 | +#### Solution |
| 104 | +Double check that the Python virtual environment is correctly setup and its |
| 105 | +path is correct. |
| 106 | +If the error still persists, apply the `--stack-trace` parser option |
| 107 | +to view a more detailed stack trace of the error. |
| 108 | + |
| 109 | +### Missing module (file = path line = number) |
| 110 | +In this case, the parser tried to parse a given Python file, however it |
| 111 | +could not find a definition for a module. |
| 112 | +Commonly, the Python file imports another module and the parser cannot locate this module. |
| 113 | +If this happens, the Python file is marked *partial* indicating that |
| 114 | +a module definition was not resolved in this file. |
| 115 | +The error message displays the module name, exact file path and line number |
| 116 | +so one can further troubleshoot this problem. |
| 117 | + |
| 118 | +#### Solution |
| 119 | +Ensure that the `--venvpath` option is correctly specified and all the required |
| 120 | +dependencies are installed in that Python virtual environment. |
| 121 | +If the imported module is part of the parsed project, use the `--syspath` option |
| 122 | +and specify the directory where the module is located in. |
| 123 | + |
0 commit comments