Skip to content

Commit 6b2ec83

Browse files
author
Barnabás Domozi
committed
Added Python plugin documentation.
1 parent 02e4981 commit 6b2ec83

File tree

1 file changed

+123
-0
lines changed

1 file changed

+123
-0
lines changed

doc/pythonplugin.md

Lines changed: 123 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,123 @@
1+
# Python Plugin
2+
3+
## Parsing Python projects
4+
Python projects can be parsed by using the `CodeCompass_parser` executable.
5+
See its usage [in a seperate document](/doc/usage.md).
6+
7+
## Python specific parser flags
8+
9+
### Python dependencies
10+
Large Python projects usually have multiple Python package dependencies.
11+
Although a given project can be parsed without installing any of its dependencies, it is strongly recommended
12+
that the required modules are installed in order to achieve a complete parsing.
13+
To install a project's dependencies, create a [Python virtual environment](https://docs.python.org/3/library/venv.html)
14+
and install the necessary packages.
15+
When parsing a project, specify the virtual environment path so the parser can successfully resolve the dependencies:
16+
```
17+
--venvpath <path to virtual environment>
18+
```
19+
20+
### Type hints
21+
The parser can try to determine Python type hints for variables, expressions and functions.
22+
It can work out type hints such as `Iterable[int]` or `Union[int, str]`.
23+
However, this process can be extremely slow, especially for functions, thus it is disabled by default.
24+
It can be enabled using the `--type-hint` flag.
25+
26+
### Python submodules
27+
Large Python projects can have internal submodules and the parser tries to locate them automatically.
28+
Specifically, it looks for `__init__.py` files and considers those folders modules.
29+
This process is called submodule discovery and can be disabled using the `--disable-submodule-discovery` flag.
30+
31+
You can also add submodules manually by adding those specific paths to the parser's syspath:
32+
```
33+
--syspath <path>
34+
```
35+
For more information, see the [Python syspath docs](https://docs.python.org/3/library/sys.html#sys.path).
36+
37+
### File references
38+
By default, the parser works out references by looking for definitions only - if nodes share the same definition
39+
they are considered references.
40+
However, this method sometimes misses a few references (e.g. local variables in a function).
41+
To extend search for references in a file context, apply the `--file-refs` flag.
42+
Note that using this option can potentially extend the total parsing time.
43+
44+
## Examples of parsing Python projects
45+
46+
### Flask
47+
We downloaded [flask 3.1.0](https://github.com/pallets/flask/releases/tag/3.1.0) source code to `~/parsing/flask/`.
48+
The first step is to create a Python virtual environment and install flask's dependencies.
49+
Create a Python virtual environment and activate it:
50+
```bash
51+
cd ~/parsing/flask/
52+
python3 -m venv venv
53+
source venv/bin/activate
54+
```
55+
Next, we install the required dependencies listed in `pyproject.toml`.
56+
```bash
57+
pip install .
58+
```
59+
Further dependencies include development packages listed in `requirements/dev.txt`.
60+
These can be also installed using `pip`.
61+
```bash
62+
pip install -r requirements/dev.txt
63+
```
64+
Finally, we can run `CodeCompass_parser`.
65+
```bash
66+
CodeCompass_parser \
67+
-n flask \
68+
-i ~/parsing/flask/ \
69+
-w ~/parsing/workdir/ \
70+
-d "pgsql:host=localhost;port=5432;user=compass;password=pass;database=flask" \
71+
-f \
72+
--venvpath ~/parsing/flask/venv/ \
73+
--label src=~/parsing/flask/
74+
```
75+
76+
### CodeChecker
77+
We downloaded [CodeChecker 6.24.4](https://github.com/Ericsson/codechecker/releases/tag/v6.24.4) source code to `~/parsing/codechecker`.
78+
CodeChecker has an automated way of creating a Python virtual environment and installing dependencies - by running the `venv` target of a Makefile:
79+
```bash
80+
cd ~/parsing/codechecker/
81+
make venv
82+
```
83+
Next, we can run `CodeCompass_parser`.
84+
```bash
85+
CodeCompass_parser \
86+
-n codechecker \
87+
-i ~/parsing/codechecker/ \
88+
-w ~/parsing/workdir/ \
89+
-d "pgsql:host=localhost;port=5432;user=compass;password=pass;database=codechecker" \
90+
-f \
91+
--venvpath ~/parsing/codechecker/venv/ \
92+
--label src=~/parsing/codechecker/
93+
```
94+
95+
## Troubleshooting
96+
A few errors can occur during the parsing process, these are highlighted in color red.
97+
The stack trace is hidden by default, and can be shown using the `--stack-trace` flag.
98+
99+
### Failed to use virtual environment
100+
This error can appear if one specifies the `--venvpath` option during parsing.
101+
The parser tried to use the specified virtual environment path, however it failed.
102+
103+
#### Solution
104+
Double check that the Python virtual environment is correctly setup and its
105+
path is correct.
106+
If the error still persists, apply the `--stack-trace` parser option
107+
to view a more detailed stack trace of the error.
108+
109+
### Missing module (file = path line = number)
110+
In this case, the parser tried to parse a given Python file, however it
111+
could not find a definition for a module.
112+
Commonly, the Python file imports another module and the parser cannot locate this module.
113+
If this happens, the Python file is marked *partial* indicating that
114+
a module definition was not resolved in this file.
115+
The error message displays the module name, exact file path and line number
116+
so one can further troubleshoot this problem.
117+
118+
#### Solution
119+
Ensure that the `--venvpath` option is correctly specified and all the required
120+
dependencies are installed in that Python virtual environment.
121+
If the imported module is part of the parsed project, use the `--syspath` option
122+
and specify the directory where the module is located in.
123+

0 commit comments

Comments
 (0)