Lot of fixes, improvements etc... (#66)

* feat: fork for pypi * fix: handle json decode errors * fix: error not defined * feat: jmiroosh feature Signed-off-by: QuentinN42 <[email protected]> * feat: duplicate query regex Signed-off-by: QuentinN42 <[email protected]> * fix: dockerhub token * feat: proxy Signed-off-by: QuentinN42 <[email protected]> * feat: ignore error Signed-off-by: QuentinN42 <[email protected]> * feat: add max retry param Signed-off-by: QuentinN42 <[email protected]> * fix: input object heuristics * fix: typing context * fix: input object for argmuments * fix: double regex * fix: new field regex * fix: did you mean * fix: did you mean * fix: tests * fix: tests * fix: typename regex * feat: skip regex * feat: fix regex * fix: was not provided regex * feat: dot in field * feat: add backoff Signed-off-by: QuentinN42 <[email protected]> * feat: multiple did you mean * fix: better regexps * feat: move to triple quotes * feat: move to triple quotes * feat: move to triple quotes * feat: move to triple quotes * feat: profiles Signed-off-by: QuentinN42 <[email protected]> * feat: field regexes above * feat: triple quote * feat: compile arg regexes * feat: compile typeref regexes * --wip-- [skip ci] * --wip-- [skip ci] * fix: final regexes * fix: remove print * docs: add help message to the doc Signed-off-by: QuentinN42 <[email protected]> * feat: V2.3 Signed-off-by: QuentinN42 <[email protected]> * feat: @jamboro fixes issue 11 Signed-off-by: QuentinN42 <[email protected]> * feat: retro compat Signed-off-by: QuentinN42 <[email protected]> * feat: update python Signed-off-by: QuentinN42 <[email protected]> * feat: lookup in stderr Signed-off-by: QuentinN42 <[email protected]> * feat: ignore aiohttp warns Signed-off-by: QuentinN42 <[email protected]> * chore: v2.3.1 Signed-off-by: QuentinN42 <[email protected]> * fix: better did you mean for type field * feat: did you mean skip field regex * fix: clairvoyance * fix: inline fragments regexps * fix: inline fragments regexps * feat: skip ever useless error message * fix: print context * fix: input object Input * fix: naming * feat: v2.4.0 Signed-off-by: QuentinN42 <[email protected]> * fix: unknown-message-log-error * fix: input flagh * fix: reccursive schemas loops * chore: remove unused import * chore: v2.4.1 Signed-off-by: QuentinN42 <[email protected]> * feat: disable ssl verification * chore: update readme * fix: don't crash when no field suggestion * docs: update readme Signed-off-by: QuentinN42 <[email protected]> * feat: some rich progress bars Signed-off-by: QuentinN42 <[email protected]> * fix: use pathlib to work with any OS Signed-off-by: QuentinN42 <[email protected]> * fix: updated an uniform tasks Signed-off-by: QuentinN42 <[email protected]> * fix: optional progress Signed-off-by: QuentinN42 <[email protected]> * feat: v2.5.0 Signed-off-by: QuentinN42 <[email protected]> * refactor: allow over python 3.11 * chore: typo Signed-off-by: QuentinN42 <[email protected]> * Revert "fix: dockerhub token" This reverts commit 200f43b. * feat: update debugger config Signed-off-by: QuentinN42 <[email protected]> * feat: revert one line Signed-off-by: QuentinN42 <[email protected]> * fix: call super in the __init__ Signed-off-by: QuentinN42 <[email protected]> * fix: drop warluses Signed-off-by: QuentinN42 <[email protected]> * docs: move comments Signed-off-by: QuentinN42 <[email protected]> * feat: fix typing Signed-off-by: QuentinN42 <[email protected]> * feat: restore main project name Signed-off-by: QuentinN42 <[email protected]> * fix: Closing angle bracket Signed-off-by: QuentinN42 <[email protected]> --------- Signed-off-by: QuentinN42 <[email protected]> Co-authored-by: Antoine Carossio <[email protected]> Co-authored-by: c3b5aw <[email protected]> Co-authored-by: nohehf <[email protected]> Co-authored-by: nohehf <[email protected]>
nikitastupin · Mar 18, 2023 · de71a0f · de71a0f
1 parent 66efa8f
commit de71a0f
Show file tree

Hide file tree

Showing 21 changed files with 1,516 additions and 1,240 deletions.
diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
@@ -28,7 +28,9 @@ jobs:
 
     steps:
     - uses: actions/checkout@v3
-    - uses: actions/setup-python@v2
+    - uses: actions/setup-python@v4
+      with:
+        python-version: '3.10'
     - name: Install dependencies
       run: |
         pip install poetry
@@ -47,7 +49,9 @@ jobs:
 
     steps:
     - uses: actions/checkout@v3
-    - uses: actions/setup-python@v2
+    - uses: actions/setup-python@v4
+      with:
+        python-version: '3.10'
     - name: Install dependencies
       run: |
         pip install poetry
@@ -89,7 +93,9 @@ jobs:
     if: false
     steps:
     - uses: actions/checkout@v3
-    - uses: actions/setup-python@v2
+    - uses: actions/setup-python@v4
+      with:
+        python-version: '3.10'
     - name: Install dependencies
       run: |
         pip install poetry

diff --git a/.vscode/launch.json b/.vscode/launch.json
@@ -11,13 +11,42 @@
             "cwd": "${workspaceFolder}",
             "module": "clairvoyance",
             "args": [
-                "-o", "/tmp/t.json",
-                "-w", "tests/data/wordlist-for-apollo-server.txt",
-                "http://localhost:4000", "--verbose"
+                "-o",
+                "/tmp/t.json",
+                "-w",
+                "tests/data/wordlist-for-apollo-server.txt",
+                "http://localhost:4000",
+                "--verbose"
+            ],
+            "console": "integratedTerminal",
+            "justMyCode": true
+        },
+        {
+            "name": "Clairvoyance on localhost:4000",
+            "type": "python",
+            "request": "launch",
+            "cwd": "${workspaceFolder}",
+            "module": "clairvoyance",
+            "args": [
+                "-o",
+                "target/t.json",
+                "http://localhost:4000",
+                "--verbose"
+            ],
+            "console": "integratedTerminal",
+            "justMyCode": true
+        },
+        {
+            "name": "Clairvoyance on rick and morty GQL application",
+            "type": "python",
+            "request": "launch",
+            "cwd": "${workspaceFolder}",
+            "module": "clairvoyance",
+            "args": [
+                "https://rickandmortyapi.com/graphql"
             ],
             "console": "integratedTerminal",
             "justMyCode": true
         }
-
     ]
 }
diff --git a/README.md b/README.md
@@ -1,60 +1,48 @@
-# clairvoyance
+# Clairvoyance
 
-Some GraphQL APIs have disabled introspection. For example, [Apollo Server disables introspection automatically if the `NODE_ENV` environment variable is set to `production`](https://www.apollographql.com/docs/tutorial/schema/#explore-your-schema).
-
-Clairvoyance allows us to get GraphQL API schema when introspection is disabled. It produces schema in JSON format suitable for other tools like [GraphQL Voyager](https://github.com/APIs-guru/graphql-voyager), [InQL](https://github.com/doyensec/inql) or [graphql-path-enum](https://gitlab.com/dee-see/graphql-path-enum).
+Obtain GraphQL API Schema even if the introspection is disabled.
 
-## Acknowledgments
+[![PyPI](https://img.shields.io/pypi/v/clairvoyance)](https://pypi.org/project/clairvoyance/)
+[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/clairvoyance)](https://pypi.org/project/clairvoyance/)
+[![PyPI - Downloads](https://img.shields.io/pypi/dm/clairvoyance)](https://pypi.org/project/clairvoyance/)
+[![GitHub](https://img.shields.io/github/license/nikitastupin/clairvoyance)](https://github.com/nikitastupin/clairvoyance/blob/main/LICENSE)
 
-Thanks to [Swan](https://github.com/c3b5aw) from [Escape-Technologies](https://github.com/Escape-Technologies) for 2.0 version.
+## Introduction
 
-## Usage
+Some GraphQL APIs have disabled introspection. For example, [Apollo Server disables introspection automatically if the `NODE_ENV` environment variable is set to `production`](https://www.apollographql.com/docs/tutorial/schema/#explore-your-schema).
 
-You may find more details on how the tool works in the second half of the [GraphQL APIs from bug hunter's perspective by Nikita Stupin](https://youtu.be/nPB8o0cSnvM) talk.
+Clairvoyance allows us to get GraphQL API schema when introspection is disabled. It produces schema in JSON format suitable for other tools like [GraphQL Voyager](https://github.com/APIs-guru/graphql-voyager), [InQL](https://github.com/doyensec/inql) or [graphql-path-enum](https://gitlab.com/dee-see/graphql-path-enum).
 
-### From PyPI
+## Contributors
 
-```bash
-pip install clairvoyance
-```
+Thanks to the [contributors](#contributors) for their work.
 
-### From Python interpreter
+- [nikitastupin](https://github.com/nikitastupin)
+- [Escape](https://escape.tech) team :
+  - [iCarossio](https://github.com/iCarossio)
+  - [Swan](https://github.com/c3b5aw)
+  - [QuentinN42](https://github.com/QuentinN42)
+  - [Nohehf](https://github.com/Nohehf)
+- [i-tsaturov](https://github.com/i-tsaturov)
+- [EONRaider](https://github.com/EONRaider)
+- [noraj](https://github.com/noraj)
+- [belane](https://github.com/belane)
 
-```bash
-git clone https://github.com/nikitastupin/clairvoyance.git
-cd clairvoyance
-pip install poetry
-poetry config virtualenvs.in-project true
-poetry install --no-dev
-source .venv/bin/activate
-```
+## Getting started
 
 ```bash
-python3 -m clairvoyance --help
-```
-
-```bash
-python3 -m clairvoyance -o /path/to/schema.json https://swapi-graphql.netlify.app/.netlify/functions/index
+pip install clairvoyance
+clairvoyance https://rickandmortyapi.com/graphql -o schema.json
+# should take about 2 minute
 ```
 
-### From Docker Image
+## Docker Image
 
 ```bash
 docker run --rm nikitastupin/clairvoyance --help
 ```
 
-```bash
-# Assuming the wordlist.txt file is found in $PWD
-docker run --rm -v $(pwd):/tmp/ nikitastupin/clairvoyance -vv -o /tmp/schema.json -w /tmp/wordlist.txt https://swapi-graphql.netlify.app/.netlify/functions/index
-```
-
-### From BlackArch Linux
-
-> NOTE: this distribution is supported by a third-party (i.e. not by the mainainters of clairvoyance)
-
-```bash
-pacman -S clairvoyance
-```
+## Advanced Usage
 
 ### Which wordlist should I use?
 
@@ -80,3 +68,7 @@ In case of question or issue with clairvoyance please refer to [wiki](https://gi
 ## Contributing
 
 Pull requests are welcome! For major changes, please open an issue first to discuss what you would like to change. For more information about tests, internal project structure and so on refer to [Development](https://github.com/nikitastupin/clairvoyance/wiki/Development) wiki page.
+
+## Documentation
+
+- You may find more details on how the tool works in the second half of the [GraphQL APIs from bug hunter's perspective by Nikita Stupin](https://youtu.be/nPB8o0cSnvM) talk.
diff --git a/clairvoyance/cli.py b/clairvoyance/cli.py
@@ -1,8 +1,9 @@
 import asyncio
 import json
 import logging
+import re
 import sys
-import os
+from pathlib import Path
 from typing import Dict, List, Optional
 
 from clairvoyance import graphql, oracle
@@ -18,6 +19,10 @@ def setup_context(
     logger: logging.Logger,
     headers: Optional[Dict[str, str]] = None,
     concurrent_requests: Optional[int] = None,
+    proxy: Optional[str] = None,
+    max_retries: Optional[int] = None,
+    backoff: Optional[int] = None,
+    disable_ssl_verify: Optional[bool] = None,
 ) -> None:
     """Initialize objects and freeze them into the context."""
 
@@ -26,12 +31,16 @@ def setup_context(
         url,
         headers=headers,
         concurrent_requests=concurrent_requests,
+        proxy=proxy,
+        max_retries=max_retries,
+        backoff=backoff,
+        disable_ssl_verify=disable_ssl_verify,
     )
     logger_ctx.set(logger)
 
 
 def load_default_wordlist() -> List[str]:
-    wl = os.path.join(os.path.dirname(__file__), 'wordlist.txt')
+    wl = Path(__file__).parent / 'wordlist.txt'
     with open(wl, 'r', encoding='utf-8') as f:
         return [w.strip() for w in f.readlines() if w.strip()]
 
@@ -45,6 +54,10 @@ async def blind_introspection(
     input_document: Optional[str] = None,
     input_schema_path: Optional[str] = None,
     output_path: Optional[str] = None,
+    proxy: Optional[str] = None,
+    max_retries: Optional[int] = None,
+    backoff: Optional[int] = None,
+    disable_ssl_verify: Optional[bool] = None,
 ) -> str:
     wordlist = wordlist or load_default_wordlist()
     assert wordlist, 'No wordlist provided'
@@ -54,6 +67,10 @@ async def blind_introspection(
         logger=logger,
         headers=headers,
         concurrent_requests=concurrent_requests,
+        proxy=proxy,
+        max_retries=max_retries,
+        backoff=backoff,
+        disable_ssl_verify=disable_ssl_verify,
     )
 
     logger.info(f'Starting blind introspection on {url}...')
@@ -65,7 +82,10 @@ async def blind_introspection(
 
     input_document = input_document or 'query { FUZZ }'
     ignored = set(e.value for e in GraphQLPrimitive)
+    iterations = 1
     while True:
+        logger.info(f'Iteration {iterations}')
+        iterations += 1
         schema = await oracle.clairvoyance(
             wordlist,
             input_document=input_document,
@@ -107,6 +127,17 @@ def cli(argv: Optional[List[str]] = None) -> None:
     wordlist = []
     if args.wordlist:
         wordlist = [w.strip() for w in args.wordlist.readlines() if w.strip()]
+        # de-dupe the wordlist.
+        wordlist = list(set(wordlist))
+
+    # remove wordlist items that don't conform to graphQL regex github-issue #11
+    if args.validate:
+        wordlist_parsed = [w for w in wordlist if re.match(r'[_A-Za-z][_0-9A-Za-z]*', w)]
+        logging.info(
+            f'Removed {len(wordlist) - len(wordlist_parsed)} items from Wordlist, to conform to name regex. '
+            f'https://spec.graphql.org/June2018/#sec-Names'
+        )
+        wordlist = wordlist_parsed
 
     asyncio.run(
         blind_introspection(
@@ -118,5 +149,9 @@ def cli(argv: Optional[List[str]] = None) -> None:
             input_schema_path=args.input_schema,
             output_path=args.output,
             wordlist=wordlist,
+            proxy=args.proxy,
+            max_retries=args.max_retries,
+            backoff=args.backoff,
+            disable_ssl_verify=args.no_ssl,
         )
     )
diff --git a/clairvoyance/client.py b/clairvoyance/client.py
@@ -1,4 +1,5 @@
 import asyncio
+import json
 from typing import Dict, Optional
 
 import aiohttp
@@ -8,20 +9,26 @@
 
 
 class Client(IClient):
-
     def __init__(
         self,
         url: str,
         max_retries: Optional[int] = None,
         headers: Optional[Dict[str, str]] = None,
         concurrent_requests: Optional[int] = None,
+        proxy: Optional[str] = None,
+        backoff: Optional[int] = None,
+        disable_ssl_verify: Optional[bool] = None,
     ) -> None:
         self._url = url
         self._session = None
 
         self._headers = headers or {}
         self._max_retries = max_retries or 3
         self._semaphore = asyncio.Semaphore(concurrent_requests or 50)
+        self.proxy = proxy
+        self.backoff = backoff
+        self._backoff_semaphore = asyncio.Lock()
+        self.disable_ssl_verify = disable_ssl_verify or False
 
         client_ctx.set(self)
 
@@ -37,7 +44,8 @@ async def post(
 
         async with self._semaphore:
             if not self._session:
-                self._session = aiohttp.ClientSession(headers=self._headers)
+                connector = aiohttp.TCPConnector(verify_ssl=(not self.disable_ssl_verify))
+                self._session = aiohttp.ClientSession(headers=self._headers, connector=connector)
 
             # Translate an existing document into a GraphQL request.
             gql_document = {'query': document} if document else None
@@ -46,6 +54,7 @@ async def post(
                 response = await self._session.post(
                     self._url,
                     json=gql_document,
+                    proxy=self.proxy,
                 )
 
                 if response.status >= 500:
@@ -57,9 +66,17 @@ async def post(
             except (
                 aiohttp.client_exceptions.ClientConnectionError,
                 aiohttp.client_exceptions.ClientPayloadError,
+                asyncio.TimeoutError,
+                json.decoder.JSONDecodeError,
             ) as e:
                 log().warning(f'Error posting to {self._url}: {e}')
 
+            if self.backoff:
+                async with self._backoff_semaphore:
+                    delay = 0.5 * self.backoff**retries
+                    log().debug(f'Waiting for backoff {delay} seconds.')
+                    await asyncio.sleep(delay)
+
         return await self.post(document, retries + 1)
 
     async def close(self) -> None:

diff --git a/clairvoyance/config.py b/clairvoyance/config.py
@@ -4,8 +4,8 @@
 
 # pylint: disable=too-few-public-methods
 class Config(IConfig):
-
     def __init__(self) -> None:
+        super().__init__()
         self._bucket_size: int = 64
 
         config_ctx.set(self)
diff --git a/clairvoyance/entities/interfaces.py b/clairvoyance/entities/interfaces.py
@@ -8,7 +8,6 @@
 
 
 class IConfig(ABC):
-
     _bucket_size: int
 
     @property

diff --git a/clairvoyance/entities/meta.py b/clairvoyance/entities/meta.py
@@ -3,7 +3,6 @@
 
 
 class MetaEnum(EnumMeta):
-
     """Meta class for Enum."""
 
     # pylint: disable=no-value-for-parameter

diff --git a/clairvoyance/entities/oracle.py b/clairvoyance/entities/oracle.py
@@ -0,0 +1,9 @@
+"""Oracle definitions."""
+
+from enum import Enum
+
+
+class FuzzingContext(str, Enum):
+    """Contexts."""
+    ARGUMENT = 'InputValue'
+    FIELD = 'Field'
diff --git a/clairvoyance/entities/primitives.py b/clairvoyance/entities/primitives.py
@@ -7,7 +7,6 @@
 
 @unique
 class GraphQLPrimitive(str, Enum, metaclass=MetaEnum):
-
     """The default GraphQL Scalar primitives.
 
     ref: https://spec.graphql.org/draft/#sec-Input-Values
@@ -22,7 +21,6 @@ class GraphQLPrimitive(str, Enum, metaclass=MetaEnum):
 
 @unique
 class GraphQLKind(str, Enum, metaclass=MetaEnum):
-
     """The default GraphQL kinds.
 
     ref: https://spec.graphql.org/draft/#sec-Types
Original file line number	Diff line number	Diff line change
Expand Up		@@ -8,7 +8,6 @@


		class IConfig(ABC):

		_bucket_size: int

		@property
Expand Down
Original file line number	Diff line number	Diff line change
Expand Up		@@ -3,7 +3,6 @@


		class MetaEnum(EnumMeta):

		"""Meta class for Enum."""

		# pylint: disable=no-value-for-parameter
Expand Down