Skip to content

Commit c5b8f57

Browse files
authored
docs(metadata-models-custom): add example script to show producing cu… (datahub-project#4681)
1 parent 9ecb785 commit c5b8f57

File tree

2 files changed

+57
-0
lines changed

2 files changed

+57
-0
lines changed

metadata-models-custom/README.md

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -129,6 +129,15 @@ results in
129129
Update succeeded with status 200
130130
```
131131

132+
The `scripts/insert_custom_aspect.py` script shows you how to accomplish the same using the Python SDK. Note that we are just using a raw dictionary here to represent the `dq_rule` aspect and not a strongly-typed class.
133+
```console
134+
cd scripts
135+
python3 insert_custom_aspect.py
136+
```
137+
results in
138+
```console
139+
Successfully wrote to DataHub
140+
```
132141

133142
## Advanced Guide
134143

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
import json
2+
3+
from datahub.emitter.rest_emitter import DatahubRestEmitter
4+
from datahub.metadata.schema_classes import (
5+
ChangeTypeClass,
6+
GenericAspectClass,
7+
MetadataChangeProposalClass,
8+
)
9+
10+
dq_aspect = {
11+
"rules": [
12+
{
13+
"field": "my_event_data",
14+
"isFieldLevel": False,
15+
"type": "isNull",
16+
"checkDefinition": "n/a",
17+
"url": "https://github.com/datahub-project/datahub/blob/master/checks/nonNull.sql",
18+
},
19+
{
20+
"field": "timestamp",
21+
"isFieldLevel": True,
22+
"type": "increasing",
23+
"checkDefinition": "n/a",
24+
"url": "https://github.com/datahub-project/datahub/blob/master/checks/increasing.sql",
25+
},
26+
]
27+
}
28+
29+
emitter: DatahubRestEmitter = DatahubRestEmitter(gms_server="http://localhost:8080")
30+
31+
dataset_urn = "urn:li:dataset:(urn:li:dataPlatform:hive,logging_events,PROD)"
32+
mcp_raw: MetadataChangeProposalClass = MetadataChangeProposalClass(
33+
entityType="dataset",
34+
entityUrn=dataset_urn,
35+
changeType=ChangeTypeClass.UPSERT,
36+
aspectName="customDataQualityRules",
37+
aspect=GenericAspectClass(
38+
contentType="application/json",
39+
value=json.dumps(dq_aspect).encode("utf-8"),
40+
),
41+
)
42+
43+
try:
44+
emitter.emit(mcp_raw)
45+
print("Successfully wrote to DataHub")
46+
except Exception as e:
47+
print("Failed to write to DataHub")
48+
raise e

0 commit comments

Comments
 (0)