To run it locally, do the following:
- Install and set up minikube v0.22.3 (google for more details)
- run
minikube start --kubernetes-version v1.7.5 --cpus 7 --memory 12360 --disk-size 100g
- run
make sidecars
(to set up dependencies) - run
make publish
(to build and publish all the docker images needed) - run
make submit JOB=<job_name>
(to submit the spark job twitter_follower_jdbc_logging or reliable_s3_parquet_logging to local minikube cluster) - run
kubectl get pods
to find the spark driver pod - run
kubectl logs -f <driver-pod-name>
to check the driver log - run
kubectl exec -it cockroachdb-0 bash
to log into cockroachdb - run
./cockroach sql --insecure --host=cockroachdb-public -e '<any_sql_query>'
query cockroachDB - have fun
- run
kubectl delete pod <driver-pod-name>
to stop the spark job - try re-installing vm and minikube if running into weird kubernetes related issues