Apache Spark based price analysis (Part of Big Data Analysis project)
From websites like - OLX, PurpleYo
- Technology - Python
- Library - Scrapy, Beautiful Soup
Apache Spark
- Spark Job Web UI
While a spark job is running, UI can be found at - http://<socket-address>
For e.g. http://192.168.1.4:4040
However, once the job is completed, one is unable to view the UI.
- History Server
Run the shell script to start the history server. It accesses the /tmp/spark-events for Event Logs.
/usr/local/spark/sbin/start-history-server.sh
Every spark job executed / failed can be found at http://172.20.10.13:18080/