Tracking Jobs Progress

Hadoop Jobs:

All the Jobs status can be seen in  YARN Web UI. 

Similar to the Web UI for Hadoop Distributed File System, Hadoop also provided YARN Web UI for Yarn Resource manager.

YARN Web UI:

Similar to the Web UI for Hadoop Distributed File System, Hadoop also provided YARN Web UI for Yarn Resource manager.

Enter the URL as http://localhost:8088/ in web browser and to point to port number 8088 on localhost host name in pseudo-distributed mode. In Fully distributed mode, replace the ‘localhost’ with actual host name of machine on cluster.For example, let us run one Map Reduce job from terminal and browse through each section present on YARN Web UI.As shown in above screen, if jobs are running for longer time, job’s progress can be monitored from the URL given as shown above for our example.

mapreduce.Job: The url to track the job: http://siva-desktop:8088/proxy/application_1396885203337_0003/

During the job run, by opening the above URL in web browser, we can monitor the progress of the job as shown below.Once the job is completed successfully,This job will added to Finished Applications section in the cluster home page.We can observe the state of the jobs as FINISHED and Final Status as SUCCEEDED for successfully completed jobs. If jobs are during their run time, these will be listed under KILLED section and any failed jobs will be listed under FAILED section. We can monitor the status of the scheduler from scheduler section in Cluster Menu.We can also trace the logs of the jobs through Tools –> Local Logs path. For detailed information of tracing of logs please refer the post Tracing of Logs.

For complete info please follow http://hadooptutorial.info/yarn-web-ui/

Spark Jobs:

Access spark jobs in Spark UI, this includes all spark jobs like streaming etc.

Spark History only shows job after is finished.

To access Spark UI while your job is running on YARN, got to YARN UI (usually port :8088) and click on Application Master link of your job.