Sunday, July 2, 2017

Production Development model

In General,

To provide access to Hadoop cluster, need to use SSH connection.

Need to create SSH to any one of the node in the cluster(usually one of the data node), but in Microsoft Azure SSH would be created for namenode. This becomes edge node. Only few machines will be configured with SSH connection.

SSH connection will give a host name to connect via Putty. This is not same as the host name of the machine.
Eg: HadoopHortonwks-ssh.azurehdinsight.net

Via Ambari/Cloudera manager any number of users can be created. All these users will access the cluster via this client machine also called as edge node.

For development:


Only Terminal access would be provided.
An IP of a datanode along with Username and password would be provided. Using Putty need to connect and get terminal access.

Any number of terminals can be launched and Using this terminal need to work on Hive, Spark, Pig, shell, HBase etc.

No Cloudera Manager.

Eg:
Putty:
10.188.193.152
User: hadoop
Password: W3b!6xfMd

To create JAR file need to install Ecllipse in Windows machine and write code. Need to transfer these JARS using Winscp tool.
For the combination of putty and filesystem viewer, use MobaXterm 8.4 version. in this putty connections can be made as well as the edgenodes filesystem.

For deployment:


HUE Access would be provided to create and schedule Jobs. A URL would be provided and this will launch HUE UI as like in Standalone mode.

Eg:

HUE Access:

http://10.188.193.152:8888/accounts/login/?next=/

user:admin
password:admin

No comments:

Post a Comment