-> Avoid shuffling in spark using spark bucketing
Overwrite Partition. http://www.ericlin.me/2015/05/hive-insert-overwrite-does-not-remove-existing-data/
-> Futures in scala and thread pool in scala
->salting
->Ratio of data processing to CPU Cores and Memory
-> Which is better fo huge data(Peta Bytes) processing Spark / Hive
-> Skewing in bigdata, while joining 2 tables NULL values going to same partition.
-> Spark Execution Model, Executors, Stage etc.
1. Spark Streaming in Python and Scala
2. Create online dashboards in Python
3. Convert CSV to Avro in NIFI
Remote Debug Java
Windowing concept
Broadcast variables and Accumulator
4. In a Dataframe convert all the values in a column from string to Integer.
4. JDBC Connection in Spark
5. save dataframe to s3
6. difference bttween ORC and Parquet interms of usecases
Parquet is best for read intensive operations.
Lamda functions and different kinds of functions in Scala
https://alvinalexander.com/scala/passing-function-to-function-argument-anonymous
http://michaelpnash.github.io/scala-lambdas/
7. ORC is best which we would like to perform OLTP operations
8. Indexing in ORC
9. HBase usecases and cassandra usecases. Which cases HBase is preferable and cases in which Cassandra is preferable.
https://www.youtube.com/watch?v=nIPPTn4IC2Y&t=20s
https://www.youtube.com/watch?v=zCOFC8IVn3Q
cassandra works well with object storage. data stored in the form of object.
HBASE is block storage
Cassandra can be confugured outside Hadoop cluster
10. passing function as input to another function in scala
function as member variable
11. Difference between class and case class. Whats the need of using caseclass
case class = class + companion object.
In a class, to access protected variables Get and Set methods are to be implemented.But we don't want to access member variables using get and set functions. In this case companion object comes into picture which can directly access class member variables.
12. Can 2 consumer groups access complete data in a kafka topic independently
Yes.
13. If there are 2 consumers in a consumer group then can both these access all data in the kafka topic
14. DAG
RDD Lineage
Impala vs SparkSQL
Nifi vs flume
Java vs Scala
can 2 spark contexts run
python vs scala
In Python accumulators are not supported
Online Dashboards in Python for kafka-spark streaming:
https://www.rittmanmead.com/blog/2017/01/getting-started-with-spark-streaming-with-python-and-kafka/
Exercise 1:
1. Fetch data from http://samples.openweathermap.org/data/2.5/weather?q=London&appid=14e8adf775bccd703001ef35eefdfe3c in NIFI and GetJSON FILE.
2. Push JSON file to Kafka.
3. Get the data in Spark streaming and create a Hive table.
Exercise 2:
sqoop
=======
sqoop2 -> kafka
sqoop -> Mysql
sqoop -> hdfs
sqoop -> hive
spark core (All inputs should give in runtime)
read different files data
1) csv
2) avro
3) json
wordcount
staticwordcount
Dataframes
perform sql querys on Dataframes (spark SQL )
Spark streaming
Source dest
kafka => Spark streaming => hdfs
kafka => Spark streaming => store in hive internal table
kafka => Spark streaming => kafka
Oozie
step-1
exe wordcount program
step-2
Put this( exe wordcount program) in shell script and run from local
step-3
put that shell script in Workflow.xml and run
1) workflow
2) coord
3) Bundil jobs
Hive
====
create internal AND external TABLES
partitons
bucketing
create static and dynamic partitons
2. RegEx
3. Apache NIFI
4. SERDEPROPERTIES in HIVE
5. Self-Join with SubQueries
6. Shell Scripting
7. Difference between ORC and Parquet
8. Go through https://www.youtube.com/watch?v=eD3LkpPQIgM for deduplication implementation in MLlib
9. How to identify the failure in spark Job?
10. What happens if there is a failure in connection between Kafka and consumer for a while
What happens next?
11 Why can't spark streaming implement consumer API's
12. What is the JAVA version used and what are its new features?
13. HBase and Cassandra Architecture
14. Kerberos Security
15. Shuffling in case of colease and re-partition
16. Schema Evolution in AVRO
17. Data has date and time values in the format dd/mm/yyyy:00:00:00. Create Hive partitioned tables partitioned by Date.Need to use date in-built functions and extract date.
18. Download traffic data and find out the location that has highest traffic
19. Table design, performance, data modelling, denormalizing
20. Spark and Spark streaming job submission.
21. Copy data from s3 to cluster.
22. Create Hive table on s3 location
AWS Access Key ID [None]: AKIAINIHCWDZMRWJEI3A
AWS Secret Access Key [None]: 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
23. hive JDBC Connection via JAVA
-> Futures in scala and thread pool in scala
->salting
->Ratio of data processing to CPU Cores and Memory
-> Which is better fo huge data(Peta Bytes) processing Spark / Hive
-> Skewing in bigdata, while joining 2 tables NULL values going to same partition.
-> Spark Execution Model, Executors, Stage etc.
1. Spark Streaming in Python and Scala
2. Create online dashboards in Python
3. Convert CSV to Avro in NIFI
Remote Debug Java
Windowing concept
Broadcast variables and Accumulator
4. In a Dataframe convert all the values in a column from string to Integer.
4. JDBC Connection in Spark
5. save dataframe to s3
6. difference bttween ORC and Parquet interms of usecases
Parquet is best for read intensive operations.
Lamda functions and different kinds of functions in Scala
https://alvinalexander.com/scala/passing-function-to-function-argument-anonymous
http://michaelpnash.github.io/scala-lambdas/
7. ORC is best which we would like to perform OLTP operations
8. Indexing in ORC
9. HBase usecases and cassandra usecases. Which cases HBase is preferable and cases in which Cassandra is preferable.
https://www.youtube.com/watch?v=nIPPTn4IC2Y&t=20s
https://www.youtube.com/watch?v=zCOFC8IVn3Q
cassandra works well with object storage. data stored in the form of object.
HBASE is block storage
Cassandra can be confugured outside Hadoop cluster
10. passing function as input to another function in scala
function as member variable
11. Difference between class and case class. Whats the need of using caseclass
case class = class + companion object.
In a class, to access protected variables Get and Set methods are to be implemented.But we don't want to access member variables using get and set functions. In this case companion object comes into picture which can directly access class member variables.
12. Can 2 consumer groups access complete data in a kafka topic independently
Yes.
13. If there are 2 consumers in a consumer group then can both these access all data in the kafka topic
14. DAG
RDD Lineage
Impala vs SparkSQL
Nifi vs flume
Java vs Scala
can 2 spark contexts run
python vs scala
In Python accumulators are not supported
Online Dashboards in Python for kafka-spark streaming:
https://www.rittmanmead.com/blog/2017/01/getting-started-with-spark-streaming-with-python-and-kafka/
Exercise 1:
1. Fetch data from http://samples.openweathermap.org/data/2.5/weather?q=London&appid=14e8adf775bccd703001ef35eefdfe3c in NIFI and GetJSON FILE.
2. Push JSON file to Kafka.
3. Get the data in Spark streaming and create a Hive table.
Exercise 2:
sqoop
=======
sqoop2 -> kafka
sqoop -> Mysql
sqoop -> hdfs
sqoop -> hive
spark core (All inputs should give in runtime)
read different files data
1) csv
2) avro
3) json
wordcount
staticwordcount
Dataframes
perform sql querys on Dataframes (spark SQL )
Spark streaming
Source dest
kafka => Spark streaming => hdfs
kafka => Spark streaming => store in hive internal table
kafka => Spark streaming => kafka
Oozie
step-1
exe wordcount program
step-2
Put this( exe wordcount program) in shell script and run from local
step-3
put that shell script in Workflow.xml and run
1) workflow
2) coord
3) Bundil jobs
Hive
====
create internal AND external TABLES
partitons
bucketing
create static and dynamic partitons
2. RegEx
3. Apache NIFI
4. SERDEPROPERTIES in HIVE
5. Self-Join with SubQueries
6. Shell Scripting
7. Difference between ORC and Parquet
8. Go through https://www.youtube.com/watch?v=eD3LkpPQIgM for deduplication implementation in MLlib
9. How to identify the failure in spark Job?
10. What happens if there is a failure in connection between Kafka and consumer for a while
What happens next?
11 Why can't spark streaming implement consumer API's
12. What is the JAVA version used and what are its new features?
13. HBase and Cassandra Architecture
14. Kerberos Security
15. Shuffling in case of colease and re-partition
16. Schema Evolution in AVRO
17. Data has date and time values in the format dd/mm/yyyy:00:00:00. Create Hive partitioned tables partitioned by Date.Need to use date in-built functions and extract date.
18. Download traffic data and find out the location that has highest traffic
19. Table design, performance, data modelling, denormalizing
20. Spark and Spark streaming job submission.
21. Copy data from s3 to cluster.
22. Create Hive table on s3 location
AWS Access Key ID [None]: AKIAINIHCWDZMRWJEI3A
AWS Secret Access Key [None]: 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
23. hive JDBC Connection via JAVA
No comments:
Post a Comment