Introduction in the big data learning, on the part of the foundation, has always been very important to everyone, the foundation is good, in order to really benefit in the subsequent development, faster growth. As a white man in the big data industry, want to become a big data engineer, you need to master some knowledge and skills, then white man to become a big data engineer, need to master what knowledge and skills? Here we will be specific to understand it.
1, learn big data, in the early stage is mainly to play the foundation, including java foundation and Linux foundation, and only after that will formally enter the stage of big data technology learning.
2, Linux learning is mainly to build big data cluster environment to prepare, so the Linux system commands and shell programming as the main content to master.
3, and Java, mainly Java
SE, involving more need to master the content, including mastering the java language variables, control structures, loops, object-oriented encapsulation and other content; mastering the object-oriented, IO streams, data structures and other content; mastering the reflection, xml parsing, sockets, threads, and databases and other content.
Java EE, the need to master the content is not much, master html, css, js, http protocol, Servlet and other content; master Maven, spring, spring
mvc, mybatis and other content basically enough.
4, with the above foundation, into the big data technology framework for learning, the use of Linux systems to build Hadoop distributed clusters, the use of Hadoop distributed program development, the use of Zookeeper to build Hadoop
HA high availability, Shell script calls and other big data technology framework has a preliminary understanding.
5, for Hadoop, involving related system components, are required to gradually learn to master, including understanding and mastering the Mapreduce framework principles, using Mapreduce on offline data analysis, using Hive on massive data storage and analysis, using MySQL database to store metadata information using regular expressions, the use of Shell scripts. Use Mapreduce and Hive to complete the microblogging project part of the function development, learn to use flume and so on.
6, to be able to hbase database different scenarios for data crud, kafka installation and cluster common commands and the use of java
api, able to use the scala language for the subsequent development of the spark project to lay the foundation, learn to use sqoop;
7, to master the core programming of spark to carry out offline batch processing, sparkSQL to do interactive query, sparkStreaming to do real-time streaming computing, in-depth understanding of the principles of spark, spark parameter tuning and operation and maintenance related knowledge.
The above is a small white to become a big data engineer skills related to the introduction, I hope you can help, of course, want to become a good big data engineer, continuous learning and enhancement is the first, I hope you cheer hard!