利用先电云iaas平台搭建apache官方大数据平台，ambari2.7+hdp3.0

一、ambari架构解析

二、基础环境配置

以两台节点为例来组件Hadoop分布式集群，这里采用的系统版本为Centos7 1511，如下表所示：

搭建所需各软件版本信息

①ambari-2.7.1.0-centos7.tar.gz

②jdk-8u151-linux-x64.tar.gz

③HDP-3.0.1.0-centos7-rpm.tar.gz

④HDP-UTILS-1.1.0.22-centos7.tar

⑤CentOS-7-x86_64-DVD-1511.iso

1.在iaas平台创建两台云主机

1. 主机名、主机名解析配置

vi /etc/hosts

1. 挂载镜像并配置本地yum源

首先将所需镜像、安装包等，通过secureFX上传至master 的/opt目录

Master节点

3.1进入/opt目录，解压ambari.tar.gz压缩包

[root@master ~]# cd /opt

[root@master opt]# ls

ambari-2.7.1.0-centos7.tar.gz HDP-3.0.1.0-centos7-rpm.tar.gz

centos HDP-UTILS-1.1.0.22-centos7.tar

[root@master opt]# tar -zxvf ambari-2.7.1.0-centos7.tar.gz

解压完成后获得一个ambari的文件夹

[root@master ~]# ls

CentOS-7-x86_64-DVD-1511.iso

[root@master ~]# mount -o loop CentOS-7-x86_64-DVD-1511.iso /mnt/

[root@master ~]# mkdir /opt/centos

[root@master ~]# cp -rvf /mnt/* /opt/centos/

[root@master ~]# cd /etc/yum.repos.d/

[root@master yum.repos.d]# ls

CentOS-Base.repo CentOS-Debuginfo.repo CentOS-Sources.repo

CentOS-CR.repo CentOS-fasttrack.repo CentOS-Vault.repo

[root@master yum.repos.d]# rm -rf *

[root@master yum.repos.d]# vi local.repo

[centos]

name=centos

baseurl=file:///opt/centos

gpgcheck=0

enable=1

[ambari]

name=ambari

baseurl=file:///opt/ambari/centos7/2.7.1.0-169/

gpgcheck=0

enable=1

保存退出

清理源，并列出源

[root@master ~]# yum clean all

Loaded plugins: fastestmirror

Cleaning repos: ambari centos

Cleaning up everything

Cleaning up list of fastest mirrors

[root@master ~]# yum list

[root@master ~]# yum install vsftpd

[root@master ~]# vi /etc/vsftpd/vsftpd.conf

添加

anon_root=/opt/

3.2配置开机启动vsftpd

[root@master ~]# systemctl enable vsftpd

[root@master ~]# systemctl start vsftpd

Slaver节点

[root@slaver ~]# cd /etc/yum.repos.d/

[root@slaver yum.repos.d]# ls

CentOS-Base.repo CentOS-Debuginfo.repo CentOS-Sources.repo

CentOS-CR.repo CentOS-fasttrack.repo CentOS-Vault.repo

[root@slaver yum.repos.d]# rm -rf *

[root@slaver yum.repos.d]# vi ftp.repo

[centos]

name=centos

baseurl=ftp://192.168.200.6/centos

gpgcheck=0

enable=1

[ambari]

name=ambari

baseurl=ftp://192.168.200.6/ambari/centos7/2.7.1.0-169/

gpgcheck=0

enable=1

保存退出

1. 配置NTP服务

Master节点

[root@master ~]# yum -y install ntp

[root@master ~]# vi /etc/ntp.conf

4.1将原有server注释

添加

server 127.127.1.0

fudge 127.127.1.0 stratum 10

[root@master ~]# systemctl enable ntpd

[root@master ~]# systemctl start ntpd

Slaver节点

[root@slaver ~]# yum -y install ntpdate

[root@slaver ~]# ntpdate master

[root@slaver ~]# systemctl enable ntpdate

1. 配置SSH 免密访问

Master节点

[root@master ~]# yum install openssh-clients

[root@master ~]# ssh-keygen

[root@master ~]# ssh-copy-id slaver.hadoop

Slaver节点

[root@slaver ~]# yum install openssh-clients

[root@slaver ~]# ssh-keygen

[root@slaver ~]# ssh-copy-id master.hadoop

1. 禁用Transparent Huge Pages

操作系统后台有一个叫做khugepaged的进程，它会一直扫描所有进程占用的内存，在可能的情况下会把4kpage交换为Huge Pages，在这个过程中，对于操作的内存的各种分配活动都需要各种内存锁，直接影响程序的内存访问性能，并且，这个过程对于应用是透明的，在应用层面不可控制,对于专门为4k page优化的程序来说，可能会造成随机的性能下降现象。

master & slaver

[root@master ~]# echo never > /sys/kernel/mm/transparent_hugepage/enabled

[root@master ~]# echo never > /sys/kernel/mm/transparent_hugepage/defrag

[root@master ~]# cat /sys/kernel/mm/transparent_hugepage/enabled

always advise [never]

重启后失效，需要再次执行

1. 安装配置jdk

Master节点

[root@master ~]# mkdir /usr/jdk64/

[root@master ~]# tar -zxvf jdk-8u151-linux-x64.tar.gz -C /usr/jdk64/

[root@master ~]# vi /etc/profile

7.1添加以下两条变量

export JAVA_HOME=/usr/jdk64/jdk1.8.0_151

export PATH=$JAVA_HOME/bin:$PATH

[root@master ~]# source /etc/profile

[root@master ~]# java -version

java version "1.8.0_151"

Java(TM) SE Runtime Environment (build 1.8.0_151-b12)

Java HotSpot(TM) 64-Bit Server VM (build 25.151-b12, mixed mode)

Slaver节点

[root@slaver ~]# mkdir /usr/jdk64/

[root@slaver ~]# scp 192.168.200.6://root/jdk-8u151-linux-x64.tar.gz .

jdk-8u151-linux-x64.tar.gz 100% 181MB 90.5MB/s 00:02

[root@slaver ~]# tar -zxvf jdk-8u151-linux-x64.tar.gz -C /usr/jdk64/

[root@slaver ~]# vi /etc/profile

export JAVA_HOME=/usr/jdk64/jdk1.8.0_151

export PATH=$JAVA_HOME/bin:$PATH

[root@slaver ~]# source /etc/profile

[root@slaver ~]# java -version

java version "1.8.0_151"

Java(TM) SE Runtime Environment (build 1.8.0_151-b12)

Java HotSpot(TM) 64-Bit Server VM (build 25.151-b12, mixed mode)

1. 安装httpd服务

8.1安装httpd

[root@master ~]# yum -y install httpd

[root@master ~]# systemctl enable httpd.service

[root@master ~]# systemctl start httpd.service

[root@master ~]# systemctl status httpd.service

8.2解压HDP

将/opt目录下的

HDP-3.0.1.0-centos7-rpm.tar.gz

HDP-UTILS-1.1.0.22-centos7.tar

解压，并复制到 /var/www/html

[root@master opt]# tar -zvxf HDP-3.0.1.0-centos7-rpm.tar.gz -C /var/www/html/

[root@master opt]# tar -zvxf HDP-UTILS-1.1.0.22-centos7.tar -C /var/www/html/

三、ambari-server 安装配置

Master节点

[root@master ~]# yum -y install ambari-server

1.安装MariaDB数据库

[root@master ~]# yum install mariadb mariadb-server mysql-connector-java

1.1启动数据库服务

[root@master ~]# systemctl enable mariadb

[root@master ~]# systemctl start mariadb

1.2配置数据库服务

[root@master ~]# mysql_secure_installation

Set root password? [Y/n] y （设置密码为bigdata）

Remove anonymous users? [Y/n] y

Disallow root login remotely? [Y/n] n

Remove test database and access to it? [Y/n] y

Reload privilege tables now? [Y/n] y

[root@master ~]# mysql -uroot -pbigdata

MariaDB [(none)]> create database ambari;

MariaDB [(none)]> grant all privileges on ambari.* to \'ambari\'@\'localhost\' identified by \'bigdata\';

MariaDB [(none)]> grant all privileges on ambari.* to \'ambari\'@\'%\' identified by \'bigdata\';

MariaDB [(none)]> use ambari;

MariaDB [ambari]> source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql

MariaDB [ambari]> quit

1. 配置ambari-server

[root@master ~]# vi /etc/profile

添加

export buildNumber=2.7.1.0

[root@master ~]# ambari-server setup

Using python /usr/bin/python

Setup ambari-server

Checking SELinux...

SELinux status is \'disabled\'

Customize user account for ambari-server daemon [y/n] (n)? n

Adjusting ambari-server permissions and ownership...

Checking firewall status...

Checking JDK...

[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8

[2] Custom JDK

==============================================================================

Enter choice (1): 2

WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts.

WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts.

Path to JAVA_HOME: /usr/jdk64/jdk1.8.0_151

Validating JDK on Ambari Server...done.

Check JDK version for Ambari Server...

JDK version found: 8

Minimum JDK version is 8 for Ambari. Skipping to setup different JDK for Ambari Server.

Checking GPL software agreement...

GPL License for LZO: https://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html

Enable Ambari Server to download and install GPL Licensed LZO packages [y/n] (n)? n

Completing setup...

Configuring database...

Enter advanced database configuration [y/n] (n)? y

Configuring database...

==============================================================================

Choose one of the following options:

[1] - PostgreSQL (Embedded)

[2] - Oracle

[3] - MySQL / MariaDB

[4] - PostgreSQL

[5] - Microsoft SQL Server (Tech Preview)

[6] - SQL Anywhere

[7] - BDB

==============================================================================

Enter choice (1): 3

Hostname (localhost):

Port (3306):

Database name (ambari):

Username (ambari):

Enter Database Password (bigdata):

Configuring ambari database...

Should ambari use existing default jdbc /usr/share/java/mysql-connector-java.jar [y/n] (y)?

Configuring remote database connection properties...

WARNING: Before starting Ambari Server, you must run the following DDL against the database to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql

Proceed with configuring remote database connection properties [y/n] (y)?

Extracting system views...

ambari-admin-2.7.1.0.169.jar

....

Ambari repo file doesn\'t contain latest json url, skipping repoinfos modification

Adjusting ambari-server permissions and ownership...

Ambari Server \'setup\' completed successfully.

[root@master ~]# ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar

1. 启动ambari-server服务

[root@master ~]# ambari-server start

DB configs consistency check: no errors and warnings were found.

Ambari Server \'start\' completed successfully.

四、ambari-agent 安装配置

master & slaver

1.安装ambari-agent

[root@master ~]# yum -y install ambari-agent

[root@slaver ~]# yum -y install ambari-agent

1. 修改变量

[root@master ~]# vi /etc/ambari-agent/conf/ambari-agent.ini

[root@slaver ~]# vi /etc/ambari-agent/conf/ambari-agent.ini

将[server]项修改为master.hadoop

hostname=master.hadoop

3.重启ambari-agent服务

[root@master ~]# ambari-agent restart

[root@slaver ~]# ambari-agent restart

五、部署Hadoop集群

1.访问WEB界面

打开浏览器输入IP:8080访问ambari的web界面

本机为http://192.168.200.6:8080

默认用户名 admin

默认密码 admin

1. 创建新的集群

3.1命名集群

3.2选择本地仓库源

选择redhat 7

http://192.168.200.6/HDP/centos7/3.0.1.0-187/

http://192.168.200.6/HDP-UTILS/centos7/1.1.0.22/

3.3配置主机名

3.4主机状态注册检查

3.5服务安装选择

这里我们选择了

YARN + MapReduce2

HDFS

ZooKeeper

Ambari Metrics

Spark2

3.6服务分配大师

默认即可。

3.7节点服务选择

设置Grafana密码

默认下一步即可

确认部署

开始部署