Ambari274+HDP314 离线安装（2） 4

发表于

「这是我参与11月更文挑战的第2天，活动详情查看：2021最后一次更文挑战」

大家好，我是怀瑾握瑜，一只大数据萌新，家有两只吞金兽，嘉与嘉，上能code下能teach的全能奶爸

如果您喜欢我的文章，可以[关注⭐]+[点赞👍]+[评论📃]，您的三连是我前进的动力，期待与您共同成长~

  1. 实现离线安装，更换yum源

4.1. 文件目录展示

4.1.1. http服务方式

1
2
3
4
csharp复制代码[root@master ~]# yum -y install httpd
[root@master ~]# service httpd restart
Redirecting to /bin/systemctl restart httpd.service
[root@master ~]# chkconfig httpd on

安装完成后，会生成/var/www/html目录（相当于Tomcat的webapps目录）

将之前下的Ambari、HDP、HDP-UTILS三个包放到 /var/www/html 下

1
2
csharp复制代码[root@master ~]# cd /var/www/html/
[root@master html]# mkdir ambari

拷贝文件到ambari下面

1
2
3
4
5
6
7
8
9
10
csharp复制代码[root@master html]# cd ambari/
[root@master ambari]# ls
ambari-2.7.4.0-centos7.tar.gz   HDP-3.1.4.0-centos7-rpm.tar.gz  HDP-UTILS-1.1.0.22-centos7.tar.gz
[root@master ambari]# tar -zxvf ambari-2.7.4.0-centos7.tar.gz
[root@master ambari]# tar -zxvf HDP-3.1.4.0-centos7-rpm.tar.gz 
[root@master ambari]# mkdir HDP-UTILS
[root@master ambari]# tar -zxvf HDP-UTILS-1.1.0.22-centos7.tar.gz -C HDP-UTILS
[root@master ambari]# rm -rf ambari--2.7.4.0-centos7.tar.gz HDP-2.6.3.0-centos7-rpm.tar.gz HDP-UTILS-1.1.0.22-centos7.tar.gz 
[root@master ambari]# ls
ambari  HDP  HDP-UTILS

4.1.2. nginx服务方式

找到nginx的配置， 在nginx server中的location中增加：

1
2
3
4
5
6
7
8
9
ini复制代码server {
listen 8001;
location / {
root /www//html/ambari/;
autoindex on;
autoindex_localtime on;
autoindex_exact_size off;
}
}

参数说明：

root /data/image/

你需要开启浏览的目录，放访问

http://IP 时候显示的就是/data/image目录下的内容

autoindex_localtime on;

默认为off，显示的文件时间为GMT时间。

改为on后，显示的文件时间为文件的服务器时间

autoindex_exact_size off;

默认为on，显示出文件的确切大小，单位是bytes。

改为off后，显示出文件的大概大小，单位是kB或者MB或者GB

listen 8001;

访问端口号

1
2
csharp复制代码[root@master ~]# cd /var/www/
[root@master html]# mkdir ambari

拷贝文件到ambari下面

1
2
3
4
5
6
7
8
9
10
csharp复制代码[root@master html]# cd ambari/
[root@master ambari]# ls
ambari-2.7.4.0-centos7.tar.gz   HDP-3.1.4.0-centos7-rpm.tar.gz  HDP-UTILS-1.1.0.22-centos7.tar.gz
[root@master ambari]# tar -zxvf ambari-2.7.4.0-centos7.tar.gz
[root@master ambari]# tar -zxvf HDP-3.1.4.0-centos7-rpm.tar.gz 
[root@master ambari]# mkdir HDP-UTILS
[root@master ambari]# tar -zxvf HDP-UTILS-1.1.0.22-centos7.tar.gz -C HDP-UTILS
[root@master ambari]# rm -rf ambari--2.7.4.0-centos7.tar.gz HDP-2.6.3.0-centos7-rpm.tar.gz HDP-UTILS-1.1.0.22-centos7.tar.gz 
[root@master ambari]# ls
ambari  HDP  HDP-UTILS

访问http://172.29.30.61/ambari/ 看是否能访问

4.2. 制作本地源

4.2.1. 安装本地源制作相关工具（主节点）

1
2
bash复制代码# yum install yum-utils createrepo yum-plugin-priorities -y
# createrepo  ./

4.2.2. 修改文件里面的源地址（主节点）

注意文件路径，以自己为准

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
ini复制代码[root@master ambari]# vi ambari/centos7/2.7.4.0-118/ambari.repo
#VERSION_NUMBER=2.7.4.0-118
[ambari-2.7.4.0]
#json.url = http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json
name=ambari Version - ambari-2.7.4.0
baseurl=http://172.29.30.61:8001/ambari/centos7/2.7.4.0-118
gpgcheck=1
gpgkey=http://172.29.30.61:8001/ambari/centos7/2.7.4.0-118/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1
[root@master ambari]# cp ambari/centos7/2.7.4.0-118/ambari.repo /etc/yum.repos.d/
[root@master ambari]# vi HDP/centos7/3.1.4.0-315/hdp.repo
#VERSION_NUMBER=3.1.4.0-315
[HDP-3.1.4.0]
name=HDP Version - HDP-3.1.4.0
baseurl=http://172.29.30.61:8001/HDP/centos7/3.1.4.0-315
gpgcheck=1
gpgkey=http://172.29.30.61:8001/HDP/centos7/3.1.4.0-315/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1



[HDP-UTILS-1.1.0.22]
name=HDP-UTILS Version - HDP-UTILS-1.1.0.22
baseurl=http://172.29.30.61:8001/HDP-UTILS/HDP-UTILS/centos7/1.1.0.22
gpgcheck=1
gpgkey=http://172.29.30.61:8001/HDP-UTILS/HDP-UTILS/centos7/1.1.0.22/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1
[root@master ambari]# cp HDP/centos7/3.1.4.0-315/hdp.repo /etc/yum.repos.d/

以上就已经创建好了

使用yum的命令清一下缓存

1
2
3
arduino复制代码# yum clean all
# yum makecache
# yum repolist

4.2.3. 将创建好的源文件拷贝到子节点（主节点）

1
2
3
4
5
6
7
8
9
shell复制代码#cd /etc/yum.repos.d
# scp ambari.repo xxx2.hadoop.com:/etc/yum.repos.d/
# scp ambari.repo xxx3.hadoop.com:/etc/yum.repos.d/
# scp ambari.repo xxx4.hadoop.com:/etc/yum.repos.d/
# scp ambari.repo xxx5.hadoop.com:/etc/yum.repos.d/
# scp hdp.repo xxx2.hadoop.com:/etc/yum.repos.d/
# scp hdp.repo xxx3.hadoop.com:/etc/yum.repos.d/
# scp hdp.repo xxx4.hadoop.com:/etc/yum.repos.d/
# scp hdp.repo xxx5.hadoop.com:/etc/yum.repos.d/

  1. 安装ambari-server

这里介绍两种模式，一种是默认postgresql数据库的安装方式，还有就是mysql方式，这里我们用到的是mysql，大家根据自身选一种即可

无论是用哪种，首先都要安装ambari-server

1
arduino复制代码# yum -y install ambari-server

5.1. 默认数据库PostgreSQL安装方式（主节点）

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
erlang复制代码[root@master yum.repos.d]# ambari-server setup
Using python  /usr/bin/python
Setup ambari-server
Checking SELinux...
SELinux status is 'disabled'
Customize user account for ambari-server daemon [y/n] (n)? n
Adjusting ambari-server permissions and ownership...
Checking firewall status...
Checking JDK...
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[2] Custom JDK
Enter choice (1): 2
WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts.
WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts.
Path to JAVA_HOME: /app/tools/java/jdk1.8.0_201
Validating JDK on Ambari Server...done.
Completing setup...
Configuring database...
Enter advanced database configuration [y/n] (n)? n
Configuring database...
Default properties detected. Using built-in database.
Configuring ambari database...
Checking PostgreSQL...
Running initdb: This may take up to a minute.
Initializing database ... OK

About to start PostgreSQL
Configuring local database...
Configuring PostgreSQL...
Restarting PostgreSQL
Creating schema and user...
done.
Creating tables...
done.
Extracting system views...
ambari-admin-2.7.4.0-118.jar
...........
Adjusting ambari-server permissions and ownership...
Ambari Server 'setup' completed successfully.

启动ambari
[root@master yum.repos.d]# ambari-server start
Using python  /usr/bin/python
Starting ambari-server
Ambari Server running with administrator privileges.
Organizing resource files at /var/lib/ambari-server/resources...
Ambari database consistency check started...
Server PID at: /var/run/ambari-server/ambari-server.pid
Server out at: /var/log/ambari-server/ambari-server.out
Server log at: /var/log/ambari-server/ambari-server.log
Waiting for server start...........................................
Server started listening on 8080

DB configs consistency check: no errors and warnings were found.
Ambari Server 'start' completed successfully.

成功启动后在浏览器输入Ambari地址：

http://172.29.30.61:8080 即可看到页面

5.2. MySql安装方式（主节点）

5.2.1. 安装MySql

1
2
3
csharp复制代码[root@master ~]# wget http://dev.mysql.com/get/mysql-community-release-el7-5.noarch.rpm
[root@master ~]# rpm -ivh mysql-community-release-el7-5.noarch.rpm
[root@master ~]# yum install mysql-community-serve

5.2.2. 启动mysql，设置开机启动

1
2
3
csharp复制代码[root@master ~]# service mysqld start
[root@master ~]# vi /etc/rc.local
#添加service mysqld start

5.2.3. 登录进mysql，初始化设置root 密码

1
2
3
4
5
6
7
8
9
csharp复制代码[root@master ~]# mysql -uroot 
设置登录密码
mysql> set password for 'root'@'localhost' = password('yourPassword');

添加远程登录用户
mysql> grant all privileges on *.* to 'root'@'%' identified by 'yourPassword';

远程登录
#mysql -uroot -h ip(远程的ip地址) -p

5.2.4. 登录mysql，执行以下语句

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
sql复制代码CREATE DATABASE ambari;  
use ambari;  
CREATE USER 'ambari'@'%' IDENTIFIED BY 'ambari';  
GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'%';  
CREATE USER 'ambari'@'localhost' IDENTIFIED BY 'ambar';  
GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'localhost';  
CREATE USER 'ambari'@'master' IDENTIFIED BY 'ambari';  
GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'master';  
FLUSH PRIVILEGES;  
source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql  
show tables;  
use mysql;  
select Host User Password from user where user='ambari';  
CREATE DATABASE hive;  
use hive;  
CREATE USER 'hive'@'%' IDENTIFIED BY 'hive';  
GRANT ALL PRIVILEGES ON *.* TO 'hive'@'%';  
CREATE USER 'hive'@'localhost' IDENTIFIED BY 'hive';  
GRANT ALL PRIVILEGES ON *.* TO 'hive'@'localhost';  
CREATE USER 'hive'@'master' IDENTIFIED BY 'hive';  
GRANT ALL PRIVILEGES ON *.* TO 'hive'@'master';  
FLUSH PRIVILEGES;  
CREATE DATABASE oozie;  
use oozie;  
CREATE USER 'oozie'@'%' IDENTIFIED BY 'oozie';  
GRANT ALL PRIVILEGES ON *.* TO 'oozie'@'%';  
CREATE USER 'oozie'@'localhost' IDENTIFIED BY 'oozie';  
GRANT ALL PRIVILEGES ON *.* TO 'oozie'@'localhost';  
CREATE USER 'oozie'@'master' IDENTIFIED BY 'oozie';  
GRANT ALL PRIVILEGES ON *.* TO 'oozie'@'master';  
FLUSH PRIVILEGES;

5.2.5. 建立mysql与ambari-server的jdbc连接

1
2
3
4
5
ini复制代码[root@master yum.repos.d]# yum install mysql-connector-java
[root@master yum.repos.d]# ls /usr/share/java
[root@master yum.repos.d]# cp /usr/share/java/mysql-connector-java.jar /var/lib/ambari-server/resources/mysql-jdbc-driver.jar
[root@master yum.repos.d]# vi /etc/ambari-server/conf/ambari.properties
添加server.jdbc.driver.path=/usr/share/java/mysql-connector-java.jar

5.2.6. 设置ambari-server

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
vbnet复制代码[root@master yum.repos.d]# ambari-server setup
Using python  /usr/bin/python
Setup ambari-server
Checking SELinux...
SELinux status is 'enabled'
SELinux mode is 'permissive'
WARNING: SELinux is set to 'permissive' mode and temporarily disabled.
OK to continue [y/n] (y)? y
Customize user account for ambari-server daemon [y/n] (n)? y
Enter user account for ambari-server daemon (root):ambari #ambari-server 账号。如果直接回车就是默认选择root用户
如果输入已经创建的用户就会显示：
Adjusting ambari-server permissions and ownership...
Checking firewall status...
WARNING: iptables is running. Confirm the necessary Ambari ports are accessible. Refer to the Ambari documentation for more details on ports.
OK to continue [y/n] (y)? y
Checking JDK... #设置JDK。输入：2
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[2] Custom JDK
Enter choice (1): 2 #输入你自己的jdk位置/usr/java/jdk1.8.0_201-amd64
WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts.
WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts.
Path to JAVA_HOME: /usr/java/jdk1.8.0_201-amd64
Validating JDK on Ambari Server...done.
Check JDK version for Ambari Server...
JDK version found: 8
Minimum JDK version is 8 for Ambari. Skipping to setup different JDK for Ambari Server.
Checking GPL software agreement...
GPL License for LZO: https://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html
Enable Ambari Server to download and install GPL Licensed LZO packages [y/n] (n)? n
Completing setup...
Configuring database...
Enter advanced database configuration [y/n] (n)? y   #数据库配置。选择：y
Configuring database...
Choose one of the following options:
[1] - PostgreSQL (Embedded)
[2] - Oracle
[3] - MySQL / MariaDB
[4] - PostgreSQL
[5] - Microsoft SQL Server (Tech Preview)
[6] - SQL Anywhere
[7] - BDB
Enter choice (1): 3  #选择mysql数据库类型。输入：3
# 设置数据库的具体配置信息，根据实际情况输入，如果和括号内相同，则可以直接回车。
Hostname (localhost): 
Port (3306): 
Database name (ambari): 
Username (ambari): ambari
Enter Database Password (bigdata): ambari
Re-enter password: ambari
Configuring ambari database...
Configuring remote database connection properties...
WARNING: Before starting Ambari Server, you must run the following DDL directly from the database shell to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
Proceed with configuring remote database connection properties [y/n] (y)? y
Extracting system views...
ambari-admin-2.7.4.0.118.jar
....
Ambari repo file contains latest json url http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json, updating stacks repoinfos with it...
Adjusting ambari-server permissions and ownership...
Ambari Server 'setup' completed successfully.
[root@master yum.repos.d]# ambari-server start
Using python  /usr/bin/python
Starting ambari-server
Ambari Server running with administrator privileges.
Organizing resource files at /var/lib/ambari-server/resources...
Ambari database consistency check started...
Server PID at: /var/run/ambari-server/ambari-server.pid
Server out at: /var/log/ambari-server/ambari-server.out
Server log at: /var/log/ambari-server/ambari-server.log
Waiting for server start...........................................
Server started listening on 8080


DB configs consistency check: no errors and warnings were found.
Ambari Server 'start' completed successfully.

如果启动失败建议查看

日志在/var/log/ambari-server/ambari-server.log里面

重置ambari-server

1
2
3
csharp复制代码[root@master ~]# ambari-server stop
[root@master ~]# ambari-server reset
[root@master ~]# ambari-server setup

如果选择的是mysql方式，就需要先执行上面的语句，然后手动将mysql里面创建的数据库进行删除

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
sql复制代码[root@master ~]# mysql -uroot -p
mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| ambari             |
| hive               |
| oozie              |
| performance_schema |
+--------------------+
rows in set (0.00 sec)
mysql> drop database ambari;
mysql> drop database hive;
mysql> drop database oozie;

如果在安装的过程中出现了错误，又想重新安装，可以在ambari-server开启的情况下，执行下面的语句来移除已安装的包，然后再通过不同的情况选择上面两种方式的一种对ambari-server进行重置

1
bash复制代码python /usr/lib/python2.6/site-packages/ambari_agent/HostCleanup.py --silent

5.3. Mysql驱动检查

为了后续安装HDP时hive可以正常安装，此处需要检查mysql驱动是否符合要求

第一：数据库、用户名、密码是否正确，安装驱动

1
复制代码yum install mysql-connector-java

第二：mysql驱动是否存在

  1. /usr/share/java/mysql-connector-java.jar 是否有mysql驱动
  2. vim /etc/ambari-server/conf/ambari.properties

配置文件添加server.jdbc.driver.path=/usr/share/java/mysql-connector-java.jar

第三：设置ambari的mysql驱动

在master主机命令行界面执行即可（前提是上面第二条件都满足）

1
css复制代码ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar

  1. 安装配置部署HDP集群

6.1. 登录过程

如果你以上安装成功

输入主机ip：8080则会看到如下界面

账户：admin 密码：admin

6.2. 安装向导

6.2.1. 配置集群名字

6.2.2. 选择版本并修改为本地源地址

Centos的选择rathat7，HDP和DHP-UTILS的url地址填上之前改好的

使用当前安装包需要修改HDP，HDP-3.1.4.0

ambari确认过了就next继续了

6.2.3. 安装配置

上传之前存好的秘钥文件id_rsa

6.2.4. 确认安装ambari的agent

确认安装ambari的agent，并检查出现的问题，这个地方经常出现错误

如图就出现了错误，点击Failed的查看错误日志

我这里的错误是ambari的8040等端口无法访问的问题，我放开了8000-9000的端口就可以了，我之前也遇到很多其他的问题，具体问题具体分析，多查谷歌，百度，国外的网站更容易解决问题

检查主机可能会发现之前漏下的问题，比如说我这里防火墙没关他就会出现提示

检查无误，NEXT→通过即可

如果这个步骤失败了错误，记得多看日志，多找问题，如果还不行的话，回档咯

1
2
3
4
csharp复制代码[root@master ~]# # ambari-server stop    #停止命令
[root@master ~]# # ambari-server reset   #重置命令
[root@master ~]# # ambari-server setup   #重新设置 
[root@master ~]# # ambari-server start   #启动命令

6.2.5. 大数据服务组件安装

勾选你所需要的

6.2.6. 节点分配

6.2.7. 分配主从

6.2.8. 安装配置

hive和oozie的数据库用户密码填上之前创建好的

如果安装了hive，ooize等，需要修改成我们本地建好的库，jdbc-mysql也要配置好

6.2.9. 概况部署

警告这里我这就忽略掉了，后期我们再修复

  1. 集群配置

7.1. hive配置更新

修改hive配置，关闭acid

1
2
3
ini复制代码hive.strict.managed.tables=false 
hive.create.as.insert.only=false 
metastore.create.as.acid=false

新增hive-site配置，关闭客户端校验

1
ini复制代码hive.metastore.client.capability.check = false

设置数据库路径访问权限（KUDU使用）

1
bash复制代码hdfs dfs -chmod -R 777  /warehouse/tablespace/managed/hive

7.2. hdfs配置更新

增加配置项，方便查看文件，新增默认用户

1
ini复制代码hadoop.http.staticuser.user=hdfs

7.3. Spark配置更新

关闭hiveacid后，访问metastore值需要配置，该配置项为spark度metastore文件默认配置，原值为spark，读自己的

1
ini复制代码metastore.catalog.default=hive

结束语

如果您喜欢我的文章，可以[关注⭐]+[点赞👍]+[评论📃]，您的三连是我前进的动力，期待与您共同成长~

可关注企鹅号【怀瑾握瑜的嘉与嘉】，获取资源下载方式

本文转载自: 掘金

开发者博客 – 和开发相关的 这里全都有

0%