kafka生產(chǎn)實(shí)例安裝-創(chuàng)新互聯(lián)

2019/3/14 星期四

創(chuàng)新互聯(lián)專注于雙峰企業(yè)網(wǎng)站建設(shè),響應(yīng)式網(wǎng)站,成都商城網(wǎng)站開(kāi)發(fā)。雙峰網(wǎng)站建設(shè)公司,為雙峰等地區(qū)提供建站服務(wù)。全流程定制網(wǎng)站，專業(yè)設(shè)計(jì)，全程項(xiàng)目跟蹤，創(chuàng)新互聯(lián)專業(yè)和態(tài)度為您提供的服務(wù)

Linux 初始化腳本（centos6 centos7 通用）Linux 初始化腳本（centos6 centos7 通用）
zookeeper生產(chǎn)環(huán)境搭建 zookeeper生產(chǎn)環(huán)境搭建

在安裝前請(qǐng)務(wù)必安裝好zookeeper 查看上面2個(gè)鏈接地址！

kafka優(yōu)化：
有時(shí)候系統(tǒng)需要應(yīng)對(duì)突如其來(lái)的高峰數(shù)據(jù)，它可能會(huì)拖慢磁盤。（比如說(shuō)，每個(gè)小時(shí)開(kāi)始時(shí)進(jìn)行的批量操作等）
這個(gè)時(shí)候需要容許更多的臟數(shù)據(jù)存到內(nèi)存，讓后臺(tái)進(jìn)程慢慢地通過(guò)異步方式將數(shù)據(jù)寫到磁盤當(dāng)中。

vm.dirty_background_ratio = 5
vm.dirty_ratio = 80

這個(gè)時(shí)候，后臺(tái)進(jìn)行在臟數(shù)據(jù)達(dá)到5%時(shí)就開(kāi)始異步清理，但在80%之前系統(tǒng)不會(huì)強(qiáng)制同步寫磁盤。這樣可以使IO變得更加平滑。
實(shí)際操作：
echo "vm.dirty_background_ratio=5" >> /etc/sysctl.conf
echo "vm.dirty_ratio=80" >> /etc/sysctl.conf
sysctl -p

kafka 生產(chǎn)環(huán)境搭建

[root@emm-kafka01-10--174 ~]# cd /opt/ins/
[root@emm-kafka01-10--174 ins]# ll
total 233044
-rwxr-xr-x 1 root root 166044032 Mar 13 15:58 jdk-8u102-linux-x64.rpm
-rw-r--r-- 1 root root  50326212 Mar 13 16:14 kafka_2.12-1.1.0.tgz
-rw-r--r-- 1 root root  22261552 Mar 13 16:14 zookeeper-3.4.8.tar.gz
[root@emm-kafka01-10--174 ins]# tar -zxvf kafka_2.12-1.1.0.tgz -C /usr/local/
[root@emm-kafka01-10--174 ins]# cd /usr/local/
[root@emm-kafka01-10--174 local]# ln -s kafka_2.12-1.1.0/ kafka
[root@emm-kafka01-10--174 local]# ll
total 4
drwxr-xr-x.  2 root root    6 Apr 11  2018 bin
drwxr-xr-x.  2 root root    6 Apr 11  2018 etc
drwxr-xr-x.  2 root root    6 Apr 11  2018 games
drwxr-xr-x.  2 root root    6 Apr 11  2018 include
lrwxrwxrwx   1 root root   17 Mar 14 09:51 kafka -> kafka_2.12-1.1.0/
drwxr-xr-x   6 root root   83 Mar 24  2018 kafka_2.12-1.1.0
drwxr-xr-x.  2 root root    6 Apr 11  2018 lib
drwxr-xr-x.  2 root root    6 Apr 11  2018 lib64
drwxr-xr-x.  2 root root    6 Apr 11  2018 libexec
drwxr-xr-x.  2 root root    6 Apr 11  2018 sbin
drwxr-xr-x.  5 root root   46 Apr 11  2018 share
drwxr-xr-x.  2 root root    6 Nov 12 13:03 src
lrwxrwxrwx   1 root root   15 Mar 13 18:20 zookeeper -> zookeeper-3.4.8
drwxr-xr-x  11 root root 4096 Mar 13 18:22 zookeeper-3.4.8

修改配置文件
我們不會(huì)對(duì) zookeeper.properties producer.properties consumer.properties 做什么特殊的配置
我們只對(duì)server.properties 做修改配置

[root@emm-kafka01-10--174 config]# pwd
/usr/local/kafka/config
[root@emm-kafka01-10--174 config]# ll
total 64
-rw-r--r-- 1 root root  906 Mar 24  2018 connect-console-sink.properties
-rw-r--r-- 1 root root  909 Mar 24  2018 connect-console-source.properties
-rw-r--r-- 1 root root 5807 Mar 24  2018 connect-distributed.properties
-rw-r--r-- 1 root root  883 Mar 24  2018 connect-file-sink.properties
-rw-r--r-- 1 root root  881 Mar 24  2018 connect-file-source.properties
-rw-r--r-- 1 root root 1111 Mar 24  2018 connect-log4j.properties
-rw-r--r-- 1 root root 2730 Mar 24  2018 connect-standalone.properties
-rw-r--r-- 1 root root 1221 Mar 24  2018 consumer.properties
-rw-r--r-- 1 root root 4727 Mar 24  2018 log4j.properties
-rw-r--r-- 1 root root 1919 Mar 24  2018 producer.properties
-rw-r--r-- 1 root root 6851 Mar 24  2018 server.properties
-rw-r--r-- 1 root root 1032 Mar 24  2018 tools-log4j.properties
-rw-r--r-- 1 root root 1023 Mar 24  2018 zookeeper.properties
[root@emm-kafka01-10--174 config]# ls -l server.properties 
-rw-r--r-- 1 root root 6851 Mar 24  2018 server.properties
[root@emm-kafka01-10--174 config]# vim server.properties
...
...
...

如下
config下的 server.properties
broker.id=10 --這個(gè)id如果部署的是kafka集群，id是不能一樣的
port=9092 --默認(rèn)kafka端口，如果一臺(tái)機(jī)器上面部署了多個(gè)kafka實(shí)例，需要兩個(gè)實(shí)例端口不一樣
log.dirs=/var/log/kafka/kafka-logs 這個(gè)需要修改，這個(gè)是存topic相關(guān)信息
zookeeper.connect=10.2.10.174:2181,10.2.10.175:2181,10.2.10.176:2181/kafkagroup

[root@emm-kafka01-10--174 config]# grep '^[a-Z]' server.properties    
broker.id=174                             //每個(gè)實(shí)例都不一樣
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/var/log/kafka/kafka-logs
num.partitions=1                         //默認(rèn)的每個(gè)topic的分區(qū) 為1
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=10.2.10.174:2181,10.2.10.175:2181,10.2.10.176:2181/kafkagroup
zookeeper.connection.timeout.ms=6000
group.initial.rebalance.delay.ms=0

注意：
kafka實(shí)現(xiàn)功能需要zookeeper調(diào)度，所以這是與zk單機(jī)或者集群的連接，上面是與集群連接的方式，也可以去掉/kafkagroup,但是zk的znode結(jié)構(gòu)就會(huì)比較混亂，所以建議加一個(gè)路徑
這樣在zk的znode下就會(huì)出現(xiàn) kafkagroup

其他2臺(tái) 的server.properties 配置文件中的 broker.id=改成 175 176
其他的不變
[root@emm-kafka01-10--174 config]# scp server.properties root@10.2.10.175:/usr/local/kafka/config/
server.properties 100% 6911 2.5MB/s 00:00
[root@emm-kafka01-10--174 config]# scp server.properties root@10.2.10.176:/usr/local/kafka/config/
server.properties 100% 6911 2.8MB/s 00:0

修改kafka啟動(dòng)腳本（調(diào)節(jié)啟動(dòng)內(nèi)存占用大小
[root@emm-kafka01-10--174 bin]# vim /usr/local/kafka/bin/kafka-server-start.sh
if [ "x$KAFKA_HEAP_OPTS" = "x" ]; then
export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G"
fi

改成
我們的內(nèi)存是8G 這里我們這樣改
export KAFKA_HEAP_OPTS="-Xmx4G -Xms1G"
大4G 初始 1G
內(nèi)存為4G 就改成
export KAFKA_HEAP_OPTS="-Xmx2G -Xms1G"

[root@emm-kafka01-10--174 bin]# scp kafka-server-start.sh root@10.2.10.175:/usr/local/kafka/bin/
kafka-server-start.sh                                                        100% 1376     1.5MB/s   00:00    
[root@emm-kafka01-10--174 bin]# scp kafka-server-start.sh root@10.2.10.176:/usr/local/kafka/bin/
kafka-server-start.sh                                                        100% 1376     1.2MB/s   00:00

4、啟動(dòng)kafka //后臺(tái)啟動(dòng) 再啟動(dòng)之前，我們先設(shè)置環(huán)境變量

kafka生產(chǎn)實(shí)例安裝

[root@emm-kafka01-10--174 bin]# vim /etc/profile
export PATH=/usr/local/kafka/bin:/usr/local/zookeeper/bin:$PATH
[root@emm-kafka01-10--174 bin]# source /etc/profile
[root@emm-kafka01-10--174 bin]# which kafka-server-start.sh 
/usr/local/kafka/bin/kafka-server-start.sh
[root@emm-kafka01-10--174 bin]# which zkServer.sh
/usr/local/zookeeper/bin/zkServer.sh

其他2臺(tái)同上

后臺(tái)啟動(dòng)kafka

[root@emm-kafka01-10--174 bin]# cd ~
[root@emm-kafka01-10--174 ~]# nohup kafka-server-start.sh /usr/local/kafka/config/server.properties 1>/dev/null 2>&1 &
[1] 26314
[root@emm-kafka01-10--174 ~]# jps
14290 QuorumPeerMain
26643 Jps
26314 Kafka
[root@emm-kafka01-10--174 ~]# ps -ef|grep kafka
root     26314 24915 35 10:12 pts/0    00:00:11 java -Xmx2G -Xms1G -server

[root@emm-kafka01-10--174 ~]# sh zkCli.sh
[zk: localhost:2181(CONNECTED) 0] ls /
[kafkagroup, zookeeper]
[zk: localhost:2181(CONNECTED) 1] ls /kafkagroup
[cluster, controller, controller_epoch, brokers, admin, isr_change_notification, consumers, log_dir_event_notification, latest_producer_id_block, config]

以上安裝成功

我們接下來(lái)就是測(cè)試kafka

[root@emm-kafka01-10--174 ~]# kafka-topics.sh --zookeeper 10.2.10.174/kafkagroup --create --topic majihui --partitions 2 --replication-factor 2
Created topic "majihui".
[root@emm-kafka01-10--174 ~]# kafka-topics.sh --zookeeper 10.2.10.174/kafkagroup --describe --topic majihui    
Topic:majihui   PartitionCount:2        ReplicationFactor:2     Configs:
        Topic: majihui  Partition: 0    Leader: 174     Replicas: 174,175       Isr: 174,175
        Topic: majihui  Partition: 1    Leader: 175     Replicas: 175,176       Isr: 175,176

[root@emm-kafka02-10--175 ~]# kafka-topics.sh --zookeeper 10.2.10.174 --list  
[root@emm-kafka02-10--175 ~]# kafka-topics.sh --zookeeper 10.2.10.174/kafkagroup --list
__consumer_offsets
majihui

啟動(dòng)生產(chǎn)者
[root@emm-kafka02-10--175 ~]# kafka-console-producer.sh --zookeeper 10.2.10.174:2181 --topic majihui           
zookeeper is not a recognized option
這個(gè)命令不對(duì)，命令改了
[root@emm-kafka02-10--175 ~]# kafka-console-producer.sh --broker-list 10.2.10.174:9092 --topic majihui
>hello

啟動(dòng)消費(fèi)者
[root@emm-kafka01-10--174 ~]# kafka-console-consumer.sh --bootstrap-server 10.2.10.174:9092 --topic majihui    
hello

刪除topic majihui
[root@emm-kafka02-10--175 ~]# kafka-topics.sh --zookeeper 10.2.10.174/kafkagroup --delete --topic majihui
Topic majihui is marked for deletion.
Note: This will have no impact if delete.topic.enable is not set to true.
[root@emm-kafka02-10--175 ~]# kafka-topics.sh --zookeeper 10.2.10.174/kafkagroup --list
__consumer_offsets

[root@emm-kafka01-10--174 ~]# kafka-topics.sh --zookeeper 10.2.10.174/kafkagroup --delete --topic __consumer_offsets
Error while executing topic command : Topic __consumer_offsets is a kafka internal topic and is not allowed to be marked for deletion.
//執(zhí)行主題命令時(shí)出錯(cuò)：主題__consumer_offsets是kafka內(nèi)部主題，不允許標(biāo)記為刪除。
[2019-03-14 11:24:17,407] ERROR kafka.admin.AdminOperationException: Topic __consumer_offsets is a kafka internal topic and is not allowed to be marked for deletion.
        at kafka.admin.TopicCommand$.$anonfun$deleteTopic$1(TopicCommand.scala:188)
        at kafka.admin.TopicCommand$.$anonfun$deleteTopic$1$adapted(TopicCommand.scala:185)
        at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:59)
        at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:52)
        at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
        at kafka.admin.TopicCommand$.deleteTopic(TopicCommand.scala:185)
        at kafka.admin.TopicCommand$.main(TopicCommand.scala:71)
        at kafka.admin.TopicCommand.main(TopicCommand.scala)
 (kafka.admin.TopicCommand$)
[root@emm-kafka01-10--174 ~]# kafka-topics.sh --zookeeper 10.2.10.174/kafkagroup --list
__consumer_offsets

由于版本的升級(jí)，以前的命令不可以用了
這里注意一下。

kafka優(yōu)化：

[root@emm-kafka01-10--174 ~]# echo "vm.swappiness=1" >> /etc/sysctl.conf
[root@emm-kafka01-10--174 ~]# echo "vm.dirty_background_ratio=5" >> /etc/sysctl.conf
[root@emm-kafka01-10--174 ~]# echo "vm.dirty_ratio=80" >> /etc/sysctl.conf
[root@emm-kafka01-10--174 ~]# sysctl -p
vm.swappiness = 1
vm.dirty_background_ratio = 5
vm.dirty_ratio = 80
[root@emm-kafka01-10--174 ~]# sysctl -p
vm.swappiness = 1
vm.dirty_background_ratio = 5
vm.dirty_ratio = 80
[root@emm-kafka01-10--174 ~]# scp /etc/sysctl.conf root@10.2.10.175:/etc/sysctl.conf  
sysctl.conf                                                                  100%  511   529.6KB/s   00:00    
[root@emm-kafka01-10--174 ~]# scp /etc/sysctl.conf root@10.2.10.176:/etc/sysctl.conf  
sysctl.conf                                                                  100%  511   497.6KB/s   00:00

kafka啟動(dòng)腳本

[root@emm-kafka01-10--174 scripts]# pwd
/opt/scripts
[root@emm-kafka01-10--174 scripts]# ll
total 16
-rwxr-xr-x 1 root root 6704 Mar 13 15:10 initialization.sh
-rw-r--r-- 1 root root  122 May 10 11:05 kafkastart.sh
-rw-r--r-- 1 root root   88 May 10 10:26 zkstart.sh
[root@emm-kafka01-10--174 scripts]# cat kafkastart.sh 
#!/bin/bash
source /etc/profile

nohup /usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties 1>/dev/null 2>&1 &

[root@emm-kafka01-10--174 scripts]# scp kafkastart.sh root@10.2.10.175:/opt/scripts/
kafkastart.sh                                                                  100%  122    70.8KB/s   00:00    
[root@emm-kafka01-10--174 scripts]# scp kafkastart.sh root@10.2.10.176:/opt/scripts/
kafkastart.sh                                                                  100%  122    61.6KB/s   00:00  
[root@emm-kafka01-10--174 scripts]# sh kafkastart.sh
[root@emm-kafka01-10--174 scripts]# jps
11185 Jps
7974 QuorumPeerMain
10856 Kafka

[root@emm-kafka01-10--174 scripts]# netstat -lntup
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1379/master         
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1110/sshd           
tcp6       0      0 ::1:25                  :::*                    LISTEN      1379/master         
tcp6       0      0 :::39549                :::*                    LISTEN      10856/java          
tcp6       0      0 :::9092                 :::*                    LISTEN      10856/java          
tcp6       0      0 :::2181                 :::*                    LISTEN      7974/java           
tcp6       0      0 10.2.10.174:3888        :::*                    LISTEN      7974/java           
tcp6       0      0 :::33264                :::*                    LISTEN      7974/java           
tcp6       0      0 :::22                   :::*                    LISTEN      1110/sshd

添加到開(kāi)機(jī)自啟動(dòng)

[root@emm-kafka03-10--176 scripts]# echo "/usr/bin/sh /opt/scripts/kafkastart.sh" >> /etc/rc.d/rc.local
[root@emm-kafka03-10--176 scripts]# cat /etc/rc.d/rc.local
#!/bin/bash
# THIS FILE IS ADDED FOR COMPATIBILITY PURPOSES
#
# It is highly advisable to create own systemd services or udev rules
# to run scripts during boot instead of using this file.
#
# In contrast to previous versions due to parallel execution during boot
# this script will NOT be run after all other services.
#
# Please note that you must run 'chmod +x /etc/rc.d/rc.local' to ensure
# that this script will be executed during boot.

touch /var/lock/subsys/local
ulimit -SHn 65535
echo 'never' >/sys/kernel/mm/transparent_hugepage/enabled
echo 'never' >/sys/kernel/mm/transparent_hugepage/defrag
/usr/bin/sh /opt/scripts/zkstart.sh
/usr/bin/sh /opt/scripts/kafkastart.sh

[root@emm-kafka03-10--176 scripts]# chmod +x /etc/rc.d/rc.local

kafka優(yōu)化參考鏈接：https://www.cnblogs.com/yinzhengjie/p/9994207.html

另外有需要云服務(wù)器可以了解下創(chuàng)新互聯(lián)scvps.cn，海內(nèi)外云服務(wù)器15元起步，三天無(wú)理由+7*72小時(shí)售后在線，公司持有idc許可證，提供“云服務(wù)器、裸金屬服務(wù)器、高防服務(wù)器、香港服務(wù)器、美國(guó)服務(wù)器、虛擬主機(jī)、免備案服務(wù)器”等云主機(jī)租用服務(wù)以及企業(yè)上云的綜合解決方案，具有“安全穩(wěn)定、簡(jiǎn)單易用、服務(wù)可用性高、性價(jià)比高”等特點(diǎn)與優(yōu)勢(shì)，專為企業(yè)上云打造定制，能夠滿足用戶豐富、多元化的應(yīng)用場(chǎng)景需求。

文章標(biāo)題：kafka生產(chǎn)實(shí)例安裝-創(chuàng)新互聯(lián)
當(dāng)前網(wǎng)址：http://jinyejixie.com/article36/jshsg.html

成都網(wǎng)站建設(shè)公司_創(chuàng)新互聯(lián)，為您提供關(guān)鍵詞優(yōu)化、網(wǎng)站建設(shè)、品牌網(wǎng)站制作、服務(wù)器托管、搜索引擎優(yōu)化、企業(yè)網(wǎng)站制作

聲明：本網(wǎng)站發(fā)布的內(nèi)容（圖片、視頻和文字）以用戶投稿、用戶轉(zhuǎn)載內(nèi)容為主，如果涉及侵權(quán)請(qǐng)盡快告知，我們將會(huì)在第一時(shí)間刪除。文章觀點(diǎn)不代表本網(wǎng)站立場(chǎng)，如需處理請(qǐng)聯(lián)系客服。電話：028-86922220；郵箱：631063699@qq.com。內(nèi)容未經(jīng)允許不得轉(zhuǎn)載，或轉(zhuǎn)載時(shí)需注明來(lái)源：創(chuàng)新互聯(lián)

猜你還喜歡下面的內(nèi)容

成人午夜视频全免费观看高清-秋霞福利视频一区二区三区-国产精品久久久久电影小说-亚洲不卡区三一区三区一区

kafka生產(chǎn)實(shí)例安裝-創(chuàng)新互聯(lián)