該樓層疑似違規(guī)已被系統(tǒng)折疊?隱藏此樓查看此樓

#me the components on this agent

a1.sources = r1

a1.sinks = k1

a1.channels = c1

# Describe/configure the source

a1.sources.r1.type = exec

a1.sources.r1.command = tail -F /usr/local/test1.txt

a1.sources.r1.fileHeader = true

a1.sources.r1.deserializer.outputCharset=UTF-8

# Describe the sink

a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink

a1.sinks.k1.brokerList = c100:9092,c200:9092,c300:9092

a1.sinks.k1.topic = test5

a1.sinks.k1.requiredAcks = 1

a1.sinks.k1.batchSize = 20

a1.sinks.k1.channel = memcnl

#a1.sinks.k1.type = avro

#a1.sinks.k1.hostname = 192.168.28.134

#a1.sinks.k1.port = 44444

# Use a channel which buffers events in memory

a1.channels.c1.type = file

a1.channels.c1.checkpointDir = /usr/local/chk

a1.channels.c1.dataDirs = /usr/local/flumeData

a1.channels.c1.checkpointOnClose = true

# Bind the source and sink to the channel

a1.sources.r1.channels = c1

a1.sinks.k1.channel = c1

把flume停掉，重啟后，數(shù)據(jù)會(huì)重復(fù)，怎么解決呢

創(chuàng)作挑戰(zhàn)賽新人創(chuàng)作獎(jiǎng)勵(lì)來(lái)咯，堅(jiān)持創(chuàng)作打卡瓜分現(xiàn)金大獎(jiǎng)

總結(jié)

以上是生活随笔為你收集整理的flume数据丢失与重复_Flume监听文件到kafka，文件通道，kafka消费重复问题的全部?jī)?nèi)容，希望文章能夠幫你解決所遇到的問(wèn)題。

如果覺(jué)得生活随笔網(wǎng)站內(nèi)容還不錯(cuò)，歡迎將生活随笔推薦給好友。