Spark 某两个节点数据分析速度慢 - hbase数据删除(分裂) 元信息未删除导致 There is an overlap in the region chain.
生活随笔
收集整理的這篇文章主要介紹了
Spark 某两个节点数据分析速度慢 - hbase数据删除(分裂) 元信息未删除导致 There is an overlap in the region chain.
小編覺得挺不錯(cuò)的,現(xiàn)在分享給大家,幫大家做個(gè)參考.
基于Hbase2.0，Spark2.2
問題描述
執(zhí)行Spark處理Hbase數(shù)據(jù)時(shí)，遇到某兩個(gè)Excutor處理速度特別慢，如圖
正常速度10多分鐘 左右處理完成， 一個(gè)多小時(shí)有另外一個(gè)處理完成，還有一個(gè)在處理中。
分析原因
1.查看hbase數(shù)據(jù)分布狀況，看看是否存在數(shù)據(jù)傾斜問題，可以看出平均為6.8G左右，但有部分?jǐn)?shù)據(jù)是1.71GB和3.4GB左右，但之前任務(wù)跑spark是80多個(gè)Executor不會(huì)導(dǎo)致時(shí)間差距這么大。
其實(shí)并非傾斜，下圖1.71GB是6.8GB剛剛分裂的Region，分裂成了四個(gè)1.71*4 = 6.84GB，大小剛剛是平均大小，
startrowkey ：bc27300~bf2a300 大小3.41+1.71.+1.71 = 6.83 都是正常的（數(shù)據(jù)存儲(chǔ)前rowkey做了處理）
| ReadRequests | WriteRequests | StorefileSize | Num.Storefiles | MemSize | Locality | Start Key | End Key |
| (1,887,103,211) | (18,159,897) | (437.56 GB) | (171) | (315 MB) | |||
| 0 | 0 | 0 B | 0 | 0 B | 0 | ? | 4003300_c126f5cc6cef33e84de0cafc9e52d44f |
| 39,112,630 | 327,294 | 6.83 GB | 3 | 5 MB | 1 | ? | 0400300_a439d4f851fa7540d87ae33454a7f892 |
| 37,563,645 | 326,734 | 6.83 GB | 2 | 6 MB | 1 | 0400300_a439d4f851fa7540d87ae33454a7f892 | 07fe300_4130b717f89f58b3b6f530eec7a54b05 |
| 37,212,501 | 327,619 | 6.84 GB | 2 | 5 MB | 1 | 07fe300_4130b717f89f58b3b6f530eec7a54b05 | 0bfd300_4a2dc6b7b1a17472484041e49382b11d |
| 37,392,603 | 57,911 | 6.84 GB | 2 | 5 MB | 1 | 0bfd300_4a2dc6b7b1a17472484041e49382b11d | 0ffd300_74fdb5eebd54ee32e19abb31bccec158 |
| 0 | 0 | 0 B | 0 | 0 B | 0 | 0ffd300_74fdb5eebd54ee32e19abb31bccec158 | 11fe300_2aa404cbc291ffd913bab13d9339085c |
| 13,882,247 | 14,525 | 1.71 GB | 2 | 1 MB | 1 | 0ffd300_74fdb5eebd54ee32e19abb31bccec158 | 10fd300_7a83ffc34c076654e69d558f1c9f1038 |
| 12,799,609 | 82,395 | 1.71 GB | 2 | 1 MB | 1 | 10fd300_7a83ffc34c076654e69d558f1c9f1038 | 11fe300_2aa404cbc291ffd913bab13d9339085c |
| 10,021,069 | 81,978 | 1.71 GB | 3 | 1 MB | 1 | 11fe300_2aa404cbc291ffd913bab13d9339085c | 12fe300_3f46699293dbeaa042e6856597d26dca |
| 8,877,982 | 16,428 | 1.71 GB | 3 | 1 MB | 1 | 12fe300_3f46699293dbeaa042e6856597d26dca | 13fd300_c223d551c26e191ffacb9981c5fb2cfd |
| 36,137,439 | 328,938 | 6.84 GB | 3 | 5 MB | 1 | 13fd300_c223d551c26e191ffacb9981c5fb2cfd | 1800300_0c10469f3ca178e9d5d81b507cc1f684 |
| 37,475,208 | 328,758 | 6.83 GB | 3 | 5 MB | 1 | 1800300_0c10469f3ca178e9d5d81b507cc1f684 | 1c00300_b0e2e6ce62897dbd1e033f88bcf22336 |
| 35,610,681 | 329,050 | 6.83 GB | 3 | 5 MB | 1 | 1c00300_b0e2e6ce62897dbd1e033f88bcf22336 | 2002300_cb4302be610bfedebaedce4a53cd0e4e |
| 36,154,352 | 328,413 | 6.86 GB | 3 | 5 MB | 1 | 2002300_cb4302be610bfedebaedce4a53cd0e4e | 2404300_806ce32f00569ec34eb1fdc008c574cc |
| 35,637,405 | 57,144 | 6.85 GB | 2 | 5 MB | 1 | 2404300_806ce32f00569ec34eb1fdc008c574cc | 2805300_3063b4db3bc49e0cd2d2e2430b8f92e8 |
| 35,617,576 | 328,429 | 6.85 GB | 3 | 5 MB | 1 | 2805300_3063b4db3bc49e0cd2d2e2430b8f92e8 | 2c04300_c3f674b39aeb8f289313f87d8d21bfe7 |
| 37,567,336 | 328,392 | 6.85 GB | 2 | 5 MB | 1 | 2c04300_c3f674b39aeb8f289313f87d8d21bfe7 | 3005300_be2766fa12b179733c9a55801367d56f |
| 33,905,338 | 329,393 | 6.84 GB | 2 | 5 MB | 1 | 3005300_be2766fa12b179733c9a55801367d56f | 3406300_ec238a180804b26ea232125ff11fcf68 |
| 33,344,294 | 328,009 | 6.84 GB | 3 | 5 MB | 1 | 3406300_ec238a180804b26ea232125ff11fcf68 | 3806300_886cf9e6502abd928a8c063ca517f54e |
| 32,536,915 | 326,671 | 6.82 GB | 2 | 5 MB | 1 | 3806300_886cf9e6502abd928a8c063ca517f54e | 3c05300_7ea6294dd2cd312d63d846bf5de5cd35 |
| 29,736,155 | 327,215 | 6.82 GB | 2 | 5 MB | 1 | 3c05300_7ea6294dd2cd312d63d846bf5de5cd35 | 4003300_c126f5cc6cef33e84de0cafc9e52d44f |
| 22,272,474 | 328,354 | 6.85 GB | 2 | 5 MB | 1 | 4003300_c126f5cc6cef33e84de0cafc9e52d44f | 4406300_18e8e63d5a80f546cdcadda551531dfc |
| 22,267,884 | 329,515 | 6.84 GB | 2 | 5 MB | 1 | 4406300_18e8e63d5a80f546cdcadda551531dfc | 4807300_8fb3f14d16049e78c52c1c1c416dda4f |
| 22,259,844 | 56,744 | 6.84 GB | 2 | 5 MB | 1 | 4807300_8fb3f14d16049e78c52c1c1c416dda4f | 4c08300_e44ffb40f82cf98c9e0802ee74070cc6 |
| 22,258,382 | 327,812 | 6.84 GB | 2 | 4 MB | 1 | 4c08300_e44ffb40f82cf98c9e0802ee74070cc6 | 500a300_fff0f934443b857109d1dbece53c770f |
| 22,269,549 | 328,870 | 6.85 GB | 3 | 5 MB | 1 | 500a300_fff0f934443b857109d1dbece53c770f | 540c300_b9bbb1361667ab3ac5669a3144a2fa71 |
| 22,250,497 | 328,277 | 6.84 GB | 2 | 5 MB | 1 | 540c300_b9bbb1361667ab3ac5669a3144a2fa71 | 580c300_c7d45406bd17aace3ad5f8586b9dff80 |
| 22,224,993 | 328,254 | 6.84 GB | 3 | 6 MB | 1 | 580c300_c7d45406bd17aace3ad5f8586b9dff80 | 5c0c300_0526a7b197b249ca24427063cf16657a |
| 22,209,147 | 327,629 | 6.83 GB | 2 | 5 MB | 1 | 5c0c300_0526a7b197b249ca24427063cf16657a | 600a300_5523694821dfcaab1fc9ff5ed40aab87 |
| 22,106,815 | 56,473 | 6.81 GB | 2 | 5 MB | 1 | 600a300_5523694821dfcaab1fc9ff5ed40aab87 | 6405300_0030bf135d4c2c20e9915c1941bd8cc7 |
| 22,136,156 | 327,095 | 6.80 GB | 3 | 5 MB | 1 | 6405300_0030bf135d4c2c20e9915c1941bd8cc7 | 67ff300_beefe04a96f547703165d057642c7a6e |
| 21,805,246 | 325,501 | 6.81 GB | 2 | 5 MB | 1 | 67ff300_beefe04a96f547703165d057642c7a6e | 6bfa300_c556fbef5490c3ca3168c7eb02015d16 |
| 21,160,315 | 56,503 | 6.81 GB | 3 | 5 MB | 1 | 6bfa300_c556fbef5490c3ca3168c7eb02015d16 | 6ff8300_471ab4d62d3bfe6ae86fe08b15524ee2 |
| 22,647,694 | 328,901 | 6.83 GB | 2 | 5 MB | 1 | 6ff8300_471ab4d62d3bfe6ae86fe08b15524ee2 | 73f7300_84b51415b091cdf93f380303f2a53658 |
| 22,209,724 | 56,882 | 6.83 GB | 2 | 5 MB | 1 | 73f7300_84b51415b091cdf93f380303f2a53658 | 77f5300_a6447ed7062c78c6f2216ec46ee47593 |
| 22,194,551 | 327,634 | 6.83 GB | 2 | 5 MB | 1 | 77f5300_a6447ed7062c78c6f2216ec46ee47593 | 7bf3300_3eab3e0f5794997bffffa596ca42d78d |
| 22,150,573 | 326,600 | 6.81 GB | 3 | 5 MB | 1 | 7bf3300_3eab3e0f5794997bffffa596ca42d78d | 7fef300_18b81d0e3245848ddfaf6a97ea868de5 |
| 0 | 0 | 0 B | 0 | 0 B | 0 | 7fef300_18b81d0e3245848ddfaf6a97ea868de5 | c02b300_52789a8b10e2d896e9be1272d9169afd |
| 37,290,422 | 328,707 | 6.87 GB | 2 | 6 MB | 1 | 7fef300_18b81d0e3245848ddfaf6a97ea868de5 | 83f3300_50ee6021166da9b30d27205aa93c7a47 |
| 35,724,649 | 329,463 | 6.87 GB | 2 | 5 MB | 1 | 83f3300_50ee6021166da9b30d27205aa93c7a47 | 87f7300_3a43c0b12b4c893d99bc9795c6fedde4 |
| 35,732,883 | 329,070 | 6.87 GB | 3 | 5 MB | 1 | 87f7300_3a43c0b12b4c893d99bc9795c6fedde4 | 8bfb300_4143081d05ed98403cf157074bd12a5d |
| 17,903,439 | 34,080 | 3.43 GB | 2 | 2 MB | 1 | 8bfb300_4143081d05ed98403cf157074bd12a5d | 8dff300_146416041229f7ddc0d92a8f3ba72e22 |
| 17,856,516 | 28,246 | 3.43 GB | 2 | 2 MB | 1 | 8dff300_146416041229f7ddc0d92a8f3ba72e22 | 9001300_1a9a07fdf1ba346228d63f9a4db9ebbf |
| 35,734,987 | 330,855 | 6.87 GB | 2 | 5 MB | 1 | 9001300_1a9a07fdf1ba346228d63f9a4db9ebbf | 9404300_f5e0017433b307e15eb645c82ba69eb1 |
| 34,355,672 | 330,896 | 6.87 GB | 2 | 5 MB | 1 | 9404300_f5e0017433b307e15eb645c82ba69eb1 | 980a300_eb3ea8174076a9343b9fefc131da4bfb |
| 35,863,280 | 57,528 | 6.89 GB | 2 | 5 MB | 1 | 980a300_eb3ea8174076a9343b9fefc131da4bfb | 9c13300_b4c65bb87a51729cdc51720b0cb0f0b7 |
| 35,775,342 | 328,782 | 6.87 GB | 3 | 5 MB | 1 | 9c13300_b4c65bb87a51729cdc51720b0cb0f0b7 | a019 |
| 35,636,124 | 328,914 | 6.86 GB | 2 | 5 MB | 1 | a019 | a41a300_3756e9713d68d9e36f1dc7fe07829779 |
| 32,629,922 | 329,001 | 6.85 GB | 3 | 5 MB | 1 | a41a300_3756e9713d68d9e36f1dc7fe07829779 | a81d300_2f82d2b7ea67878b8c1ef86bd0e1c169 |
| 33,134,529 | 328,822 | 6.84 GB | 2 | 5 MB | 1 | a81d300_2f82d2b7ea67878b8c1ef86bd0e1c169 | ac1e300_0bd8ddff51d175087f6cda84b271b0eb |
| 33,375,442 | 328,668 | 6.85 GB | 2 | 5 MB | 1 | ac1e300_0bd8ddff51d175087f6cda84b271b0eb | b01e300_c52d0d2a792286a1b88d7b91c488813b |
| 33,476,456 | 329,592 | 6.86 GB | 3 | 5 MB | 1 | b01e300_c52d0d2a792286a1b88d7b91c488813b | b423300_8a37f3ac542446d2c3e16b069b08e09d |
| 31,318,752 | 328,389 | 6.85 GB | 3 | 5 MB | 1 | b423300_8a37f3ac542446d2c3e16b069b08e09d | b826300_1cdc89eeef046475883578cf0fc1d950 |
| 30,799,781 | 329,079 | 6.85 GB | 2 | 5 MB | 1 | b826300_1cdc89eeef046475883578cf0fc1d950 | bc27300_2b9315f844b8d5e1a253f18c4d2eabe1 |
| 14,482,007 | 29,199 | 3.42 GB | 3 | 2 MB | 1 | bc27300_2b9315f844b8d5e1a253f18c4d2eabe1 | be28300_f3a448e233baeddf68cc03027bf45a74 |
| 7,271,606 | 82,854 | 1.71 GB | 2 | 1 MB | 1 | be28300_f3a448e233baeddf68cc03027bf45a74 | bf2a300_5023e51cd969429bc3f008d780b46f61 |
| 7,269,349 | 82,563 | 1.71 GB | 2 | 1 MB | 1 | bf2a300_5023e51cd969429bc3f008d780b46f61 | c02b300_52789a8b10e2d896e9be1272d9169afd |
| 20,762,362 | 325,903 | 6.82 GB | 2 | 5 MB | 1 | c02b300_52789a8b10e2d896e9be1272d9169afd | c429300_6c3de304911e18799fb01d20f69be03f |
| 22,220,126 | 327,000 | 6.82 GB | 3 | 5 MB | 1 | c429300_6c3de304911e18799fb01d20f69be03f | c828300_c50e85412bf2cdfcb005505377a0ba73 |
| 21,767,188 | 327,870 | 6.83 GB | 2 | 4 MB | 1 | c828300_c50e85412bf2cdfcb005505377a0ba73 | cc27300_2e0fea9d4e083026bf18ea43642a73d0 |
| 21,537,300 | 326,781 | 6.83 GB | 2 | 4 MB | 1 | cc27300_2e0fea9d4e083026bf18ea43642a73d0 | d023300_fa22e502bb83df688bb331aaeafc4ee6 |
| 11,061,319 | 28,100 | 3.40 GB | 3 | 2 MB | 1 | d023300_fa22e502bb83df688bb331aaeafc4ee6 | d221300_4eb87797b5869a19a7bf18100abf50ec |
| 11,081,110 | 162,680 | 3.40 GB | 3 | 2 MB | 1 | d221300_4eb87797b5869a19a7bf18100abf50ec | d41e300_961e28d64cbf6e460820a7d47b9efa12 |
| 20,708,894 | 326,154 | 6.81 GB | 3 | 5 MB | 1 | d41e300_961e28d64cbf6e460820a7d47b9efa12 | d81a300_5229559bbe763cc7ebd875db357002cd |
| 22,158,241 | 327,000 | 6.81 GB | 2 | 5 MB | 1 | d81a300_5229559bbe763cc7ebd875db357002cd | dc17300_7ac1eebea0c79430e0b0b0af4dfeffc2 |
| 22,058,558 | 327,673 | 6.82 GB | 2 | 5 MB | 1 | dc17300_7ac1eebea0c79430e0b0b0af4dfeffc2 | e014300_ec7ef53bab65a01697e2c51175c52435 |
| 0 | 0 | 0 B | 0 | 0 B | 0 | e014300_ec7ef53bab65a01697e2c51175c52435 | e813300_d1e7c088b528940a93a289f9e66e88d7 |
| 37,529,102 | 57,605 | 6.83 GB | 2 | 5 MB | 1 | e014300_ec7ef53bab65a01697e2c51175c52435 | e416300_0edad48cf0a12f8e2b1370bd6e3fc151 |
| 35,533,233 | 327,217 | 6.83 GB | 3 | 5 MB | 1 | e416300_0edad48cf0a12f8e2b1370bd6e3fc151 | e813300_d1e7c088b528940a93a289f9e66e88d7 |
| 22,219,616 | 328,019 | 6.83 GB | 2 | 5 MB | 1 | e813300_d1e7c088b528940a93a289f9e66e88d7 | ec12300_f9576788ea38623c324f1fd1ff84385c |
| 22,188,307 | 327,943 | 6.82 GB | 2 | 5 MB | 1 | ec12300_f9576788ea38623c324f1fd1ff84385c | f010300_2053a69a8c06dd64c1a7d749ba16ef65 |
| 0 | 0 | 0 B | 0 | 0 B | 0 | f010300_2053a69a8c06dd64c1a7d749ba16ef65 | f808300_3bfbe199c6b2cd26acc4a94e98a410d1 |
| 37,715,743 | 327,565 | 6.81 GB | 2 | 5 MB | 1 | f010300_2053a69a8c06dd64c1a7d749ba16ef65 | f40d300_152e1c39039240e45a842b1e6bf955d4 |
| 9,448,945 | 82,183 | 1.70 GB | 2 | 1 MB | 1 | f40d300_152e1c39039240e45a842b1e6bf955d4 | f50c300_0cc62bd5f3b5d1fcdba2ff0cc95fff37 |
| 9,236,841 | 81,378 | 1.70 GB | 2 | 1 MB | 1 | f50c300_0cc62bd5f3b5d1fcdba2ff0cc95fff37 | f60a300_e9bd0585d0aff0c3cbd93b1cf59b14b6 |
| 17,719,497 | 163,774 | 3.40 GB | 2 | 2 MB | 1 | f60a300_e9bd0585d0aff0c3cbd93b1cf59b14b6 | f808300_3bfbe199c6b2cd26acc4a94e98a410d1 |
| 20,755,049 | 327,957 | 6.81 GB | 2 | 5 MB | 1 | f808300_3bfbe199c6b2cd26acc4a94e98a410d1 | fc05300_e476fad884176fe964420de97e5cdaaf |
| 20,761,793 | 326,047 | 6.80 GB | 2 | 5 MB | 1 | fc05300_e476fad884176fe964420de97e5cdaaf | ? |
2.查看Spark日志看看有什么問題
?發(fā)現(xiàn)日志,可以看到startRow=0000，stopRow=fffg ，這個(gè)Spark掃描Hbase設(shè)置的條件
19/04/15 09:29:41 INFO rdd.NewHadoopRDD: Input split: HBase table split(table name: zh_ams_ns:wechat_article, scan: {"loadColumnFamiliesOnDemand":null,"startRow":"0000","stopRow":"fffg","batch":-1,"cacheBlocks":true,"totalColumns":2,"maxResultSize":-1,"families":{"fn":["article_type","download_type"]},"caching":-1,"maxVersions":1,"timeRange":[0,9223372036854775807]}, start row: 7fef300_18b81d0e3245848ddfaf6a97ea868de1, end row: c02b300_52789a8b10e2d896e9be1272d9169af1, region location: hostname1, encoded region name: 8943fc0bd38fd292d9acb1c6bb4b7a6) 19/04/15 09:29:41 INFO rdd.NewHadoopRDD: Input split: HBase table split(table name: zh_ams_ns:wechat_article, scan: {"loadColumnFamiliesOnDemand":null,"startRow":"0000","stopRow":"fffg","batch":-1,"cacheBlocks":true,"totalColumns":2,"maxResultSize":-1,"families":{"fn":["article_type","download_type"]},"caching":-1,"maxVersions":1,"timeRange":[0,9223372036854775807]}, start row: 6405300_0030bf135d4c2c20e9915c1941bd8cc3, end row: 67ff300_beefe04a96f547403165d057642c7a6e, region location: hostname1, encoded region name: 96c9963bdd044ebdf2bd883435735d5) 19/04/15 09:29:41 INFO broadcast.TorrentBroadcast: Started reading broadcast variable 0這個(gè)節(jié)點(diǎn)的具體處理數(shù)據(jù)信息為
start row: 7fef300_18b81d0e3245848ddfaf6a97ea868de5, end row: c02b300_52789a8b10e2d896e9be1272d9169afd, region location: hostname1 start row: 6405300_0030bf135d4c2c20e9915c1941bd8cc3, end row: 67ff300_beefe04a96f547403165d057642c7a6e, region location: hostname1明顯看出7fef300~c02b300，范圍遠(yuǎn)大于6405300~67ff300，查看其他Executor，也發(fā)現(xiàn)跨度不會(huì)這么大，查看Hbase表的數(shù)據(jù)狀態(tài)發(fā)現(xiàn)7fef300~c02b300的region沒有數(shù)據(jù)，所有信息都為零。
可以初步判斷出region分裂后，region的元信息沒有進(jìn)行刪除，?導(dǎo)致spark讀取時(shí)把rowkey 7fef300~c02b300的數(shù)據(jù)重新讀取一遍，其中包含多個(gè)region導(dǎo)致Executor處理速度過慢。
為什么發(fā)現(xiàn)很多個(gè)region信息分裂后，元信息都沒有刪除，但是只有兩個(gè)是非常慢的，可以從hbase數(shù)據(jù)中發(fā)現(xiàn)只有兩個(gè)元信息跨度比較大，其他的rowkey范圍寬度較小，所以沒有明顯感覺出來，如下圖只跨了兩個(gè)region
?
判斷完成后，我們通過hbase hbck查看一下表的狀態(tài)，截取報(bào)異常的一段，提示tableName,,1539422767596.259a2349c4f07b7d625142d323bede56.? ?region與其他region有重復(fù)區(qū)域，驗(yàn)證了我們剛才的想法
ERROR: (regions tableName,,1539422767596.259a2349c4f07b7d625142d323bede56. and tableName,0ffd300_74fdb5eebd54ee32e19abb31bccec158,1542774878705.a860b3230759d601187e9dcf6a4bdad0.) There is an overlap in the region chain. ERROR: (regions tableName,,1539422767596.259a2349c4f07b7d625142d323bede56. and tableName,10fd300_7a83ffc34c076654e69d558f1c9f1038,1550065067486.62a2f5707f49a536755d8a357d30bfd0.) There is an overlap in the region chain. ERROR: (regions tableName,0ffd300_74fdb5eebd54ee32e19abb31bccec158,1542774878705.a860b3230759d601187e9dcf6a4bdad0. and tableName,10fd300_7a83ffc34c076654e69d558f1c9f1038,1550065067486.62a2f5707f49a536755d8a357d30bfd0.) There is an overlap in the region chain. ERROR: (regions tableName,,1539422767596.259a2349c4f07b7d625142d323bede56. and tableName,11fe300_2aa404cbc291ffd913bab13d9339085c,1543086502643.f4c52419535c5574f7b79c05f82a3bd2.) There is an overlap in the region chain. ERROR: (regions tableName,,1539422767596.259a2349c4f07b7d625142d323bede56. and tableName,12fe300_3f46699293dbeaa042e6856597d26dca,1543086502643.dcfff4e27445fd7ef7d54e931fc3fee1.) There is an overlap in the region chain. ERROR: (regions tableName,,1539422767596.259a2349c4f07b7d625142d323bede56. and tableName,13fd300_c223d551c26e191ffacb9981c5fb2cfd,1542770662897.f0c75de535b324f02267b2e2d0020c75.) There is an overlap in the region chain. ERROR: (regions tableName,,1539422767596.259a2349c4f07b7d625142d323bede56. and tableName,1800300_0c10469f3ca178e9d5d81b507cc1f684,1548215404844.8546360f0bb0b7d0dcbd2c07772127ab.) There is an overlap in the region chain. ERROR: (regions tableName,,1539422767596.259a2349c4f07b7d625142d323bede56. and tableName,1c00300_b0e2e6ce62897dbd1e033f88bcf22336,1548215404844.b12cda6b9cb167c54ed47f62cc0aa7a2.) There is an overlap in the region chain. ERROR: (regions tableName,,1539422767596.259a2349c4f07b7d625142d323bede56. and tableName,2002300_cb4302be610bfedebaedce4a53cd0e4e,1549815723592.9b62e6522d5c687a088d95fbd1ca0c1b.) There is an overlap in the region chain. ERROR: (regions tableName,,1539422767596.259a2349c4f07b7d625142d323bede56. and tableName,2404300_806ce32f00569ec34eb1fdc008c574cc,1549815723592.dada7d00670779d1aaf6c0d47ebb50c7.) There is an overlap in the region chain. ERROR: (regions tableName,,1539422767596.259a2349c4f07b7d625142d323bede56. and tableName,2805300_3063b4db3bc49e0cd2d2e2430b8f92e8,1551035005714.2a6ad9c50e10527e2f08cad1d5189e4a.) There is an overlap in the region chain. ERROR: (regions tableName,,1539422767596.259a2349c4f07b7d625142d323bede56. and tableName,2c04300_c3f674b39aeb8f289313f87d8d21bfe7,1551035005714.df8ba688e677572214f75b270dedcd1b.) There is an overlap in the region chain. ERROR: (regions tableName,,1539422767596.259a2349c4f07b7d625142d323bede56. and tableName,3005300_be2766fa12b179733c9a55801367d56f,1551240890445.e479bcacf255f34bd0a98540e6e02ac5.) There is an overlap in the region chain. ERROR: (regions tableName,,1539422767596.259a2349c4f07b7d625142d323bede56. and tableName,3406300_ec238a180804b26ea232125ff11fcf68,1551240890445.21a2461c5242d970cc46486b14817e12.) There is an overlap in the region chain. ERROR: (regions tableName,,1539422767596.259a2349c4f07b7d625142d323bede56. and tableName,3806300_886cf9e6502abd928a8c063ca517f54e,1548178323143.01d1784953e10b53ed9a41953302b3bc.) There is an overlap in the region chain. ERROR: (regions tableName,,1539422767596.259a2349c4f07b7d625142d323bede56. and tableName,3c05300_7ea6294dd2cd312d63d846bf5de5cd35,1548178323143.f861eb4119dc4dc3f7c5e73555824552.) There is an overlap in the region chain. 19/04/16 13:28:43 WARN util.HBaseFsck: reached end of problem group: 4003300_c126f5cc6cef33e84de0cafc9e52d44f?
?
重大問題：統(tǒng)計(jì)完成時(shí)發(fā)現(xiàn)，會(huì)導(dǎo)致數(shù)據(jù)重復(fù)讀取，正常數(shù)據(jù)量為142825979條? ，結(jié)果查詢出來224847552條。
解決方法
1.運(yùn)行過程中，不但慢后來因?yàn)镋xecutor處理數(shù)據(jù)過大，內(nèi)存不足導(dǎo)致異常，解決方式
https://blog.csdn.net/zhangshenghang/article/details/89313245
2.刪除分裂前的region
根據(jù)region name進(jìn)行刪除，元信息在hbase:meta表中，查看狀態(tài)
get 'hbase:meta','tableName,7fef300_18331d0e3245848ddfa16a97ea868de5,1539422555091.dee970bfd95a946d9a9268db01f7ec77.'刪除
deleteall 'hbase:meta','tableName,7fef300_18b81d0e3245848ddfaf6a97ea868de5,1539422555091.dee970bfd95a946d9a9268db01f7ec77.'刪除后再次跑spark即可過濾掉這個(gè)region，解決該問題。
?
?但是存在一個(gè)隱藏問題，將表enable之后，刪除的region又出現(xiàn)了，查看zookeeper也沒有改配置，如何恢復(fù)的呢？后來經(jīng)查閱相關(guān)資料發(fā)現(xiàn)master是有緩存存儲(chǔ)這些信息的，刪除元信息（目錄 hdfs? ?/hbase/MasterProcWALs ）后重啟hbase Master集群元信息就可以了。
?
問題：
千萬不要?jiǎng)h除有數(shù)據(jù)或者正常的region，刪除并重啟后，表數(shù)據(jù)異常無法讀取數(shù)據(jù)，提示如下
hbase(main):062:0> count 'tableName'ERROR: Unknown table tableName!使用 hbase hbck查看狀態(tài)，提示region rowkey 范圍開始應(yīng)該為 '' 空字符串，如果刪除正常的region會(huì)導(dǎo)致異常
ERROR: (region tableName_201615,201604097053999999_38599d53219f66c8605abbe144b33844,1494968976496.08e0b835edffba9c7f47c68d800f297a.) First region should start with an empty key. You need to create a new region and regioninfo in HDFS to plug the hole. ERROR: Found inconsistency in table tableName_201615正常 如下圖所示
?
?
?
總結(jié)
以上是生活随笔為你收集整理的Spark 某两个节点数据分析速度慢 - hbase数据删除(分裂) 元信息未删除导致 There is an overlap in the region chain.的全部內(nèi)容，希望文章能夠幫你解決所遇到的問題。
- 上一篇： 低效程序员的7个坏习惯
- 下一篇： Linux平台下C++编程