zookeeper客户端库curator分析
zookeeper客戶端庫curator分析
- 前言
- 綜述
- zookeeper保證
- 理解zookeeper的順序一致性
- 之前使用zookeeper客戶端踩到的坑
- curator 連接保證
- 連接狀態監控以及重試機制
- 實例管理
- Recipes 場景支持
- 基本操作
- 監聽watch
- 實現的recipes
- Elections 選舉
- locks 鎖
- counters 計數器
- caches 緩存
- Nodes/Watchers
- Queues 隊列
- 事務
- tech note
- 參考鏈接
前言
筆者在日常工作中主要使用的編程語言是C++，但從事互聯網行業總離不開要和分布式共識協議下的注冊中心打交道。筆者所在的公司主要用zookeeper。最初是裸著用，出現了很多問題，后來痛定思痛，決定研究curator，并根據其思路開發一套c++的客戶端。下面是筆者在閱讀curator代碼和設計文檔的過程中的筆記。
本文對java源碼探究不深，原因一是筆者本身java水平不高，二是筆者閱讀curator代碼的目的只是為了學習其設計思路，啟發自己設計c++版本庫。
綜述
zookeeper不是為高可用性設計的，但它使用ZAB協議達到了極高的一致性。所以它經常被選作注冊中心、配置中心、分布式鎖等場景。zookeeper是最終一致性系統，而很多實際應用需要保證強一致。
官方文檔這樣描述Curator存在的意義：ZooKeeper is a very low level system that requires users to do a lot of housekeeping. See: Zookeeper FAQ. The Curator Framework is designed to hide as much of the details/tedium of this housekeeping as is possible.
目前看到有兩套兩款比較好的開源客戶端，對zookeeper的原生API進行了包裝：zkClient和curator。后者是Nexflix的開源項目，目前運作在Apache基金會名下，也是spring全家桶的選擇。
zookeeper保證
根據zookeeper官方文檔，zookeeper提供了如下保證：
- Sequential Consistency - Updates from a client will be applied in the order that they were sent.
- Atomicity - Updates either succeed or fail. No partial results.
- Single System Image - A client will see the same view of the service regardless of the server that it connects to. i.e., a client will never see an older view of the system even if the client fails over to a different server with the same session. 如果client首先看到了新數據，再嘗試重連到存有舊數據的follower，該follower會拒絕該連接（client的zxid高于follower）
- Reliability - Once an update has been applied, it will persist from that time forward until a client overwrites the update.
- Timeliness - The clients view of the system is guaranteed to be up-to-date within a certain time bound.
根據我的實踐，認為zookeeper只是一個最終一致性的分布式系統，并且歷史上zookeeper還經常爆出違反分布式共識的bug，比如expired ephemeral node reappears after ZK leader change這個，session expired之后，臨時節點仍然存在
理解zookeeper的順序一致性
ZooKeeper Programmer’s Guide提到：
Sometimes developers mistakenly assume one other guarantee that ZooKeeper does not in fact make. This is:
Simultaneously Conistent Cross-Client Views
ZooKeeper does not guarantee that at every instance in time, two different clients will have identical views of ZooKeeper data. Due to factors like network delays, one client may perform an update before another client gets notified of the change. Consider the scenario of two clients, A and B. If client A sets the value of a znode /a from 0 to 1, then tells client B to read /a, client B may read the old value of 0, depending on which server it is connected to. If it is important that Client A and Client B read the same value, Client B should should call the sync() method from the ZooKeeper API method before it performs its read.
So, ZooKeeper by itself doesn’t guarantee that changes occur synchronously across all servers, but ZooKeeper primitives can be used to construct higher level functions that provide useful client synchronization.
就是說zookeeper并不保證每次從其一個server讀到的值是最新的，它只保證這個server中的值是順序更新的，如果想要讀取最新的值，必須在get之前調用sync()(zoo_async)
之前使用zookeeper客戶端踩到的坑
- 忽略了connecting事件，client與server心跳超時之后沒有將選主服務及時下線掉，導致雙主。
- 多個線程處理zk的連接狀態，導致產生了多套zk線程連接zkserver。
- zk超時時間不合理，導致重連頻率太高，打爆zkserver。
- 所有的zkserver全部重置（zk server全部狀態被重置），這種情況下客戶端不會受到expired事件，我之前實現的客戶端也不會重新去建立zk session。導致之前的zkclient建立的session全部不可用，陷入無限重連而連不上的窘境。
- zk自己的線程do_completion會調用watcher的回調函數，和業務線程產生競爭，導致core dump。
- 同步API沒有超時時間，如果zkserver狀態不對，會導致調用同步zk API的線程卡死。
- 供業務使用的api設計不當，導致初始化時調用的同步版本api造成死鎖。
curator 連接保證
Curator會監控所有的zookeeper連接，并且所有的操作都會有重試機制，因此curator可以保證：
連接狀態監控以及重試機制
ConnectionStateListener
Curator will set the LOST state when it believes that the ZooKeeper session has expired. ZooKeeper connections have a session. When the session expires, clients must take appropriate action. In Curator, this is complicated by the fact that Curator internally manages the ZooKeeper connection. Curator will set the LOST state when any of the following occurs: a) ZooKeeper returns a Watcher.Event.KeeperState.Expired or KeeperException.Code.SESSIONEXPIRED; b) Curator closes the internally managed ZooKeeper instance; c) The session timeout elapses during a network partition. It is possible to get a RECONNECTED state after this but you should still consider any locks, etc. as dirty/unstable.checkSessionExpiration如果一定時間內收不到zkserver的任何時間，則認為當前連接已經expire
實例管理
類構造函數
/*** @param ensembleProvider the ensemble provider 連接ipstring* @param sessionTimeoutMs session timeout 就是我們設置的sessiontimeout超時時間* @param connectionTimeoutMs connection timeout 連接超時，這么久還沒有連上就不連了* @param watcher default watcher or null* @param retryPolicy the retry policy to use retryforever*/public CuratorZookeeperClient(EnsembleProvider ensembleProvider, int sessionTimeoutMs, int connectionTimeoutMs, Watcher watcher, RetryPolicy retryPolicy){this(new DefaultZookeeperFactory(), ensembleProvider, sessionTimeoutMs, connectionTimeoutMs, watcher, retryPolicy, false);}狀態轉換時調用ConnectionStateListener
Recipes 場景支持
curator有不同的Recipes執行不同的功能，并且都集成了zookeeper很多底層語句，比如節點選舉，會首先注冊整合path，再注冊和watch選舉znode。
Most Curator recipes will autocreate parent nodes of paths given to the recipe as CreateMode
基本操作
curator采用fluent風格api，提供同步和異步(BackgroundCallback)兩種
監聽watch
addListener加入的監聽器不用重復添加
Zookeeper原生支持通過注冊Watcher來進行事件監聽，但是開發者需要反復注冊(Watcher只能單次注冊單次使用)。Cache是Curator中對事件監聽的包裝，可以看作是對事件監聽的本地緩存視圖，能夠自動為開發者處理反復注冊監聽。Curator提供了三種Watcher(Cache)來監聽結點的變化。可以兼用以下狀態的變化：
- zk掛掉type=CONNECTION_SUSPENDED,，一段時間后type=CONNECTION_LOST
- 重啟zk：type=CONNECTION_RECONNECTED, data=null
- 更新子節點：type=CHILD_UPDATED
- 刪除子節點type=CHILD_REMOVED
實現的recipes
curator提供了各種recipes提供各種功能直接為上層業務使用。
Elections 選舉
- LeaderSelector：只要takeLeadership不退出，當前節點就一直是leader。實際上是用InterProcessMutex做的
- LeaderLatch：一旦選舉出Leader，除非有客戶端掛掉重新觸發選舉，否則不會交出領導權。
locks 鎖
分布式鎖
counters 計數器
由于zk的寫是遞交到leader去寫的，而讀是follower就可以讀，所以不知道這個計數器會不會引起stale read
caches 緩存
有不同級別的緩存，比如node、path、tree。并且它會注冊watcher，如果節點有變更，curator會及時更新cache
Nodes/Watchers
這個主要指創建一些persist node，和與之對應的watcher
Queues 隊列
zookeeper順序節點本身就可以作為隊列使用
事務
略
tech note
- zookeeper有1M的消息大小限制
- zookeeper的children太多會極大的影響性能
- znode太大也會影響性能
- znode太大會導致重啟zkserver耗時10-15分鐘
- zookeeper僅使用內存作為存儲，所以不能存儲太多東西。
- CONNECTED：第一次建立連接成功時收到該事件
- READONLY：標明當前連接是read-only狀態
- SUSPENDED：連接目前斷開了(收到KeeperState.Disconnected事件，也就是說curator目前沒有連接到任何的zk server)，leader選舉、分布式鎖等操作遇到SUSPENED事件應該暫停自己的操作直到重連成功。Curator官方建議把SUSPENDED事件當作完全的連接斷開來處理。意思就是把收到SUSPENDED事件的時候就當作自己注冊的所有臨時節點已經掉了。
- LOST：如下幾種情況會進出LOST事件
- curator收到zkserver發來的EXPIRED事件。
- curator自己關掉當前zookeeper session
- 當curator斷定當前session被zkserver認為已經expired時設置該事件。在Curator 3.x，Curator會有自己的定時器，如果收到SUSPENDED事件一直沒有沒有收到重連成功的事件，超時一定時間（2/3 * session_timeout）。curator會認為當前session已經在server側超時，并進入LOST事件。
- RECONNECTED：重連成功
對于何時進入LOST狀態，curator的建議：
When Curator receives a KeeperState.Disconnected message it changes its state to SUSPENDED (see TN12, errors, etc.). As always, our recommendation is to treat SUSPENDED as a complete connection loss. Exit all locks, leaders, etc. That said, since 3.x, Curator tries to simulate session expiration by starting an internal timer when KeeperState.Disconnected is received. If the timer expires before the connection is repaired, Curator changes its state to LOST and injects a session end into the managed ZooKeeper client connection. The duration of the timer is set to the value of the “negotiated session timeout” by calling ZooKeeper#getSessionTimeout().
The astute reader will realize that setting the timer to the full value of the session timeout may not be the correct value. This is due to the fact that the server closes the connection when 2/3 of a session have already elapsed. Thus, the server may close a session well before Curator’s timer elapses. This is further complicated by the fact that the client has no way of knowing why the connection was closed. There are at least three possible reasons for a client connection to close:
- The server has not received a heartbeat within 2/3 of a session
- The server crashed
- Some kind of general TCP error which causes a connection to fail
In situtation 1, the correct value for Curator’s timer is 1/3 of a session - i.e. Curator should switch to LOST if the connection is not repaired within 1/3 of a session as 2/3 of the session has already lapsed from the server’s point of view. In situations 2 and 3 however, Curator’s timer should be the full value of the session (possibly plus some “slop” value). In truth, there is no way to completely emulate in the client the session timing as managed by the ZooKeeper server. So, again, our recommendation is to treat SUSPENDED as complete connection loss.
curator默認使用100%的session timeout時間作為SUSPENDED到LOST的轉換時間，但是用戶可以根據需求配置為33%的session timeout以滿足上文所說的情況的場景
參考鏈接
總結
以上是生活随笔為你收集整理的zookeeper客户端库curator分析的全部內容，希望文章能夠幫你解決所遇到的問題。
- 上一篇： 一致性协议raft详解（四）：raft在
- 下一篇： 分布式键值系统Amazon Dynamo