What is the best way to monitor the number of connections a ZooKeeper server is servicing?
Well – I think the correct answer to that depends on what type of brain you are. For example if you are a programmer – you will probably be more interested in the number of watches or subscriptions on a per key basis. If you are an operations brain then you are probably more interested in knowing about the number of connections coming through the service as a whole – to validate that the service is generally healthy.
If you run your ZooKeeper ensemble in an environment where collectd is running or supported then there is actually a really fast and easy solution for you.
Use the collectd plugin called TCPConns. http://collectd.org/wiki/index.php/Plugin:TCPConns
Using this plugin you can get very valuable data on the number of connections that are active on your server. In this case this simple single server installation of ZooKeeper is able to show that it can handle 1.5K connections without any issue.
Do bear in mind that this is simply a reporting based monitor. This will not actual display any information on ceilings or performance caps. Depending on your use of ZooKeeper – 1K connections may be too many – or maybe not…