codehaus


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Help in understanding strange cassandra CPU usage


Sounds like over time you’re ending to doing something odd - maybe you’re leaking cql connections or something and it gets more and more intensive to manage them until you invoke the breaker, then it drops

Will probably take someone going through a heap dump to really understand what’s going on, which is unfortunate because it’s a fair amount of effort. 

-- 
Jeff Jirsa


> On Dec 9, 2018, at 2:09 AM, Devaki, Srinivas <me@eightnoteight.space> wrote:
> 
> Hi Guys,
> 
> Since the start of our org, cassandra used to be a SPOF, due to recent priorities we changed our code base so that cassandra won't be SPOF anymore, and during that process we made a kill switch within the code(PHP), this kill switch would ensure that no connection is made to the cassandra for any queries.
> 
> During the testing phase of kill switch we have identified a strange behaviour that CPU and Load Average would go down from 400%(cpu), 14-20(load on a 16 core machine) to 20%(cpu), 2-3(load)
> 
> and even if the kill switch is activated only for 30 secs, then cpu would go down from 400 to 20, and maintain at 20% for atleast 24 hrs before it starts to increase back to 400 and stay consistent from then. and this is for all the nodes but not just a few.
> 
> Details:
> Cassandra Version: 2.2.4
> Number of Nodes: 8
> AWS Instance Type: c4.4xlarge
> Number of Open Files: 30k to 50k (depending on number of auto scaled php nodes)
> 
> Would be grateful for any explanation regarding this strange behaviour
> 
> Thanks & Regards
> Srinivas Devaki
> SRE/SDE at Zomato
> 
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@xxxxxxxxxxxxxxxxxxxx
For additional commands, e-mail: user-help@xxxxxxxxxxxxxxxxxxxx