[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Re: How to gracefully decommission a highly loaded node?

On Mon, Dec 17, 2018 at 11:44 AM Riccardo Ferrari <ferrarir@xxxxxxxxx> wrote:
I am having "the same" issue.
One of my nodes seems to have some hardware struggle, out of 6 nodes (same instance size) this one is likely to be makred down, it consntantly compacting, high system load, it's just a big pain.

My idea was to add nodes and decommission all the one running on old hardware (m1.xlarge), however this very specific "bad" node is causing trouble to the whole cluster and decided to decommission it first.

The node is simply stuck in "LEAVING" - Not sending any stream. I already have disabled binary and autocompactions and tried to restart the decommission process couple of times with no luck.
Any suggestions?
assassinate vs removenode?
Any tuning that could help?

If it's stuck that badly, then I would consider it lost and just do replacenode.  Hope it's not too late if you started to decommission?