I have planned out our org's cassandra upgrade plan and want to make sure if it seems fine.
Details Existing Cluster:
* Cassandra 2.2.4
* 8 nodes with 32G ram and 12G max heap allocated to cassandra
* 4 nodes in each rack
1. Ensured all clients to use LOCAL_* consistency levels and all traffic to "old" dc
2. Add new cluster as "new" dc with cassandra 2.2.4
2.1 update conf on all nodes in "old" dc
2.2 rolling restart the "old" dc
3. Alter tables with similar replication factor on the "new" dc
4. cassandra repair on all nodes in "new" dc
5. upgrade each node in "new" dc to cassandra 3.11.3 (and upgradesstables)
6. switch all clients to connect to new cluster
7. repair all new nodes once more
8. alter tables to replication only on new dc
9. remove "old" dc
and I have some doubts on the same plan
D1. can i just join 3.11.3 cluster as "new" dc in the 2.2.4 cluster?
D2. how does rolling upgrade work, as in within the same cluster how can 2 versions coexist?
Will be grateful if you could review this plan.
PS: following this plan to ensure that I can revert back to old behaviour at any step
SRE/SDE at Zomato