When I was 20 I’d have been gung ho for removing it—the Civil War was about slavery and I don’t want to honor that.
I’m not 20 any more. The Confederate powers-that-were wanted slavery, but seriously, did the soldiers actually say "Ma, I’m going off to fight the Yankees so our neighbor Jeff can buy himself a slave if he ever saves up enough money"?
One of the points of the war in the first place was "these Southerners are part of us." You can get away with damning the soldiers because of the cause of their rulers, provided you don’t want to be part of the same tribe with them afterwards.
I get the impression that those calling for the universal destruction of all such monuments (*) aren’t interested in living together afterwards.
(*) Though you can melt Jeff Davis down for scrap and I won’t mind.