To be perfectly honest, my personal wish is that the term "Zionism" be consigned to history. I think it ceased to have much material grip in 1948, when its historical project was accomplished. What has it meant since then? Support for the continued existence of the Israeli state, in any form? Support for its continued existence as a Jewish state? Support for its colonial project in the Palestinian territories? Some combination of all of these things? Something else entirely?
Much of what is spoken about today as "Zionism" (by people who are actually trying to make the word means something, rather than crypto-antisemites simply using it as a code-word for "Jews") would be more accurately termed Israeli-Jewish chauvinism or ultra-nationalism. Those terms are clunkier, perhaps, but they also avoid the flattening out of a complex historical category and the provocative and fruitless affront to the complex, historically-developed identity of (probably) most Jews alive.
Imagine a Jew who is for two states, who opposes the occupation, who wants to end settlement building, who supports civil rights and equality for Israeli Arabs, etc., but for reasons of historical affinity sees themselves, in however loose a sense, as a "Zionist" (as in, they support the existence of the Israeli state and would "defend" that existence against those who oppose it). There are many such Jews in Britain, and I imagine in America too. Theirs is a "Zionism" one hopes would "wither away", as it were, "in struggle". But if the far left they encounter maintains an "absolute anti-Zionism" that insists they immediately jettison this complex aspect of their historically-developed identity, and tells them, "you might think you're a Zionist, but you're not really: Zionism is colonialism, Zionism is racism! Unless you accept that you are not, in fact, a Zionist, you must be a colonialist and a racist!", then I think they are more likely to end up retreating behind the cordons of ethno-cultural identity that growing beyond them.
You're right, I think, that context, both geographical and historical, matters a great deal. In Israel itself I suspect I'd have less of a problem with straightforwardly describing myself as "anti-Zionist". I also have no problem identifying as a "pre-WW2 anti-Zionist". But in the contemporary context in Britain, where something called or understood as "Zionism" both a) forms an integral aspect of the historically-developed identity of most Jews, based on inherited experiences and cultural memories of oppression, persecution, and most crucially, the Holocaust, and b) is perhaps the key term used by antisemites, on both the right and the left, to attack not just the Israeli state or government but all Jews, and who make "anti-Zionism" an exceptionalised point of departure for their politics in a way they don't with the nationalism of any other group; then I prefer "non-Zionist".