Including the year CJ5, engine, and carburetor in your question would help us to better answer your question.
The CJ5 was fitted with several different Carter carbs, on the (F-134), (232), (258), engines, also the Rochester, (225) and MC2100, (304).
[/quote
I think the float valve is sticking.
There is a good chance the synthetic tipped needle is dirty and isn't seating. Outside chance there could be a crack in the carb but it’s not typical.
Any easy remedy for that, or am I forced into rebuilding the carb?"
If it’s the needle the easiest remedy would be to tap the side of the float bowl with the handle end of a screwdriver. However, I would only do this to verify the needle wasn't seating. I would suggest removing the air horn (4 screws) take a good look to make sure nothing is amiss then spry some carb cleaner on the needle and seat assembly.
I hope this helps.
