A story on Rocketnews 24 gives a rundown of TripAdvisor's "20 Best Castles" in Japan, and I'm surprised to find I've visited even as many as four of them. While four isn't an impressive number at all, it's the inclusion of Kakegawa Castle in the number 20 spot that makes it possible. I went to Kakegawa Castle on a date years and years ago. At that point, the only castles I'd visited were Hamamatsu Castle, Nijo-jo and Nagoya Castle. For a castle in a smaller town, Kakegawa Castle impressed me a lot more than Hamamatsu-jo. I'm struggling to remember the specifics, but I believe I liked the gardens and the general layout more than the one in Hamamatsu.
Also, I enjoyed the way the girl I was with strode up to the main gate-- which was closed-- pounded on it and shouted, "Konbanwa!" as if we were expected guests.
I've since been to Edo Castle twice, and I'm very interested in visiting Inuyama Castle now. It's not far and it's an original rather than a concrete and steel replica.