Do I remember it wrong that the ring has many downsides?
This is one of those "it's up to interpretation" things, since Tolkien tended to be pretty vague about what his magic
does.
However, the way I interpret it is that the Ring
itself has only one drawback (the fact that it's sentient, malicious, and desires to be reunited with Sauron). Everything else is actually just a side effect of its core temptation (which is enhancing your natural abilities and tendencies, not invisibility
per se — remember that all three of the characters that we see turn invisible when they put it on are hobbits, who are noted for being naturally stealthy). If you actually look at the story, there's
surprisingly little fighting over the Ring itself (outside of the central conflict, which can be summarized as "Sauron would like the bling that he shoved a good portion of his natural power into, please and thank you") — you essentially just have Smeagol killing Deagol and Boromir demanding the Ring from Frodo. There are some other minor instances (from Bilbo, Frodo, and Smeagol), but in all of those cases they each had the Ring for
years... but those are the only two instances of people fighting over it.
The Ring is fundamentally
about the temptation of evil/power, which doesn't really
work if the drawbacks come into play immediately. Heck, it's important to consider that the mechanic is called
tempted by the Ring and not, say,
possession of the Ring. It's offering power so you do stupid stuff and take it closer to Sauron.