I had the same problem and after reading these comments, I kind of like the approach you take. Still it is not intuitive, so maybe you should use something like Cmd + click for that.