By allowing selection in a UI can lead to a huge impact in what you expect — not really rocket science… In an app that only allows for the addition (+undo for removal) your mental model is fairly simple. The moment you add the concept of selection all of a sudden your expectations jump exponentially. Think of the differences between Paint and Draw. In a paint app you don’t really think about a whole class of operations, but in your draw app all of a sudden you feel like you could move where a brush stroke is.
It’s really funny how drawing a little blue box around something to indicate selection, has such an enormous change.