Currently all video annotations are unlisted, but public. In other words, if someone knows the exact URL to your video then it will be visible.

We will be releasing a premium plan to allow for private annotations in the near future (mid-June).