If it’s just off the coast as in “territorial waters”, there’s the concept of innocent passage in international law (https://en.wikipedia.org/wiki/Innocent_passage) - although the US are not part of this UN convention. If the criteria for innocent passage are not met, “the coastal State may take the necessary steps in its territorial sea to prevent passage”.

Edit: I guess in more tricky situations e.g. the ship isn’t obviously armed or spying, it comes down to questions similar to probable cause.