In "Scandal in Belgravia" Mycroft said it to John in the cafe, near the end of the episode. They try to figure out Sherlock, with his brain of a scientist and a heart of a philosopher (or something like that), but a decision to become a detective. Mycroft added, "initially, he wanted to be a pirate". :)

Sherlock and a parrot! :D

EDIT
Oh, no parrot. But excentric as he is, he can have anything :)