Escaping the history expansion character

[Matthew Walker]

From the bash info page:
  Only `\' and `'' may be used to escape the history expansion
  character.

I have tried the following:
  echo '!'   # works fine, displays "!"
  echo "!"   # errors as expected with "event not found"
  echo "\!"  # produces "\!" --- this is my problem

Why is it that "\!" doesn't output just "!"?

For every other scenario "\x" produces "x".  Why does this fail for an 
exclamation mark?

This time I remembered to check the FAQ, so hopefully this isn't a 
stupid question :o)

Matthew