This will set the variable $hosts to an array containing
all the hosts in ~/.ssh/known_hosts and ~/.ssh/known_hosts2
that do not start with a digit.

hosts=(${${${${(f)"$(<$HOME/.ssh/known_hosts <$HOME/.ssh/known_hosts2)"}:#[0-9]*}%%\ *}%%,*})

This will set the variable $hosts to an array containing
all the hosts in ~/.ssh/known_hosts that do not start with
a digit.

hosts=(${${${${(f)"$(<$HOME/.ssh/known_hosts)"}:#[0-9]*}%%\ *}%%,*})

If you are using the new completion system, you can then place

zstyle ':completion:*:hosts' hosts $hosts

after compinit is autoloaded to use those anywhere hosts would be
completed, or

zstyle ':completion:*:complete:ssh:*:hosts' hosts $hosts

to use those hosts to complete only ssh.

An explanation of the $hosts assignment, written by Peter Stephenson,
follows.

$(<$HOME/.ssh/known_hosts)

is a standard substitution: it simply takes the file and sticks it onto the
command line at that point.

"$(<$HOME/.ssh/known_hosts)"

Now it's quoted, it doesn't do word splitting; we have the complete file as
one word.  From now on, we do nested substitutions: you just have to
remember that ${${...}}, or ${${...}}, essentially does nothing but an
ordinary parameter expansion --- the whole point is the extra bits tacked
on with each extra set of braces.  For example, we're now going to do

${(f)"$(<$HOME/.ssh/known_hosts)"}

so we get the same answer, but with the effect of putting the (f) flag at
the start, which splits the result of that into lines.  So we now have the
entire file as an array, one line per element.

${${(f)"$(<$HOME/.ssh/known_hosts)"}:#[0-9]*}
(Clint says the ^ shouldn't be there) says take the array elements (= lines
of the original file) which completely match [0-9]*, i.e. elements
beginning with a digit, and remove them, which is what ${...:#...} is for. 

${${${(f)"$(<$HOME/.ssh/known_hosts)"}:#[0-9]*}%%\ *}

takes the result of that, and strips off from the end the largest pattern
matching ' *', i.e. a space followed by anything else, in other words it   
leaves the largest initial string with no whitespace, which is a hostname
(this is a standard ${...%%...} which even ordinary shells do, although not
nested).

${${${${(f)"$(<$HOME/.ssh/known_hosts)"}:#[0-9]*}%%\ *}%%,*}

does another strip at the end, this time for everything from the first
comma on.  If there wasn't a comma, nothing changes.  You could have
combined the last two as ${...%%[[:blank:],]*}, or something.