The problem is mainly with firefox, who don't care about the font-size.
I've mainly found my information here : http://groups.google.com/group/mozilla.dev.tech.svg/browse_thread/thread/1d574c2690e37c7b
I know it's already a bug (827), but I don't know how to make it up ! Sorry about that...
Thanks if you can do something !
[ellson] Perhaps you can use -Tsvg:cairo instead? Cairo renders text as glyphs, and so totally ignores firefox's overrides. Unfortunately, cairo does not yet provide support for anchors, so the images are unclickable.
Some days you just can't win