Drew McLellan has posted a very good run down of how to improve blog (and general website) performance. It covers caching, outsourcing of feeds, and using something like Amazon’s S3 to host large files, just as an example.
Heck, there are (amazingly enough) 24 different articles on the 24ways site that deal with all sorts of web design quirks. Very good stuff there, check it out.