You can download the software such as openoffice.org and many commercial products (quicken, microsoft office, etc) allow you to download the software rather than purchase the CD (or you can optionally download it immediately and purchase a CD that will be sent later from some electronic storefronts).

To purchase an external USB drive you are talking $40-$60 (there are more expensive ones of course that have dual layer DVD writing, or lightscribe). Check the big-box stores or big online stores.