You can install Standard Applications using the standard windows tools, as the USB drive appears to be a normal drive to Windows. Note: Apps installed this way will probably only work on the PC you installed them on.
Also, some Applications (such as Firefox) are available in "portable" versions. These can be copied to any device and just run straight from the device. They will run from the device, but you may not be able to store preferences/configuration info.
Also, if your USB drive has a U3 logo on it, you'll be able to get a whole load of software than you can install on the device, and it will function exactly like a normal Windows Appplication.
http://www.u3.com/
Finally, yes, as long as your computer supports booting from USB, you can install an operating system on the device. There are various versions of Linux that will boot from USB stick, and you can install a heavily cut down version of Windows using the tools and instructions at
http://www.nu2.nu/pebuilder/