Windows Server reserves half of its available RAM for the operating system. Of the remaining RAM, by default IIS will only allow 60% to be allocated to applications (in v1.0 I think it was actually 40%, in later versions its more generous). This setting is configurable, and can be set in your machine.config of the relevant framework path (%system% . I recommend not reserving more than 80%.
There is a ton to do here, and more than I can address in one post. Benchmarking these settings in development is a necessity, so do the needful and check out this overview here: http://msdn.microsoft.com/en-us/library/ff647813.aspx