What this should tell you is that I do 99% of my world building within my outline.

Interesting--it usually happens the other way around in my case. My worldbuilding notes usually contain bits and pieces that eventually produce the plot when I join them up together and let them ferment for a while.