Just to wrap this up. I was never able to reproduce the error on my test environment, so it is possible it is caused with an environmental conflict. However, putting the assembly in the GAC did solve my problem on the production environment and I was able to verify that very clearly.
Around 110 active web sites using the DLL caused an "Unable to initialize assembly error". When I moved the assembly into the GAC, more than 200 sites came up without any problem.
So for now my problem seems solved, thanks for your help Robert!
Joshua
http://geographicalmedia.org