Erratic or Totally Missing eMails with Windows SMTP Server

This week I was doing configuration work on a client's new SharePoint Server 2010 farm to get inbound & outbound mail set-up.  This project had a slightly "one-off" requirement in that their business uses Microsoft Office365 for their employees' email and Office suite.  Ergo, there is no on-premise Exchange or other mail server and instead we needed to set-up SharePoint to route mail via their mail domain inside Office365.  Fortunately, John White has two great blog posts covering this topology (though it references Office365's "predecessor," BPOS):

http://whitepages.unlimitedviz.com/2010/09/integrating-sharepoint-on-premises-with-bpos-and-exchange-online-part-1outbound/

http://whitepages.unlimitedviz.com/2010/10/integrating-sharepoint-on-premises-with-bpos-and-exchange-online-part-2-inbound/

I had the configuration done and had started unit testing the setup.  Outbound email was working fine but as we completed more and more inbound email tests and confirmed the SharePoint site's mail-enabled document library was getting the emails it became apparent that not ALL emails were making it in.  Time to put on the troubleshooting faceshield, crap-waders, leather gloves and get out the proper tools for computer troubleshooting:

 

Spam filter folders/logs showed nothing amiss. Examination of the SMTP log files showed that each test message resulted in the Office365 mailservers connecting to our SMTP server and seemingly delivering the email messages successfully (e.g. 200-series status codes, bytes-transferred relative to the test email's attachments).  No error messages could be found in the SharePoint ULS nor the Windows Event Logs.  Then, looking in the SMTP server's Badmail folder (default location is C:\inetpub\mailroot\Badmail) we discovered where all the missing emails were going.

For each missing message sent, three files existed in the Badmail folder; a .bad, a .bdr, and a .bdp.

This MS KB article describes the three file types in more detail:  http://support.microsoft.com/kb/306043

But looking inside the .bdr files for an explanation gave only the following error information:

Unable to deliver this message because the follow error was encountered:This message is a delivery status notification that cannot be delivered.The specific error code was 0xC00402C7.

Sadly, that is a fairly generic error message and Googling it lead to many false leads (including the KB article above).


Over the course of several days we sent multiple messages from different email addresses (e.g. hotmail, gmail, internal to the client, etc.) and the puzzle only grew.  There seemed to be no absolute pattern to the failures. E.g. attachment v. no attachment, type of attachment, size of attachment, time of day, one sender v. another, one email service v. another, text in the body of the message.  Eventually, though, it started to seem that emails sent via corporate email servers (e.g. our Exchange server) had a higher failure rate.

So here is the short and skinny of it all:  The default setting in the MS SMTP Service (for IIS7.5...and probably all the way back to the first offering of SMTP Server) for the Maximum Hop Count is 15.

You can see that setting at the top of the Advanced Delivery tab as screenshotted in John White's coverage of SMTP configuration in his blog post (see URL above). Ignore his red circle around the Smart host parameter.....

In the old days, that was probably a pretty generous limit.  But, now days with outbound & inbound spam filters, corporate IP trackers, and global Web mail-provider networks (e.g. GMail), it seems it isn't that hard to have your email message passed through 15 machines on its zippity-do-da way around the big ole Internet as it goes to a server across the street from your office.  Depending on the gods-of-TCP/IP-routing and the current sum of the airspeed of all airborne English Sparrows, test messages that seemed to be identical tests resulted in different pass/fail results as some messages took slightly different (i.e. longer routes, they'd exceed the allowed Maximum Hop Count and the SMTP Server would do as configured and shove them into the Badmail folder.

 Raising the Maximum Hop Count to a more generous 99 (max allowed value is 255) solved the erratic-emails problem.

One other thing to note while your in the SMTP configuration is to ensure the message size and session limits on the Messages tab reflect modern email expectations...especially for a SharePoint server that will be accepting documents into a document library via email.  If nothing else, raise the "Limit Message Size To (KB)" parameter from its default value to something with real MB meat - e.g. 51200KB = 50MB (but don't forget that the default IIS & SharePoint file upload limits might still need to be addressed for your document library's needs, as well).

Sadly, the server never seemed to use the email address in the "Send Copy of Non-Delivery Report to:" parameter (aka Postmaster) to send an email notifying a human of the non-deliverable emails during our testing.

Oh, and here is a nice single-page overview sheet of the MS SMTP Server settings: http://www.tech-faq.com/understanding-and-managing-smtp-virtual-servers.html

Determine Which Version of SharePoint You're Running

This isn't anything new, just a summation of the best sources I've found for different ways to determine:

1) Am I running WSS v3 (Windows SharePoint Services v3) versus MOSS 2007 (Microsoft Office SharePoint Server 2007).
2) Which version / build / hotfix / CU / SP of the respective software am I running?
3) Which edition of MOSS 2007 (standard or enterprise) am I running?

So try these references:

http://insomniacgeek.com/how-to-determine-the-installed-sharepoint-version/


http://www.heathersolomon.com/blog/archive/2007/05/11/SharePoint-Tip--How-to-tell-if-your-WSS-site.aspx

http://www.sharepointoverflow.com/questions/1395/determine-which-edition-of-sharepoint-is-installed

And if you have access to the SharePoint server's SQL Server:

SELECT [VersionId]



,[Version]


,[Id]


,[UserName]


,[TimeStamp]


,[FinalizeTimeStamp]


,[Mode]


,[ModeStack]


,[Updates]


,[Notes]


FROM [SharePoint_Config].[dbo].[Versions]

Increasing the Maximum File Upload Size in WSS v3/MOSS 2007

So there are three related issues when it comes to trying to allow larger files to be upload to a SharePoint WSS v3 or MOSS 2007 system:

1    1)      Several settings that specify the maximum file size allowed for uploading. We’ll call these settings “Max File Upload Size” settings though the actual property/attribute/setting label may be different in each area you’ll configure. Details are below….

2    2)      Several settings that specify the maximum timeout period (usually in seconds) for either the upload period or the http session.  We’ll call these settings the “timeout” settings though the actual property/attribute/setting label may be different in each area you’ll configure. Details are below….

3    3)      Several settings that are related to the SharePoint site/document indexer for the search shared service provider.  WE’ll call these settings “search crawler” settings though the actual property/attribute/setting label may be different in each area you’ll configure. Details are below….

The relationship between these items is as follows:  the bigger the document/file that you want to upload, the longer it will take, especially for users on low-bandwidth connections.  First, to control how much stuff can be shoved into the SharePoint server, WSS3/MOSS2007 (and 2010, too!) ship with caps on the uploadable size.  When we raise that value we increase the odds that some users will appear to the server to have “gone missing” while uploading large (i.e. slowly) files.  Thus the need to raise the timeout settings.  Also, because of the long lags during file uploads, the user’s security authentication may expire during the time they wait for the file to upload.  Increasing the security validation period can prevent forcing them to login again to see the result of their upload attempt.  Obviously there are security ramifications for changing this (at the Web Application’s zone. Note that one could forego these timeout config changes until/if users start complaining about failed uploads.  The (minor) ramifications of these changes to timeout settings are beyond the scope of this doc so for now we are just going to assume it’s easier to be proactive and do the timeout settings at the same time as the max file upload size setting mods, but do note that the timeout setting is optional until users have problems with the upload of the now allowed larger limit.  Finally, there is typically much desire for uploaded documents to be crawled by the SharePoint Search indexer so documents can be located by users using the search tools in SharePoint.  But just as there are problems introduced by allowing big files to be uploaded, similar problems are introduced by not putting some sort of cap on the size of a file that the crawler/indexer will try to “chew.”  16MB is the limit out of the box for WSS3/MOSS20007.  Often times (e.g. video file uploads, CAD drawings), it is not necessary or desired to raise this limit. But if the documents are text based, you may want to raise this limit to allow the data in the file beyond the 16MB point to be indexed.  I have had problems with raising the value though.  Let me know if you get it to work. J  For now, I have ended up leaving it at the default value and been ok with the note that gets put in the ShPt log file about a too large file being unable to be indexed.

All the changes we are going to make can easily be rolled back AS LONG AS YOU MAKE BACKUPS OF THE ITEMS PRIOR TO MODIFYING THEM – i.e. create a copy of *.config files and rename them (e.g. “Web.Config” copied to become “Web-ORIG as of yyyy-dd-mm hhmm.configOLD”), backup a registry node to a reg file prior to making mods, record ShPt settings in the Central Admin or SSP Search GUI with screenshots stored in a WordPad file.  Also be aware that there are typically SEVERAL Web.config files on a SharePoint server AND there is a difference between a Web.config and an Application.config file.  You may need to modify several Web.Config files and the exact path cannot be provided as it changes depending on how the ShPt server has been configured.  Ergo, when you see a path that mentions a virtual directory (e.g.) you must do the translating for your server AND make sure you alter the proper virtual directory for the ShPt site(s) you  are making the mods for.  Go slow, take notes of before/after and THINK!!!! I.e. the ass you save may be your own.

Max File Upload Size (MFUS) limit mods:

1    1)      First, calculate the value in bytes the size you want to raise the limit to.  E.g. 500MB = (500 * 1028 * 1024) = 524288000 Bytes.

2    2)      Open up Central Admin and for each Web application that contains the site(s) you are increasing the MFUS limit, open the “Web Application General  Settings” page. Increase the “Maximum Upload Size” setting to your desired MFUS limit in MB.

3    3)      Using the integer amount in bytes for your new MUFS limit, add the maxAllowedContentLength attribute and value in the Web.config for each virtual directory that maps to a site, zone, or Web application (extended or base WA) in SharePoint, locate the Web.config file.   A typical (EXAMPLE) path for such a Web.config looks like:

a.      C:\inetpub\wwwroot\wss\VirtualDirectories\80

b.     C:\inetpub\wwwroot\wss\VirtualDirectories\11001

e.g. this... web-ORIG as of 2010-08-02.config

               </configSections>
               <SharePoint>

 

…gets turned into this...

  </configSections>
  <system.webServer>
       <security>
  <requestFiltering>
  <requestLimits maxAllowedContentLength="524288000"/>
  </requestFiltering>
  </security>
  </system.webServer>
  <SharePoint>

Note that if doing timeout setting mods, you will return to this file (but I’m breaking this into functional sections for clarity of what affects what).

4    4)      If you are running on Windows Server 2008 (i.e. Internet Information Server 7.0) or R2 (i.e. IIS7.5), the following (optional) proactive change may  also need to be made.

a.      Add the same value for the maxAllowedContentLength attribute IF IT EXISTS.  The attribute, its element (or even the whole section ) may be missing. If not, you can create the attribute or make it look like <requestLimits maxAllowedContentLength="524288000"/> for the element and attribute (and any missing parent elements as shown in the example above) to the application.config file, located at:

                                                    i.     %windir%\system32\inetsrv\config\applicationhost.config

Timeout mods:

1    1)      If desired, to prevent user’s from having to re-authenticate after a long file upload, increase the time that a logged-in user is allowed to sit “idle” by going to ShPt Central Admin and on the appropriate Web Application(s)’s “Web Application General Settings” page (i.e. http://YourSharePointServer[:CAport]/_admin/vsgeneralsettings.aspx ) and increasing the “Web Page Security Validation” settings “Security validation expires’ value. Default is 30 minutes.  60 has generally worked for us in the past.

2    2)      for each  Web site (aka Virtual Server) in IIS Manager , change the connection timout limit (in Seconds) as appropriate for you new MFUS limit. E.g. default "120" (i.e. 2 minutes) to "3600" (i.e. 60 minutes). I.e. …

a.      Click Start, point to All Programs, point to Administrative Tools, and then click Internet Information Services (IIS) Manager.

b.      Right-click the virtual server that you want to configure, and then click Properties.

c.      Click the Web Site tab. Under Connections, type the number of seconds that you want in the Connection time-out box, and then click OK.

3    3)      Modify the “layouts” Web.config (i.e.  "C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\12\TEMPLATE\LAYOUTS\web.config" by changing/adding the executionTimeout attribute and a max allowed value of 999999 in the Web.config’s httpRuntime element like so...

  <location path="upload.aspx">
  <system.web>
  <httpRuntime maxRequestLength="2097151" />
    </system.web>
  </location>

to this...

  <location path="upload.aspx"> 
  <system.web> 
    <httpRuntime executionTimeout="999999" maxRequestLength="2097151" /> 
    </system.web> 
 </location>

Note that the value in maxRequestLength will probably be different.  This value seems to be overridden by the <system.webServer> <security> <requestFiltering> <requestLimits maxRequestLength<  attribute’s value.  If you don’t find that to be true, you can change it to match your MUFS limit but IN Kilobytes, NOT Bytes. E.g.  524288000 Bytes  = 512000KB.

4    4)      add the executionTimeout attribute and a max allowed value of 999999 in the Web.config for each virtual directory that maps to a site, zone, or Web application (extended or base WA) in SharePoint, locate the Web.config file.   A typical (EXAMPLE) path for such a Web.config looks like:

a.      C:\inetpub\wwwroot\wss\VirtualDirectories\80

b.     C:\inetpub\wwwroot\wss\VirtualDirectories\11001

e.g. this...

<httpRuntime maxRequestLength="51200" />

becomes this...

<httpRuntime executionTimeout="999999" maxRequestLength="51200" />

Note that the value in maxRequestLength will probably be different.  This value seems to be overridden by the    attribute’s value.  If you don’t find that to be true, you can change it to match your MUFS limit but IN Kilobytes, NOT Bytes. E.g.  524288000 Bytes  = 512000KB.