Customer Questions – Email to SharePoint List Scale, Managing Inbound Email, and Migration

Great turn out for the webinar turned into so many questions that I couldn’t answer them all.

I’ve started sharing further detailed information on the information shared on the webinar on the OfflineSharePoint.com SharePoint blog on Email Management with SharePoint.

If you are interested in seeing a replay of the webinar, you can view the webinar "Demystifying Email Management with SharePoint."

Here are the questions I answered plus the ones I didn’t get to.  As the Colligo ones are answered I’ll point to them from here as well.

Q) How do I use SharePoint to replace my PF servers when they have replication and SharePoint doesn’t?

Replication is a feature in Public Folders, and not in SharePoint.  Although SharePoint is often centralized or in regional hubs many companies focus on optimized access to the servers and go without replication.  Others opt to purchase third party software from any number of partners from Syntergy to Infonic or others.  I did a post a while back on Global WAN Solution SharePoint partners that include replication.  Note the Colligo the host for the webinar does fit in the the category of offline cache and sync in the background when online.

I actually used this tool when trying to get the audio to Colligo.  I had 130MB Of content, and each file was about 30MB.  Obviously email isn’t the right answer.  I wanted to get it up to SharePoint.  A multi file upload would fail since it won’t let you upload more than 50MB at a time, and individual upload would take a ton of time since I’d have to do it serially.  Using Colligo Contributor I got the audio up with a couple of clicks and was able to walk away.  It was a great solution for me.

Q) Where can I get these whitepapers you mentioned? And the exchange blog links? 
All of the resources are all in the debunking email management with SharePoint post.

Q) Does SharePoint finally overlap in functionality with Public Folders in the next version?  How?

I can’t speak to the next version beyond what is already known.  I don’t think we can make any assumptions at this point.

Q) When should I migrate?  When will Microsoft have supported migration tools?

If you are on Lotus Notes, there are a bunch of tools in the Lotus Transport Suite.  If you are in Public Folders and looking to move to SharePoint, I included some links to migration tools and information in Debunking email management with SharePoint.

Q) How do you prevent some sites from using incoming email? I thought if you turned it on at the farm level, it enabled it for all sites.

It is off by default.  First you do configure incoming email in central admin.  If you never give the SharePoint service account the proper creation rights to an AD OU, but setup connections manually then you won’t have this issue.  If you want to selectively enable it choose . 

When incoming email has been enabled you will see in list settings the incoming email settings link.  Click on it, and you’ll see the ability to "allow this list to receive e-mail."  Unfortunately if in central admin you enable it for one you’ll see this option on all lists, which could be a support issue.

When configuring the email you choose

Here’s the best references on configuring the inbound email (Thanks Steve Smith Combined Knowledge)

Q) Is that true – EMLs cannot be opened in Outlook?
.eml is commonly an Outlook express format.  Here’s a KB on problems with associations and using Outlook express to open it.  http://support.microsoft.com/kb/312355

Other interesting results:

*.Eml *.Email are essentially plain text SMTP/MIME messages, while *.msg is a binary MAPI message [built on COM/structure-storage].  FYI there is an ifilter for .eml (reading text based objects in the box)

Well, at home I use Outlook Express and Express will store emails in the .EML format which is just plain text. The Outlook .MSG file format however is a binary format

Here’s steps for importing .eml into Outlook using an Outlook express trick.

http://bitdaddys.com/Import-EML-To-Outlook.html

Q) Are there any issues with interfacing SharePoint with PostPath?

I haven’t tried, but it’s possible if it uses web services to connect to the data with BDC from the SharePoint side.  It should be ADO.NET compliant.  If going the other way, all of the SharePoint lists are exposed as web services.  I’d need more info and likely need to look at the data source and know how you’re trying to integrate them.

Q) Is it possible to send email from SPS – this is a normal action for project mailboxes (project document controllers)

Send mail from SharePoint as in outbound?  Yes, through alerts and workflows and various timer jobs.  Using the manage users interface to send mail will open your default email client.

Q) In a list with, say, 10K items and you configure it to view 100 items per page, will you still experience performance degradation on the first page of the All Items view?

First page if I understand you correctly is about the "Default" view.  I’d say make sure that allitems is not the default view then it wouldn’t be the first page.  So let me start with the facts…

Allitems displays allitems.  You can modify that view and limit the number of items, but based on the filtering this query could still be poor performing based on the filtering and indexed columns.  Note the second option to limit the total number of items returned.  This will give you your performance back.  The batches of 100 per page only helps some, but the query is still retrieving all items on each page and then determining what to display.  (Yep it’s inefficient.  Hopefully we’ll see some changes.)

 

These two questions sound like Bob Fox questions…
Q) Does Joel drink anything else besides Pineapple Juice?

Yes, but sprite gets old and makes the stomach expand when you’re trying to keep up with the rounds.  In Europe I prefer to drink Orangina.  Good stuff.
Q) When can we expect Joel to do a Fountain performance again?

LOL.  I think it would take getting Joel and Bob Fox back together and getting them in the right mood.  Probably wouldn’t take too many pineapple juices 🙂
Q) how would you propose to move current folders and mail items into SharePoint now

There are a ton of migration tools, but let me start with someone trying to phase it out over 3 years…
I’d say essentially turn off PF as a service essentially make the permissions read only. Then turn on date archiving. Don’t worry about moving anything. And handle all NET NEW requests with SharePoint, if you have a DL archiving service you’ll need to work out that situation and make some tradeoffs one way or the other.

If you want to keep the data that’s in PF and are cool with migrating to SharePoint, you can use Quest Public Folder migrator for SharePoint, Tsunami, Avepoint, etc… or the codeplex.com/pfmigrator or look at the Lotus Transport Suite, their is some stuff on PF to SharePoint. 

Q) What size attachment should consider max size for email enabled lists?

Microsoft Exchange limits mail sizes of 10MB to encourage people to save on SharePoint.  It use to be at 2MB, but then you’d get funny things happening in the extranet.  SharePoint’s default 50MB size should be fine.  I wouldn’t recommend changing the size upwards for SharePoint, I have no problem with SharePoint at 100MB or even some larger for people doing CAD drawings and the like, but special timeouts need to be setup and BW and Latency have to be setup properly to achieve that.  I wouldn’t suggest pushing files over 100MB through email.

Q) I have heard that lists will support direct SQL tables. Will that change the lists on limits?

Bill was speculating.  Let’s wait to see what it looks like. 

Q) I think one of Joel’s fundamental points is that some content belongs in SharePoint and some belongs in email. How does an individual decide which email he/she should move to SharePoint? Or maybe I am asking what an enterprise strategy for getting these types of content into the right places might look like. "

I think Managed Folders plus training is the way to get specific scenarios like legal archiving to be better managed.  For collaboration scenarios, I think it’s training and cultural… People need to learn not to send the files in email, but to send links.  That’s a very cultural experience and starts with your SharePoint User Champions.  Also saving emails to SharePoint requires people to see the benefit.  Colligo Outlook-Add in is one of those third party things that could help encourage it, but even then people need to see the benefits.

Q) Joel: Where should I allow users to "dump" email data? Users at my firm insist on keeping everything and we hate .PST files. I was hoping to replace them with sharepoint but you worried me with scalability concerns."

You need to be careful here.  It does require planning to scale to the millions of items.  If you decide to keep everything in SharePoint, you should decide how you’ll setup a structure that will scale.  Don’t put more than 5 million items in one list, even with special views for example.  That will push you way too far.  If you can design your archiving across site collections and across databases you will be able to scale and use search for retrieval.  Make sure you have all the right ifilters based on how it’s being saved.  I’d recommend testing to where you plan it to reach.  You may want to consider a partner like Knowledge Lake.  Owen Allen may have some other ideas on partners.

Q) Joel, what is the advice for standardizing naming convention for email enabled contacts ?

Workflow.  I’d suggest that when users want email enabled lists they should fill out a form to provide business justification, and IT (whether automated through workflows or manual) would setup the necessary steps.

Q) I have always felt that one of the drawbacks to using SharePoint for email is the proliferation of copies of the attachments. Would you care to comment on this?

It’s a great point.  That’s actually the problem with email.  Files get sent back and forth full file.  I think corporate culture is what needs to change here.  People need to be incented to save the file to SharePoint and send links in email, especially on the Intranet.  You do have the option of not saving the attachments that get sent to SharePoint, but you’d need to understand the scenario…  Maybe this would encourage people to use links?

Q) When you say don’t return too many items in a view, are you referring to how many are actually shown on the page using paging, or are you referring to the whole result set? Right now we have a AllItems view that includes over 2000 items, but they are displayed on the page only 100 at a time."

You should worry about the pagination as well.  The view that returns more than 2000 items will query all items for the pagination even if it displays 100 at a time.  Performance will get worse.  The efficiency of the query is important.

Q) Can you briefly mention the pros and cons to enabling SDMS (SharePoint Directory Management Services)?

Here’s an article on Incoming Email on Tech Target on doing it, different than Steve Smith’s whitepaper referred to above.

Pros:

Self Service on email enabled lists, timer job works through the Directory Management services to setup the proper objects as the users request them through list settings and enable their lists for incoming email.

Cons:

Unknown quanity of contact objects getting created in an OU. 

No naming convention enforcement

It’s the contact objects that bother me, seeing them when I’m trying to find groups in AD.  Which is the right group when there is five with similar names and no naming convention?

I’m a bit worried about lifecycle as well.  How do I know what ones are in use, which ones are trash.  How do you get usage information on a contact object?  How do you manage a few thousand?  The SharePoint tools just aren’t there.

Q) Great article on SP scalability you might want to mention: Using Microsoft® Office SharePoint® Server to Implement a Larger Scale Content Storage Scenario with Rapid Search Availability. http://go.microsoft.com/fwlink/?LinkID=120891"

It is a great article.  I included it in my follow up resources post, but didn’t get a chance to mention it on the call.  I’m just trying to encourage people to be cautious and aware as they scale.  Some objects scale well (Site collections), others require lots of care and feeding (like lists).

Q) Can you explain the point about searching archived distribution through sharepoint. How did MS do this?

Steve Smith has another article on Crawling Exchange 2003 and Exchange 2007 PFs.  If you are using MOSS or even SharePoint Search Express you’ll find there are options to configure for indexing Exchange/ Public Folders as a content source (the protocol handlers are built in).

Q) Is performance impacted if the view dataset is large (e.g. 5000)but paging is used to limit to 100 items/page?

Yes, see other related questions above.  Query performance is the problem.
Q) What do you mean by "queries" in SharePoint? "

It’s the SQL Query where the Web Front end Requests data from the Backend that I’m referring to.
Q) In the future, do you think Sharepoint will evolve into a Business Intelligence Application ?

Yes.  If you look at the marketing slides, there is clearly a piece of pie for BI.  It is not Cognos, but it is a ton easier and has some of that functionality.  I see BI as one of the big opportunities for growth.  Also a great place to hang your hat if you’re looking to get more money as a SharePoint person.

Q) Can IT monitor the growth of a SharePoint list with SCOM?

Not with any out of the box ones.  We should get Bill Baer or Corey to explain how they monitor list scalability.

Q) any comments on using scanning to email-enabled list?
A) With the right kind of governance, it’s a great scenario. File size and list scalability is what you need to be aware of an plan for.  I know Knowledge Lake has done some work in this area.  See the whitepaper they did with MCS on large scale content storage.

Q) in outlook there is connection between contact list and the mail box and automatically all the mail correspondence. how you do it in moss

Not sure I understand this question fully.  SharePoint isn’t designed as a mail client.  You can add your SharePoint calendar you can connect to it in Outlook, and you can import your contacts into a SharePoint Contact list for easy web based access or for extending and managing the custom attributes… sharing cell numbers or Twitter accounts, whatever.  SharePoint is about the collaboration.  From a correspondence perspective everyone would be communicating as normal, but it would be archived to their team or project workspace and essentially searchable all in the context of the team.  That’s the feature design.  Contextual.

Q) what i did in the moss i gave every user a macro that transfer the email from the outlook to he user or department folder

In a small company this might work where you don’t have hundreds of thousands to millions of items.  I still caution you on the long term scale of the list.

Q) What size file would you consider a max for sharing within SharePoint?

2GB is the hard limit. 50MB is default which I believe is preferable, but it is surely possible to do it upwards of 100-200MB.

Q) One thing I do not see in this list is emailbox sharing. So you can’t do this in Sharepoint, Correct?

Right.  SharePoint is more for an archive or a discussion list or simply retrieving attachments into a list.  Calendar sharing and contact sharing definitely, sharing the items sure, but not really an emailbox or inbox type scenario.

Q) its posible to use sharepoint list as a team mail / in-out folder ? / And if it’s a good idea todo so?
In, yeah, it’s possible.  That’s what inbound email is configured for, but outbound it’s more about workflows and alerts.  Not really a mailbox.

Q) Should we stop all PST archiving, to avoid, all legal problems?

If I was a company that was concerned about liability about what employees were saying in email then yes.  I would look at reducing liability of emails with some limited life cycle of 1 year or so.  Telling employees to save their important emails to SharePoint or save the contents into word as a document if it’s corporate information that needs to be retained… something like that.  It requires getting policies that legal or corporate affairs can look at and approve, and HR can support in their employee handbook and training.

Q) What about the handling of encrypted and digitally signed emails?

The signed mails shouldn’t be a problem for reading in the binary formats, but encryption and even rights protection could pose some challenges.  I’d recommend testing it.

Q) can you please elaborate on AD objects created by email-enabling? This isn’t the case in our environment.

To turn on email enabled lists for self service, you give SharePoint write access to an OU where it creates contacts so people can email it. Refer to the answers above where I show the instructions for Inbound email where SharePoint uses it’s SharePoint Directory Management Service (a timer job).

Q) I assume the underlying structure is SQL server?

For SharePoint yes, the repository is SQL.
Q) We are considering migrating an in-house built messaging solution (storing messages on an SQL 2005 backend and having windows forms application as a client). This database holds about a million emails. Given all of the scalability issues is sharepoint the right way to go?"

Maybe.  If you design the app to spread across more than one list, then it should should give you the scale you need for a million items.  I’ve seen farms that manage 20+ million items, it’s not the number of items, its the containers and views that are the issue.  Optimizing the query without touching SQL.  That’s the challenge.

Q) In using Sharepoint for Team calendars, I don’t believe there is the ability to create meeting requests. What would be a work around for this?

The person creating the meeting would create the meeting and invite the contact that is the SharePoint calendar.  A meeting workspace is a good example of how meetings start in Outlook and move into SharePoint and not visa versa.

Q) How is the compatibility migrating from GroupWise?

That’s an Exchange Question.  Most of the Groupwise email solutions would be covered in Exchange.  SharePoint would cover the applications like group calendaring, and custom forms, shared task lists and the like.

I found this Groupwise/SharePoint integration response from a partner:

Omni Technology Solutions

Riva synchronises SharePoint and GroupWise contacts, appointments, groups and project tasks. Use Riva to add GroupWise Web Parts to the SharePoint Portal interface. Transparent, server-side integration means end-users do not need to install any client plug-ins.

The first option is to synchronize SharePoint calendar, task, contact list information directly to GroupWise and provides bi-directional synchronization for contacts. Calendar information is read-only because a SharePoint calendar is a shared calendar. The second solution allows the GroupWise mailbox, day, week and month calendar view and address books to be added to SharePoint as Web Parts.

For migration resources look at Groupwise Migration Tool for Exchange from Quest Software.  I didn’t see much on migration to SharePoint.  Probably not too hard to migrate to SharePoint once in Exchange.

Q) What are the intended usages of Email-Enabled lists?

Tons of usage scenarios…  Email archiving, email to a discussion lists, archiving attachments into a document library, receiving resumes into a common inbox, managing invoices, managed folders legal to legal repository.  I wasn’t trying to push it though, despite it being an interesting feature.

Q) What version of sharepoint is needed to manage email? MOSS or sharepoint services 3.0

Either WSS 3.0 or MOSS 2007 since the functionality is built into WSS the platform.

In WSS 2.0 and SPS 2003 you had some limited abilities around archiving email to Public Folders (Ironic huh?)

Q) is the OOB functionality to be "self Service", can this easily be changed to only have IT enable this feature on a list by list basis?

I would say the design leans toward self service.  It can be limited by IT, but the UI on the list may be confusing when people try to enable it and it errors.

Q) Is it only a copy to SharePoint, or will deleting from the managed folders also delete from SharePoint?

I believe it is sent to SharePoint.  So what you see through the managed folder would be SharePoint.

Q) replication? what??

Public folders has replication built in for replicating the folders.

SharePoint does not.  It requires a third party product or solution.

Q) would like info on templates that "look like notes" – have not seen that

I saw a demo at ITForum in Barcelona last year.  I haven’t been able to track it down, but I know what I saw 🙂

Q) an idiot’s record management system

SharePoint is definitely easy to use for sharing documents, but the email management part of this discussion does pose challenges.  The records management features are limited.

Q) good job, joel. I could argue with certain things, but that’s why we are all here, thanks.

Thanks.

Q) Joel, do you provide, or know of, additional webinars for SharePoint developers focusing on tech aspects and best practices (say, on Web Part dev)?

Yes, track down Andrew Connell.  Also MSDN webcasts and a number of partner vendors do put on webinars.  I did a recent blog on videos, webcasts, and podcasts

Entertain Me! SharePoint Videos, Screencasts, and Podcasts