Some SharePoint people might come away from Teched and say… meh the content was ok, but those that are honest will say it was way more than the content that made it a rich experience. Teched this year so far, not only in Africa, but also the U.S. was like meeting up with a friend whose been pregnant for 8-9 months and hoping to see the baby. It was good seeing the friend, but they hadn’t had the baby yet. We know SharePoint 2010 disclosure information is due in October. So we continue to count down the days. In my sessions I showed small snippets of the 2010 sneak peak videos and continue to encourage all SharePoint fans to check them out if they haven’t. Richard Riley an old office mate from my MS days remotely performed more than one demo for me this week. Thanks buddy!
As well you may have heard I’m setup to do a webcast on PREPARING to Upgrade to SharePoint 2010 Today. Exciting stuff, not to be confused… this session essentially provides a masterful level of guidance based on tools in SP2, the team blog system requirements, links to the sneak peak videos, and a major emphasis on the pre-upgrade checker tool and its reports and output. (No NDA information is revealed in this webcast – period.)
If you haven’t been following my tweets on my adventure through Southern Africa with Eric, I won’t get a chance to include too much of that, but do suffice to say that we had an incredible journey.
Eric Harlan, principal SharePoint Architect/Consultant got a really good taste of what it’s like to see the world with me. It was a blast taking him out of his comfort zone on occasion and going into places where neither of us speak a world of Portuguese, animals that would prefer us for breakfast (lion shots and elephant bulls), and days with little concrete plans with God as our guide. We captured a just few memories with his massive lens and my HD flip, but only we and the charging elephant will remember how close we came to this.
(pictures to come)
You have to check out our first video – SharePoint Wild Boys in Africa a compilation of the big 5 a snippet from the night safari and traditional dancing. It makes more sense when you see the video. Before we arrived at Teched we 3 nights at Kruger National Park including a night safari, and morning walk to hunt (photo shoot) a pair of white rhinos. We saw hundreds of elephants, thousands of impala, buffalo, zebra, and many illusive animals like hyenas, lions, rhinos, and dangerous hippos. After our time at the park we went across the border into Mozambique to Maputo for a night having an adventure we’ll both remember for our lifetimes. Then an adventure in Swaziland sleeping in traditional huts that look like beehives and doing traditional dancing. The video captures just a few highlights in HD in only 5 minutes.
I did 5 sessions at Teched Africa below are links that are currently available to the decks on Slideshare.net. If you want to use them in your company that’s fine. If you want to redeliver them, or use any of the slides in your decks please give me a heads up through twitter. Some of those slides are used with permission from Mike Watson or Paul Culmsee for example, and the right people need credit.
Teched Africa 2009
OFC304 – Demystifying Backup and Restore and Disaster Recovery in SharePoint 2007
OFC401 – SharePoint Foundations & Best Practices to Physical and Information Architecture
OFC301 – 10 Steps to Successful SharePoint Deployments
SharePoint Architects Panel (Chalk Talk) – Joint session Zlatan Dzinic and Eric Harlan, recorded by Hilton (recording to be available soon on the MOSS Show Podcast)
Teched Online – “Twitter and the SharePoint Community” – Joint session with Eric Harlan
I’ve also been uploading my pictures of this adventure to facebook. I’ll likely get up more of the bulk of them to flickr when I get a solid connection and time. LOL, I wonder when that will happen.
Above: Elephant poo. A featured item in a sack (shellacked elephant poo) that resulted in a webcam being given away. Can you believe the first official answer was that poo was in the sack? I couldn’t believe it. The Physical and Information Architecture session I did included a youtube video of Bear Grills showing us how to survive without water by squeezing elephant poo. My analogy with all this was trying to help people get through the poo to understand the difference between Sites and Site Collections and get the guidance and core foundation they need for successful scalable SharePoint deployments.
Beyond all the cool animals which were many many many. I have to say it was the cool people that really made this trip stand out. First off I need to call out Veronique. The IW User Group lead for the South African Information Worker community in Johannesburg or Jo’berg as they call it here. Veronique was the first to meet us at the airport and help us feel welcome. She helped us get where we needed to go and even packed us a lunch and let us use her phone! The pack of biltong (pretty much jerky sticks) came in handy as well when the picnic area that was suppose to have lunch wasn’t there.
Here are a few of the people I met, (below) many of which I don’t remember their names, but I do remember how they made me feel like a friend. I know I have many friends around Southern Africa.
Even the random Canadians we met in Maputo who we shared dinner with. Really nice people that were donating their time to help the people of Swazi better deal with the disease situation.
I had some great jam sessions at Teched Africa that apparently our Rock Band Pearl Jam Star performance made it to youtube, it was even played during a session. Don’t believe what you hear, the poor recording equipment and speakers were to blame for anything that doesn’t sound awesome. It definitely sounded a lot better in person, and yeah we were rocking it with XBOX 360 RockBand in the Community Lounge. Zlatan with his axe and our star drummer with me on vocals and various guest bass guitarists were ready to start a band right there.
I’ve been able to experience 17 Tech Ed’s across 5+ continents… Europe, Japan, Malaysia (South East Asia one of my most favorite), Australia, NZ and nearly a decade at Teched US. Teched continues to be the best geek party on the planet and brings together people from all industries and helps inspire us geeks to be rockstars (best of the best). The networking at Tech Ed continues to be my favorite thing. It’s not the Teched dancers or Jam sessions or a beauty among geeks (last year’s Teched U.S. that featured a Beauty & the Geek Photo Shoot) that bring us together. It’s a common interest in cutting edge technology that unites us and our passions that inspires us to do it for yet another year. From what I was able to see, the attendees of Teched Africa were able to get that inspiration and ignite their passions. Thanks for letting me share this experience with you! I’m already looking forward to next year… If it all works out. As well looks like I’ll be speaking at Teched to SharePoint fans in Europe in Berlin. See you then!
Thank you and See you later!
(Special thanks to Michael O’Donovan, Zlatan Dzinic SharePoint MVP, and Veronique Palmer for making all this happen.)
I had to put in a short blurb about one of the greatest achievements a product can make. Today here at Teched Africa I was talking to a guy who is the account specialist at Microsoft for Namibia. In our conversations he talked about the evolution of SharePoint and how in the past it took a while to explain SharePoint Server and why it was important in an organization.
Now, he tells me it is simply infrastructure. As mail servers, authentications servers, so is collaboration. Next might be OCS as pure infrastructure rather than add ons. Just as the LAN phone became a utility so is SharePoint as core IT infrastructure.
It’s awesome to see this day. I have memories of it’s infancy, and I do agree with my Namibian friend. I do believe we have passed the hump where not only collaboration and workgroup storage solutions are mainstream, but SharePoint as a collaboration infrastructure has become a staple, a utility, a commodity, a core set of infrastructure that is expected by the business. Sure there are many who would say that deployments aren’t yet that mature. If our Namibia friends are finding this so in Zimbabwe and more and it’s confirmed in Chile and Argentina (based on conversations a few months ago), then I’m sticking my neck out and saying, yep. It’s not simply saturation with Core CALS or simply signed enterprise agreements, but developed countries and districts across the globe are seeing SharePoint as core infrastructure.
I’ve just scheduled the latest webcast to help people as we build our roadmaps for the future. As you budget and plan for next year and beyond it’s important to take into consideration all of the valuable information that has been shared by the product teams around system requirements, client and browser requirements, and planning your customizations and ultimately documenting all those changes over time. Essentially I’ve pulled together this guidance and a lot more for getting your broader environment prepared for the upgrade to SharePoint 2010.
As many look to the future upgrade, I want to make sure people are properly budgeting for what really need to happen to be prepared. It’s not all about just the logistics of upgrade, it’s making sure everything is lined up for a smooth upgrade.
Join me for the upcoming webcast mid August. Details below.
Webcast: Preparing for Your Upgrade to SharePoint 2010 Today
Date/Time: Thursday, August 13, 2009 at 8:00 a.m. PDT / 11:00 a.m. EDT / 4:00 p.m. GMT
NOTE: There is no NDA material in this session.
Part of the reason for my last post was me dealing with my huge comment list. Sure it’s a lot of spam, but I have some serious lessons learned in dealing with it.
1. Comment moderation can do 2 things. 1) Prevent unwanted comments from showing up 2) Make you the bottleneck
But did you also know that when you turn on moderation you get read-only datagrid views?
2. Limiting the item limit view is one of my new favorite features in dealing with large lists resets back to all items viewable every time you edit/save the list! Very much a pain since it is collapsed by default and not noticeable. This is likely a bug, but now that I know, I can work around it, until it’s fixed.
3. I was editing a list with more than 10,000 items and the best way to manage it was to 1) Use the datagrid view 2) Limit the items to groups of a few hundred. A thousand was doable, but more than that I felt like I was waiting. 3) Limiting the columns to those that I needed to see also helped improve the query response time
4. I’ve now closed my new comments, and replaced it with DISQUS. It’s a full blown comment solution that is really designed for community and already has a ton of online support. I was able to simply create an account, and drop in the snippet of code into a content editor web part and close the existing “new comment.” It gives me the ability to support all sorts of interesting things yet to be seen, but things like integrated twitter responses, facebook responses, video, and trackbacks… I’m a huge trackback fan, so that’s a huge bonus.
5. One thing I also have to note from a practicality standpoint. My servers are in Texas and I’m in Seattle area. My servers are really fast, but when I was trying to edit my huge list with Access, I was having all sorts of troubles with timeouts and hanging. A lot of this had to do with latency, but also the query. Limiting the item limit for the query itself was still giving me issues, and turning off moderation of the list.
When using the export option it does respect the item limit, but it still might feel like it’s hung and give you the “Not responding,” but don’t give up if your query is reasonable like 1000 or 2000 or so. For more, I’d recommending walking away so you’re not tempted to kill the process.
6. When using dataview be sure to note the little progress bar in the bottom right hand corner. If you ever delete like 700 rows at once it can take a while. It will let you continue doing other operations, but if you refresh it will warn you… if you ignore the warning you will find out later it wasn’t done. Bad news. Pay attention to “Pending Changes” Also pay attention to conflicts, locks, and “Read-only”
7. I wanted to know how many items I had in much huge list. What’s the easiest way to get that? Go add totals on my view. Right? Wrong… I found no matter what I did to my view to get it to give me the total, it would either hang, or go read only and give me an error. The totals are great when you have limited the list, so you can see what’s going on, but the all items view for example was a joke. It would only show me 1-100 the simple page to page view. Which would render, but didn’t give me the data I was looking for. Using an external tool like the SPDiag or stsadm –o PreUpgradeCheck (in SP2) is a much better means of getting your list count.
8. When editing my super large list I found I had to turn off moderation, that was the beginning, that would at least give me the flexibility i needed. As well in the dataviews, I limited my columns to only those “required” during editing, and when deleting, I’d even go for less. I know I mentioned some of this above, but ultimately as well, the “Versioning” settings of the list… that’s where you go to change moderation. Despite the fact that you’re an admin on the box, an admin on the site collection, and an owner of the items, sorry you’re likely going to have difficulty editing in datagrid unless moderation is off.
Toggling the moderation from on to off immediately made a huge difference. for me.
9. I learned that all other views, other than dataview are a pain to delete items, and even editing large quantities of items going from page to page in all other views, was simply not worth my time. Either external clients like Excel, Access, Outlook, or the browser datagrid was the way to go. Note the datagrid didn’t work when Excel wasn’t installed. I’m sure there’s a version dependency, but didn’t get a chance to check that. Often in Dataview the pending changes might last for up to 5 minutes on large operations like huge bulk deletes. You can imagine what that’s doing to the database, and as a server admin, I can just imagine what’s going on when someone connects access to a large list 🙂 and it looks like it hangs for 5-10 minutes. Some serious SQL action going on.
Deleting was interesting to figure out. With moderation it was obviously grayed out, and no matter what I did in datagrid I couldn’t get it on. It was all read only. After addressing that I was able to get the multi select in datagrid to work where I could select multiple rows very quickly and the wheel on the mouse was very responsive as well. I was able to select hundreds to thousands of items very quickly and delete them. While deleting I could continue to select and delete other rows, and despite the pending as long as I didn’t refresh or navigate to another page, it would continue to queue the operations.
10. One of the things I’ve been missing for years is a HUGE set of Office integration tools with the datagrid view. If you click on the very edge you’ll find a huge set of integration links including richer filtering, sorting, cutting, and undo! Why are all those tools so hidden? Am I blind? Maybe others have been using the print and chart with excel options for your list and creating pivot reports on the fly.
Now that I’m done editing and cleaning out my spam out of my list. I’ll turn moderation back on…
One of the biggest challenges for a document management system built on SharePoint is scaling lists. There is actually quite a bit of knowledge spread across the help files, technet documentation, in whitepapers and on blogs… but it’s still quite difficult to navigate this challenge. I also see this as a step in the progress toward upgrade. People that are looking to optimize their environments should be looking at large lists sooner than later… here are some of the impacts…
Managing large lists is not necessarily only an end user or power user problem it is a developer issue and an IT issue….
Did you know that Microsoft IT scans for lists over 5000 and assists in either cleaning up or helps the users optimize them? See: SharePoint Performance Optimization Microsoft IT
Did you know that PreUpgradeCheck (in SP2) and SPDiag (in the Admin Toolkit) will help you identify large lists? For PreUpgradeCheck I’ve been telling people to make copies of their environments, but if you’ve already got SP2 you can run it today. It’s read only and an STSADM command that’s actually pretty quick and gives you amazing reports. Ultimately this PreUpgradeCheck is a best practices Analyzer.
While the planning software boundaries Technet article mentions you can scale a list to 5 million documents, it is extremely important how you do it. “Item – 2,000 per view – Testing indicates a reduction in performance beyond two thousand items. Using indexing on a flat folder view can improve performance.” I’ve mentioned SQL blocking that can happen not only on other areas of the site or site collection, but on the server and blocking in the database. Blocking SPIDs… ever heard of em? That’s why you have to be careful ever killing a blocking SPID from SQL on a SharePoint box. Why? You could end up with data or site or web orphans. Ever killed a blocking SPID in SharePoint? Better pull out your trusty SP2 featured PreUpgradeCheck (Don’t be afraid of it. It doesn’t write, and it will save your life in identifying best practices.)
I noticed some content in help and online info has changed over time to help clarify help and assistance on this topic. The content is very important, I’m not sure why the content isn’t as well understood in broad SharePoint circles. It’s very common to have someone rattle off responses to the question… “What methods can you use to optimize a large SharePoint list or library?”
There are many more options you might not be aware of:
Recommended custom views & indexes:
Office Online on large lists “Each additional column index consumes extra resources in the database. Therefore, you should add indexes only to columns that will be used actively in views on the list or library.”
Recently changed (Index based on “Modified” includes new documents added, create filter based on Last 7 or last 30, etc…)
My Files – (Index based on Created by, sort & filter based on this or create a search based on this attribute)
Due in the next week (Indexed based on Date Due, such as in a task list)
Contact list – Use Outlook to manage and edit the list it’s much more efficient for editing
Task List – use Outlook to manage and edit the task list… much more efficient for editing
Custom Lists – Consider using Excel 2003 or Access 2007 for editing large custom lists as well as datasheet view with limited max “Item limit”
Office Online “You might want to remove the Manage Personal Views permission from contributors for a large list or library. By removing this permission, you can prevent users from creating a view that spans all of the items and that might adversely affect the performance of the rest of the site.”
Ultimately the best information written on the top of large lists is the whitepaper done by Steve Peshka which provides charts and detail on the best retrieval methods. I quickly extrapolate from that paper, that the out of the box queries and pages are the worst way to work with SharePoint and that search is the best method. Reza’s interpretation of the paper I believe, do make the paper itself much easier to understand.
20 key Points Arising, or Inferred, From “Working with large lists in MOSS 2007” Paper (PDF)
<Update July 16, 09> RE: Copy/paste mistake in Fastest methods to query large lists in MOSS. Anyway,I have corrected the issue and updated the docuement so please g o ahead and update your post with the following section (just copy and paste below into your blog post please) </Update>
The Fastest Methods to query Large Lists in WSS & MOSS based on Reza’s Dev Horizon blog.
Fastest methods to query large lists in WSS (Not explicitly mentioned in the paper) :
I) Looping through SPListItemCollection of SPList.GetItems(SPQuery)
II) Lists Web Service
III) Looping through each DataRow of SPListItemCollection.GetDataTable()
Fastest methods to query large lists in MOSS :
I) PortalSiteMapProvider. Depending on the column you use in the WHERE clause
of the SPQuery object used in this method and whether or not that column is
indexed there might be some performance degradation. More about this in tip #10.
II) Search API
III) Looping through each DataRow of SPListItemCollection.GetDataTable (). Pleas e
be note d that, for smaller sets of data, for example 1500 list items, this approach
may outperform querying using Search API technique.
Reza’s book just got published on Professional SharePoint & Reporting Services.
Datagrid View
Some may think that using datagrid view will allow you to continue to scale with your data for editing. Unfortunately despite that you can begin reading the data in your list with dataview, the query continues to “retrieve” data and keep it read only until the query is complete. If it can’t get a lock or meet various other critera you’ll get a read-only view.
Using list max “Item Limit” and then Limiting the columns “display”-ed will help return the results for much easier editing.
Note the bottom right hand corner says “ready-only”
Folders
Folders are seen as very old school, but in the past this was the most common means of managing large lists. Now with the recommended 2000 max per folder it’s straight forward to explain to users who manage these lists. Employing this strategy with the max limit Item limit of somewhere between 100 or 2000. You’re better off. You’ll have to play with this, it isn’t as simple as it sounds.
My users commonly use Web Dav to access my document library, what can I do to optimize my list?
While it may be easy to say, don’t use ‘em. You can actually potentially satisfy both types by using folders and views.
More and more I see people blocking SharePoint Designer. It wouldn’t be difficult to block the explorer view. You could turn off the “browse directories” right which would disable viewing folders in Web DAV interfaces such as Explorer View and SharePoint Designer in one go.
Here’s a Quote from the Comments from GetThePoint Blog on “Manage SharePoint Large Lists for Performance”
“Hi Dina,
Do you have any tips to increase the performance of the "Save As" dialog in the Office clients, when working with large lists?
Let’s say you’ve got 10,000 documents in a document library, and you are careful to display them only 100 at a time in the web browser, using a 100 item limit on all views. Indexed columns are also in use.
When you "Save As" in Word, the item limit on your views appears to be ignored; Word’s "Save As" dialog will try to display all 10,000 documents and performance will degrade.
Is there any way to work around this, to make the Office clients only use the carefully crafted views?
Thanks
Pete
Your best option here is to folderize. Having less than 2,000 items per folder should improve your performance significantly for the Office "Save As" dialog, which does not utilize SharePoint views and does not support paging. So, it should be smoother if you split up the items inside your document library into at least 5 folders.
Dina”
I found this Q&A at askville helpful on improving client performance with WebDav Web Folders. There are third party clients that can help you map and connect to https WebDav for example.
Confused about why web folders are so odd and often tough to troubleshoot? Here are some resources:
While dated I found this KB article 838028 on “How documents are opened from a Web site in Office 2003” to be the most technical resource on the topic, of check out 841215 for common WebDav Client Auth issues.
There are a lot more methods that can be used, and ultimately both of the solutions above can oversimplify the problem and solution.
Here’s an example of a list with more than 2000 items. First off you can see… This list or library contains a large number of items.
When you click on the link you get some help content that explains that “When the number of items in a list or library grows to more than one thousand (MOSS lists two thousand in help), the performance of the list or library may begin to slow down or adversely affect other areas of the site.” Essentially this list scale issue explains that not only is the list slow to load, but can affect the performance of other areas of the site.
As well the article explains “[large list] issues can be caused by user actions such as sorting or retrieving a large number of items at the same time.” It does go onto explain that as long as you plan and configure the list or library properly, you can scale to millions of items.
The root of the list performance problem is actually in the view. When you closely read the capacity boundaries document you’ll see that recommend list performance is recommended at a maximum of 2000 per view. The default all items view and actually all views created by default have no limit by default.
Configure all Views with Item Limit
I see a new best practice to simply either configure a limit of 2000 less… your choice on all large lists. If you’re creating a template, list template, site tempate, or feature that creates a list… would be great to see more use of this feature.
If you edit all of your views you can actually limit the total number of items returned
More Resources many of which are referred to above:
Joel’s Blog: Key Capacity Planning Info, Resources, and Links (I’ve been updating this over time)
Case Study by MCS and Knowledge Lake on Extreme Scalability –
Office Online – SharePoint Server: Manage Lists and Libraries with many items
GetThePoint Blog Manage Large Lists for better performance
Whitepaper: Working with Large Lists in Office SharePoint Server 2007
Three Will’s Community Wiki a snippet from the TechNet article on SP1
Office Online: WSS Manage Lists and Libraries with many items (smaller set of ideas)
White paper: Performance recommendations for storage planning and monitoring (newer)
Whitepaper: SharePoint Performance Optimization Microsoft IT (IT Showcase)