Collabshow.com

One of the biggest challenges you’ll face as a SharePoint administrator is moving content around.  If you followed the head to head migration stand off that happened today you’d have seen users wanted to see their workflows migrate.  Initially I was thinking.  I really want to see a single list of the out of the box limitations with the API and with the built in migration methods in SharePoint 2007 and 2010. (You’d be amazed how little changed in this area. Guess we didn’t yell loud enough.)

There are a lot of reasons why you’d migrate, but preserving your content is the key.  The reason SharePoint migration tools are popular is because of limitations in the Content Migration API (formerly known as PRIME API), Import/Export in STSADM and Import/Export in Powershell.  PowerShell Export-SPWeb and Import-SPWeb you can migrate sites, site collections, lists and document libraries between entirely different SharePoint 2010 farms. (If this post is too deep and your just looking for basic commands to do the migration look at SharePoint Monitors overview of the commands and powershell command.

The closest to full fidelity is first stsadm backup/restore then database attach.  Even database attach will loose the settings that are in the config database, so be careful when you hear full fidelity migration from anyone.

I’ve found it extremely difficult to find a single clear list of what limitations are in the box and what I should be looking for in a SharePoint Migration tool.  Special kudos to conversations on twitter with @glapointe @toddklindt and information from migration expert Alex Kirillov @alexkirillov from Quest Software (PM of Quest’s Migration Manager) and added to this list based on my own experience, what I’ve found on blogs and MSDN/TechNet and would love to see the limitations all in one place.  Ultimately it would be great to have a single list and ultimately get this fixed… especially as people are restructuring their data and both getting ready to upgrade, during migration, and post migration cleanup and further restructuring.

What Doesn’t Migrate with SharePoint Built in Import/Export and other known issues:

• Item created by, created date, last modified date
• Workflows (Definitions)
• Workflow State & Tasks
• Workflow History
• Check in/check out state
• Audit logs
• Usage Logs that have not been processed
• List configuration/app configuration
• Site collection settings (Site Collection Ownership, Quota, email notification settings and a lot more)
• Personalized views
• Alerts
• Recycle bin state/items including admin and user Recycle bin items
• Lookup lists have issues due to remapping of object GUIDs
• Object GUIDS are not preserved (This causes all sorts of weird things and can cause issues down the road. Empty folders may not migrate)  This includes using import/export in STSADM or Windows Powershell, SOAP or Web services as well both result in loss of object id consistency including reparenting. The only way to avoid this is using the Content Migration object model.
• Empty objects may not migrate (Discussion lists may have some problems related to this, and some folder structures may be remapped.)
• Users that no longer exist are removed during import/export (related to ownership being re-written)
• Surveys that are not completed will not be exported
• Global Navigation inheritance issues (parenting challenges)
• Known issues with conflicts when used with content deployment
• No migration of features, solutions or assemblies associated with site/web (Must be manually migrated if separate farm and exist on destination)
• Content overwritten if exists
• Site dependencies on Site Collection Gallery lists (example if you migrate a web the parent site collection may have galleries that the web uses i.e. galleries include some look and feel, masterpages, images, features, and webparts.) known to have issues (Need more detail)
• No list level migration (granularity is site level in 2007; Windows Powershell SharePoint Cmdlet in 2010 gives you list level) Thanks @toddklindt
• Scale limitations (reliability goes down the larger the site <25GB Note: if you add the "no compression" parameter it increases the likelihood that large exports will work. Best case is dedicated databases as database migration is higher fidelity)
• Portal Template Global Nav known to have issues (Need detail. Problem appears to be associated with site collection property loss)
• Security remapping AD group challenges for groups between 2 untrusted forests
• Versions of aspx pages (Need more detail)
• Can’t change site template during migration
• Can’t consolidate sites during migration
• No good list of what fails to import
• blocked file types
• files over max file size
• Objects that don’t contain objects may not be migrated (See comments in Gary LaPointe’s Export lists blog)
• Post migration sync (No way to migrate again. If template already chosen or not blank)
• No support between farms of different builds and versions. Have to be exact same version number (This may be different in 2010).
• No ability to target a specific content without manual intervention (limiting all others or setting capacity high)

This list is in progress.  Feel free to ping me on twitter to update or add things on this list.  I’ve added text such as “need more detail” where I don’t know the exact issue and could use clarification.

Question any vendor tool that says they do full fidelity migrations since much of this list is not available in the Content Migration API.  How are they doing? That’s the follow up question.  You do have the ability to retain the GUIDs the Object IDs and as a developer using the content migration API you have the ability to target migrations at the list level.  What about SharePoint Designer?  It uses the same API and has all the limitations above plus it’s own scale limitations for size of CMP files. Some of the known issues around SPD scale are listed won’t fix.

Both WSS and MOSS in 2007 and as well in 2010 you’ll see a lot of use of this API at various different places:

• Content Deployment
• STSADM -o export and import
• Copy/Move operations in Site Manager
• Variations
• Windows Powershell cmdlet import/export
• Unattached recovery uses a form of the migration API in SharePoint 2010

What’s fixed with content migration in 2010?

Well, the unattached recovery gives us a new API for interacting with a database that’s not apart of the farm, but ultimately I’ve found a lot of the same issues.  Try to recover a site and a workflow for example.  You do get more granularity in recovery if you can find what was lost.  Powershell export can give you more granularity, but the data faces a lot of the same issues.

Excellent Resources and Blogs on different aspects of Import/Export Content Migration and Content Deployment

Gary LaPointe

Automation Blog post on Import Export Copy & Delete Lists with some of his fabulous powershell cmdlets

Chris O’Brien UK SharePoint MVP

STSADM Import & Export and Content Deployment

SharePoint Designer

SharePoint Designer and STSADM Import Export

Stefan Gossner from Content Management Team at Microsoft

Data protection and recovery for Office SharePoint Server (White paper)

Deep dive into the SharePoint Deployment and Migration API (Multi-Part!)

Here’s some info from MSDN on the Content Migration API detail on SharePoint 2010 (but also applies to 2007):

What the Content Migration APIs Are Not

“The content migration APIs are designed to move content from a source location to a specific destination; these APIs are not designed for backup and restore purposes. Following are some of the limitations of the content migration APIs with respect to backing up and restoring data.

• No configuration or application data can be exported or imported.

• The largest object you can export is a SharePoint Foundation site (SPWeb object).

• The following content cannot be exported or imported: alerts, audit trail, change log history, check in/check out state, recycle bin items, recycle bin state, security state, workflow tasks, and workflow state.”

.One of the most exciting part of how the internet has been transforming to effect lives with social networking and true crowdsourcing.  Wikipedia (a great example of crowdsourcing itself) today suggests “Crowdsourcing is the act of outsourcing tasks, traditionally performed by an employee or contractor, to a large group of people or community (a crowd), through an open call”

One of the popular examples of the power of crowdsourcing in 2007 when Researcher Jim Gray was lost at sea. 12,000 people came to the rescue with a combination of maps and a new idea of web based software for searching the seas.  It’s amazing how much we’ve evolved from the crowd based SETI project where you had to do nothing the app would take advantage of your processor cycles to process data to look for intelligent life. 

Answering Dreams and Prayers through Crowdsourcing at Kiva.org 

I was introduced to  by my cousin who said she and her boys would spend the evening as a family evening as a family picking their candidate for their $25.  What a great gift, what a great way to teach your children, what a great way to help make contributions personal.  I had a chance to try it out for my son’s birthday.  He picked a guy who was looking to buy a scooter so he could make more money and spend less time getting to and from work and spend more time with his family.

I previously tried out the site and made a loan that has nearly been repaid, but about a week ago, I was thinking… How could we help our more people?  As I travel around the world I think man I wish I could be doing more to help people.  I think Kiva gives us that option.  What it really needs is awareness.  The platform is really set up.  You can fund in small $25 incremental loans also referred to as Microloans, you can fund with paypal or other easy methods.  Loans repayment can go directly back to allow you to then lend the money to someone else or cash out. 

At first I was a little cautious about posting on facebook that I was loaning money to people with their enterprising projects, buying a goat, a motorcycle, or sewing machine, but then I thought.  Hey this isn’t like the chartity contributions that should be in secret.  This is crowdsourcing.  My excitement could encourage others do the same.  Why not share my passion!!???

I created a group on Kiva called SharePoint http://www.kiva.org/team/sharepoint where we can simply track how we’re doing and invite others to join.  You don’t have to list all your loans with the group, but it’s a mechanism to encourage people to join.  I have been extremely impressed with the 98.98% return rate and stats like 1 loan every 19 seconds.  The stories themselves are very compelling.  Thinking of starting a small business? You’ll actually see right through the site you can sign up for a loan as well.

If you hadn’t heard.  I like to travel.  I’m now on a quest to see if I can lend to someone in every country possible.

Crowdsourcing Searching the Stars

A recent example is a schoolteacher in Holland recently made a major discovery on http://galaxyzoo.org, of an unusual gas cloud that might help explain the life cycle of quasars. (More detail in an article on The Rise of Crowd Science) Search for Hanny’s Voorwerp.  You can also get more info from Hanny’s blog.

I recommend checking out http://zooniverse.org/ for more on the crowdsourcing project of scanning. 

The idea of this Crowd Science is leveraging the power of the crowd of people who are motivated.  The software simply allows people to then use a simple interface to perform a task.  The task could be looking for a person on an image of sea, inspecting stars, or making microloans.

FamilySearch.org

My mom spent thousands of volunteer hours a few years ago transcribing text on birth and marriage certificates that was hard to read for FamilySearch.org project.  Some could be automated with OCR, but where it was making mistakes, my mom would fix it, and even verify work from other volunteers.  She is a trained Graph analyst so it takes advantage of her skills and the crowdsourced thousands around the globe that were trying to help.  People who had a few moments could take as little or as much on as they had time for.

Make Monkey by doing tasks or get work done with crowds at MechanicalTurk.com

I recently was at a SharePoint dev shop, and they said they were working on a cloud project with Mechanical Turk a project of Amazon.  They call the tasks HITs, Human Intelligence Tasks.

The design there is two fold.  On one hand you can choose to make money doing tasks at home or whereever you are on your hours with legitimate work.  The other side of the coin is creating tasks such as tagging a photo for a penny, translating Malay words into English at 5 cents, copy editing a Recipe for 40 cents each recipe, or taking a personality survey for 50 cents.  There are now over 65,000 different tasks!  

From the Mech Turk requester site the value you get is:

• Have access to a global, on-demand, 24 x 7 workforce
• Get thousands of HITs completed in minutes
• Pay only when you’re satisfied with the results

To get the HITs into the cloud based system, the devs would build against the API to load the tasks.

That’s exciting stuff.  Blows my mind what’s possible here especially as this continues to grow.

Source to the specialized SharePoint technical community to build your webpart, Architect your farm or write a blog

I had mentioned Elance.com to Mike Watson and he was saying.  That’s old school.  Now you need to look at http://odesk.com (2000+ SharePoint resources) or http://guru.com (250,000+ Freelancers). People are spec’ing SharePoint work and individuals or companies pick it up.  The work is rated and the social features we’d expect are built in.  Very slick stuff.  Post jobs and search contractors and visa versa.  You can then filter the results by category, feedback, and hourly rate.  Mike was explaining how you could write out a web part you’re looking for and the people could bid on it. 

As an example Ajay has 5 stars as a portfolio of 10 things you can look at.  I can imagine listing myself to do some blogging, webcasts, social media marketing, or tech writing. 🙂 (assuming no conflict of interest)

Mike Watson’s got an account.  Check him out.  Great to see the ability to crowdsource a SharePoint Architect.

I can imagine my Office365 vendor support friend taking off hours SharePoint support calls, my blogger friends sharing their blogging skills, and independent consultants and even small companies listing their skills.  Guys like Owen Allen could list his SharePoint and IT Business Development skills.  It takes the resume to a very practical level.  Obviously you should make sure you have a moonlighting agreement if you already work somewhere, and ensure you don’t have a conflict of interest on the projects you work.  There are other practical considerations, but you get the idea.

I hope this has helped you explore the world of crowd sourcing and how it works.  Love the idea of allowing people to use their skills where they are ultimately benefiting each other in an extreme scale kind of way.

I’ve found when I’m reading an article that I like, I want to share it with my friends.  I’ll tweet it or share it on facebook.  Now there are tons of ways to do this without forcing people to open up their twitter client or potentially navigate away from your page to Facebook.

These tips work with widgets that work with SharePoint, WordPress, or other blogging platforms.  The implementation may be slightly different.  I was able to setup all these things with SharePoint.

1. Add the tweet button to share this on twitter – This one is nice and tight.

On a SharePoint blog you want to put this on the actual posts page, not the default homepage of your blog.  You’ll add this in a content editor webpart.

I like that it can add the “via username” so people can recognize where the page content is coming from. It also allows the user to customize what is being tweeted and add commentary.

2. Add the facebook like button (or like box I do see them as different) with or without faces – It’s fun to see the profile pics of your friends that like your blog.  It also encourages people to get more involved.  Facebook recommends the like button over the share button capabilities.

Simply add your URL and choose whether or not you want to show faces and click get code.  I recommend putting this one on your home page or significant landing page since it’s working with a static URL.

3. Build community around your blog with a facebook page – this will allow you to communicate with your readers and promote events, activities, posts and articles.

4. Consider other widgets such as the Facebook page Like Box which will help your blog readers find your page, and you’ll benefit by having the popular pages linked.

Try Recommendations which uses your friends shares to determine which posts you might be interested in.  This takes a lot of the logic that you’d have to do with counting up comments which would be less accurate than using likes/shares.

5. Social Comments: Now that you have a mechanism for people to share, like, and tweet your content make sure people will see the content in those networks as well through – I’ve been surprised how much more interaction I have with people as a result of not just automating the blogs such as with twitterfeed.com, but actually posting them myself and responding to questions.  Definitely don’t be afraid to put yourself out there on twitter.  But even if you don’t want to add all your fans as friends on facebook, you can still interact with them on pages, and with facebook apps.  My favorite blog network is networked blogs.  It can import posts automatically into facebook and creates a special place where people can favorite your blog and has a great catalog and listing of blogs for discovery.  Don’t skip out on enhancing your comments with facebook comment widget or with Disqus which is designed to interact with both facebook and twitter and has open id login and comment filtering.

• Syndicate your blog’s RSS to twitter or facebook or more with twitterfeed
• Syndicate your tweets to LinkedIn (be sure you want to do this!)
• Syndicate your blog to your Facebook blog page and/or profile

There are a ton of other things you can do, but make sure you take performance into consideration.  Every few months I look at the weight of the page and look at the new widgets and keep what’s working and rotate out what doesn’t.

Watch your traffic reports to see if twitter and facebook and your social interaction is working.  I think you’d be surprised.  Search still rules, so make sure you’re being indexed and SEO is all good, but interacting with your users will only encourage loyalty.

ARCast.TV

At TechEd 2010, Joel Oleson, Sr. Product Manager and SharePoint Evangelist at Quest Software, shared with Zhiming Xue “Z” his insights in designing and deploying SharePoint solutions for a global audience. During this interview, he discussed common scenarios and challenges for SharePoint global deployment, improved experience SharePoint 2010 has brought into the deployment process in several key areas such as SharePoint workspace (aka Groove), service applications, federated search, and multi-lingual user interface. Joel also discussed the pros and cons of the two common SharePoint, including the built-in MySite, deployment options — centralized deployment vs. regionalized deployment. Throughout the discussion Joel emphasized the importance of planning for SharePoint deployment and working with your IT teams. Check out the interview and find out what Joel has to share with you and why he thinks you can help with the SharePoint community.

Check out the details and videos on Channel9 ARCast.TV

ARCast.TV

Next week I’m doing two free webcasts.  On Tuesday I’ll be introducing the tool I designed around SharePoint Operational Efficiencies and Change Management called Quest Server Administrator for SharePoint.  Thursday I’ll be presenting on “Building operational governance for SharePoint 2010.

SharePoint Server Insight: What’s Lurking Beneath?

When: Tuesday, November 09, 2010 11:00 AM-12:00 PM (GMT-05:00) Eastern Time (US & Canada), 8 AM Pacific Time

Note: The GMT offset above does not reflect daylight saving time adjustments.

Now is the time to find out what’s lurking beneath your SharePoint’s underlying platforms.

Join Joel Oleson and Curtis Kelly for an insightful one-hour webcast and learn how Server Administrator gives you total visibility into your SharePoint server, even enabling you to compare settings across farms!

Find out how get the most from your SharePoint server, and keep it secure and available.

Register Now!

Building Operational Governance for SharePoint 2010

When: Thursday, November 11, 2010 11:00 AM-12:00 PM (GMT-05:00) Eastern Time (US & Canada).

Note: The GMT offset above does not reflect daylight saving time adjustments.

The more your organization is collaborating using SharePoint 2010, the more critical a good governance service strategy becomes.

During this free webcast, SharePoint expert Joel Oleson will explore common frameworks such as MOF, ITIL and Scrum you how to get better quality in your day-to-day management of SharePoint 2010 with an operations plan that includes governance and change management policies.

Register Now!