Troubleshooting SharePoint: System Log Manager

In a guest post on the Bamboo Solutions team blog titled "How do I troubleshoot SharePoint, So many Logs!" I explored the various logs and what is important for troubleshooting issues on the Front end (Windows, SharePoint, IIS, etc…) and backend (SQL Logs), with a few links to the Index/Search logging aspects such as gatherer logs and change logs (SharePoint ULS logs, Windows (App, Security, System), and even IIS logs may be relevant and apply as well). 

Bamboo Solutions pushed a web based solution to address this common challenge of hunting and gathering and keeping track of these various logs with the key purpose of putting the power in the hands of the administrator even if they don’t have access to the physical server.  They’ve done this through a central admin interface that intuitively integrates an administrative web based log viewing console for viewing more than the most common logs, and even providing a filtered interface for narrowing down the relevant IIS, SharePoint ULS, SQL logs, and common Windows logs.

In a critical analysis of Bamboo’s SharePoint System Log Manager in a blog titled "SharePoint Troubleshooting: A Critical Analysis of the Bamboo System Log Manager" I break this new solution down bit by bit and tell you what I like and share where I think there’s room for improvement.  (As well dispersed across the two blogs you’ll find info where you may want to look beyond the product for certain types of troubleshooting… for example you should consider creating trace logs using IIS 7, or using SQL profiler and looking at perfmon or netmon for a more detailed analysis.)

Essentially the goal here is to help you understand the tool for what it is… face value.  This tool was requested by various SharePoint Admins and SharePoint MVPs (I’ve even asked for this!) and is frequently heard when SharePoint Ops find out how many logs are needed for troubleshooting a typical even common issue.  I say in my log blog, that there are more logs in SharePoint than any other Microsoft app with the exception of those built on it.  The way I look at it, if we want these tools to exist, we need to evaluate them and give our feedback or the tools won’t exist.  These tools aren’t for the business directly, they are for IT.  A tight feedback loop and broad awareness is a key principal of our community values.  Did Bamboo hit the mark?  I think they got pretty close, and my evaluation should give you a good idea of how close.  Feel free to check out the trial for yourself and give them the feedback to nail it.  I want to see tools to make it quicker and help us provide higher availability through systematic tools and interfaces.

After you’ve seen these posts… You should evaluate for yourself (you don’t have to take my word for it) and download the trial of Bamboo’s SharePoint System Log Manager.  If you decide to take it, be sure to use my discount code TABJO08 and save yourself 5% (if it looks like the discount didn’t work, it probably didn’t.  They are working on making it easier to add the code). 

image

Disclosure: If you weren’t aware I am working with Bamboo as a Technical Advisor to give them product feedback and community exposure, but I’m being honest here and I do see this as good for the community.  Give it to me straight if you see it any different.

Enjoy,

Joel Oleson

Secrets of Independent Consulting – What they won’t tell you

After I left Microsoft at the end of March, I had a few offers for big consulting gigs and had some horrible experiences.  Let me share what they won’t tell you…

1) Enteprise Company Politics are Slow (TONS of red tape) – The customer’s SharePoint folks were very excited to have me, but because of the size of the company and company politics, I needed to come on as a sub contractor through MCS.  So talking to the MCS account rep, he setup a call and we discussed it.  The whole account team was very excited.  We can have you start on Monday.  Monday comes, and I’m still there and I send mails with delayed responses.  Oh, sorry, we need to get upper level management approval and your rate is concerning them.  What?  I thought we were past this…  So tell me what they want.  Oh, MCS is adding a cushion?  Sure I expect that.  So will each layer.

2) Statement of Works are a fact of life and completely ridiculous – We know what you’re going to do, but we need a statement of work.  The statement of work might be approved, but it will take a review sometime in the next few weeks.  It might happen as soon as tomorrow or as late as next budget.  Template for the SOW?  Just write something up and we’ll let you know if it works.

3) To be a sub, you need to choose sub contracting company – So I need to get in a company to get in another company?  And we wonder why the rates are a problem.  Independent consulting really doesn’t exist at the enterprise level.

4) What sounds real is fake – the people who want to do things are often completely not enabled to make decisions.  This is very, very much a common problem.  I’m sure SharePoint ISVs see this all the time.

5) Life is completely random – You might get a call in the middle of the night from a client that has to have you on the ground tomorrow, and by the time you get yourself together.  It’s gone.  Something happened, but you don’t know what.  You just know they stopped calling and they don’t return your calls… if you even have their number.  Urgent has been reprioritized.

Lessons Learned – If you want to do consulting, then you likely are better off picking a company to hang your hat on, even if your relationship with them is loose.  Getting insurance can be hard if you have existing conditions that have medication.  Some alumni organizations either from your previous company or from your school may have options for you.  Small and medium companies are actually more agile and can make decisions much faster.  They often are willing to pay the rate and can pay it because they don’t have the multiple layers and red tape.  They want the expert and boom they can have them.

Long story short.  I never ended up at the big enterprise job that had to have me that made me drop other things off my plate.  They wasted a TON of my time and I wouldn’t even give them my time if they doubled my rates due to the way I was treated.  I had an awesome summer doing a ton of fun things from short high impact consulting gigs, training, workshops, and conferences all over the globe.  It was awesome.

Consulting at an independent level is ultimately sales.  So if you don’t like sales you may not like independent consulting.  You have to sell your skills, then you have to sell your solution, then you have to deal with account management.  No guarantees at all along the way.  No matter how much time the customer has invested there is no guarantee it will go through.  Those 2 months where you could make TONS of money could be 2 months of wasted time trying to get the job that gets reprioritized and you’re out all that time.  SAD.

There are upsides to independence don’t get me wrong, but that’s for another post.

SharePoint Plastic Revisited

I was in Burma a few weeks ago and came across this scary Robot Barbie.  It made me realize just how crazy we can make things out of plastic.  Ironically I’ve heard of some pretty crazy SharePoint deployments as well.
 
Robot Barbie
If you’re doing SharePoint to do SharePoint you could end up with a Robot Barbie that won’t sell on any market.

Automating and Scripting SharePoint Administration

First off let me thank Todd Klindt, my good buddy who’s shown me the light on the fact that IT Pro geeks really get into STSADM and the fact that you can get good scores showing off the black command line screen for over an hour.
 
I’ve used a few of his slides as inspiration (plus a bit, thanks buddy) for my talk here in Teched South East Asia.  Wish he was here doing this himself.  Maybe next year.
 
Here’s my Teched SEA Automating SharePoint Administration with STSADM deck.  Click "cancel" if you get prompted.
 
For my Scripted Administration talk I divided it up into 3 main sections.  1) Understanding the command line SharePoint Tools and other cmd line tools, 2) Understanding the containment hierarchy and it’s relationship to the stsadm commands 3) Things you can only do in STSADM and STSADM Extensions and Powershell
 
SharePoint Command Line Tools (Scripted Deployment more in previous post.)
  • Setup.exe
    • Install bits
    • Use Answer file to provide needed details to automate.
  • Psconfig.exe
    • Create or Connect to config db
       Start or stop services
       Change server roles
  • STSAdm.exe
    • Create Web Apps
    • Create Site Collections and sites
    • Backup/Restore
    • Import/Export
    • Maintenance and management
Non SharePoint Command Line Tools
 
IISWeb – query/create/delete empty IIS Web Applications (IIS Web Sites)
 
IISApp – List application pools with process id and use to recycle
 
Copy/Xcopy/Robocopy – Copy and Manage files.  There are various ways to connect and bulk upload.  Bamboo solutions has some nice very cheap import tools.  Beyond this there are a ton of third party tools for migrating data into SharePoint from BinaryTree, Tzunami, Quest, AvePoint and on and on.  Don’t forget Excel and Access and interfaces for getting list data into SharePoint.
 
Containment Hierarchy Creation and Management with Cmline tools
 
Farm
      (Setup.exe and Psconfig.exe) Setconfigdb, AddSolution (for farm solutions/features and below)
Servers
      (Setup.exe and Psconfig.exe) setconfigDB, addServer
Web Applications
      extendVS, extendVSinfarm, createAdminVS, createSSP, addcontentDB, AddZoneUrl, Managepolicypermissionlevel
Databases
     createDATABASE, createSSP, setconfigDB,
Site Collections (Site, SPSite)
     CreateSITE, CreateSITEinnewdb, AddUser, AddGroup
 Sites (Web, SubWeb, SPWeb)
     createWEB
Lists
     (automate list creation through list and site templates) (or with STSADM extensions or Access or Excel to add entire custom lists)
Items
    (copy, xcopy, and robocopy can be used to add documents, use Excel and Access to insert or update items.)
Maintenance/Operations
 
Farm
   Quiesefarm, listlogginglevels, setlogginglevel, backuphistory
Servers
   renameServer
Web Applications
   preparetomove, unextendVS, addpath, editSSP
Databases
   DATABASErepair, preparetomove, AddContentDB
Site Collections (Site, SPSite)
   CreateSITE, CreateSITEinnewdb, GetSITElock, enumSITES
Sites (Web, SubWeb, SPWeb)
   createWEB, renameWeb, enumSUBWEBS
Lists
   forcedeleteLIST
Items
   blockedFILElist
 
Advanced Cmdline Admin
 
STSADM Extensions – Gary Lapoint’s STSADM extensions are the most verbose list of extensions anywhere.  Tons of em.  My favorite… Createsiteindb (create site collection in existing database without all that unnecessary jumping around.)
 
Powershell – There are lots of powershell references out there now.  There’s one codeplex community project at http://www.codeplex.com/pssharepoint  and Darrin Bishop (blog) did a session at TechEd called "ehancing automation with SharePoint" that I recommend.  There are quite a few other powershell projects on codeplex.
 
Caution: All those dispose concerns you might have heard about in development applies to anything you’re doing in Powershell.  There can be common mistakes with not properly disposing when walking the site tree.
 
Also check out SP1 and the Infrastructure update and future updates to include additional things… such as MergeContentdbs the most useful STSADM command for managing your databases.  You might not have heard of a SharePoint Admin Toolkit built by Microsoft.  The Move, Lock, and Delete in Batch in a web UI is very nice.
 
Other SharePoint Admin Tools should also be considered if you’re having manageability issues.  There is a lot of good work being done by third parties to make admin easier.
 
Additional References
 

TechNet STSADM Reference and Poster

Minsharp STSADM References (premium content) Mindsharp.com

Todd Klindt MVP blog

Ben Curry’s “SharePoint Admin Companion” (MS Press Book)

Gary Lapoint’s STSADM extensions

SharePoint Backup and Disaster Recovery Updated Resources for Teched South East Asia

I’m here in Kuala Lumpur doing some last minute prepping for my SharePoint Backup and Disaster Recovery session and figured I should share some of my content.