Best Practices for Enterprise User Scalability in SharePoint

Best Practices for Overcoming the 64K ACL, Security Principal Limits, and Future of Identity Management

One of the oldest, but more hidden and illusive capacity boundaries is the 64K ACL limit.  It’s gotten some attention lately, and is bound to scare a few people.  It’s one that takes some thought, and for enterprises it should be part of your security, identity and permissions strategy.  Eli Robilard SharePoint MVP, recently included some best practices down in a post titled "SharePoint Security: Hard Limits and Best Practices."  I may clarify a few of the items in his list in this post.  Todd Klindt, Keith Richie, Eli and I spent some time discussing this on the SharePoint MVP super valuable restricted DL (I couldn’t even respond to, they’d have to forward my responses).  An imortant customer ran into this issue due to lack of use of security groups and we want to make sure it’s clear for both planning and scale.  I also spent some time recently drilling in on a similar vein on the Indexing and Authentication aspects of this thread with the SharePoint CAT (Customer Advisory Team) team getting clarification, discussing Search/Indexing details and providing feedback.  I expect to see more clarity on the software boundaries doc and in various TechNet storage documents on the SharePoint site.

Here’s a quoted copy of the recently updated table on TechNet in the "Software Boundaries" doc.  I also refer to it as the capacity boundaries article if you follow my blog.  You’ll see more limits in this TechNet article than in any of the others.  Need more information than what you see in this post?  I put all of the scale and performance stuff in one resource post – Key Capacity SharePoint Info and with some minimal overlap, resources on Top SharePoint Storage Resources.

People	Guideline	Notes
Users in groups	2 million per Web site	You can add millions of people to your Web site by using Microsoft Windows security groups to manage security instead of using individual users.
User profile	5 million per farm	This number represents the number of profiles which can be imported from a directory service, such as Active Directory, into the people profile store.
Security principal	Approximately 2,000 per ACL (Access Control List) on any securable object (scope)	The total size of the ACL on scopes can be no larger than 64kb. Because each security principal is approximately 32 bytes in size, there can be no more than approximately 2,000 security principals or less for each scope.  If this limit is reached, indexing of items in that scope, and all items below that scope, to fail. Also, because SharePoint Groups are expanded during the indexing process, having more than 2,000 users or Directory Groups in a SharePoint group and using that group for securing scopes may cause indexing of items secured with these groups, and all items below them, to fail. This limit only occurs when Windows Integrated Authentication is utilized.

 

Let me break down this table section by section…

Users in groups – first off, the first "best practice" and logical example here for scale is keyword "ACTIVE DIRECTORY" Security Groups.  SharePoint Cross site groups and permissions levels will NOT allow you to scale.  They are simple containers for ease of management.  You WILL have serious issues if you do not use active directory and you try to add thousands of users using only the SharePoint related groups and permissions levels. 

If you have a site you want to use enterprise wide, but don’t want to use anonymous, use Authenticated Users (which is much better than "everyone") or even better for more security use "domaindomain users" for all of your corporate domains.  This will reduce the risk of trusts that may be setup outside your organization that would put someone in an "authenticated" state.  Many would say it’s a poor practice to use authenticated users.  Here’s a simple table that walk you through common examples with a rating to help you understand best practices.  If you don’t use groups at all you may find the built in groups very handy, but moving to domain groups will assist with better security and moves toward a better practice.

 

Group	Rating for Intranet Content
Anonymous	Worst
Built inGuest	Bad
Built inEveryone	Bad
Built inAuthenticated Users	Ok
ForestDomain Users (Group Designed to includes all domaindomain users)	Good solution
DomainDomain Users	Good
DomainNamed Security Group with users	Better
DomainDomain FTE (Custom Dynamic Group of FTEs)	Best Practice for broad sensitive content

 

From a scalability perspective all of these groups will fit the bill in being able to secure content.  They each differ in what is better use.

To overcome the ACL issue using AD groups will reduce the number of security principals.  Each named account or when named accounts are in a SharePoint permission level or SharePoint group (or cross site groups), each of the users of those SharePoint groups or individuals count as a security principal.  Each AD group counts as one, and if you use built in security groups such as authenticated users, that only counts as one. Each security principal is 32 bytes, and you need to make sure you keep way under 64K.

While you can nest groups, it’s better not to go past a couple of nestings.  You’ll find SharePoint will stop working at somewhere around 4 or more.  I haven’t seen this written anywhere, but I have seen nested DLs in SGs that do fail, and they will.  That is not an acceptable solution.  DG in SG not viable.  SG in SG ok.  SG in SG in SG in SG… pushing it. More below on best practices on proper nesting from AD.

KEY: (DG = Distribution Group, DL = Distribution List (same thing), SG = Security Group, UG = Universal Group very powerful, but lots of replication (AD guys don’t like to create these))

User Info Doesn’t Scale — MYTH!

Some would say the userinfo list has a max of 2000 items, or that you’re limited to 2000 contributors to your site.  This is incorrect.  The Userinfo list is a very special list designed to dynamically scale to for millions of items.  For example, you want to do a company wide survey with a giveaway of a BMW.  You simply secure the survey list with contributor rights to "domaindomain users" with a minimal ACL footprint of a few Kb, and as people fill out the survey they will be added to the Userinfo list the item author of their list entries into the survey.  The index will not have a problem indexing that site collection, and performance while survey’s do not support folders can be exported to Excel or various reports and views can be generated off hours.

The Userinfo lookups are already optimized with indexed columns and optimized for query time.

Don’t confuse the Userinfo list with other SharePoint lists, it is special.

User Profile

The User Profile is a very special list with special views built in.  While the timer jobs to synchronize users will take some very special care and feeding.  There are some optimization techniques of putting like site collections in similar databases to optimize queries, but from a planning perspective the profile scales very well.

Security Principal

What is a security principal?  "A security principal is an entity that can be positively identified and verified via a technique known as authentication.  Security principal accounts are Active Directory objects that are assigned unique security identifiers (SIDs).  A security principal account can be defined as a user account, group account[…] and is also assigned permissions to access certain network resources or Active Directory objects. " (Tech FAQ)

This new detail in the table above has been a problem for companies that have flat user and group structures, no groups, or restricted access to group creation and management for active directory.  What I mean by Flat is a User OU with hundreds of thousands of users and little to no use of groups in active directory.  File servers in the past often had the problem of users being added individually with permissions on items.  Inheritance, broken inheritance in folder structures with massive complexity or rules upon rules as you work your way down the tree.  With SharePoint and granular security, it is easy to fall into the same trap of messy and confusing security principals that are difficult to understand. 

With the features of item level security with security trimmed UI, you’d think it would be simple to see what you have rights to.  Not so.  Many parts of the UI are not security trimmed.  Also when you setup granular security you are making real trade offs with caching, and performance.  All three of these things don’t fit well together.  If you want to cache your page and all your web parts and items, you trade off the use of granular security and security trimmed UI.  Also your performance takes a hit when you setup granular security.  It has a place, but you should limit the use of it.

Using Inheritance for easily managing permissions below is often a best practice for easing administration.  The challenges of broken inheritance, branching in folders, and on items, and differences across various site collections in a haphazard manner can be extremely confusing and difficult to manage.  Delegation is awesome, but often has trade offs with supportability and security.

The Real ACL issue

The ACL issue is a Windows Maximum size for reading the security principals.  It has been around for a long time.  You can see KB 885482: The content index is not updated successfully and error messages are logged to the gatherer log… site contains a large number of groups and users

"This behavior occurs if the size of the access control list (ACL) is larger than 64 kilobytes (KB). The maximum buffer size of the InitializeAcl function is 64 KB. Therefore, the maximum size of an ACL in Windows, including the access control entries (ACEs) that are contained in the ACL, is 64 KB."

You can see this limit is inherited from Windows.  See KB 166348 which applies to all Windows Operating systems. 

"This issue occurs when you reach the maximum size of the access control list (ACL). The size of an ACL varies with the number and size of its access control entries (ACEs). The maximum size of an ACL is 64K, or approximately 1,820 ACEs. However, for performance reasons, the maximum size is not practical."

The Most recent SharePoint specific KB article which again reiterates this spells it out even more clearly with a couple of recommendations… KB 953132

The maximum size of a SharePoint discretionary access control list (DACL) is 64 kilobytes (KB) or approximately 1,820 access control entries (ACEs). Exceeding the 64 KB limit can have a negative effect on the performance of features such as Search and Alerts. We recommend that you do not target the maximum size in your planning.

To add lots of users to a site, to a list, or to a document library, we recommend that you add the users by using domain groups instead of adding users directly or adding users to a SharePoint group. You can use domain groups to reduce the number of ACEs in the SharePoint DACL for the site, for the list, or for the document library to optimize performance.

Did you catch the best practices?  Use Domain Groups instead of adding users directly or adding directly to a SharePoint group.  The other is the explanation that it affects indexing which impacts the availability of search results.  You will see errors in the gatherer log if you hit this limit, but only if you’re looking.  If you’re trying to narrow down a site that is misbehaving or hitting this you can look through your gatherer log for these errors: The address could not be found (0x80040DB4 – The filtering process has been terminated). The address could not be found (0x80042617 – Error in the Site Data Web Service) External component has thrown an exception.  The first can be quite common, so you may need to do some sleuthing.

Need more detail on ACLs and Security Descriptors?  See this TechNet article on how ACLs work.  More on how indexing works in a post I did called Anatomy of Indexing.

FAQ

Q. Wouldn’t my site stop working, or will I have a security issue if I hit this limit? Is it only the crawler?

A. As mentioned in the KB above there is both performance and indexing issues.  The Index will fail to index a scope where there are more than 64K in security principals.  No the WSS API for doing site authorization does not use do it the way that Windows does, and does not have this limit.

Q. What about other types of authentication like Forms Based Authentication?

A. If you have more than 2000 users with forms based authentication you may have issues in the display of those users, but since the index continues to use Windows Authentication and would be indexing with one principal (unless you use security descriptors that map to 64K of ACLed Windows Users in one scope) you won’t experience this.  This shouldn’t be the reason to switch from Windows to something else.

Q. I want the members web part to show my users.  If I use AD groups the membership web part won’t work??

A. For small team and project sites or workspaces, it isn’t as critical.  Its those where thousands of users are in site groups or added individually.  It’s not the thousands that you need in your members web part.

Q. We are doing this in the extranet and using Basic over SSL and hence using Windows and AD.  How can we avoid doing this when we need to add thousands of partners?

A. There are many third party products to do user and group management for an OU.  You may consider doing user management with a tool whose UI sits right in SharePoint to manage AD security groups.  Both Bamboo Solutions and Idevfactory, think even bigger and look at Microsoft’s upcoming Identity Lifecycle Management 2 (currently in beta, RC coming soon, more below).

Q. Why are SharePoint Groups not good enough?

A. SharePoint’s site groups do provide ways of easily adding users across the various sites and managing permissions, but you should add AD security groups when you’re managing hundreds and thousands of users again for similar manageability reasons.  The important thing to note is the Index will expand the SharePoint site groups and each individual will count against the 64K max and 18400 or somewhere around there will cause it to bomb.  Here’s a reminder from the TechNet article "Also, because SharePoint Groups are expanded during the indexing process, having more than 2,000 users or Directory Groups in a SharePoint group and using that group for securing scopes may cause indexing of items secured with these groups, and all items below them, to fail."

Q. Can I use distribution groups?  This would solve my problems…

A. No, distribution groups can’t be used for securing content.  Even in WSS 2.0 when it would expand a DL it would dangerously throw tons of users individually on a site, and wouldn’t maintain the group going forward.  You can work with your AD team to convert the DL (distribution list) to a UG (Email enabled Universal Group).

Q. I’m confused by all these new permissions levels, cross site groups, site groups, etc… Where can I get some real info to help me understand this better.

A. I did a post with detailed info and resources called SharePoint Groups, Permissions, Site Security, and Depreciated Site Groups.  I think a lot of people learned how to do security in SharePoint from the old Windows 4.0 methods.  Here’s how things changed with AD and how it applies to SharePoint.  Here’s a clip from that post…

From Windows security on what to use when creating your AD groups:

"For each group, you need to know what objects it can contain, as well as the overall purpose of the group.

For the group scope, you are determining where the group should be used within the Active Directory enterprise. Your group selection here determines a lot about how you want to use the group within the overall assignment of permissions. Before we discuss each group specifically, the overall picture of group and user nesting is designed to be as follows:

Users go into Global Groups, Global Groups go into Domain Local Groups, and Domain Local Groups are listed on the Access Control List (ACL) of the resource.

Domain Local Groups: (DG)

• Members From Anywhere

• Used in Local Domain

Global Groups: (GG)

• Members From Local Domain

• Used Anywhere

Universal Groups: (UG)

• Members From Anywhere

• Used Anywhere

I highly, highly recommend avoiding using member or server local groups anywhere anymore minus a few such as the built in administrator group.  Otherwise it’s just confusing and painful.

If Universal Groups are used, then the following nesting rules apply:

Users go into Global Groups, Global Groups go into Universal Groups, Universal Groups go into Domain Local Groups, and Domain Local Groups are listed on the Access Control List (ACL) of the resource."

Mantra is AGULP.  Accounts go into Global Groups go into Universal Groups go into Domain Local Groups where they are applied as permissions.

More detail on "How to Nest Users and Groups for Permissions" and good visuals on Ask the MCT on Groups, Groups and more Groups

So what’s the rule for SharePoint?

Just follow your AD best practices above, and during the Permissions part you simply add those groups to permissions levels or roles.  Example, you want to add a couple of users individually as admins, then add your team which should be an existing security group as contributors, and for your broader readers as authenticated users or domain users (best practice above.)  For a small workspace where you want to get membership, you may add a few users individually, this is up to your discretion. 

Essentially you add your existing AD security groups (Global, Domain Local, or Universal (DG and UG preferred) to the SharePoint Group which have the appropriate Roles which have role definitions (permissions levels) which have role assignments (granularity).  Then manage those AD security groups.

So I wanted to give you a heads up on the Future of Identity Management, User and Group Lifecycle and AD.  The future of User Management/Identity Management is management on SharePoint. (http://www.crn.com/software/208403182)  I saw a demo at Tech Ed Australia… awesome demo, very cool!

"There’s a lot of challenges with identities," Muglia told the estimated 10,000 attendees in a keynote. "One of the key things we’re doing with [Identity Lifecycle Management 2] is automating the process of managing the life cycle of identity management. This is a product that’s focused on a problem that effectively all organizations have."

The public beta of Identity Lifecycle Management 2 follows two limited beta releases earlier this year. Microsoft is shooting to debut a release candidate beta of the software in the fourth quarter, with the final release to manufacturing version slated for early 2009, said Douglas Leland, general manager of Microsoft’s identity and access business group, in a press conference following Muglia’s keynote.

Identity management is a hot topic, not just for security but also for complying with company policies and government regulations, and for reducing IT management costs. "Today there is a significant burden on, primarily, IT organizations and help desks for managing identities and access privileges," Leland said. "The state of the art in the market today for identity and access management is, quite frankly, underdelivered."

Leland said Microsoft’s strategy is to provide identity management solutions for on-premise and on-demand applications running either in physical or virtual environments. Microsoft also will build an administration console into SharePoint for managing identities and add user self-service capabilities, such as profile management and password reset, to Microsoft Office.

Clearly spelled out on the Microsoft ILM 2 product page: (Currently in Open Beta!)

• User Management. One of the most important things Microsoft is delivering from a business standpoint is automated, codeless, user provisioning. ILM "2" delivers tools for integrated user management and self-service across enterprise applications without the costly coding of business rules or recoding of the target systems.

• Group Management. ILM ”2” provides powerful capabilities out of the box that help increase the productivity of end users, frees up IT from repetitive tasks and provide better security and compliance outcomes.