WCF Cache Refresh during high CheckAccess Activity

May 13, 2009 at 5:51 PM

Hi Andrea,

If we have a high rate of CheckAccess activity going on against the WCF Cache with large volumes, and an Invalidate Cache gets issued - after the cache is built and during the time when the new cache is being switched - what if some clients are in the middle of the CheckAccess code? Will it mess them up or will the cache switch wait for them to finish, as well as keep new requests from starting?

Thanks.

Coordinator
May 13, 2009 at 7:23 PM
Edited May 13, 2009 at 7:28 PM

Hi,

 

the WCF Cache Service is designed to work in multi-user, multi-access and stressed environment.

 

This is the core method (look for Design Feature remarks):

                 internal volatile static bool buildingCache;

         internal static void startStorageBuildCache(string storeName, string applicationName)

        {

            //Design Feature 1: If Not already building cache ... ignore new InvalidateCache

            if (!CacheService.buildingCache)

            {

                CacheService.buildingCache = true;

                WindowsCacheService.writeEvent(String.Format("Invalidate Cache invoked from user '{0}'. Store: '{1}' - Application: '{2}'.", ((System.Threading.Thread.CurrentPrincipal.Identity as WindowsIdentity) ?? WindowsIdentity.GetCurrent()).Name, storeName ?? String.Empty, applicationName ?? String.Empty), System.Diagnostics.EventLogEntryType.Information);

               

                //Design Feature 2: Async Cache Building

                System.Threading.ThreadPool.QueueUserWorkItem(new System.Threading.WaitCallback(o =>

                    {

                        try

                        {

                            StorageCache sc = new StorageCache(Properties.Settings.Default.NetSqlAzManStorageCacheConnectionString);

                            sc.BuildStorageCache(storeName, applicationName);

                            if (CacheService.storageCache != null)

                            {

                                //Design Feature 3: When Build ... replace the old cache with new one

                                //3.1) This means that while building ... User can invoke CheckAccess on the OLD cache

                                //3.2) Replacement is thread safe

                                lock (CacheService.storageCache)

                                {

                                    CacheService.storageCache = sc;

                                }

                            }

                            else

                            {

                                CacheService.storageCache = sc;

                            }

                            WindowsCacheService.writeEvent("Cache Built.", System.Diagnostics.EventLogEntryType.Information);

                        }

                        catch (Exception ex)

                        {

                            WindowsCacheService.writeEvent(String.Format("Cache building error:\r\n{0}\r\n\r\nStack Track:\r\n{1}", ex.Message, ex.StackTrace), System.Diagnostics.EventLogEntryType.Error);

                        }

                        finally

                        {

                            CacheService.buildingCache = false;

                        }

 

                    }

                        ));

            }

            else

            {

                WindowsCacheService.writeEvent("Invalidate Cache invoked while building cache. Command ignored.", System.Diagnostics.EventLogEntryType.Warning);

            }

        }

 

Also the WCF Cache Service was tested using several stress test tools.

Regards,

Andrea.

 

__________________________________
Andrea Ferendeles
NetSqlAzMan - Project Coordinator

http://netsqlazman.codeplex.com

 

 

 

 volatile static bool buildingCache;

May 14, 2009 at 8:04 PM
Edited May 14, 2009 at 8:07 PM

Hi Andrea,

I created a simple application that loops thru the following CheckAccess command. It goes against the WCF Cache using http:

wid = WindowsIdentity.GetCurrent();
string  users = wid.User.Value;
string[] groups = (from x in wid.Groups select  x.ToString()).ToArray();
auth = _cacheServiceClient.CheckAccessForWindowsUsersWithoutAttributesRetrieve(storeName, appName, itemName, users, groups,
DateTime.Now, false, new KeyValuePair<string, object>[] { });

I had 12 separate instances of these loop apps running. When I do repeated WCF cache refreshes, every once in awhile some of the CheckAccess calls will throw one of the following exceptions:

 - ExecuteReader requires an open and available Connection. The connection's current state is open.
 - Invalid attempt to call Read when reader is closed.
 - There is already an open DataReader associated with this Command which must be closed first.
 - An item with the same key has already been added.
 - Invalid attempt to call FieldCount when reader is closed.
 - Unable to cast object of type 'System.Boolean' to type 'System.String'.

Any ideas?

Thanks.

 

Coordinator
May 14, 2009 at 9:29 PM

Hi usood,

I have fixed some WCF Cache Service bugs in the latest release (3.5.4.2).
(Some properties like SqlAzManStorage.Mode, LogErrors, LogOnDb and so on was reader more then once while building the cache in a multi-thread environment … this could be the reason).

Have you tried first with this WCF Cache Service Release ?


Please let me know.

Best Regards,

Andrea.

__________________________________
Andrea Ferendeles
NetSqlAzMan - Project Coordinator

http://netsqlazman.codeplex.com

May 14, 2009 at 9:43 PM

I was using 3.5.2.0 for my test. 

I will upgrade to 3.5.4.2 and try again. Will let you know how that goes.

Thanks.

May 15, 2009 at 9:49 PM
So far it looks good with 3.5.4.2 - I have not encountered any of the previous exceptions. Thanks.
May 20, 2009 at 5:40 PM

Hi Andrea,

The installed 3.5.4.2 version of WCF Cache works fine but when I compile the source code and then run it, it still has the same problem with the exceptions on CheckAccess calls.

Does the source code 3.5.4.2 RTM (23906) have all the latest updates?

Coordinator
May 20, 2009 at 7:57 PM
Sorry I have Checked-In a wrong version.
Please use the change-set 24145 (latest version).
Regards,
Andrea.

Feb 20, 2010 at 11:25 AM

Hi Andrea
I am using the latest version of NetSqlAzMan (V 3.6.0.5)
but I Merged the asp.net membership and NetSqlAzMan Role provider. I Want to Use the aspnet_membership users In NetsqlAzman,therefore I Changed the "netsqlazman_GetDBUsers" function in NetSqlAzManStorage Database to read the Users from AspNet Membership DataBase.

this function After Change :

ALTER FUNCTION [dbo].[netsqlazman_GetDBUsers] (@StoreName nvarchar(255), @ApplicationName nvarchar(255), @DBUserSid VARBINARY(85) = NULL, @DBUserName nvarchar(255) = NULL)
RETURNS TABLE
AS
RETURN

SELECT TOP 100 PERCENT CONVERT(VARBINARY(85), UserId) AS DBUserSid, UserName AS DBUserName FROM [AspNetDataBaseName].[dbo].[aspnet_Users]
WHERE
(@DBUserSid IS NOT NULL AND CONVERT(VARBINARY(85), UserId) = @DBUserSid OR @DBUserSid IS NULL)
AND
(@DBUserName IS NOT NULL AND UserName = @DBUserName OR @DBUserName IS NULL)
ORDER BY UserName


But some times An Error Is Occur. "System.InvalidOperationException: There is already an open DataReader associated with this Command which must be closed first."

In the Next time I Merged 2 Database together and test it but I see this error again

Any ideas?

Thanks.

2010-02-20 13:17:22,611 [11] DEBUG UserControls_MenuAdminAction:System.Web.Services.Protocols.SoapException: System.Web.Services.Protocols.SoapException: Server was unable to process request. ---> System.InvalidOperationException: There is already an open DataReader associated with this Command which must be closed first.
at System.Data.SqlClient.SqlInternalConnectionTds.ValidateConnectionForExecute(SqlCommand command)
at System.Data.SqlClient.SqlConnection.ValidateConnectionForExecute(String method, SqlCommand command)
at System.Data.SqlClient.SqlCommand.ValidateCommand(String method, Boolean async)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, DbAsyncResult result)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)
at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method)
at System.Data.SqlClient.SqlCommand.ExecuteDbDataReader(CommandBehavior behavior)
at System.Data.Common.DbCommand.System.Data.IDbCommand.ExecuteReader(CommandBehavior behavior)
at System.Data.Common.DbDataAdapter.FillInternal(DataSet dataset, DataTable[] datatables, Int32 startRecord, Int32 maxRecords, String srcTable, IDbCommand command, CommandBehavior behavior)
at System.Data.Common.DbDataAdapter.Fill(DataTable[] dataTables, Int32 startRecord, Int32 maxRecords, IDbCommand command, CommandBehavior behavior)
at System.Data.Common.DbDataAdapter.Fill(DataTable dataTable)
at NetSqlAzMan.LINQ.NetSqlAzManStorageDataContext.GetDBUsersEx(String storeName, String applicationName, Byte[] dBUserSid, String dBUserName)
at NetSqlAzMan.SqlAzManStorage.GetDBUser(String userName)
at ePoll_Engine.NetSqlAzManSecurity.Security.CheckAccess(String itemName, String dbUserName, Boolean operationsOnly, KeyValuePair`2[] contextParameters)
at ePoll_Engine.NetSqlAzManSecurity.Security.GetAuthorizationType(ItemName_NetSqlAzManSecurity m_ItemName_NetSqlAzManSecurity, String dbUserName, KeyValuePair`2[] contextParameters)
at ePoll_Engine.NetSqlAzManSecurity.Security.CheckAccess(ItemName_NetSqlAzManSecurity m_ItemName_NetSqlAzManSecurity, String dbUserName, KeyValuePair`2[] contextParameters)
at epollService.CheckAccess(Guid guidProviderUserId, Boolean blnIsUserAuthenticated, ItemName_NetSqlAzManSecurity m_ItemName_NetSqlAzManSecurity, Boolean& blnCheckAccess, Int32& intErrorCode, epollService_Exception& objEpollException)
--- End of inner exception stack trace ---
at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall)
at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters)
at epollService.epollService.CheckAccess(Guid guidProviderUserId, Boolean blnIsUserAuthenticated, ItemName_NetSqlAzManSecurity m_ItemName_NetSqlAzManSecurity, Boolean& blnCheckAccess, Int32& intErrorCode, epollService_Exception& ExceptionInfo)
...............

 

 

 

 

Coordinator
Feb 22, 2010 at 7:21 AM

Hi,

the problem in not in your GetDBUsers sql function but in the NetSqlAzManRoleProvider (open / close connection).

I will check it.

TY.

Andrea.

__________________________________
Andrea Ferendeles
NetSqlAzMan Project Coordinator  
E-mail aferende@hotmail.com Web http://netsqlazman.codeplex.com

Feb 22, 2010 at 7:35 PM
Edited Feb 22, 2010 at 7:35 PM

I was JUST looking into this problem today. I think it is related to LINQ and DataReaders staying open. Here is a link I used to understand what I believe to be the problem. http://www.netknowledgenow.com/blogs/onmaterialize/archive/2006/09/20/fixing-the-_2200_there-is-already-an-open-datareader-associated-with-this-command-which-must-be-closed-first_2e002200_-exception-in-entity-framework.aspx

The solution at this post (MARS setting) seemed to clear up the problem, but I'm still finding that high activity is still giving me strange results that somehow seems to then be invalidating the cache store. Once an initial error occurs, it seems all subsequent calls fail as well.

Coordinator
Feb 24, 2010 at 8:02 AM

Hi,

the problem should be the “shared” connection used in your web service to CheckAccess.

Try to:

1) Dispose() the NetSqlAzManStorage object after each web service call

2) Create a new NetSqlAzManStorage instance foreach web service call

Let me know.

Regards,

Andrea.

__________________________________
Andrea Ferendeles
NetSqlAzMan Project Coordinator  
E-mail aferende@hotmail.com Web http://netsqlazman.codeplex.com

Feb 24, 2010 at 1:35 PM

I'm not sure I understand. I use the StorageCache so that the database doesn't need to be hit everytime (for checkaccess anyway). Isn't the purpose to NOT get rid of the storage on each request? (BTW, are your referring to the NetSqlAzManStorageDataContext?)

One other note I should make for my scenario. I'm not actually checking access at this time (we're not fully implemented yet), but we are using the framework to manage our setup of users, groups, roles, and tasks and their associations (authorizations). So, I noticed that the database is getting hit on every call anyway, which is actually what we want for authorization setup activities since we want the changes to be persisted on every call. It is during this type of activity that I'm seeing the troubles as noted previously in this ticket.

Once I change the MARS setting in the connection string, I'm no longer getting the "There is already an open DataReader..." error, but I am getting the following error at random times. On a call to GetGroups for example, it will work one time, then not work the next (and throw this error) and then work again the next time it is called. It appears to be losing reference to the store when calling the GetStore method:

System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection.
Parameter name: index
   at System.Data.Linq.SqlClient.SqlProvider.Execute(Expression query, QueryInfo queryInfo, IObjectReaderFactory factory, Object[] parentArgs, Object[] userArgs, ICompiledSubQuery[] subQueries, Object lastResult)
   at System.Data.Linq.SqlClient.SqlProvider.ExecuteAll(Expression query, QueryInfo[] queryInfos, IObjectReaderFactory factory, Object[] userArguments, ICompiledSubQuery[] subQueries)
   at System.Data.Linq.SqlClient.SqlProvider.System.Data.Linq.Provider.IProvider.Execute(Expression query)
   at System.Data.Linq.DataQuery`1.System.Linq.IQueryProvider.Execute[S](Expression expression)
   at System.Linq.Queryable.FirstOrDefault[TSource](IQueryable`1 source)
   at NetSqlAzMan.SqlAzManStorage.GetStore(String storeName) in C:\DEV\NetSqlAzMan\SqlAzManStorage.cs:line 403

 

 

Coordinator
Feb 25, 2010 at 6:04 AM

Sorry,

I have confused your issue with saeid_mh83 issue (same thread). J

Could you post the code that use StorageCache ?

Regards,

Andrea.

__________________________________
Andrea Ferendeles
NetSqlAzMan Project Coordinator  
E-mail aferende@hotmail.com Web http://netsqlazman.codeplex.com

Feb 25, 2010 at 3:46 PM

I keep an internal map (hashtable) that holds each application in it's own StorageCache. Here is that code:

private Dictionary<long, StorageCache> m_storageCacheMap = new Dictionary<long,StorageCache>();

internal IAzManApplication GetApplication( long applicationID )
{
    StorageCache storage = this.GetStorageCache( applicationID );
    return storage.Storage.GetStore( "Store1" ).GetApplication( applicationID.ToString() );
}

private StorageCache GetStorageCache( long applicationID )
{
    StorageCache storage = null;

    if( !m_storageCacheMap.ContainsKey( applicationID ) )
    {
        lock( m_mutex )
        {
            if( !m_storageCacheMap.ContainsKey( applicationID ) )
            {
                storage = new StorageCache( m_dbConnectionString );
                storage.BuildStorageCache( "Store1", applicationID.ToString() );
                m_storageCacheMap.Add( applicationID, storage );
            }
        }
    }
    else
    {
        storage = m_storageCacheMap[ applicationID ];
    }

    return storage;
}


the "Index was out of range" Exception that I last posted occurs here:

internal IAzManApplication GetApplication( long applicationID )
{
    StorageCache storage = this.GetStorageCache( applicationID );
    return storage.Storage.GetStore( "Store1" ) [<---exception occurs during this access] .GetApplication( applicationID.ToString() );
}

on this line of the GetStore method in SqlAzManStorage
if ((sr = (from t in this.db.Stores() where t.Name == storeName select t).FirstOrDefault())!=null)

Feb 25, 2010 at 3:47 PM

and... Thank you for your help!!

Coordinator
Feb 25, 2010 at 7:24 PM

Humm,

I suspect that have some thread-unsafe problems because when you are on this line:

if( !m_storageCacheMap.ContainsKey( applicationID ) )

the m_storageCacheMap dictionary is not locked.

Try this:

           Hashtable m_storageCacheMap = Hashtable.Synchronized(new Hashtable()); //Thread-safe

     lock (m_storageCacheMap.SyncRoot)

           {

                if (!m_storageCacheMap.ContainsKey(applicationId))

                {

                    //...

                    m_storageCacheMap.Add(applicationID, storage);

                }

            }

Hashtable is also faster than Dictionary and it is thread safe (if created with the Synchronized method).

Please try this way and let me know if something change.

Regards,

Andrea.

__________________________________
Andrea Ferendeles
NetSqlAzMan Project Coordinator  
E-mail aferende@hotmail.com Web http://netsqlazman.codeplex.com

Mar 4, 2010 at 8:09 PM

Thanks for your response. I've been on some fires the past few days so haven't been able to get back to this yet. I plan on doing some more testing in the next few days.

I'm not sure that threading is the issue since this was occurring on a single user (dev unit test) environment. Just a single request, although there MAY have been some latency during multiple requests in a rapid fire testing scenario.

Let me ask this question. Is it important to keep a copy of the StorageCache available in a static property, or singleton pattern for CheckAccess activity? Also, am I correct in that even through using the StorageCache, any updates to authorizations, roles, groups, etc; the database will be hit? And, any CheckAccess type requests will NOT hit the database? I'm looking back to your reponse about getting rid of the DataStorage on each webservices call, but it seems to me that would not be good since I don't want to be loading up the authorizations every request. Can you clarify?

Thanks again. I look forward to your response about my questions. I hope to post more about the "Index was out of range" issue I was having in the next few days.

Coordinator
Mar 5, 2010 at 1:59 PM

Hi,

In my opinion the best way is that you download the source of the WCF Cache Service and make essentially the same thing but without using WCF.

Inside you'll find all the existing mechanisms to share the statically StorageCache to read it every so often and to ensure continuous operation even when you're rebuilding the StorageCache.

This is a portion of the WCF Cache Service source code:

using System;

using System.Collections.Generic;

using System.Diagnostics;

using System.Linq;

using System.Runtime.Serialization;

using System.ServiceModel;

using System.Text;

using NetSqlAzMan;

using NetSqlAzMan.Interfaces;

using System.Security.Principal;

using System.Configuration;

namespace NetSqlAzMan.Cache.Service

{

    [ServiceBehavior(ConcurrencyMode = ConcurrencyMode.Multiple, InstanceContextMode = InstanceContextMode.Single)]

    public sealed class CacheService : ICacheService

    {

        internal static StorageCache storageCache;

        internal volatile static bool buildingCache;

        internal static Object locker = new Object();

        public CacheService()

        {

            CacheService.buildingCache = false;

        }

        internal static void startStorageBuildCache()

        {

            CacheService.startStorageBuildCache(ConfigurationManager.AppSettings["StoreNameFilter"], ConfigurationManager.AppSettings["ApplicationNameFilter"]);

        }

       

        internal static void startStorageBuildCache(string storeName)

        {

            CacheService.startStorageBuildCache(storeName, String.Empty);

        }

        internal static void startStorageBuildCache(string storeName, string applicationName)

        {

            //Design Feature 1: If Not already building cache ... ignore new InvalidateCache

            lock (CacheService.locker)

            {

                if (!CacheService.buildingCache)

                {

                    CacheService.buildingCache = true;

                    WindowsCacheService.writeEvent(String.Format("Invalidate Cache invoked from user '{0}'. Store: '{1}' - Application: '{2}'.", ((System.Threading.Thread.CurrentPrincipal.Identity as WindowsIdentity) ?? WindowsIdentity.GetCurrent()).Name, storeName ?? String.Empty, applicationName ?? String.Empty), System.Diagnostics.EventLogEntryType.Information);

                    //Design Feature 2: Async Cache Building

                    System.Threading.ThreadPool.QueueUserWorkItem(new System.Threading.WaitCallback(o =>

                        {

                            try

                            {

                                StorageCache sc = new StorageCache(Properties.Settings.Default.NetSqlAzManStorageCacheConnectionString);

                                sc.BuildStorageCache(storeName, applicationName);

                                if (CacheService.storageCache != null)

                                {

                                    //Design Feature 3: When Build ... replace the old cache with new one

                                    //3.1) This means that while building ... User can invoke CheckAccess on the OLD cache

                                    //3.2) Replacement is thread safe

                                    lock (CacheService.storageCache)

                                    {

                                        CacheService.storageCache = sc;

                                    }

                                }

                                else

                                {

                                    CacheService.storageCache = sc;

                                }

                                WindowsCacheService.writeEvent("Cache Built.", System.Diagnostics.EventLogEntryType.Information);

                            }

                            catch (Exception ex)

                            {

                                WindowsCacheService.writeEvent(String.Format("Cache building error:\r\n{0}\r\n\r\nStack Track:\r\n{1}", ex.Message, ex.StackTrace), System.Diagnostics.EventLogEntryType.Error);

                            }

                            finally

                            {

                                lock (CacheService.locker)

                                {

                                    CacheService.buildingCache = false;

                                }

                            }

                        }

                            ));

                }

                else

                {

                    WindowsCacheService.writeEvent("Invalidate Cache invoked while building cache. Command ignored.", System.Diagnostics.EventLogEntryType.Warning);

                }

            }

        }

        #region ICacheService Members

        public void InvalidateCache()

        {

            this.InvalidateCache(true);

        }

        public void InvalidateCache(bool invalidateCacheOnServicePartners)

        {

            Debug.WriteLine("InvalidateCache invoked.");

            if (invalidateCacheOnServicePartners)

            {

                System.Threading.ThreadPool.QueueUserWorkItem(new System.Threading.WaitCallback(

                    delegate(object o)

                    {

                        string[] partnerEndpoints = (string[])o;

                        foreach (string partnerEndpoint in partnerEndpoints)

                        {

                            try

                            {

                                using (WCFCacheServicePartnerServiceReference.CacheServiceClient csc = new NetSqlAzMan.Cache.Service.WCFCacheServicePartnerServiceReference.CacheServiceClient())

                                {

                                    csc.Endpoint.Address = new EndpointAddress(partnerEndpoint);

                                    csc.Open();

                                    csc.BeginInvalidateCacheOnServicePartners(false, null, null);

                                    WindowsCacheService.writeEvent(String.Format("Invalidate Cache invoked on WCF Cache Service Partner: '{0}'.", partnerEndpoint), System.Diagnostics.EventLogEntryType.Information);

                                }

                            }

                            catch (Exception ex)

                            {

                                WindowsCacheService.writeEvent(String.Format("WCF Cache Service Partner error.\r\n Endpoint: '{0}'\r\nError: {1}.", partnerEndpoint, ex.Message), System.Diagnostics.EventLogEntryType.Warning);

                            }

                        }

                    }

                ), ConfigurationManager.AppSettings["WCFCacheServicePartners"].Split(new[] { ';' }, StringSplitOptions.RemoveEmptyEntries));

            }

            CacheService.startStorageBuildCache();

        }

        public void InvalidateStoreCache(string storeName)

        {

            Debug.WriteLine(String.Format("InvalidateStoreCache invoked for Store '{0}'.", storeName));

            CacheService.startStorageBuildCache(storeName);

        }

        public void InvalidateStoreApplicationCache(string storeName, string applicationName)

        {

            Debug.WriteLine(String.Format("InvalidateStoreApplicationCache invoked for Store '{0}' - Application '{1}'.", storeName, applicationName));

            CacheService.startStorageBuildCache(storeName, applicationName);

        }

__________________________________
Andrea Ferendeles
NetSqlAzMan Project Coordinator  
E-mail aferende@hotmail.com Web http://netsqlazman.codeplex.com

Apr 12, 2010 at 2:38 PM

Andrea,

Thanks so much for your response, and so sorry for my delay. I was pulled off on other projects for a while. I'm back to investigate this issue and I have some more detail for you.

I changed my code to ensure no thread access issues are occurring. Also, I'm running a test with a single user, with single requests. I'm sure there is no thread safety issue occurring here.

What I am doing is instantiating a NetSqlAzMan.Cache.StorageCache, and then storing this in a static variable so that on any subsequent requests, it uses the StorageCache that has already been loaded in memory so that the database is not hit on every request. This seems to work fine for the most part. However, randomly on a request, I get an error within NetSqlAzMan when it tries to access the Store. I'm going to past the NetSqlAzMan code here and highlight the line where the error occurs. The error that I get is:

System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection.
Parameter name: index
   at System.Data.Linq.SqlClient.SqlProvider.Execute(Expression query, QueryInfo queryInfo, IObjectReaderFactory factory, Object[] parentArgs, Object[] userArgs, ICompiledSubQuery[] subQueries, Object lastResult)
   at System.Data.Linq.SqlClient.SqlProvider.ExecuteAll(Expression query, QueryInfo[] queryInfos, IObjectReaderFactory factory, Object[] userArguments, ICompiledSubQuery[] subQueries)
   at System.Data.Linq.SqlClient.SqlProvider.System.Data.Linq.Provider.IProvider.Execute(Expression query)
   at System.Data.Linq.DataQuery`1.System.Linq.IQueryProvider.Execute[S](Expression expression)
   at System.Linq.Queryable.FirstOrDefault[TSource](IQueryable`1 source)
   at NetSqlAzMan.SqlAzManStorage.GetStore(String storeName) in C:\DEV\NetSqlAzMan\SqlAzManStorage.cs:line 403

Here is the NetSqlAzMan code where the error occurs:

<font size="2">

 /// <summary>
        /// Opens the specified store name.
        /// </summary>
        /// <param name="storeName">Name of the store.</param>
        /// <returns></returns>
        public IAzManStore GetStore(string storeName)
        {
            StoresResult sr;
            if ((sr = (from t in this.db.Stores() where t.Name == storeName select t).FirstOrDefault())!=null)  //<----------------This line right here is where the index error occurs
            {
                int storeId = sr.StoreId.Value;
                string name = sr.Name;
                string description = sr.Description;
                byte netsqlazmanFixedServerRole = 0;
                if (this.IAmAdmin)
                {
                    netsqlazmanFixedServerRole = 3;
                }
                else
                {
                    var res1 = this.db.CheckStorePermissions(storeId, 2);
                    var res2 = this.db.CheckStorePermissions(storeId, 1);
                    if (res1.HasValue && res1.Value)
                        netsqlazmanFixedServerRole = 2;
                    else if (res2.HasValue && res2.Value)
                        netsqlazmanFixedServerRole = 1;
                }
                IAzManStore result = new SqlAzManStore(this.db, this, storeId, name, description, netsqlazmanFixedServerRole, this.ens);
                this.raiseStoreOpened(result);
                if (this.ens!=null)
                    this.ens.AddPublisher(result);
                return result;
            }
            else
            {
                throw SqlAzManException.StoreNotFoundException(storeName,null);
            }
        }

</font>

 

Any thoughts?  My deadline on this is actually approaching quickly, and I'm hoping I don't have to go to plan B. I really want to be able to use NetSqlAzMan!!!

Thanks again Andrea! I appreciate your help!

Kevin

Apr 12, 2010 at 4:06 PM

This link appears to describe this problem exactly as I'm experiencing it. I'm going to research it further, but if you have any insight with NetSqlAzMan's model and how it might be affected I'd be interested in knowing your thoughts.  http://stackoverflow.com/questions/191690/strange-linq-exception-index-out-of-bounds

Apr 12, 2010 at 7:00 PM

The classes for this library all attempt to use the same data context everywhere.

It is created and then passed around.  So anytime multiple threads (or request in website) are involved there are going to be issues.

Andrea recently resolved the issue with roleprovider by having it instantiate new objects for each request and those objects get a context on every request.  This works, but I think a better solution to resolve that and any other concurrency issue would be to not store and reuse these contexts.   Here is a link to an example factory to get a context that respects both multi-threading and multiple requests.

http://www.west-wind.com/weblog/posts/246222.aspx

I would propose something like this be integrated into the library and the context would no longer be stored in any of the storage/cache/etc classes.   Thoughts?

Coordinator
Apr 14, 2010 at 1:47 PM

Hi All,

I cannot use the “per-thread connection” because Storage and all child-objects could be in a transaction … so 1 transaction … 1 connection.

Regarding khohnbaum issue I believe that everything is solved without the use of static variables. If you need to store everything you need a competition manager (web request with a single user) by locking techniques.

Have you tried with the latest changes I've made?

The problem is it anyway?

__________________________________
Andrea Ferendeles
NetSqlAzMan Project Coordinator
E-mail aferende@hotmail.com Web http://netsqlazman.codeplex.com

Apr 14, 2010 at 2:26 PM

Andrea,

Admittedly, I am on an older verion: 3.5.4.1.  I've made some customizations (pretty minor, only additive work) and I'm just fearful of the amount of work required to upgrade at this point. In the end, I may be forced to upgrade.

Based on your last post, are you saying that the StorageCache is not thread safe?

Kevin

 

Coordinator
Apr 14, 2010 at 2:28 PM

Yes,

- StorageCache is NOT thead-safe

- NetSqlAzManRoleProvider will be (next release but already if you get latest source code)  thread-safe

- NetSqlAzMan WCF Cache Service IS thread-safe (high recommended !!!)

Regards,

Andrea.

__________________________________
Andrea Ferendeles
NetSqlAzMan Project Coordinator
E-mail aferende@hotmail.com Web http://netsqlazman.codeplex.com

Apr 17, 2010 at 7:28 PM

"I cannot use the “per-thread connection” because Storage and all child-objects could be in a transaction … so 1 transaction … 1 connection."

 

You have a transaction that is across threads??   That would not be good anyway.   If you would use a factory like the one I linked to, then your transactions will all use the same connection and work fine.  You start transactions in one thread and finish them in another??

 

Why is storagecache not threadsafe when built?

Coordinator
Apr 26, 2010 at 9:51 AM

Exactly the opposite.

I can not use multiple threads just because I use transactions.

The StorageCache object was not designed thread-safe because the transaction is guaranteed by the WCF Service Cache Service

__________________________________
Andrea Ferendeles
NetSqlAzMan Project Coordinator
E-mail aferende@hotmail.com Web http://netsqlazman.codeplex.com

Apr 26, 2010 at 1:22 PM

Not sure what you are saying..  I am  not saying use multiple threads in the storagecache, but allow the storage cache to be used by multiple threads.

The changes are easy, I have a fully thread safe storage cache working and it works great.  I will contribute my changes if you are interested.

Coordinator
Apr 26, 2010 at 2:24 PM

Ok.

Yes I’m intersted. Have you upload some patch or what ?

Apr 26, 2010 at 2:57 PM

Sure, I will prepare a patch.

 

Apr 26, 2010 at 3:56 PM
Edited Apr 26, 2010 at 3:56 PM

I would be interested as well. Where can I find the patch if you upload it? Wayne, could you email me your thread safe storage cache code? My email starts with kevin_hohnbaum and the domain is yahoo.com.

Apr 26, 2010 at 6:33 PM

khohnbaum,

   The patch involves changes to the StorageCache class, which is in NetSqlAzMan.dll.    I will get a patch prepared shortly.

 

Jun 7, 2010 at 5:29 PM
Sorry for the delay guys. I will get this patch uploaded.
Jun 21, 2010 at 2:01 PM

Hello, Wayne.  Could you provide your version of a thread safe storage cache.  I too, would like to use.

Thanks.

Jun 24, 2010 at 2:50 PM
Edited Jun 24, 2010 at 2:51 PM

Ok guys, it only took me two months to get these changes uploaded...but I have done it.

I have been using it very successfully.

You will find the patch in a new issue.

http://netsqlazman.codeplex.com/workitem/6342

Any comments or improvements are welcome.

 

 

Coordinator
Jun 28, 2010 at 7:02 AM

Hi WayneBrantley,

thx for your collaboration.

 

I have analyzed your solution (patch) and I think that use of ToDictionary(…) and Hashtable may reduce the scalability as well as to use a large amount of memory.

My considerations to solve this issue:

1)      Make StorageCache class thread-safe except for Storage Property

2)      Adopt StorageCache inside NetSqlAzManRoleProvider for all “read-operations” (thread-safe)

3)      Cut “Storage” property from NetSqlAzManRoleProvider (because Storage is not thread-safe, because Linq2Sql (used by Storage) is not thread-safe

4)      Add a thread-safe version of the GetDBUsers inside NetSqlAzManRoleProvider instead of cache all db users (not good for scalability) (agains a new db connection)

5)      Make the NetSqlAzManRoleProvider thread-safe by doing all write-operations against a new db connection

 

I’m working on this solution.

 

Any considerations will be appreciated.

 

Regards,

Andrea.

 

 

__________________________________
Andrea Ferendeles
NetSqlAzMan Project Coordinator  
E-mail aferende@hotmail.com Web http://netsqlazman.codeplex.com

 

Jun 29, 2010 at 2:59 PM

Andrea,
Thanks for investigating.

- Not sure where I used ToDictionary(). The hashtable is created brand new for each request - so that should be safe. Not sure why you are thinking that creating that hashtable on each call is not going to scale, but I am sure there are other solutions. I used that because you had a 'class scoped' hashtable that was used in methods. By making that declared in each method call and still passing it to the methods that need it - made this work across threads. It is working and scaling so far - with no issues of any kind.
- I am storing the cached users in a dictionary - but that is thread safe wrapped and what is stored is very small, just sid and username.

1) You know best.
2) Yes, I only use the role provider for READ ONLY - so i do not use the 'add role' and such from the provider. If I want to do that, I will use your API, which is why I was not worried about that. The reason for caching in the provider is to speed up the constant reading/lookups - not for wirting, so this may be good solution.
3) The storage I did is thread safe for all the reading parts....but it may not be for the writing, which I think is what you are talking about.
4) I am caching the dbusers on purpose - I had previously contributed this code and others are using it successfully too. The same 'give me SID for this username' query is called all the time in the provider - on every request. So caching this actually is great. Caching all the dbusers actually scales quite nicely considering I am only storing a SID and username - which is very small.
5) Probably a good idea.

With the patch I submitted, users can hit my website and there is NO role/rights database activity - and this is a great performance boost (visually noticeable)

Again, thanks for your work.

 

Coordinator
Jun 30, 2010 at 6:40 AM

Hi,

so our two solutions seem quite similar.

But those who do not understand of your solution is that when a user uses the method AddUserToRole (for example) the cache must be invalidated.

You do this automatically or the user must invoke the method InvalidateCache?

If the cache must be invalidated manually ... how other threads (other users) are noticed about the changes?

If you look at my NetSqlAzManRoleProvider code of latest version (3.6.0.6) I should have handled this problem too ... and asynchronously.

Let me know.

Regards,

Andrea.

__________________________________
Andrea Ferendeles
NetSqlAzMan Project Coordinator
E-mail aferende@hotmail.com Web http://netsqlazman.codeplex.com

Jun 30, 2010 at 1:30 PM

The cache must be invalidated manually (it could be automatically done from the AddUseToRole, but as I said I only use the read parts of RoleProvider).  As you noticed, I did not use the cached storage when read/writing, but instead created a new store - so that made it thread safe with writing.   I just did not call the invalidate when that happened - so that is good you did that.

The other threads/users do not need to be notified of the change to the cache.  On EACH request to the website, asp.net checks the users permissions through the role provider.  So, on the next request (after the cache is invalidated) the new permissions just go into effect.

 

I see you did not commit my changes to storagecache, but instead put locks around where you add to the hashtable.   That will not work.  You cannot have a class level variable that is populated by methods which are called from multiple threads.  You need to take it off the class and put it local, passing it into those methods like in my implementation.

 

 

Coordinator
Jun 30, 2010 at 1:56 PM

I have answered inline.

The cache must be invalidated manually (it could be automatically done from the AddUseToRole, but as I said I only use the read parts of RoleProvider). As you noticed, I did not use the cached storage when read/writing, but instead created a new store - so that made it thread safe with writing. I just did not call the invalidate when that happened - so that is good you did that.

[In my version cache is invalidated automatically]

The other threads/users do not need to be notified of the change to the cache. On EACH request to the website, asp.net checks the users permissions through the role provider. So, on the next request (after the cache is invalidated) the new permissions just go into effect.

[What you say is partially correct. Some ASP.NET pages could run asynchronously to the main thread … so they are different threads on the same process (w3wp.exe); carefully; the cache must be invalidated where something changes !]

I see you did not commit my changes to storagecache, but instead put locks around where you add to the hashtable. That will not work. You cannot have a class level variable that is populated by methods which are called from multiple threads. You need to take it off the class and put it local, passing it into those methods like in my implementation.

[I don’t understand this paragraph, please explain with some code fragments. TY]

Regard,
Andrea.
__________________________________
Andrea Ferendeles
NetSqlAzMan Project Coordinator
E-mail aferende@hotmail.com Web http://netsqlazman.codeplex.com

Jun 30, 2010 at 2:11 PM

ASP.NET pages 'could run asynchronously' - on ASP.NET they absolutely will run on multiple threads...each request from the web comes in and takes a thread from the thread pool.

 

Looking at what you commited for storagecache.cs vs what I submitted, the "protected Hastable itemResultCache" variable must be removed from class level scope.

See my changes in the patch file (hope the patch file was done right)...

Thanks.

Coordinator
Jun 30, 2010 at 2:20 PM

Yes … from thread pool but ThreadPool is a “pool” of threads J … so … multiple threads !

http://msdn.microsoft.com/en-us/magazine/cc163725.aspx

Regarding the “protected Hastable itemResultCache” … this is a caller level cache (and not multiuser).

What’s hap if a User call multiple StorageCache methods using multiple threads ? The lock is needed.

Andrea.

__________________________________
Andrea Ferendeles
NetSqlAzMan Project Coordinator
E-mail aferende@hotmail.com Web http://netsqlazman.codeplex.com

Jun 30, 2010 at 2:49 PM

Andrea,

"

Regarding the “protected Hastable itemResultCache” … this is a caller level cache (and not multiuser).

What’s hap if a User call multiple StorageCache methods using multiple threads ? The lock is needed.

"

This is not right.  You have one instance of this class in the RoleProvider.  That role provider is called from multiple threads inside the web server process (this is different than async pages).   That means this itemResultCache is SHARED among the threads.  A lock does not help anything - you cannot share - as the results from one call will mix with another.   Making the itemResultCache a local variable passed to those methods (as I did in my code) makes the lock unnecessary AND makes it threadsafe.

Not sure where you are confused.  Please review my implementation again.

Coordinator
Jun 30, 2010 at 2:57 PM

If you look only at RoleProvider yes … but I’m talking about other Users that does not use NetSqlAzManRoleProvider … but only the StorageCache class in a multi-thread environment.

When I refused your request to join the project was also on that ground.

As you can see we need a vision for most users when produce a change.

However, I am always open to suggestions from the users.

Regards,

Andrea.

Jun 30, 2010 at 3:04 PM

Andrea,

   With my implementation, it does not matter what environment you are in - it is thread safe and will work as designed.  With your implementation, it will not.

 

It really is that simple and consider this my last contribution. 

1)  I do not need to be attacked by you.

2)  I do not need you to refuse my contributions based on your 'supreme vision'.  A contribution is just that - additional features, especially if it does not hurt other users or cause other problems.

3)  Clearly you needed help making this threadsafe or you would have already done it.  Now, I supply code to make it thread safe and you are suddenly an expert and actually undo my work.  

Yours is not thread safe, but you can do whatever you like.  I was trying to help this project and offer users code they needed, but clearly you do not want my help.

Sorry for the trouble my contributions have caused you.

Coordinator
Jun 30, 2010 at 3:08 PM

The beauty of life is that each of us can think freely.

Jun 30, 2010 at 3:09 PM

Ok, thanks.