Fetching all the items for an application -again

Jul 17, 2009 at 9:57 AM

hi andrea,

I mentioned abt this issue before in some discussions, couldnt get an answer yet. I had a test with abt 4 million item records. Application.GetItems() method consumed all the memory I have. It will be working on a server machine and this "fetching all the items" will be executed once when my security service is being started. But I have to consider that I may have 10 millions items.. Then, application.GetItems()  will take too much time and what is more critical is the memory it keeps. I really dont think its very good idea to play with all the items in the memory, items are mostly not needed alltogether for the functional requirements.. So at least it must be optional as per my understanding.

pls let me know what u think ?

regards..

Jul 21, 2009 at 8:54 AM

Hi,

The WCF Service Cache Service, as the term is a service that makes Caching and therefore aims to open a connection to the database only once.

If you have 3 million Items not likely you'll want to use the caching service but:

1) Do not cache anything (make a CheckAccess directly – IAzManStorage.CheckAccess() method)

2) Write your own custom service for caching

Regards,

Andrea.

__________________________________
Andrea Ferendeles
NetSqlAzMan - Project Coordinator

http://netsqlazman.codeplex.com

Jul 27, 2009 at 9:34 AM

hi,

I am not talking abt cache service or any caching needs, the problem is not about querying the storage (checkaccess).

The problem is, when I insert an item over those 3 million items, application.GetItems() method is being executed somehow and it takes those 3 million items (if it didnt get the items already)  into an array (application.Items ) . And then we have a very fat array in the memory from this point.

regards..

 

 

Aug 13, 2009 at 8:07 AM

Hi,

application.GetItems() in never called when you create a new item (at run-time).

It is called only if you are inserting a new Item from the MMC Snap In.

Let me know your steps to reproduce this behavior.

Regards,

Andrea.

__________________________________
Andrea Ferendeles
NetSqlAzMan Project Coordinator  
E-mail aferende@hotmail.com Web http://netsqlazman.codeplex.com

Aug 13, 2009 at 8:56 AM

hi andrea,

I think I wrote the last message wrong..  It was before one bugfix was done, its ok now.

its called when sqlAzmanItem.GetMembers(), GetItemsWhereIAmAMember() methods are called. They get only hierarchy records from db and with ID info, get other info from application.Items property. (Because it knows, application.Items must be there for sure, so for that method its good idea to get those info's from application.Items which was already fetched, but I never want to fetch all the items of an application at once)

Finally, I commented out where application.GetItems() is called, as below

public Dictionary<string, IAzManItem> Items
        {
            get
            {
                if (this.items == null)
                {
                    //this.GetItems();
                }
                return this.items;
            }
        }

 

and in sqlAzmanItem.GetMembers and GetItemsWhereIamAMember methods, made some modification.. to get wht it needs from the db, as below

 

 

public IAzManItem[] GetMembers()
        {
            //var dt = (from v in this.db.ItemsHierarchyView
            //          where v.ItemId == this.itemId
            //          select v.MemberName).ToList();
            //IAzManItem[] result = new IAzManItem[dt.Count];
            //int index = 0;
            //this.members = new Dictionary<string, IAzManItem>();
            //foreach (var row in dt)
            //{
            //    result[index] = this.application.Items[row];
            //    this.members.Add(result[index].Name, result[index]);
            //    index++;
            //}
            //return result;


            //////////////////////////////////////////////////////////////


            IAzManItem[] items;
            var ds = (from tf in this.db.Items()
                      join v in this.db.ItemsHierarchyView on tf.ItemId equals v.MemberItemId
                      where v.ItemId == this.itemId
                      orderby tf.Name
                      select tf).ToList();


            items = new SqlAzManItem[ds.Count];
            int index = 0;
            this.members = new Dictionary<string, IAzManItem>();
            foreach (var row in ds)
            {
                string bizRule = String.Empty;
                NetSqlAzMan.BizRuleSourceLanguage? bizRuleScriptLanguage = null;
                if (row.BizRuleId.HasValue)
                {
                    var bizrule = (from tf in this.db.BizRules()
                                   where tf.BizRuleId == row.BizRuleId
                                   select tf).First();
                    bizRule = bizrule.BizRuleSource;
                    bizRuleScriptLanguage = (NetSqlAzMan.BizRuleSourceLanguage)bizrule.BizRuleLanguage.Value;
                }
                items[index] = new SqlAzManItem(this.db, this.application, row.ItemId.Value, row.Name, row.Description, (ItemType)row.ItemType.Value, bizRule, bizRuleScriptLanguage, this.ens);
                this.members.Add(items[index].Name, items[index]);
                if (this.ens != null)
                    this.ens.AddPublisher(items[index]);
                index++;
            }

            return items;
        }

 

this is enough for me for now. u may want to think to implement something like this. its looking inevitable for such scenerios where u have big number of items

 

thanks.