IRC log of #zope for Wednesday, 2012-05-30

*** andreypopp has joined #zope00:17
*** frapell has quit IRC00:42
*** frapell has joined #zope00:42
*** frapell has joined #zope00:42
*** menesis has quit IRC00:51
*** do3cc has quit IRC01:09
*** evilbungle has quit IRC01:24
*** evilbungle has joined #zope01:25
*** sp0cksbeard has quit IRC01:49
*** daMaestro|isBack has joined #zope01:50
*** daMaestro has quit IRC01:50
*** daMaestro|isBack is now known as daMaestro01:50
*** sp0cksbeard has joined #zope01:51
*** J1m has quit IRC01:53
*** andreypopp has quit IRC02:00
*** daMaestro has quit IRC02:13
*** daMaestro has joined #zope02:13
*** daMaestro has joined #zope02:13
*** Spanktar has joined #zope02:26
*** sp0cksbeard has quit IRC02:28
*** evilbungle has quit IRC02:43
*** _mup_ has quit IRC03:00
*** _mup_ has joined #zope03:00
*** Spanktar has quit IRC03:04
*** zenwryly has quit IRC03:06
*** zenwryly has joined #zope03:07
*** daMaestro has quit IRC03:19
*** J1m has joined #zope03:20
*** tiwula has quit IRC03:44
*** thetet has quit IRC03:47
*** mr_jolly has quit IRC04:26
*** J1m has quit IRC04:35
*** J1m has joined #zope05:25
*** J1m has quit IRC05:26
*** Arfrever has quit IRC05:27
*** dayne has joined #zope06:43
*** dayne1 has joined #zope07:25
*** dayne has quit IRC07:27
*** tiwula has joined #zope07:31
*** tisto has joined #zope07:35
*** nande has joined #zope08:01
nandehi guys i have a question i havent found the answer. I'm using zodb3, and i'm wondering if a PersistentList is lazy loading the contained items or not. In short, should i use a Btree instead of a PersistentList if i have lots of items? only thing i've found is this, but it is not very clear08:03
koshhow many items do you have?08:04
nandeand i'd rather avoid keys if i could08:06
koshlet me look just a bit more but it looks like they won't lazy load08:06
nandebut mostly is like a big question i had since i started using zodb, and in the doc isnt very clear. is something i'd love to know from a zodb dev or something that knows it for sure. because it impact on all  my apps design08:08
koshthe code definitely does not look lazy08:08
koshI am not a zodb dev but I have been programming zope apps for probably about 12-14 years now08:09
koshwhat is your usage case? what are you storing? how often are items added? removed? modified?08:09
nandeas i've said, is something i want to get out of my head, because that question strikes a lot of my design decisions.08:11
koshit looks like it will work like a regular list where the list itself is loaded as a single large unit and all changes are written back as a single large thing but if you store persistent items in them those items are not loaded which is pretty normal08:11
nande i c, that's awesome explanation, you are my hero :D08:12
nandebtw, this case is that i'm doing a simple POS soft, it stores data from clients, products, distributors, etc, and the models are linked08:12
koshI just wondered why you have order but don't care about keys08:13
koshnormally someone would use an OOBTree or if you are inside the management interface use a BTreeFolder208:13
koshoverall I would just stay away from PersistentList, PersistentDict etc08:14
nandewell, i dont like keys.08:14
nandethanks, i'll keep that hint :)08:14
nandeIOBtree is slightly faster than OOBtree right?08:15
kosha list has order so I wondered what you needed the order for or how you find items in it08:15
koshIOBTree is slightly faster and you use an integer as an index08:15
koshjust wondered what you data is that you would not want a key, how do you find items? do you linear scan all 3000 items to find what you want?08:15
nandewell... basically, yes. when the user chooses to link something he would scan. i think there's no way to avoid that, (unless the user knows the code but i dont have "codes" for certain objects)08:17
nandethen i just use the reference to the object itself.08:17
koshthat seems to be a problem08:17
koshhow is a user going to find what they want among 3000 items without any real order to it? especially as you add more08:17
kosha human has a hard enough time trying to find a single item among a few dozen08:18
*** tisto has quit IRC08:18
nandehaha, yeah, well that's why i tried to say, when the user needs to choose or find something, they always enter a fuzzy code, sometimes inexact sometimes they dont remember, so, i have to scan all the objects anyway to find the correct one.08:19
nandefor _some_ objects i do use the code as a key for an OOBtree but .. cant enforce it to every entity08:19
koshhmm I would try to key it by something reasonable (orderid, something like that) and then use a catalog to index the items by a bunch of different fields so you can search08:19
nandethat's interesting, the project is really in an early stage yet. so i'll try that.08:20
nandei just try to avoid forcing keys where there arent. i've used zodb like that once and it was really a paradigm shift from sql08:20
kosheven with sql you try to index your data and set it up so you don't linear scan data08:21
koshlinear scans are expensive!08:21
nandeyeeeeeap, i know :)08:21
koshif you are doing linear scans then who cares if you system is lazy since it won't be lazy in practice since you will load them all anyways to find what you want08:21
koshI have taken systems from doing queries in a few minutes to doing them in under a second by figuring out better ways to index the data so more was not loaded08:22
koshloads are expensive in any database08:22
nandehehe, i know, but since what i'm looking isnt the key, the key is useless to me, because i must scan every object to know which one i'm looking for. anyway dont worry i'll try to do as you say08:22
nandebut, as i've said, i use scanning only in parts where i cant avoid it. in the rest i use direct reference of the object. i asume that isnt as expensive.08:23
koshdirect reference is not expensive on a BTree it does have a much larger expensive with a PersistentList08:24
nandethat's a great hint :) thanks08:24
nandebtw, my system is not web, so i dont need to reference my entities from outside my app. that's why i dont need keys that much either08:27
koshthat is fine I just find that useful keys can save you loading a bunch of other items08:27
koshI did one system where I gave the keys customername_orderid08:28
koshwhich means you can search keys() of the BTree and find what you want, that is still a linear scan if you don't index it but you are only scanning keys not loading the objects08:29
koshmy favorite one though is I use a timestamp for a key08:29
nandeoh i see what are you talking about :)08:29
nandei'll try to think my models more carefully08:29
koshso I can do a search on a BTree.values(min=someminddata, max=somemaxdate)  and find all items in that time period08:29
koshand that is extremely useful08:29
koshsince many items are useful to find by time08:30
nandeyeap it is, i can use that with some of the entities here.08:30
koshI use the file floating point timestamp since then you can use a DataTime object as your min and max and it all works08:30
koshif you use the builtin python date type I would use whatever its full floating point value is so you can get the same effect08:31
*** tisto has joined #zope08:31
nandebtw, when you do min= , the internal search that zodb does is not a linear search right? if its a btree then it does some sort of optimization like balanced tree search?08:31
koshmin and max on a BTree are not linear searches08:32
koshthey are actualy completely lazy at that point even, until you actually try to access the items08:33
koshand it is based on python comparisons08:33
koshso you can do it with any python object that can be ordered08:33
*** andreypopp has joined #zope08:34
nandegreat, for performance then, you recommend to use a fpoint instead of a datetime object?08:34
koshI normally use an OOBTree with floating point as the key for when I will need to order by date08:35
koshon some of my systems I also create a catalog and index certain values in my BTree and give a record id for the catalog as the floating point value08:36
koshwith that you can search a catalog and find all the ids which are your floating point keys you want to load and then load only those keys08:37
nandejust to be sure. IOBT are for integer indexes only right? not fpoint?08:37
nandethat's great, you do your catalog yourself or use some module?08:38
koshI then use BTrees.OOBTree.Intersection so I can grab all the values I want like a set operation between my ids from the catalog and the ids in my BTree08:38
koshyeah IO is only integer08:38
koshfor catalog I actually use ZCatalog in zope, I suspect there are ways you can use it without caring about the web just not sure exactly how08:39
nandei see, i wont mind to build my own, but i bet it wont be as half as good.08:40
koshI would see if you can use ZCatalog or Catalog without the rest of zope08:40
koshmaybe ask on the zope-dev list and see if anyone knows08:41
nandeok, thanks :)08:41
koshdon't reinvent the wheel if you don't have to08:41
koshyour wheel won't be very round, won't roll very well and will probably explode ;)08:41
nandehahaa, very wise :) in fact i do try that, but sometimes i cant find a wheel of my size08:42
*** tiwula has quit IRC08:43
koshhave fun I am heading out08:47
nandeok, thanks a lot :) good luck08:47
*** __mac__ has joined #zope08:58
*** agroszer has joined #zope09:06
*** juh has joined #zope09:49
*** juh has quit IRC09:52
*** juh has joined #zope09:53
*** andreypopp has quit IRC09:59
*** mitchell`off is now known as mitchell`10:05
*** nande has left #zope10:06
*** avoinea has joined #zope10:18
*** tisto has quit IRC10:31
*** tisto has joined #zope10:32
*** maurits has joined #zope10:37
*** maurits has quit IRC11:02
*** do3cc has joined #zope11:03
*** maurits has joined #zope11:07
*** menesis has joined #zope11:26
*** MrTango has joined #zope11:52
*** RichyB has joined #zope12:14
*** fredvd has joined #zope12:33
*** goschtl has joined #zope12:36
*** mr_jolly has joined #zope12:39
*** __mac__ has quit IRC12:51
*** tisto has quit IRC13:11
*** teix has joined #zope13:22
*** tisto has joined #zope13:24
*** mr_jolly_ has joined #zope13:27
*** mr_jolly has quit IRC13:27
*** mr_jolly_ is now known as mr_jolly13:27
*** evilbungle has joined #zope13:35
*** thetet has joined #zope13:35
*** J1m has joined #zope13:40
*** J1m has quit IRC13:44
*** sunew has joined #zope13:58
*** sunew has quit IRC13:59
*** sunew has joined #zope14:04
*** __mac__ has joined #zope14:12
*** menesis has quit IRC14:13
*** moo-_- has quit IRC14:22
*** moo-_- has joined #zope14:29
*** mcdonc has joined #zope14:30
*** mcdonc has quit IRC14:31
*** menesis has joined #zope14:51
*** J1m has joined #zope15:07
*** dayne1 has quit IRC15:36
*** tisto is now known as tisto|lunch15:48
*** sp0cksbeard has joined #zope15:53
*** mcdonc has joined #zope16:00
*** ericof has joined #zope16:11
*** goschtl has quit IRC16:23
*** goschtl has joined #zope16:24
*** moo-_- has quit IRC16:25
*** J1m has quit IRC16:26
*** mcdonc has quit IRC16:29
*** moo-_- has joined #zope16:33
*** J1m has joined #zope16:34
*** mcdonc has joined #zope17:09
*** mcdonc has quit IRC17:19
*** mcdonc has joined #zope17:20
*** tisto|lunch is now known as tisto17:39
*** mr_jolly_ has joined #zope17:40
*** mr_jolly has quit IRC17:40
*** mr_jolly_ is now known as mr_jolly17:40
*** mcdonc has quit IRC17:41
*** goschtl has quit IRC18:01
*** daMaestro has joined #zope18:05
*** maurits has quit IRC18:10
koshhail creatures of evil, pain and suffering!18:13
*** mcdonc has joined #zope18:15
*** tiwula has joined #zope18:19
*** fredvd has quit IRC18:29
*** MrTango has quit IRC18:33
*** zenwryly has quit IRC18:44
*** mitchell` is now known as mitchell`off18:46
*** runyaga has joined #zope18:48
*** runyaga has quit IRC18:48
*** runyaga has joined #zope18:48
*** agroszer has quit IRC18:55
*** RichyB has quit IRC19:01
*** avoinea has quit IRC19:04
*** thetet has quit IRC19:09
*** Spanktar has joined #zope19:23
*** juh has quit IRC19:36
*** f10w has quit IRC19:40
*** f10w has joined #zope19:44
*** moo-_- has quit IRC19:50
*** avoinea has joined #zope19:51
*** do3cc has quit IRC19:51
*** moo-_- has joined #zope19:52
*** mcdonc has quit IRC19:55
*** avoinea has quit IRC20:02
*** urbanape has joined #zope20:04
*** ericof has quit IRC20:07
*** moo-_- has quit IRC20:21
*** tisto has quit IRC20:21
*** f10w has quit IRC20:25
*** f10w has joined #zope20:27
*** moo-_- has joined #zope20:28
*** fredvd has joined #zope21:08
*** zenwryly has joined #zope21:23
*** thetet has joined #zope21:35
*** moo-_- has quit IRC21:36
*** moo-_- has joined #zope21:53
*** menesis has quit IRC22:19
*** mcdonc has joined #zope22:22
*** runyaga has quit IRC22:22
*** mcdonc has quit IRC22:32
*** menesis has joined #zope23:00
*** Arfrever has joined #zope23:21
*** alecm has quit IRC23:22
mgedminzope/server/ has only two untested methods -- so of course the one I need to change is one of them23:28
* mgedmin ponders switching to waitress23:28
*** urbanape has left #zope23:35
*** teix has quit IRC23:35

Generated by 2.15.1 by Marius Gedminas - find it at!