| *** andreypopp has joined #zope | 00:17 | |
| *** frapell has quit IRC | 00:42 | |
| *** frapell has joined #zope | 00:42 | |
| *** frapell has joined #zope | 00:42 | |
| *** menesis has quit IRC | 00:51 | |
| *** do3cc has quit IRC | 01:09 | |
| *** evilbungle has quit IRC | 01:24 | |
| *** evilbungle has joined #zope | 01:25 | |
| *** sp0cksbeard has quit IRC | 01:49 | |
| *** daMaestro|isBack has joined #zope | 01:50 | |
| *** daMaestro has quit IRC | 01:50 | |
| *** daMaestro|isBack is now known as daMaestro | 01:50 | |
| *** sp0cksbeard has joined #zope | 01:51 | |
| *** J1m has quit IRC | 01:53 | |
| *** andreypopp has quit IRC | 02:00 | |
| *** daMaestro has quit IRC | 02:13 | |
| *** daMaestro has joined #zope | 02:13 | |
| *** daMaestro has joined #zope | 02:13 | |
| *** Spanktar has joined #zope | 02:26 | |
| *** sp0cksbeard has quit IRC | 02:28 | |
| *** evilbungle has quit IRC | 02:43 | |
| *** _mup_ has quit IRC | 03:00 | |
| *** _mup_ has joined #zope | 03:00 | |
| *** Spanktar has quit IRC | 03:04 | |
| *** zenwryly has quit IRC | 03:06 | |
| *** zenwryly has joined #zope | 03:07 | |
| *** daMaestro has quit IRC | 03:19 | |
| *** J1m has joined #zope | 03:20 | |
| *** tiwula has quit IRC | 03:44 | |
| *** thetet has quit IRC | 03:47 | |
| *** mr_jolly has quit IRC | 04:26 | |
| *** J1m has quit IRC | 04:35 | |
| *** J1m has joined #zope | 05:25 | |
| *** J1m has quit IRC | 05:26 | |
| *** Arfrever has quit IRC | 05:27 | |
| *** dayne has joined #zope | 06:43 | |
| *** dayne1 has joined #zope | 07:25 | |
| *** dayne has quit IRC | 07:27 | |
| *** tiwula has joined #zope | 07:31 | |
| *** tisto has joined #zope | 07:35 | |
| *** nande has joined #zope | 08:01 | |
| nande | hi guys i have a question i havent found the answer. I'm using zodb3, and i'm wondering if a PersistentList is lazy loading the contained items or not. In short, should i use a Btree instead of a PersistentList if i have lots of items? only thing i've found is this, http://stackoverflow.com/questions/8934530/are-zodb-persistentlists-lazily-loaded but it is not very clear | 08:03 |
|---|---|---|
| kosh | how many items do you have? | 08:04 |
| nande | 3K+ | 08:05 |
| nande | and i'd rather avoid keys if i could | 08:06 |
| kosh | let me look just a bit more but it looks like they won't lazy load | 08:06 |
| nande | but mostly is like a big question i had since i started using zodb, and in the doc isnt very clear. is something i'd love to know from a zodb dev or something that knows it for sure. because it impact on all my apps design | 08:08 |
| kosh | the code definitely does not look lazy | 08:08 |
| kosh | I am not a zodb dev but I have been programming zope apps for probably about 12-14 years now | 08:09 |
| kosh | what is your usage case? what are you storing? how often are items added? removed? modified? | 08:09 |
| nande | as i've said, is something i want to get out of my head, because that question strikes a lot of my design decisions. | 08:11 |
| kosh | it looks like it will work like a regular list where the list itself is loaded as a single large unit and all changes are written back as a single large thing but if you store persistent items in them those items are not loaded which is pretty normal | 08:11 |
| nande | i c, that's awesome explanation, you are my hero :D | 08:12 |
| nande | btw, this case is that i'm doing a simple POS soft, it stores data from clients, products, distributors, etc, and the models are linked | 08:12 |
| kosh | I just wondered why you have order but don't care about keys | 08:13 |
| kosh | normally someone would use an OOBTree or if you are inside the management interface use a BTreeFolder2 | 08:13 |
| nande | order? | 08:14 |
| kosh | overall I would just stay away from PersistentList, PersistentDict etc | 08:14 |
| nande | well, i dont like keys. | 08:14 |
| nande | thanks, i'll keep that hint :) | 08:14 |
| nande | IOBtree is slightly faster than OOBtree right? | 08:15 |
| kosh | a list has order so I wondered what you needed the order for or how you find items in it | 08:15 |
| kosh | IOBTree is slightly faster and you use an integer as an index | 08:15 |
| kosh | just wondered what you data is that you would not want a key, how do you find items? do you linear scan all 3000 items to find what you want? | 08:15 |
| nande | well... basically, yes. when the user chooses to link something he would scan. i think there's no way to avoid that, (unless the user knows the code but i dont have "codes" for certain objects) | 08:17 |
| nande | then i just use the reference to the object itself. | 08:17 |
| kosh | that seems to be a problem | 08:17 |
| kosh | how is a user going to find what they want among 3000 items without any real order to it? especially as you add more | 08:17 |
| nande | why? | 08:17 |
| kosh | a human has a hard enough time trying to find a single item among a few dozen | 08:18 |
| *** tisto has quit IRC | 08:18 | |
| nande | haha, yeah, well that's why i tried to say, when the user needs to choose or find something, they always enter a fuzzy code, sometimes inexact sometimes they dont remember, so, i have to scan all the objects anyway to find the correct one. | 08:19 |
| nande | for _some_ objects i do use the code as a key for an OOBtree but .. cant enforce it to every entity | 08:19 |
| kosh | hmm I would try to key it by something reasonable (orderid, something like that) and then use a catalog to index the items by a bunch of different fields so you can search | 08:19 |
| nande | that's interesting, the project is really in an early stage yet. so i'll try that. | 08:20 |
| nande | i just try to avoid forcing keys where there arent. i've used zodb like that once and it was really a paradigm shift from sql | 08:20 |
| kosh | even with sql you try to index your data and set it up so you don't linear scan data | 08:21 |
| kosh | linear scans are expensive! | 08:21 |
| nande | yeeeeeap, i know :) | 08:21 |
| kosh | if you are doing linear scans then who cares if you system is lazy since it won't be lazy in practice since you will load them all anyways to find what you want | 08:21 |
| kosh | I have taken systems from doing queries in a few minutes to doing them in under a second by figuring out better ways to index the data so more was not loaded | 08:22 |
| kosh | loads are expensive in any database | 08:22 |
| nande | hehe, i know, but since what i'm looking isnt the key, the key is useless to me, because i must scan every object to know which one i'm looking for. anyway dont worry i'll try to do as you say | 08:22 |
| nande | but, as i've said, i use scanning only in parts where i cant avoid it. in the rest i use direct reference of the object. i asume that isnt as expensive. | 08:23 |
| kosh | direct reference is not expensive on a BTree it does have a much larger expensive with a PersistentList | 08:24 |
| nande | that's a great hint :) thanks | 08:24 |
| nande | btw, my system is not web, so i dont need to reference my entities from outside my app. that's why i dont need keys that much either | 08:27 |
| kosh | that is fine I just find that useful keys can save you loading a bunch of other items | 08:27 |
| kosh | I did one system where I gave the keys customername_orderid | 08:28 |
| kosh | which means you can search keys() of the BTree and find what you want, that is still a linear scan if you don't index it but you are only scanning keys not loading the objects | 08:29 |
| kosh | my favorite one though is I use a timestamp for a key | 08:29 |
| nande | oh i see what are you talking about :) | 08:29 |
| nande | i'll try to think my models more carefully | 08:29 |
| kosh | so I can do a search on a BTree.values(min=someminddata, max=somemaxdate) and find all items in that time period | 08:29 |
| kosh | and that is extremely useful | 08:29 |
| kosh | since many items are useful to find by time | 08:30 |
| nande | yeap it is, i can use that with some of the entities here. | 08:30 |
| kosh | I use the file floating point timestamp since then you can use a DataTime object as your min and max and it all works | 08:30 |
| kosh | if you use the builtin python date type I would use whatever its full floating point value is so you can get the same effect | 08:31 |
| *** tisto has joined #zope | 08:31 | |
| nande | btw, when you do min= , the internal search that zodb does is not a linear search right? if its a btree then it does some sort of optimization like balanced tree search? | 08:31 |
| kosh | min and max on a BTree are not linear searches | 08:32 |
| kosh | they are actualy completely lazy at that point even, until you actually try to access the items | 08:33 |
| kosh | and it is based on python comparisons | 08:33 |
| kosh | so you can do it with any python object that can be ordered | 08:33 |
| *** andreypopp has joined #zope | 08:34 | |
| nande | great, for performance then, you recommend to use a fpoint instead of a datetime object? | 08:34 |
| kosh | yes | 08:35 |
| kosh | I normally use an OOBTree with floating point as the key for when I will need to order by date | 08:35 |
| kosh | on some of my systems I also create a catalog and index certain values in my BTree and give a record id for the catalog as the floating point value | 08:36 |
| kosh | with that you can search a catalog and find all the ids which are your floating point keys you want to load and then load only those keys | 08:37 |
| nande | just to be sure. IOBT are for integer indexes only right? not fpoint? | 08:37 |
| nande | that's great, you do your catalog yourself or use some module? | 08:38 |
| kosh | I then use BTrees.OOBTree.Intersection so I can grab all the values I want like a set operation between my ids from the catalog and the ids in my BTree | 08:38 |
| kosh | yeah IO is only integer | 08:38 |
| kosh | for catalog I actually use ZCatalog in zope, I suspect there are ways you can use it without caring about the web just not sure exactly how | 08:39 |
| nande | i see, i wont mind to build my own, but i bet it wont be as half as good. | 08:40 |
| kosh | I would see if you can use ZCatalog or Catalog without the rest of zope | 08:40 |
| kosh | maybe ask on the zope-dev list and see if anyone knows | 08:41 |
| nande | ok, thanks :) | 08:41 |
| kosh | don't reinvent the wheel if you don't have to | 08:41 |
| kosh | your wheel won't be very round, won't roll very well and will probably explode ;) | 08:41 |
| nande | hahaa, very wise :) in fact i do try that, but sometimes i cant find a wheel of my size | 08:42 |
| *** tiwula has quit IRC | 08:43 | |
| kosh | have fun I am heading out | 08:47 |
| nande | ok, thanks a lot :) good luck | 08:47 |
| *** __mac__ has joined #zope | 08:58 | |
| *** agroszer has joined #zope | 09:06 | |
| *** juh has joined #zope | 09:49 | |
| *** juh has quit IRC | 09:52 | |
| *** juh has joined #zope | 09:53 | |
| *** andreypopp has quit IRC | 09:59 | |
| *** mitchell`off is now known as mitchell` | 10:05 | |
| *** nande has left #zope | 10:06 | |
| *** avoinea has joined #zope | 10:18 | |
| *** tisto has quit IRC | 10:31 | |
| *** tisto has joined #zope | 10:32 | |
| *** maurits has joined #zope | 10:37 | |
| *** maurits has quit IRC | 11:02 | |
| *** do3cc has joined #zope | 11:03 | |
| *** maurits has joined #zope | 11:07 | |
| *** menesis has joined #zope | 11:26 | |
| *** MrTango has joined #zope | 11:52 | |
| *** RichyB has joined #zope | 12:14 | |
| *** fredvd has joined #zope | 12:33 | |
| *** goschtl has joined #zope | 12:36 | |
| *** mr_jolly has joined #zope | 12:39 | |
| *** __mac__ has quit IRC | 12:51 | |
| *** tisto has quit IRC | 13:11 | |
| *** teix has joined #zope | 13:22 | |
| *** tisto has joined #zope | 13:24 | |
| *** mr_jolly_ has joined #zope | 13:27 | |
| *** mr_jolly has quit IRC | 13:27 | |
| *** mr_jolly_ is now known as mr_jolly | 13:27 | |
| *** evilbungle has joined #zope | 13:35 | |
| *** thetet has joined #zope | 13:35 | |
| *** J1m has joined #zope | 13:40 | |
| *** J1m has quit IRC | 13:44 | |
| *** sunew has joined #zope | 13:58 | |
| *** sunew has quit IRC | 13:59 | |
| *** sunew has joined #zope | 14:04 | |
| *** __mac__ has joined #zope | 14:12 | |
| *** menesis has quit IRC | 14:13 | |
| *** moo-_- has quit IRC | 14:22 | |
| *** moo-_- has joined #zope | 14:29 | |
| *** mcdonc has joined #zope | 14:30 | |
| *** mcdonc has quit IRC | 14:31 | |
| *** menesis has joined #zope | 14:51 | |
| _mup_ | Branch | 15:03 |
| _mup_ | Branch | 15:05 |
| *** J1m has joined #zope | 15:07 | |
| *** dayne1 has quit IRC | 15:36 | |
| *** tisto is now known as tisto|lunch | 15:48 | |
| *** sp0cksbeard has joined #zope | 15:53 | |
| *** mcdonc has joined #zope | 16:00 | |
| *** ericof has joined #zope | 16:11 | |
| *** goschtl has quit IRC | 16:23 | |
| *** goschtl has joined #zope | 16:24 | |
| *** moo-_- has quit IRC | 16:25 | |
| *** J1m has quit IRC | 16:26 | |
| *** mcdonc has quit IRC | 16:29 | |
| *** moo-_- has joined #zope | 16:33 | |
| *** J1m has joined #zope | 16:34 | |
| *** mcdonc has joined #zope | 17:09 | |
| *** mcdonc has quit IRC | 17:19 | |
| *** mcdonc has joined #zope | 17:20 | |
| *** tisto|lunch is now known as tisto | 17:39 | |
| *** mr_jolly_ has joined #zope | 17:40 | |
| *** mr_jolly has quit IRC | 17:40 | |
| *** mr_jolly_ is now known as mr_jolly | 17:40 | |
| *** mcdonc has quit IRC | 17:41 | |
| *** goschtl has quit IRC | 18:01 | |
| *** daMaestro has joined #zope | 18:05 | |
| *** maurits has quit IRC | 18:10 | |
| kosh | hail creatures of evil, pain and suffering! | 18:13 |
| *** mcdonc has joined #zope | 18:15 | |
| *** tiwula has joined #zope | 18:19 | |
| *** fredvd has quit IRC | 18:29 | |
| *** MrTango has quit IRC | 18:33 | |
| *** zenwryly has quit IRC | 18:44 | |
| *** mitchell` is now known as mitchell`off | 18:46 | |
| *** runyaga has joined #zope | 18:48 | |
| *** runyaga has quit IRC | 18:48 | |
| *** runyaga has joined #zope | 18:48 | |
| *** agroszer has quit IRC | 18:55 | |
| *** RichyB has quit IRC | 19:01 | |
| *** avoinea has quit IRC | 19:04 | |
| *** thetet has quit IRC | 19:09 | |
| *** Spanktar has joined #zope | 19:23 | |
| *** juh has quit IRC | 19:36 | |
| *** f10w has quit IRC | 19:40 | |
| *** f10w has joined #zope | 19:44 | |
| *** moo-_- has quit IRC | 19:50 | |
| *** avoinea has joined #zope | 19:51 | |
| *** do3cc has quit IRC | 19:51 | |
| *** moo-_- has joined #zope | 19:52 | |
| *** mcdonc has quit IRC | 19:55 | |
| *** avoinea has quit IRC | 20:02 | |
| *** urbanape has joined #zope | 20:04 | |
| *** ericof has quit IRC | 20:07 | |
| *** moo-_- has quit IRC | 20:21 | |
| *** tisto has quit IRC | 20:21 | |
| *** f10w has quit IRC | 20:25 | |
| *** f10w has joined #zope | 20:27 | |
| *** moo-_- has joined #zope | 20:28 | |
| *** fredvd has joined #zope | 21:08 | |
| *** zenwryly has joined #zope | 21:23 | |
| *** thetet has joined #zope | 21:35 | |
| *** moo-_- has quit IRC | 21:36 | |
| *** moo-_- has joined #zope | 21:53 | |
| *** menesis has quit IRC | 22:19 | |
| *** mcdonc has joined #zope | 22:22 | |
| *** runyaga has quit IRC | 22:22 | |
| *** mcdonc has quit IRC | 22:32 | |
| *** menesis has joined #zope | 23:00 | |
| *** Arfrever has joined #zope | 23:21 | |
| *** alecm has quit IRC | 23:22 | |
| mgedmin | zope/server/taskthreads.py has only two untested methods -- so of course the one I need to change is one of them | 23:28 |
| * mgedmin ponders switching to waitress | 23:28 | |
| *** urbanape has left #zope | 23:35 | |
| *** teix has quit IRC | 23:35 | |
Generated by irclog2html.py 2.15.1 by Marius Gedminas - find it at mg.pov.lt!