IRC log of #zope3-dev for Wednesday, 2008-03-19

*** regebro has left #zope3-dev00:01
*** lucielejard has quit IRC00:01
*** ktwilight has joined #zope3-dev00:05
*** alecm has quit IRC00:11
*** ktwilight_ has quit IRC00:18
*** rcrafton has quit IRC00:28
*** gimni has quit IRC00:29
*** acsr__ has quit IRC00:31
*** acsr has joined #zope3-dev00:37
*** reco has quit IRC00:50
*** whit has quit IRC00:57
*** b52laptop has joined #zope3-dev00:58
*** d2m has quit IRC00:58
*** greenman has joined #zope3-dev01:03
*** natea_ has quit IRC01:10
*** J1m has quit IRC01:17
*** reedobrien has quit IRC01:19
*** b52laptop has quit IRC01:27
*** rmarianski has quit IRC01:28
*** dunny has quit IRC01:34
*** ccomb has left #zope3-dev01:41
*** harobed has quit IRC01:44
*** dunny has joined #zope3-dev01:55
*** norro has quit IRC01:57
*** RaFromBRC has joined #zope3-dev02:02
*** mcdonc has joined #zope3-dev02:07
*** MrTopf has quit IRC02:16
*** natea_ has joined #zope3-dev02:24
*** ignas has quit IRC02:24
*** mcdonc has quit IRC02:26
*** pcardune has quit IRC02:33
*** mcdonc has joined #zope3-dev02:39
*** projekt01 has joined #zope3-dev02:42
*** hexsprite_ has quit IRC02:43
*** CSWookie has joined #zope3-dev02:51
*** natea_ has quit IRC02:51
CSWookieIs there a project that does for function what zope.schema does for classes?  I'd like to be able to define what types of arguments a function ought to take and what type of return value it should return.  Is there something in zope.schema I haven't seen?02:53
*** natea_ has joined #zope3-dev02:54
*** mcdonc has quit IRC02:55
*** edgordon has joined #zope3-dev02:55
*** jpcw2002 has quit IRC02:55
*** philiKON has quit IRC03:03
*** georgy has joined #zope3-dev03:07
*** nathany has quit IRC03:07
*** febb has quit IRC03:21
*** alecm has joined #zope3-dev03:24
*** dtremea has joined #zope3-dev03:32
*** dtremea is now known as deo03:37
*** hexsprite has joined #zope3-dev03:44
*** greenman has quit IRC03:53
*** georgy has quit IRC03:54
*** greenman has joined #zope3-dev03:55
*** natea_ has quit IRC04:05
*** natea_ has joined #zope3-dev04:06
*** srichter has quit IRC04:07
*** derek|home has joined #zope3-dev04:08
*** tarek has quit IRC04:09
*** niemeyer has quit IRC04:18
*** stub has joined #zope3-dev04:52
*** RaFromBRC has quit IRC04:58
*** ignas has joined #zope3-dev05:09
*** sm has quit IRC05:10
*** deo has quit IRC05:28
*** derek|home has quit IRC05:34
*** greenman has quit IRC05:34
*** rmarianski has joined #zope3-dev05:52
*** bigkevmcd has quit IRC06:03
*** redir has joined #zope3-dev06:08
*** alecm has quit IRC06:24
*** afd_ has joined #zope3-dev06:41
*** RaFromBRC has joined #zope3-dev06:46
*** afd_ has quit IRC07:11
*** srichter has joined #zope3-dev07:13
*** ChanServ sets mode: +o srichter07:13
*** rmarianski has quit IRC07:17
*** srichter has quit IRC07:20
*** stub has quit IRC07:21
*** srichter has joined #zope3-dev07:21
*** timte has quit IRC07:23
*** yvl has joined #zope3-dev07:24
*** redir has quit IRC07:25
*** ChanServ sets mode: +o srichter07:32
*** sm has joined #zope3-dev07:43
*** projekt01 has quit IRC07:48
*** redir has joined #zope3-dev07:55
*** srichter has quit IRC07:57
*** mcdonc has joined #zope3-dev08:02
*** menesis has joined #zope3-dev08:05
*** jayaraj has joined #zope3-dev08:09
*** ignas has quit IRC08:09
*** dobee__ has joined #zope3-dev08:17
*** afd_ has joined #zope3-dev08:23
*** jukart has joined #zope3-dev08:25
*** djohnson has quit IRC08:27
*** sm has quit IRC08:39
*** sorin has joined #zope3-dev08:57
*** sorin is now known as sorindregan08:57
*** RaFromBRC has quit IRC08:57
*** stub has joined #zope3-dev08:59
*** dobee__ has quit IRC08:59
*** redir has quit IRC09:00
*** hdima has joined #zope3-dev09:01
*** greenman has joined #zope3-dev09:04
*** __mac__ has joined #zope3-dev09:05
*** dobee_ has joined #zope3-dev09:06
*** dobee_ is now known as dobee09:09
*** huajie has joined #zope3-dev09:34
*** jukart has quit IRC09:36
*** jukart has joined #zope3-dev09:36
*** goschtl has joined #zope3-dev09:39
*** philiKON has joined #zope3-dev09:43
*** malthe has quit IRC09:45
*** malthe has joined #zope3-dev09:45
*** markusleist has quit IRC09:56
*** tarek has joined #zope3-dev09:59
*** bigkevmcd has joined #zope3-dev10:00
*** quodt has joined #zope3-dev10:05
*** tarek has quit IRC10:11
*** tarek has joined #zope3-dev10:12
*** markusleist has joined #zope3-dev10:16
*** d2m has joined #zope3-dev10:18
*** jodok has joined #zope3-dev10:23
*** harobed has joined #zope3-dev10:31
*** jpcw2002 has joined #zope3-dev10:33
*** __mac__ has left #zope3-dev10:33
*** maurits has joined #zope3-dev10:34
*** zagy has joined #zope3-dev10:43
*** vaish has joined #zope3-dev10:44
*** philiKON has quit IRC10:44
*** jodok has quit IRC10:47
*** philiKON has joined #zope3-dev11:00
*** malthe has quit IRC11:02
*** MJ has joined #zope3-dev11:11
*** vaish has left #zope3-dev11:27
*** b52laptop has joined #zope3-dev11:34
*** dunny has quit IRC11:36
*** norro has joined #zope3-dev11:38
*** thruflo has joined #zope3-dev11:40
*** QuyetNextG has joined #zope3-dev11:46
*** QuyetNextG has quit IRC11:51
*** malthe has joined #zope3-dev11:56
*** toutpt has joined #zope3-dev12:12
*** rocky has quit IRC12:13
*** sunew has joined #zope3-dev12:16
*** rocky has joined #zope3-dev12:18
*** gimni has joined #zope3-dev12:27
*** timte has joined #zope3-dev12:27
*** regebro has joined #zope3-dev12:30
*** thruflo is now known as devilsadvocate12:30
*** devilsadvocate is now known as thruflo12:34
*** pelle_ has joined #zope3-dev12:40
*** andy-chen has joined #zope3-dev12:43
*** menesis has quit IRC12:45
andy-chenhi,all. what's the best practice to communicate between two zope3 instance run on their own server? xml-rpc ?json-rpc?or something else12:53
andy-chenwe need use something based on http...12:54
*** greenman has quit IRC12:56
*** ghendi_ has joined #zope3-dev12:57
*** timte has quit IRC12:58
*** thruflo has left #zope3-dev12:59
*** timte has joined #zope3-dev12:59
*** andy-chen has quit IRC13:00
*** djohnson_ has joined #zope3-dev13:02
*** b52laptop has quit IRC13:10
*** b52lap has joined #zope3-dev13:10
*** zagy has quit IRC13:11
*** zagy has joined #zope3-dev13:12
*** mkerrin has joined #zope3-dev13:16
*** timte has quit IRC13:17
*** toutpt has quit IRC13:32
*** tarek_ has joined #zope3-dev13:34
*** tarek has quit IRC13:40
*** MJ is now known as MJ|lunch13:48
*** sunew has quit IRC13:51
*** FFighter has joined #zope3-dev13:55
*** benji has joined #zope3-dev14:15
*** djohnson_ has quit IRC14:17
*** sunew has joined #zope3-dev14:19
*** timte has joined #zope3-dev14:19
*** timte has quit IRC14:20
*** timte has joined #zope3-dev14:21
*** gimni has quit IRC14:23
*** malthe has quit IRC14:34
*** malthe has joined #zope3-dev14:36
*** niemeyer has joined #zope3-dev14:36
*** ccomb has joined #zope3-dev14:52
*** b52lap has quit IRC14:59
*** b52lap has joined #zope3-dev15:03
*** sorindregan has quit IRC15:10
*** projekt01 has joined #zope3-dev15:11
*** lucielejard has joined #zope3-dev15:13
*** lucielejard has quit IRC15:14
*** lucielejard has joined #zope3-dev15:14
*** J1m has joined #zope3-dev15:15
*** alga has quit IRC15:17
*** srichter has joined #zope3-dev15:23
*** MJ|lunch is now known as MJ15:28
*** rcrafton has joined #zope3-dev15:30
*** afd__ has joined #zope3-dev15:42
*** srichter has quit IRC15:55
*** afd_ has quit IRC15:59
*** sorin has joined #zope3-dev16:11
*** sorin is now known as sorindregan16:11
*** djohnson has joined #zope3-dev16:13
*** pcardune has joined #zope3-dev16:18
*** toutpt has joined #zope3-dev16:24
*** afd__ has quit IRC16:24
*** alga has joined #zope3-dev16:24
*** hdima has quit IRC16:42
*** nathany has joined #zope3-dev16:43
*** djohnson has quit IRC16:44
*** jayaraj has quit IRC16:50
*** mgedmin has joined #zope3-dev16:50
*** huajie has quit IRC17:00
*** sm has joined #zope3-dev17:07
*** afd_ has joined #zope3-dev17:14
*** mgedmin has quit IRC17:16
*** stub has quit IRC17:17
*** gimni has joined #zope3-dev17:18
*** srichter has joined #zope3-dev17:24
*** ChanServ sets mode: +o srichter17:25
*** ignas has joined #zope3-dev17:26
*** regebro has quit IRC17:27
afd_malthe: did you had time to benchmark running zope without object cache?17:30
*** salfield has quit IRC17:30
maltheafd_: I've only done preliminary tests; with 17 objects I had a 20% penalty, with 10,000 I had a 800% penalty. This is very approximate.17:33
afd_any way to configure the system to improve this?17:33
maltheso the key is to a) limit the number of persistant object in use, b) keep module-level read-only caches.17:33
afd_"module level read-only" cache, what does it mean?17:34
*** timte has quit IRC17:34
malthewell if such a cache is to be shared among threads, it must be under some write control.17:34
afd_sorry to bug you, I'm just curios17:34
maltheand you probably don't want per-thread caches (that would defeat the purpose).17:35
afd_what's the idea? or purpose...17:35
malthethe motivation is to be able to limit the memory consumption per thread.17:36
afd_I see17:36
malthetwo reasons: a) cheap hosting, b) network-intensive application that are waiting for I/O all the time.17:36
malthe(b) could be an application that has, say, 20 GB of data17:37
maltheit'll always have to pull data in at every request17:37
*** yvl has quit IRC17:37
malthethis causes that thread to stay idle while it does I/O---what's the CPU going to do meanwhile?17:37
*** whit has joined #zope3-dev17:38
maltheand it can't spawn more than a few threads, because each thread take up too much memory.17:38
afd_I see17:38
maltheif the threads aren't allowed to use memory, they operate slowly.17:38
malthethat's status quo.17:38
maltheso I was just investigating how to have them operate faster without using memory.17:39
afd_I see17:39
*** reco has joined #zope3-dev17:39
afd_what about an app that uses the catalog?17:39
afd_I understand the catalog is fast only if it loads its btree indexes in memory17:39
maltheyes that's also correct; so that will have to be dealt with somehow.17:40
maltheI think Lucene or Xapian is the answer here17:40
malthesimply take it out of the equation altogether17:41
malthemakes sense, too, because a catalog result is a very formalized dataset.17:41
maltheit's a good fit for an external database17:41
afd_I see17:41
*** mgedmin has joined #zope3-dev17:41
afd_ok, thanks for your thoughts :)17:42
*** alga_ has joined #zope3-dev17:42
*** hexsprite has quit IRC17:49
*** alga has quit IRC17:50
*** menesis has joined #zope3-dev17:51
*** hexsprite has joined #zope3-dev17:54
*** jsadjohnson has joined #zope3-dev17:54
*** agroszer has joined #zope3-dev18:00
*** MJ has quit IRC18:15
*** dobee has quit IRC18:17
*** romanofski has quit IRC18:18
*** pelle_ has quit IRC18:20
*** pelle_ has joined #zope3-dev18:22
tarek_Hi J1m, to avoid easy_install hitting pypi (in an intranet that does not have a web connection) i use the --allow-host option. It does not seem to get caught into buildout though.18:25
tarek_is there a way to prevent some hosts to be called in buildout (in online mode) ?18:25
tarek_we have a local apache there, that has the eggs18:26
srichtertarek_: you can use the index option18:26
srichteror "find-links"18:26
*** jukart has quit IRC18:27
tarek_srichter, hi18:27
tarek_srichter, the find-links is just adding new sources, bu will not prevent from hiting pypi.python.org18:27
tarek_this is the --allow-host option, (-H) in easy_install18:27
srichterusing  "index"18:28
srichterit allows you to point to a different index, such as PPIX18:28
tarek_i have a flat folder with eggs, this would mean I need to create an index structure then, such as PPIX right ?18:30
*** edgordon has quit IRC18:30
srichtertarek_: well, what is your goal?18:37
*** jodok has joined #zope3-dev18:38
*** sunew has quit IRC18:39
*** natea_ is now known as natea18:39
tarek_srichter, not bathering creating an index page, and using the allow-host option18:40
tarek_the use case is quite simple:18:40
tarek_having a local apache that provides all eggs through find-links18:41
tarek_and preventing buildout to hit pypi.18:41
tarek_this is straight forward with easy_install :18:42
srichterthen I think you need to create an index in buildout18:42
srichteror point to a non-working index and use find-links18:42
tarek_easy_install -H *.local.server -f http://local.server/eggs my.package18:42
tarek_ah... let me try the non working index ...18:43
srichterI think it is okay pointing index to http://local.server/eggs18:43
tarek_if it's a flat folder, it will fail afaik:18:43
* tarek_ is trying something again :)18:44
*** edgordon has joined #zope3-dev18:45
*** menesis has left #zope3-dev18:48
*** salfield has joined #zope3-dev18:55
*** b52lap has quit IRC18:55
*** J1m has quit IRC19:07
*** rock1 has joined #zope3-dev19:13
*** gimni has quit IRC19:13
*** harobed has quit IRC19:15
*** rocky has quit IRC19:17
*** rock1 is now known as rocky19:17
*** derek|office has quit IRC19:22
*** sorindregan has quit IRC19:26
*** quodt has quit IRC19:27
*** RaFromBRC has joined #zope3-dev19:30
*** edgordon has quit IRC19:31
*** sunew has joined #zope3-dev19:37
*** maurits has quit IRC19:38
*** pelle_ has quit IRC19:39
*** dobee has joined #zope3-dev19:41
*** projekt01 has quit IRC19:44
*** toutpt has quit IRC19:48
*** sunew has quit IRC19:54
*** malthe has quit IRC19:55
*** goschtl has quit IRC19:58
*** sunew has joined #zope3-dev20:03
*** sunew has quit IRC20:10
*** antiretorte has joined #zope3-dev20:12
*** quodt has joined #zope3-dev20:12
*** antiretorte has quit IRC20:13
*** whit has quit IRC20:16
*** alga_ has quit IRC20:22
*** agroszer_ has joined #zope3-dev20:28
*** mkerrin has quit IRC20:28
*** djohnson has joined #zope3-dev20:34
*** regebro has joined #zope3-dev20:35
*** agroszer has quit IRC20:40
*** quodt_ has joined #zope3-dev20:44
*** quodt__ has joined #zope3-dev20:46
*** quodt has quit IRC20:46
*** whit has joined #zope3-dev20:47
*** jodok has quit IRC20:48
*** jodok has joined #zope3-dev20:50
*** jodok has quit IRC20:52
*** jodok has joined #zope3-dev20:54
afd_don't checkout zam.demo, it's a trap! :-)20:57
* afd_ is watching the checkout of 'externals/zamplugin.control/externals/zam.api/externals/'20:57
*** pelle_ has joined #zope3-dev20:57
srichterafd_: I know, we will fix the external mess once we create releases20:58
*** dunny has joined #zope3-dev20:58
afd_srichter: np, it's just funny20:59
*** J1m has joined #zope3-dev21:04
*** quodt_ has quit IRC21:07
*** lucielejard has quit IRC21:07
J1mtarek_, you can use a flat index structure as an index.21:09
J1mtarek_, you can use a flat file structure as an index.21:09
*** malthe has joined #zope3-dev21:13
*** lucielejard has joined #zope3-dev21:13
*** romanofski has joined #zope3-dev21:19
*** salfield has quit IRC21:20
*** zagy has quit IRC21:25
*** jodok has quit IRC21:29
*** agroszer_ is now known as agroszer21:30
*** jodok has joined #zope3-dev21:30
*** alga has joined #zope3-dev21:30
*** agroszer has quit IRC21:30
*** jpcw2002 has left #zope3-dev21:31
*** greenman has joined #zope3-dev21:33
maltheJ1m: When the ZODB ghostifies objects in order to keep the object cache at a given size after a transaction boundary, would it not make sense to move those object to some *global object cache* such that another connection could "check out" that object without cost?21:41
maltheI was investigating the cost of having a very small (maybe 0) object cache size, and not get the cost of loading objects from the ZODB.21:42
J1mNo, because other objects in the original connection still have references to the object.21:42
*** hexsprite_ has joined #zope3-dev21:42
malthewhat if we expunge all of them? keeping nothing after a transaction.21:42
*** dobee has quit IRC21:42
J1mThat wouldn't be a very good optimization.21:43
*** hexsprite has quit IRC21:43
malthewell let's say it was a premise to not keep any objects in memory after a request; would it then not be a viable strategy to minimize loads?21:43
maltheI'm wondering if we're doomed to keep a per-connection object cache21:44
J1mThe whole point of keeping objects in memory is to avoid object loads.21:44
*** b52laptop has joined #zope3-dev21:44
J1mwhat would be the point of not having a per-connection cache?21:44
J1mbtw, I don't have time for a lengthy discussion on this.21:44
maltheto lower memory requirement with more threads21:44
J1mBut if you have multiple threads, they must be doing something and they each need access to data.21:45
J1mThe easiest way to reduce the memory requirement is to reduce the number of threads.21:45
J1mif you are compute bound, then multiple threads is counter productive.21:46
malthebut then you'll just waste CPU cycles when I/O is required21:46
malthewe're never compute bound21:46
J1mWe usually are.21:46
maltheprobably because you're developing slim applications21:46
J1mif you aren't compute bound, then use multiple threads and smaller cache sizes.21:46
malthewith an app like Plone I find the CPU does nothing21:46
J1mNo our apps are pretty big.21:47
malthea smaller cache size will cause many loads---perhaps a global object cache could avoid those loads21:47
*** jodok has quit IRC21:47
J1mIt's likely that you are doing so much IO because your caches are too small.21:47
*** markusleist has quit IRC21:47
maltheright that may be true21:47
J1mThe only good reason to not be CPU bound is if you are using RDBMSs or other external systems.21:48
maltheright: ZEO21:48
mcdoncmalthe: is your database on a SAN?21:48
J1mfor some definitioon of "good reason". :)21:48
J1mYou may want a bigger zeo client cache, a bigger object cache, or both.21:48
J1mNote that we use big caches and 1 thread per process.21:49
malthethat might be the way to go21:49
mcdoncmalthe: just fyi...
* malthe looks21:49
J1malthough to do what we do kind of requires multiple processes.21:49
mcdoncthat came from putting database on SAN21:49
*** afd_ has quit IRC21:50
malthemcdonc: this is what we have now21:50
malthe15% utilization21:50
maltheand the whole thing feels very slow21:50
maltheit's very strange21:50
maltheThe performance of a Plone site is a mystery21:51
mcdoncmalthe: do iostat -x 521:51
mcdonc(if on linux)21:51
mcdoncthe closer "svctime" is to "await", the better the device is performing21:51
mcdoncthe further they are away, the worse.21:51
malthehmm I must be look into this21:52
mcdoncwe found that when "only one thing" was happening (*just* database access, or *just* rsync) against the device, everything was ok21:52
mcdoncbut if "more than one thing" was happening, things got quite slow.21:52
maltheso that calls for specialization21:53
maltheI just find that a) ZODB-apps require too much memory, b) ZEO is very slow.21:53
mcdoncor local disks21:53
mcdoncmalthe: b) is not really true unless you have slow I/O, at least i find.21:53
mcdonca) is policy mostly.. except when it isn't ;-)21:54
malthewell (b) makes the system slow because there's a minimum time for everything, i.e. the network latency.21:54
maltheif you're just running locally, the machine will at least have a chance to work without break21:55
mcdoncJ1m: missed you at pycon21:55
J1myeah, didn't have time to go.21:55
maltheof course, this is the case for any such arrangement21:55
mgedminmalthe: what version of ZODB?21:56
malthemgedmin: 3.821:56
mgedminso it's not the ZEO cond.wait(1.0) timeout problem21:56
mgedminor is it?  was that fixed in 3.8 or 3.9?21:57
malthemgedmin: hey wait a second21:57
malthemaybe that's it21:57
maltheam I perhaps chasing a ghost here21:57
mgedminif your server is completely idle for a long time (not cpu-bound, not io-bound), then that's it21:57
mgedmincuriously, the faster server hardware you have, the more pronounced this is21:57
*** pcardune has quit IRC21:58
mgedminor maybe not21:58
malthefunny; I read that news, enjoyed the discovery and magically thought that this fix would propagate automatically21:58
mgedminmaybe just my old server had HZ of 100021:58
malthebut of course, a Zope 2.9-instance won't get the fix21:58
mgedminmalthe: what's even funnier was that I read that news, backported the fix to ZODB 3.6 and then *did not upgrade my production servers for a year*21:58
malthemgedmin: is there a 2.9-release that *does* have this fix?21:59
* malthe checks production servers21:59
mgedminzope 2? dunno21:59
malthebtw this fact still does not invalidate the discussion on how much memory Zope should use, and if it's a reasonable strategy to keep an object cache21:59
mcdoncmalthe: it's unreasonable not to, usually22:00
malthemcdonc: PHP-apps don't do it22:00
malthethey leave it to MySQL (or whoever)22:00
mcdonc(although i must admit i dont quite understand the exact purpose of the 2nd-level ZEO cache)22:00
malthemcdonc: I'm just investigating stateless ZODB-apps; is it possible, what's the cost etc.22:01
mcdoncmalthe: you've dialed down the cache-size parameter on the zodb_db (number of objects)?22:02
malthemcdonc: yes I'm doing a nasty self._cache.minimize() in the ``root``-method of the Connection-class.22:02
mcdonci suspect there is a case for having no in-mem object cache, but i also suspect plone isn't a good representative app.22:03
maltheno Plone is an awful candidate22:03
malthethe Zope 3 CMS I'm toying with (vudo) loads about 20 objects per request22:03
maltheif they're unpickled at every juncture, there's a 20ms penalty.22:04
mcdoncmalthe: you say you're minimizing, but did you actually dial down the (declarative) config in the config file for cache-size?22:04
*** dobee has joined #zope3-dev22:04
malthemcdonc: no I just put it right in the code; this is just for experimental purposes.22:04
malthethe effect would be the same22:04
maltheI'm not seriously pursuing the idea22:05
J1mmalthe, the cond.wait bug was fixed in 3.7 (probably some dot release)22:05
mcdoncmalthe: ah.  so i'll stop talking about it then. ;-)22:05
malthemcdonc: well22:05
maltheit is serious in a way; I do think PHP-based apps have a deployment advantage22:05
malthethey're really cheap to run22:05
maltheJ1m: thanks22:05
mcdoncmalthe: well, er... what about dialing down the cache-size in that case.22:06
mcdoncand if that doesn't do what you expect, then figure out why not.22:06
malthemcdonc: right---I'm expecting it will do what I think it will; only it's too expensive to unpickle all the items.22:07
malthewhich is why I'm toying with the idea of a "global object cache"22:07
malthei.e. a magic place where all connections can pull objects without cost (of course, only one thread can get it, then it's over).22:07
mcdoncmalthe: ic... well, why not use one thread?22:08
malthemcdonc: because ZEO causes I/O overhead---I'd like to have other threads use the CPU22:08
maltheit's a catch 22 perhaps22:08
mcdoncmalthe: set up multiple zope processes...22:08
mcdoncthey mostly share mem22:08
mcdonc(for code segments anyway)22:09
malthemcdonc: but they will each keep an object cache22:09
maltheeven if 90% objects in that cache are only used rarely22:09
malthewhy not pool all those "idle" objects together22:09
maltheand just check them out like a library22:09
maltheif the book is already checked out, ask the ZODB to clone it22:10
mcdoncmalthe: there is probably some optimization laying around in there, but i suspect it's pretty low-level... i wouldn't know how to do it.22:12
mcdoncthe connection cache stuff is magic to me, especially how it keeps a copy per-connection of the same object, i never did manage to wrap my head around how it's done22:13
*** pcardune has joined #zope3-dev22:13
malthemcdonc: it's all in cPickleCache.c22:13
malthewhich has nothing to do with pickles22:13
mcdonci know where it is, i just dont know why it is22:13
maltheif it has something to do with pickles, then I didn't get what it was about at least.22:14
*** ajs has joined #zope3-dev22:15
mcdoncmy muddled understanding of it is that it keeps a cache per connection because those objects get modified by code simultaneously, and making them separate means that app code doesn't need to worry about stomping on changes made via other connections at the same time.22:15
malthemcdonc: right that's it---but also it implemenents mechanisms to keep it under size control22:16
J1mmcdonc, right22:16
J1mmalthe, wrong -- keeping separate data per connection has nothing to do w size control.22:17
*** ajs has quit IRC22:17
mcdoncmalthe: if there's any optimization hanging around in there, it might be a copy-on-write sort of thing i guess22:18
maltheJ1m: right but it's certainly a responsibility of PickleCache-class?22:19
*** ajsmith has joined #zope3-dev22:19
mcdoncJ1m: whew.  10 years later, i still wasnt sure. ;-)22:19
*** dobee has quit IRC22:19
J1mYes, it is part of the implementation of the object cache to free space for objects that are no-longer used.22:19
maltheI believe in the global object cache like Ian Curtis believed in Joy Division.22:21
*** ajsmith has quit IRC22:21
*** ajsmith has joined #zope3-dev22:22
maltheI'll think about it some more; thanks for the feedback at any rate.22:22
*** ajsmith is now known as ajs22:22
*** ajs is now known as alex_smith22:23
*** alex_smith has quit IRC22:24
mcdoncmalthe: you might think about it in terms of staged invalidation, moving a copy of the data into the global cache when it was no longer needed (based on the cache expiry policy) in any per-connection cache and trying to pull a copy back out of the global cache when a given connection needed it... you might be able to be more aggressive in invalidating the per-connection caches then22:24
mcdoncbut then you need to figure out if all that actually gets you anything on any given platform as far as memory consumption, because it's up to the OS22:25
*** alex_smith has joined #zope3-dev22:25
malthemcdonc: right the assumption would have to be that we're really able to free up the memory.22:28
maltheI wouldn't move over a copy of the object though, I would move the *actual* object (if we didn't already have a copy in the global cache).22:28
mcdoncmalthe: sys.exit(0) does the trick ;-)22:29
malthemcdonc: right so supervisord would have to kill the process every x minutes to clean up.22:29
maltheno I don't think cleanup is necessary; certainly Python's garbage collector works well enough.22:29
maltheunless we're stocking up references where we shouldn't22:30
mcdoncmalthe: well, as i understand it, it's heap memory, and it gets fragmented, so sometimes even if python does the right thing, the OS cant get it back in usable chunks.22:30
mcdonctim peters worked on an "arena allocator" that i *think* landed in py25 that's better at it22:31
maltheisn't there a gsoc project to move Zope to py25?22:32
*** RaFromBRC is now known as RaFromBRC|lunch22:32
mcdoncso maybe one concrete task would be to make zope2 run on py25, as i suspect it's better at it... yeah, although i'm not sure where the guy landed on it22:32
J1mI fear that the new memory allocator sucks.22:32
J1mwhile working on a better packer, I tried to use py25 hoping the new allocator would make memory management better.22:33
J1mPython 2.5 packed much slower.22:33
J1mThat is just anecdotal at this point.22:33
maltheprogress is futile22:33
J1mBut I'm a little worried.22:33
mcdonctim wasnt at pycon either22:34
J1m(file storage packer)22:34
mcdoncwe need a tim-come-home-please effort22:34
mcdoncfrankly the sys.exit(0) thing sounds pretty good to me, esp. if it's staged shutdown.. it might be possible to convince graham to put some sort of request buffer in mod_wsgi that would hold on to requests as the process restarted22:36
mcdonc(at least for the zope-is-consuming-1GB-of-ram-and-i-dont-have-time-to-figure-out-why case)22:37
malthemcdonc: good call22:38
mcdoncthen the action becomes making plone start in under 35 seconds22:39
mcdoncplain old zope2 and zope3 apps start plenty fast22:39
maltheright 4-5 seconds22:39
*** dunny has quit IRC22:40
*** djohnson has quit IRC22:43
*** pcardune has quit IRC22:43
J1mI have a largish zope2 app that starts in under a second on moderately current hw.22:44
J1mMy zope3 apps take closer to 2, but I'm hopeful I can get that down quite a bit.22:44
*** alecm has joined #zope3-dev22:44
J1mWe have some zope3 appls that take much longer.22:45
srichterBy pickling ZCML actions, we can gain 30% in startup time; (Marius and I tested this during the foliage sprint)22:45
J1mI bet we could do a lot better by computing most actions in Python, which would be a lot more appropriate in many cases.22:46
srichterJ1m: how do you want to do that?22:47
J1mI'm also shidting to a model that requires far fewer registrations in the first place. Web 2 (as I define it;) is a wonderful thing.22:47
srichter(but right, the time is spent converting the unicode values to python objects)22:47
J1ms/shidting/shifting :)22:48
srichterhe he22:48
srichterJ1m: I'd love to hear your thoughts on that22:48
J1mwell, in the way I'm doing things, I have views that are more like desktop apps.22:49
J1mThey provide rish functionality.22:49
J1mThe view provides a single page with sub-pages providing ajax methods.22:50
J1mThese methods are just python methods on the view with jssonpage decorators.22:50
J1mso much less to register.22:50
* srichter notes that Jim should have a look at z3c.formjs for a cool way to do ajax sub-views22:50
srichterz3c.formjs does that22:51
J1mbasically, an adapter and a zc.resourcelibrary library for the resources.22:51
* J1m should look at z3c.formjs22:51
srichterit is a bit misleading, zince z3c.formjs contains generically useful features22:52
srichter(Paul and I would be willing to split those into a separate package)22:52
J1mI'm gonna be releasing this sometime soon, as Rob Page thinks it's OK to check something that depends on Ext into the z.o repo.22:52
*** kev_ has joined #zope3-dev22:53
J1mI've also integrated extjs w zope.formlib. So I can generate ext forms from formlib definitions.22:53
srichtercool, then I can port this to z3c.form :-)22:53
J1msrichter, were you refering to the ajax helper business?23:03
*** reco has quit IRC23:04
srichterJ1m: no, the extjs form generation23:05
*** quodt__ has quit IRC23:05
srichterJ1m: I am very interested in this23:05
srichterJ1m: z3c.formjs does the ajax helper business already23:05
srichterit also does some nice event hookup23:05
J1msrichter, I was trying to take a quick look at z3c.formjs and wondering what to focus on.23:06
*** djohnson has joined #zope3-dev23:06
*** quodt has joined #zope3-dev23:06
J1mwrt extjs forms, I'm documenting this now and will be releasing it in the next few days.23:06
*** quodt has quit IRC23:07
*** RaFromBRC|lunch is now known as RaFromBRC23:09
*** whit has quit IRC23:14
mcdoncmgedmin: nice post on testing...23:16
mgedminthings had already quietened down, and here I come in and rekindle the flame :-)23:17
*** whit has joined #zope3-dev23:17
mcdoncnaw, it was well-reasoned23:18
*** romanofski has quit IRC23:28
*** greenman has quit IRC23:32
*** whit has quit IRC23:34
*** kev_ has quit IRC23:38
*** mgedmin has quit IRC23:41
malthemcdonc: "module got 5 new untested lines of code"23:44
malthefrom megdmin's post23:44
mcdoncoh.. heh23:44
srichterJ1m: let me find you the low-level text files that should be of most interest to you23:45
*** faassen has joined #zope3-dev23:47
faassenoops. :)23:47
*** harobed has joined #zope3-dev23:52
*** timte has joined #zope3-dev23:56

Generated by 2.15.1 by Marius Gedminas - find it at!