| *** regebro has left #zope3-dev | 00:01 | |
| *** lucielejard has quit IRC | 00:01 | |
| *** ktwilight has joined #zope3-dev | 00:05 | |
| *** alecm has quit IRC | 00:11 | |
| *** ktwilight_ has quit IRC | 00:18 | |
| *** rcrafton has quit IRC | 00:28 | |
| *** gimni has quit IRC | 00:29 | |
| *** acsr__ has quit IRC | 00:31 | |
| *** acsr has joined #zope3-dev | 00:37 | |
| *** reco has quit IRC | 00:50 | |
| *** whit has quit IRC | 00:57 | |
| *** b52laptop has joined #zope3-dev | 00:58 | |
| *** d2m has quit IRC | 00:58 | |
| *** greenman has joined #zope3-dev | 01:03 | |
| *** natea_ has quit IRC | 01:10 | |
| *** J1m has quit IRC | 01:17 | |
| *** reedobrien has quit IRC | 01:19 | |
| *** b52laptop has quit IRC | 01:27 | |
| *** rmarianski has quit IRC | 01:28 | |
| *** dunny has quit IRC | 01:34 | |
| *** ccomb has left #zope3-dev | 01:41 | |
| *** harobed has quit IRC | 01:44 | |
| *** dunny has joined #zope3-dev | 01:55 | |
| *** norro has quit IRC | 01:57 | |
| *** RaFromBRC has joined #zope3-dev | 02:02 | |
| *** mcdonc has joined #zope3-dev | 02:07 | |
| *** MrTopf has quit IRC | 02:16 | |
| *** natea_ has joined #zope3-dev | 02:24 | |
| *** ignas has quit IRC | 02:24 | |
| *** mcdonc has quit IRC | 02:26 | |
| *** pcardune has quit IRC | 02:33 | |
| *** mcdonc has joined #zope3-dev | 02:39 | |
| *** projekt01 has joined #zope3-dev | 02:42 | |
| *** hexsprite_ has quit IRC | 02:43 | |
| *** CSWookie has joined #zope3-dev | 02:51 | |
| *** natea_ has quit IRC | 02:51 | |
| CSWookie | Is there a project that does for function what zope.schema does for classes? I'd like to be able to define what types of arguments a function ought to take and what type of return value it should return. Is there something in zope.schema I haven't seen? | 02:53 |
|---|---|---|
| *** natea_ has joined #zope3-dev | 02:54 | |
| *** mcdonc has quit IRC | 02:55 | |
| *** edgordon has joined #zope3-dev | 02:55 | |
| *** jpcw2002 has quit IRC | 02:55 | |
| *** philiKON has quit IRC | 03:03 | |
| *** georgy has joined #zope3-dev | 03:07 | |
| *** nathany has quit IRC | 03:07 | |
| *** febb has quit IRC | 03:21 | |
| *** alecm has joined #zope3-dev | 03:24 | |
| *** dtremea has joined #zope3-dev | 03:32 | |
| *** dtremea is now known as deo | 03:37 | |
| *** hexsprite has joined #zope3-dev | 03:44 | |
| *** greenman has quit IRC | 03:53 | |
| *** georgy has quit IRC | 03:54 | |
| *** greenman has joined #zope3-dev | 03:55 | |
| *** natea_ has quit IRC | 04:05 | |
| *** natea_ has joined #zope3-dev | 04:06 | |
| *** srichter has quit IRC | 04:07 | |
| *** derek|home has joined #zope3-dev | 04:08 | |
| *** tarek has quit IRC | 04:09 | |
| *** niemeyer has quit IRC | 04:18 | |
| *** stub has joined #zope3-dev | 04:52 | |
| *** RaFromBRC has quit IRC | 04:58 | |
| *** ignas has joined #zope3-dev | 05:09 | |
| *** sm has quit IRC | 05:10 | |
| *** deo has quit IRC | 05:28 | |
| *** derek|home has quit IRC | 05:34 | |
| *** greenman has quit IRC | 05:34 | |
| *** rmarianski has joined #zope3-dev | 05:52 | |
| *** bigkevmcd has quit IRC | 06:03 | |
| *** redir has joined #zope3-dev | 06:08 | |
| *** alecm has quit IRC | 06:24 | |
| *** afd_ has joined #zope3-dev | 06:41 | |
| *** RaFromBRC has joined #zope3-dev | 06:46 | |
| *** afd_ has quit IRC | 07:11 | |
| *** srichter has joined #zope3-dev | 07:13 | |
| *** ChanServ sets mode: +o srichter | 07:13 | |
| *** rmarianski has quit IRC | 07:17 | |
| *** srichter has quit IRC | 07:20 | |
| *** stub has quit IRC | 07:21 | |
| *** srichter has joined #zope3-dev | 07:21 | |
| *** timte has quit IRC | 07:23 | |
| *** yvl has joined #zope3-dev | 07:24 | |
| *** redir has quit IRC | 07:25 | |
| *** ChanServ sets mode: +o srichter | 07:32 | |
| *** sm has joined #zope3-dev | 07:43 | |
| *** projekt01 has quit IRC | 07:48 | |
| *** redir has joined #zope3-dev | 07:55 | |
| *** srichter has quit IRC | 07:57 | |
| *** mcdonc has joined #zope3-dev | 08:02 | |
| *** menesis has joined #zope3-dev | 08:05 | |
| *** jayaraj has joined #zope3-dev | 08:09 | |
| *** ignas has quit IRC | 08:09 | |
| *** dobee__ has joined #zope3-dev | 08:17 | |
| *** afd_ has joined #zope3-dev | 08:23 | |
| *** jukart has joined #zope3-dev | 08:25 | |
| *** djohnson has quit IRC | 08:27 | |
| *** sm has quit IRC | 08:39 | |
| *** sorin has joined #zope3-dev | 08:57 | |
| *** sorin is now known as sorindregan | 08:57 | |
| *** RaFromBRC has quit IRC | 08:57 | |
| *** stub has joined #zope3-dev | 08:59 | |
| *** dobee__ has quit IRC | 08:59 | |
| *** redir has quit IRC | 09:00 | |
| *** hdima has joined #zope3-dev | 09:01 | |
| *** greenman has joined #zope3-dev | 09:04 | |
| *** __mac__ has joined #zope3-dev | 09:05 | |
| *** dobee_ has joined #zope3-dev | 09:06 | |
| *** dobee_ is now known as dobee | 09:09 | |
| *** huajie has joined #zope3-dev | 09:34 | |
| *** jukart has quit IRC | 09:36 | |
| *** jukart has joined #zope3-dev | 09:36 | |
| *** goschtl has joined #zope3-dev | 09:39 | |
| *** philiKON has joined #zope3-dev | 09:43 | |
| *** malthe has quit IRC | 09:45 | |
| *** malthe has joined #zope3-dev | 09:45 | |
| *** markusleist has quit IRC | 09:56 | |
| *** tarek has joined #zope3-dev | 09:59 | |
| *** bigkevmcd has joined #zope3-dev | 10:00 | |
| *** quodt has joined #zope3-dev | 10:05 | |
| *** tarek has quit IRC | 10:11 | |
| *** tarek has joined #zope3-dev | 10:12 | |
| *** markusleist has joined #zope3-dev | 10:16 | |
| *** d2m has joined #zope3-dev | 10:18 | |
| *** jodok has joined #zope3-dev | 10:23 | |
| *** harobed has joined #zope3-dev | 10:31 | |
| *** jpcw2002 has joined #zope3-dev | 10:33 | |
| *** __mac__ has left #zope3-dev | 10:33 | |
| *** maurits has joined #zope3-dev | 10:34 | |
| *** zagy has joined #zope3-dev | 10:43 | |
| *** vaish has joined #zope3-dev | 10:44 | |
| *** philiKON has quit IRC | 10:44 | |
| *** jodok has quit IRC | 10:47 | |
| *** philiKON has joined #zope3-dev | 11:00 | |
| *** malthe has quit IRC | 11:02 | |
| *** MJ has joined #zope3-dev | 11:11 | |
| *** vaish has left #zope3-dev | 11:27 | |
| *** b52laptop has joined #zope3-dev | 11:34 | |
| *** dunny has quit IRC | 11:36 | |
| *** norro has joined #zope3-dev | 11:38 | |
| *** thruflo has joined #zope3-dev | 11:40 | |
| *** QuyetNextG has joined #zope3-dev | 11:46 | |
| *** QuyetNextG has quit IRC | 11:51 | |
| *** malthe has joined #zope3-dev | 11:56 | |
| *** toutpt has joined #zope3-dev | 12:12 | |
| *** rocky has quit IRC | 12:13 | |
| *** sunew has joined #zope3-dev | 12:16 | |
| *** rocky has joined #zope3-dev | 12:18 | |
| *** gimni has joined #zope3-dev | 12:27 | |
| *** timte has joined #zope3-dev | 12:27 | |
| *** regebro has joined #zope3-dev | 12:30 | |
| *** thruflo is now known as devilsadvocate | 12:30 | |
| *** devilsadvocate is now known as thruflo | 12:34 | |
| *** pelle_ has joined #zope3-dev | 12:40 | |
| *** andy-chen has joined #zope3-dev | 12:43 | |
| *** menesis has quit IRC | 12:45 | |
| andy-chen | hi,all. what's the best practice to communicate between two zope3 instance run on their own server? xml-rpc ?json-rpc?or something else | 12:53 |
| andy-chen | we need use something based on http... | 12:54 |
| *** greenman has quit IRC | 12:56 | |
| *** ghendi_ has joined #zope3-dev | 12:57 | |
| *** timte has quit IRC | 12:58 | |
| *** thruflo has left #zope3-dev | 12:59 | |
| *** timte has joined #zope3-dev | 12:59 | |
| *** andy-chen has quit IRC | 13:00 | |
| *** djohnson_ has joined #zope3-dev | 13:02 | |
| *** b52laptop has quit IRC | 13:10 | |
| *** b52lap has joined #zope3-dev | 13:10 | |
| *** zagy has quit IRC | 13:11 | |
| *** zagy has joined #zope3-dev | 13:12 | |
| *** mkerrin has joined #zope3-dev | 13:16 | |
| *** timte has quit IRC | 13:17 | |
| *** toutpt has quit IRC | 13:32 | |
| *** tarek_ has joined #zope3-dev | 13:34 | |
| *** tarek has quit IRC | 13:40 | |
| *** MJ is now known as MJ|lunch | 13:48 | |
| *** sunew has quit IRC | 13:51 | |
| *** FFighter has joined #zope3-dev | 13:55 | |
| *** benji has joined #zope3-dev | 14:15 | |
| *** djohnson_ has quit IRC | 14:17 | |
| *** sunew has joined #zope3-dev | 14:19 | |
| *** timte has joined #zope3-dev | 14:19 | |
| *** timte has quit IRC | 14:20 | |
| *** timte has joined #zope3-dev | 14:21 | |
| *** gimni has quit IRC | 14:23 | |
| *** malthe has quit IRC | 14:34 | |
| *** malthe has joined #zope3-dev | 14:36 | |
| *** niemeyer has joined #zope3-dev | 14:36 | |
| *** ccomb has joined #zope3-dev | 14:52 | |
| *** b52lap has quit IRC | 14:59 | |
| *** b52lap has joined #zope3-dev | 15:03 | |
| *** sorindregan has quit IRC | 15:10 | |
| *** projekt01 has joined #zope3-dev | 15:11 | |
| *** lucielejard has joined #zope3-dev | 15:13 | |
| *** lucielejard has quit IRC | 15:14 | |
| *** lucielejard has joined #zope3-dev | 15:14 | |
| *** J1m has joined #zope3-dev | 15:15 | |
| *** alga has quit IRC | 15:17 | |
| *** srichter has joined #zope3-dev | 15:23 | |
| *** MJ|lunch is now known as MJ | 15:28 | |
| *** rcrafton has joined #zope3-dev | 15:30 | |
| *** afd__ has joined #zope3-dev | 15:42 | |
| *** srichter has quit IRC | 15:55 | |
| *** afd_ has quit IRC | 15:59 | |
| *** sorin has joined #zope3-dev | 16:11 | |
| *** sorin is now known as sorindregan | 16:11 | |
| *** djohnson has joined #zope3-dev | 16:13 | |
| *** pcardune has joined #zope3-dev | 16:18 | |
| *** toutpt has joined #zope3-dev | 16:24 | |
| *** afd__ has quit IRC | 16:24 | |
| *** alga has joined #zope3-dev | 16:24 | |
| *** hdima has quit IRC | 16:42 | |
| *** nathany has joined #zope3-dev | 16:43 | |
| *** djohnson has quit IRC | 16:44 | |
| *** jayaraj has quit IRC | 16:50 | |
| *** mgedmin has joined #zope3-dev | 16:50 | |
| *** huajie has quit IRC | 17:00 | |
| *** sm has joined #zope3-dev | 17:07 | |
| *** afd_ has joined #zope3-dev | 17:14 | |
| *** mgedmin has quit IRC | 17:16 | |
| *** stub has quit IRC | 17:17 | |
| *** gimni has joined #zope3-dev | 17:18 | |
| *** srichter has joined #zope3-dev | 17:24 | |
| *** ChanServ sets mode: +o srichter | 17:25 | |
| *** ignas has joined #zope3-dev | 17:26 | |
| *** regebro has quit IRC | 17:27 | |
| afd_ | malthe: did you had time to benchmark running zope without object cache? | 17:30 |
| *** salfield has quit IRC | 17:30 | |
| malthe | afd_: I've only done preliminary tests; with 17 objects I had a 20% penalty, with 10,000 I had a 800% penalty. This is very approximate. | 17:33 |
| afd_ | any way to configure the system to improve this? | 17:33 |
| malthe | so the key is to a) limit the number of persistant object in use, b) keep module-level read-only caches. | 17:33 |
| afd_ | "module level read-only" cache, what does it mean? | 17:34 |
| *** timte has quit IRC | 17:34 | |
| malthe | well if such a cache is to be shared among threads, it must be under some write control. | 17:34 |
| afd_ | sorry to bug you, I'm just curios | 17:34 |
| malthe | and you probably don't want per-thread caches (that would defeat the purpose). | 17:35 |
| afd_ | what's the idea? or purpose... | 17:35 |
| malthe | the motivation is to be able to limit the memory consumption per thread. | 17:36 |
| afd_ | I see | 17:36 |
| malthe | two reasons: a) cheap hosting, b) network-intensive application that are waiting for I/O all the time. | 17:36 |
| afd_ | right | 17:37 |
| malthe | (b) could be an application that has, say, 20 GB of data | 17:37 |
| malthe | it'll always have to pull data in at every request | 17:37 |
| *** yvl has quit IRC | 17:37 | |
| malthe | this causes that thread to stay idle while it does I/O---what's the CPU going to do meanwhile? | 17:37 |
| malthe | nothing. | 17:38 |
| *** whit has joined #zope3-dev | 17:38 | |
| malthe | and it can't spawn more than a few threads, because each thread take up too much memory. | 17:38 |
| afd_ | I see | 17:38 |
| malthe | if the threads aren't allowed to use memory, they operate slowly. | 17:38 |
| malthe | that's status quo. | 17:38 |
| malthe | so I was just investigating how to have them operate faster without using memory. | 17:39 |
| afd_ | I see | 17:39 |
| *** reco has joined #zope3-dev | 17:39 | |
| afd_ | what about an app that uses the catalog? | 17:39 |
| afd_ | I understand the catalog is fast only if it loads its btree indexes in memory | 17:39 |
| malthe | yes that's also correct; so that will have to be dealt with somehow. | 17:40 |
| malthe | I think Lucene or Xapian is the answer here | 17:40 |
| malthe | simply take it out of the equation altogether | 17:41 |
| malthe | makes sense, too, because a catalog result is a very formalized dataset. | 17:41 |
| malthe | it's a good fit for an external database | 17:41 |
| afd_ | I see | 17:41 |
| *** mgedmin has joined #zope3-dev | 17:41 | |
| afd_ | ok, thanks for your thoughts :) | 17:42 |
| malthe | sure | 17:42 |
| *** alga_ has joined #zope3-dev | 17:42 | |
| *** hexsprite has quit IRC | 17:49 | |
| *** alga has quit IRC | 17:50 | |
| *** menesis has joined #zope3-dev | 17:51 | |
| *** hexsprite has joined #zope3-dev | 17:54 | |
| *** jsadjohnson has joined #zope3-dev | 17:54 | |
| *** agroszer has joined #zope3-dev | 18:00 | |
| *** MJ has quit IRC | 18:15 | |
| *** dobee has quit IRC | 18:17 | |
| *** romanofski has quit IRC | 18:18 | |
| *** pelle_ has quit IRC | 18:20 | |
| *** pelle_ has joined #zope3-dev | 18:22 | |
| tarek_ | Hi J1m, to avoid easy_install hitting pypi (in an intranet that does not have a web connection) i use the --allow-host option. It does not seem to get caught into buildout though. | 18:25 |
| tarek_ | is there a way to prevent some hosts to be called in buildout (in online mode) ? | 18:25 |
| tarek_ | we have a local apache there, that has the eggs | 18:26 |
| srichter | tarek_: you can use the index option | 18:26 |
| srichter | or "find-links" | 18:26 |
| *** jukart has quit IRC | 18:27 | |
| tarek_ | srichter, hi | 18:27 |
| tarek_ | srichter, the find-links is just adding new sources, bu will not prevent from hiting pypi.python.org | 18:27 |
| tarek_ | t | 18:27 |
| tarek_ | this is the --allow-host option, (-H) in easy_install | 18:27 |
| srichter | using "index" | 18:28 |
| srichter | it allows you to point to a different index, such as PPIX | 18:28 |
| tarek_ | i have a flat folder with eggs, this would mean I need to create an index structure then, such as PPIX right ? | 18:30 |
| *** edgordon has quit IRC | 18:30 | |
| srichter | tarek_: well, what is your goal? | 18:37 |
| *** jodok has joined #zope3-dev | 18:38 | |
| *** sunew has quit IRC | 18:39 | |
| *** natea_ is now known as natea | 18:39 | |
| tarek_ | srichter, not bathering creating an index page, and using the allow-host option | 18:40 |
| tarek_ | the use case is quite simple: | 18:40 |
| tarek_ | having a local apache that provides all eggs through find-links | 18:41 |
| tarek_ | and preventing buildout to hit pypi. | 18:41 |
| tarek_ | this is straight forward with easy_install : | 18:42 |
| srichter | then I think you need to create an index in buildout | 18:42 |
| srichter | or point to a non-working index and use find-links | 18:42 |
| tarek_ | easy_install -H *.local.server -f http://local.server/eggs my.package | 18:42 |
| tarek_ | ah... let me try the non working index ... | 18:43 |
| srichter | I think it is okay pointing index to http://local.server/eggs | 18:43 |
| tarek_ | if it's a flat folder, it will fail afaik: | 18:43 |
| * tarek_ is trying something again :) | 18:44 | |
| *** edgordon has joined #zope3-dev | 18:45 | |
| *** menesis has left #zope3-dev | 18:48 | |
| *** salfield has joined #zope3-dev | 18:55 | |
| *** b52lap has quit IRC | 18:55 | |
| *** J1m has quit IRC | 19:07 | |
| *** rock1 has joined #zope3-dev | 19:13 | |
| *** gimni has quit IRC | 19:13 | |
| *** harobed has quit IRC | 19:15 | |
| *** rocky has quit IRC | 19:17 | |
| *** rock1 is now known as rocky | 19:17 | |
| *** derek|office has quit IRC | 19:22 | |
| *** sorindregan has quit IRC | 19:26 | |
| *** quodt has quit IRC | 19:27 | |
| *** RaFromBRC has joined #zope3-dev | 19:30 | |
| *** edgordon has quit IRC | 19:31 | |
| *** sunew has joined #zope3-dev | 19:37 | |
| *** maurits has quit IRC | 19:38 | |
| *** pelle_ has quit IRC | 19:39 | |
| *** dobee has joined #zope3-dev | 19:41 | |
| *** projekt01 has quit IRC | 19:44 | |
| *** toutpt has quit IRC | 19:48 | |
| *** sunew has quit IRC | 19:54 | |
| *** malthe has quit IRC | 19:55 | |
| *** goschtl has quit IRC | 19:58 | |
| *** sunew has joined #zope3-dev | 20:03 | |
| *** sunew has quit IRC | 20:10 | |
| *** antiretorte has joined #zope3-dev | 20:12 | |
| *** quodt has joined #zope3-dev | 20:12 | |
| *** antiretorte has quit IRC | 20:13 | |
| *** whit has quit IRC | 20:16 | |
| *** alga_ has quit IRC | 20:22 | |
| *** agroszer_ has joined #zope3-dev | 20:28 | |
| *** mkerrin has quit IRC | 20:28 | |
| *** djohnson has joined #zope3-dev | 20:34 | |
| *** regebro has joined #zope3-dev | 20:35 | |
| *** agroszer has quit IRC | 20:40 | |
| *** quodt_ has joined #zope3-dev | 20:44 | |
| *** quodt__ has joined #zope3-dev | 20:46 | |
| *** quodt has quit IRC | 20:46 | |
| *** whit has joined #zope3-dev | 20:47 | |
| *** jodok has quit IRC | 20:48 | |
| *** jodok has joined #zope3-dev | 20:50 | |
| *** jodok has quit IRC | 20:52 | |
| *** jodok has joined #zope3-dev | 20:54 | |
| afd_ | don't checkout zam.demo, it's a trap! :-) | 20:57 |
| * afd_ is watching the checkout of 'externals/zamplugin.control/externals/zam.api/externals/z3c.menu.ready2go' | 20:57 | |
| *** pelle_ has joined #zope3-dev | 20:57 | |
| srichter | afd_: I know, we will fix the external mess once we create releases | 20:58 |
| *** dunny has joined #zope3-dev | 20:58 | |
| afd_ | srichter: np, it's just funny | 20:59 |
| *** J1m has joined #zope3-dev | 21:04 | |
| *** quodt_ has quit IRC | 21:07 | |
| *** lucielejard has quit IRC | 21:07 | |
| J1m | tarek_, you can use a flat index structure as an index. | 21:09 |
| J1m | tarek_, you can use a flat file structure as an index. | 21:09 |
| *** malthe has joined #zope3-dev | 21:13 | |
| *** lucielejard has joined #zope3-dev | 21:13 | |
| *** romanofski has joined #zope3-dev | 21:19 | |
| *** salfield has quit IRC | 21:20 | |
| *** zagy has quit IRC | 21:25 | |
| *** jodok has quit IRC | 21:29 | |
| *** agroszer_ is now known as agroszer | 21:30 | |
| *** jodok has joined #zope3-dev | 21:30 | |
| *** alga has joined #zope3-dev | 21:30 | |
| *** agroszer has quit IRC | 21:30 | |
| *** jpcw2002 has left #zope3-dev | 21:31 | |
| *** greenman has joined #zope3-dev | 21:33 | |
| malthe | J1m: When the ZODB ghostifies objects in order to keep the object cache at a given size after a transaction boundary, would it not make sense to move those object to some *global object cache* such that another connection could "check out" that object without cost? | 21:41 |
| malthe | I was investigating the cost of having a very small (maybe 0) object cache size, and not get the cost of loading objects from the ZODB. | 21:42 |
| J1m | No, because other objects in the original connection still have references to the object. | 21:42 |
| *** hexsprite_ has joined #zope3-dev | 21:42 | |
| malthe | what if we expunge all of them? keeping nothing after a transaction. | 21:42 |
| *** dobee has quit IRC | 21:42 | |
| J1m | That wouldn't be a very good optimization. | 21:43 |
| *** hexsprite has quit IRC | 21:43 | |
| malthe | well let's say it was a premise to not keep any objects in memory after a request; would it then not be a viable strategy to minimize loads? | 21:43 |
| malthe | I'm wondering if we're doomed to keep a per-connection object cache | 21:44 |
| J1m | The whole point of keeping objects in memory is to avoid object loads. | 21:44 |
| *** b52laptop has joined #zope3-dev | 21:44 | |
| J1m | what would be the point of not having a per-connection cache? | 21:44 |
| J1m | btw, I don't have time for a lengthy discussion on this. | 21:44 |
| malthe | to lower memory requirement with more threads | 21:44 |
| malthe | lk | 21:45 |
| malthe | k | 21:45 |
| J1m | But if you have multiple threads, they must be doing something and they each need access to data. | 21:45 |
| J1m | The easiest way to reduce the memory requirement is to reduce the number of threads. | 21:45 |
| J1m | if you are compute bound, then multiple threads is counter productive. | 21:46 |
| malthe | but then you'll just waste CPU cycles when I/O is required | 21:46 |
| malthe | we're never compute bound | 21:46 |
| J1m | We usually are. | 21:46 |
| malthe | probably because you're developing slim applications | 21:46 |
| J1m | if you aren't compute bound, then use multiple threads and smaller cache sizes. | 21:46 |
| malthe | with an app like Plone I find the CPU does nothing | 21:46 |
| J1m | No our apps are pretty big. | 21:47 |
| malthe | ok | 21:47 |
| malthe | a smaller cache size will cause many loads---perhaps a global object cache could avoid those loads | 21:47 |
| *** jodok has quit IRC | 21:47 | |
| J1m | It's likely that you are doing so much IO because your caches are too small. | 21:47 |
| *** markusleist has quit IRC | 21:47 | |
| malthe | right that may be true | 21:47 |
| J1m | The only good reason to not be CPU bound is if you are using RDBMSs or other external systems. | 21:48 |
| malthe | right: ZEO | 21:48 |
| mcdonc | malthe: is your database on a SAN? | 21:48 |
| J1m | for some definitioon of "good reason". :) | 21:48 |
| J1m | You may want a bigger zeo client cache, a bigger object cache, or both. | 21:48 |
| J1m | Note that we use big caches and 1 thread per process. | 21:49 |
| malthe | interesting | 21:49 |
| malthe | that might be the way to go | 21:49 |
| mcdonc | malthe: just fyi... http://www.plope.com/Members/chrism/easy_solution | 21:49 |
| * malthe looks | 21:49 | |
| J1m | although to do what we do kind of requires multiple processes. | 21:49 |
| malthe | hehe | 21:49 |
| mcdonc | that came from putting database on SAN | 21:49 |
| *** afd_ has quit IRC | 21:50 | |
| malthe | mcdonc: this is what we have now | 21:50 |
| malthe | 15% utilization | 21:50 |
| malthe | and the whole thing feels very slow | 21:50 |
| malthe | it's very strange | 21:50 |
| malthe | The performance of a Plone site is a mystery | 21:51 |
| mcdonc | malthe: do iostat -x 5 | 21:51 |
| mcdonc | (if on linux) | 21:51 |
| malthe | cool | 21:51 |
| mcdonc | the closer "svctime" is to "await", the better the device is performing | 21:51 |
| mcdonc | the further they are away, the worse. | 21:51 |
| malthe | hmm I must be look into this | 21:52 |
| mcdonc | we found that when "only one thing" was happening (*just* database access, or *just* rsync) against the device, everything was ok | 21:52 |
| mcdonc | but if "more than one thing" was happening, things got quite slow. | 21:52 |
| malthe | so that calls for specialization | 21:53 |
| malthe | I just find that a) ZODB-apps require too much memory, b) ZEO is very slow. | 21:53 |
| mcdonc | or local disks | 21:53 |
| mcdonc | malthe: b) is not really true unless you have slow I/O, at least i find. | 21:53 |
| mcdonc | a) is policy mostly.. except when it isn't ;-) | 21:54 |
| malthe | :-) | 21:54 |
| malthe | well (b) makes the system slow because there's a minimum time for everything, i.e. the network latency. | 21:54 |
| malthe | if you're just running locally, the machine will at least have a chance to work without break | 21:55 |
| mcdonc | J1m: missed you at pycon | 21:55 |
| J1m | yeah, didn't have time to go. | 21:55 |
| malthe | of course, this is the case for any such arrangement | 21:55 |
| mgedmin | malthe: what version of ZODB? | 21:56 |
| malthe | mgedmin: 3.8 | 21:56 |
| mgedmin | so it's not the ZEO cond.wait(1.0) timeout problem | 21:56 |
| mgedmin | or is it? was that fixed in 3.8 or 3.9? | 21:57 |
| malthe | mgedmin: hey wait a second | 21:57 |
| malthe | maybe that's it | 21:57 |
| malthe | am I perhaps chasing a ghost here | 21:57 |
| mgedmin | if your server is completely idle for a long time (not cpu-bound, not io-bound), then that's it | 21:57 |
| mgedmin | curiously, the faster server hardware you have, the more pronounced this is | 21:57 |
| *** pcardune has quit IRC | 21:58 | |
| mgedmin | or maybe not | 21:58 |
| malthe | funny; I read that news, enjoyed the discovery and magically thought that this fix would propagate automatically | 21:58 |
| mgedmin | maybe just my old server had HZ of 1000 | 21:58 |
| malthe | but of course, a Zope 2.9-instance won't get the fix | 21:58 |
| mgedmin | malthe: what's even funnier was that I read that news, backported the fix to ZODB 3.6 and then *did not upgrade my production servers for a year* | 21:58 |
| malthe | hah | 21:58 |
| malthe | mgedmin: is there a 2.9-release that *does* have this fix? | 21:59 |
| * malthe checks production servers | 21:59 | |
| mgedmin | zope 2? dunno | 21:59 |
| malthe | btw this fact still does not invalidate the discussion on how much memory Zope should use, and if it's a reasonable strategy to keep an object cache | 21:59 |
| mcdonc | malthe: it's unreasonable not to, usually | 22:00 |
| malthe | mcdonc: PHP-apps don't do it | 22:00 |
| malthe | they leave it to MySQL (or whoever) | 22:00 |
| mcdonc | (although i must admit i dont quite understand the exact purpose of the 2nd-level ZEO cache) | 22:00 |
| malthe | mcdonc: I'm just investigating stateless ZODB-apps; is it possible, what's the cost etc. | 22:01 |
| mcdonc | malthe: you've dialed down the cache-size parameter on the zodb_db (number of objects)? | 22:02 |
| malthe | mcdonc: yes I'm doing a nasty self._cache.minimize() in the ``root``-method of the Connection-class. | 22:02 |
| mcdonc | i suspect there is a case for having no in-mem object cache, but i also suspect plone isn't a good representative app. | 22:03 |
| malthe | no Plone is an awful candidate | 22:03 |
| malthe | the Zope 3 CMS I'm toying with (vudo) loads about 20 objects per request | 22:03 |
| malthe | if they're unpickled at every juncture, there's a 20ms penalty. | 22:04 |
| mcdonc | malthe: you say you're minimizing, but did you actually dial down the (declarative) config in the config file for cache-size? | 22:04 |
| *** dobee has joined #zope3-dev | 22:04 | |
| malthe | mcdonc: no I just put it right in the code; this is just for experimental purposes. | 22:04 |
| malthe | the effect would be the same | 22:04 |
| malthe | I'm not seriously pursuing the idea | 22:05 |
| J1m | malthe, the cond.wait bug was fixed in 3.7 (probably some dot release) | 22:05 |
| mcdonc | malthe: ah. so i'll stop talking about it then. ;-) | 22:05 |
| malthe | mcdonc: well | 22:05 |
| malthe | it is serious in a way; I do think PHP-based apps have a deployment advantage | 22:05 |
| malthe | they're really cheap to run | 22:05 |
| malthe | J1m: thanks | 22:05 |
| mcdonc | malthe: well, er... what about dialing down the cache-size in that case. | 22:06 |
| mcdonc | and if that doesn't do what you expect, then figure out why not. | 22:06 |
| malthe | mcdonc: right---I'm expecting it will do what I think it will; only it's too expensive to unpickle all the items. | 22:07 |
| malthe | which is why I'm toying with the idea of a "global object cache" | 22:07 |
| malthe | i.e. a magic place where all connections can pull objects without cost (of course, only one thread can get it, then it's over). | 22:07 |
| mcdonc | malthe: ic... well, why not use one thread? | 22:08 |
| mcdonc | (only) | 22:08 |
| malthe | mcdonc: because ZEO causes I/O overhead---I'd like to have other threads use the CPU | 22:08 |
| malthe | it's a catch 22 perhaps | 22:08 |
| mcdonc | malthe: set up multiple zope processes... | 22:08 |
| mcdonc | they mostly share mem | 22:08 |
| mcdonc | (for code segments anyway) | 22:09 |
| malthe | mcdonc: but they will each keep an object cache | 22:09 |
| malthe | even if 90% objects in that cache are only used rarely | 22:09 |
| malthe | why not pool all those "idle" objects together | 22:09 |
| malthe | and just check them out like a library | 22:09 |
| malthe | if the book is already checked out, ask the ZODB to clone it | 22:10 |
| mcdonc | malthe: there is probably some optimization laying around in there, but i suspect it's pretty low-level... i wouldn't know how to do it. | 22:12 |
| mcdonc | the connection cache stuff is magic to me, especially how it keeps a copy per-connection of the same object, i never did manage to wrap my head around how it's done | 22:13 |
| *** pcardune has joined #zope3-dev | 22:13 | |
| malthe | mcdonc: it's all in cPickleCache.c | 22:13 |
| malthe | which has nothing to do with pickles | 22:13 |
| mcdonc | i know where it is, i just dont know why it is | 22:13 |
| malthe | if it has something to do with pickles, then I didn't get what it was about at least. | 22:14 |
| *** ajs has joined #zope3-dev | 22:15 | |
| mcdonc | my muddled understanding of it is that it keeps a cache per connection because those objects get modified by code simultaneously, and making them separate means that app code doesn't need to worry about stomping on changes made via other connections at the same time. | 22:15 |
| malthe | mcdonc: right that's it---but also it implemenents mechanisms to keep it under size control | 22:16 |
| J1m | mcdonc, right | 22:16 |
| J1m | malthe, wrong -- keeping separate data per connection has nothing to do w size control. | 22:17 |
| *** ajs has quit IRC | 22:17 | |
| mcdonc | malthe: if there's any optimization hanging around in there, it might be a copy-on-write sort of thing i guess | 22:18 |
| malthe | J1m: right but it's certainly a responsibility of PickleCache-class? | 22:19 |
| *** ajsmith has joined #zope3-dev | 22:19 | |
| mcdonc | J1m: whew. 10 years later, i still wasnt sure. ;-) | 22:19 |
| *** dobee has quit IRC | 22:19 | |
| J1m | Yes, it is part of the implementation of the object cache to free space for objects that are no-longer used. | 22:19 |
| malthe | right | 22:20 |
| malthe | I believe in the global object cache like Ian Curtis believed in Joy Division. | 22:21 |
| *** ajsmith has quit IRC | 22:21 | |
| *** ajsmith has joined #zope3-dev | 22:22 | |
| malthe | I'll think about it some more; thanks for the feedback at any rate. | 22:22 |
| *** ajsmith is now known as ajs | 22:22 | |
| *** ajs is now known as alex_smith | 22:23 | |
| *** alex_smith has quit IRC | 22:24 | |
| mcdonc | malthe: you might think about it in terms of staged invalidation, moving a copy of the data into the global cache when it was no longer needed (based on the cache expiry policy) in any per-connection cache and trying to pull a copy back out of the global cache when a given connection needed it... you might be able to be more aggressive in invalidating the per-connection caches then | 22:24 |
| mcdonc | but then you need to figure out if all that actually gets you anything on any given platform as far as memory consumption, because it's up to the OS | 22:25 |
| *** alex_smith has joined #zope3-dev | 22:25 | |
| malthe | mcdonc: right the assumption would have to be that we're really able to free up the memory. | 22:28 |
| malthe | I wouldn't move over a copy of the object though, I would move the *actual* object (if we didn't already have a copy in the global cache). | 22:28 |
| mcdonc | malthe: sys.exit(0) does the trick ;-) | 22:29 |
| malthe | mcdonc: right so supervisord would have to kill the process every x minutes to clean up. | 22:29 |
| malthe | no I don't think cleanup is necessary; certainly Python's garbage collector works well enough. | 22:29 |
| malthe | unless we're stocking up references where we shouldn't | 22:30 |
| mcdonc | malthe: well, as i understand it, it's heap memory, and it gets fragmented, so sometimes even if python does the right thing, the OS cant get it back in usable chunks. | 22:30 |
| malthe | true | 22:31 |
| mcdonc | tim peters worked on an "arena allocator" that i *think* landed in py25 that's better at it | 22:31 |
| malthe | isn't there a gsoc project to move Zope to py25? | 22:32 |
| *** RaFromBRC is now known as RaFromBRC|lunch | 22:32 | |
| mcdonc | so maybe one concrete task would be to make zope2 run on py25, as i suspect it's better at it... yeah, although i'm not sure where the guy landed on it | 22:32 |
| J1m | I fear that the new memory allocator sucks. | 22:32 |
| mcdonc | gah | 22:32 |
| J1m | while working on a better packer, I tried to use py25 hoping the new allocator would make memory management better. | 22:33 |
| J1m | Python 2.5 packed much slower. | 22:33 |
| J1m | That is just anecdotal at this point. | 22:33 |
| malthe | progress is futile | 22:33 |
| J1m | But I'm a little worried. | 22:33 |
| mcdonc | tim wasnt at pycon either | 22:34 |
| J1m | (file storage packer) | 22:34 |
| mcdonc | we need a tim-come-home-please effort | 22:34 |
| malthe | :-) | 22:34 |
| mcdonc | frankly the sys.exit(0) thing sounds pretty good to me, esp. if it's staged shutdown.. it might be possible to convince graham to put some sort of request buffer in mod_wsgi that would hold on to requests as the process restarted | 22:36 |
| mcdonc | (at least for the zope-is-consuming-1GB-of-ram-and-i-dont-have-time-to-figure-out-why case) | 22:37 |
| malthe | mcdonc: good call | 22:38 |
| mcdonc | then the action becomes making plone start in under 35 seconds | 22:39 |
| mcdonc | plain old zope2 and zope3 apps start plenty fast | 22:39 |
| malthe | right 4-5 seconds | 22:39 |
| *** dunny has quit IRC | 22:40 | |
| *** djohnson has quit IRC | 22:43 | |
| *** pcardune has quit IRC | 22:43 | |
| J1m | I have a largish zope2 app that starts in under a second on moderately current hw. | 22:44 |
| J1m | My zope3 apps take closer to 2, but I'm hopeful I can get that down quite a bit. | 22:44 |
| *** alecm has joined #zope3-dev | 22:44 | |
| J1m | We have some zope3 appls that take much longer. | 22:45 |
| srichter | By pickling ZCML actions, we can gain 30% in startup time; (Marius and I tested this during the foliage sprint) | 22:45 |
| J1m | I bet we could do a lot better by computing most actions in Python, which would be a lot more appropriate in many cases. | 22:46 |
| srichter | J1m: how do you want to do that? | 22:47 |
| J1m | I'm also shidting to a model that requires far fewer registrations in the first place. Web 2 (as I define it;) is a wonderful thing. | 22:47 |
| srichter | (but right, the time is spent converting the unicode values to python objects) | 22:47 |
| J1m | s/shidting/shifting :) | 22:48 |
| srichter | he he | 22:48 |
| srichter | J1m: I'd love to hear your thoughts on that | 22:48 |
| J1m | well, in the way I'm doing things, I have views that are more like desktop apps. | 22:49 |
| J1m | They provide rish functionality. | 22:49 |
| J1m | The view provides a single page with sub-pages providing ajax methods. | 22:50 |
| srichter | right | 22:50 |
| J1m | These methods are just python methods on the view with jssonpage decorators. | 22:50 |
| J1m | jsonpage | 22:50 |
| J1m | so much less to register. | 22:50 |
| * srichter notes that Jim should have a look at z3c.formjs for a cool way to do ajax sub-views | 22:50 | |
| srichter | right | 22:50 |
| srichter | z3c.formjs does that | 22:51 |
| J1m | basically, an adapter and a zc.resourcelibrary library for the resources. | 22:51 |
| * J1m should look at z3c.formjs | 22:51 | |
| srichter | it is a bit misleading, zince z3c.formjs contains generically useful features | 22:52 |
| srichter | (Paul and I would be willing to split those into a separate package) | 22:52 |
| J1m | I'm gonna be releasing this sometime soon, as Rob Page thinks it's OK to check something that depends on Ext into the z.o repo. | 22:52 |
| srichter | cool | 22:52 |
| *** kev_ has joined #zope3-dev | 22:53 | |
| J1m | I've also integrated extjs w zope.formlib. So I can generate ext forms from formlib definitions. | 22:53 |
| srichter | cool, then I can port this to z3c.form :-) | 22:53 |
| J1m | srichter, were you refering to the ajax helper business? | 23:03 |
| *** reco has quit IRC | 23:04 | |
| srichter | J1m: no, the extjs form generation | 23:05 |
| *** quodt__ has quit IRC | 23:05 | |
| srichter | J1m: I am very interested in this | 23:05 |
| srichter | J1m: z3c.formjs does the ajax helper business already | 23:05 |
| srichter | it also does some nice event hookup | 23:05 |
| J1m | srichter, I was trying to take a quick look at z3c.formjs and wondering what to focus on. | 23:06 |
| *** djohnson has joined #zope3-dev | 23:06 | |
| *** quodt has joined #zope3-dev | 23:06 | |
| J1m | wrt extjs forms, I'm documenting this now and will be releasing it in the next few days. | 23:06 |
| *** quodt has quit IRC | 23:07 | |
| *** RaFromBRC|lunch is now known as RaFromBRC | 23:09 | |
| *** whit has quit IRC | 23:14 | |
| mcdonc | mgedmin: nice post on testing... | 23:16 |
| mgedmin | things had already quietened down, and here I come in and rekindle the flame :-) | 23:17 |
| *** whit has joined #zope3-dev | 23:17 | |
| mcdonc | naw, it was well-reasoned | 23:18 |
| *** romanofski has quit IRC | 23:28 | |
| *** greenman has quit IRC | 23:32 | |
| *** whit has quit IRC | 23:34 | |
| *** kev_ has quit IRC | 23:38 | |
| *** mgedmin has quit IRC | 23:41 | |
| malthe | mcdonc: "module foo.bar got 5 new untested lines of code" | 23:44 |
| mcdonc | eh? | 23:44 |
| malthe | from megdmin's post | 23:44 |
| mcdonc | oh.. heh | 23:44 |
| srichter | J1m: let me find you the low-level text files that should be of most interest to you | 23:45 |
| *** faassen has joined #zope3-dev | 23:47 | |
| faassen | br | 23:47 |
| faassen | oops. :) | 23:47 |
| *** harobed has joined #zope3-dev | 23:52 | |
| *** timte has joined #zope3-dev | 23:56 | |
Generated by irclog2html.py 2.15.1 by Marius Gedminas - find it at mg.pov.lt!