IRC log of #zope3-dev for Wednesday, 2007-11-14

* J1m looks at the transacript above more closely.00:00
*** norro has quit IRC00:00
* philiKON tries to install one of the failing packages00:00
srichterJ1m: could we make buildout ignore the version specification of
srichter(if so, I would not mind the versions in that much anymore)00:01
J1msrichter, no00:01
J1mphiliKON's explanation above doesn't sound right,00:01
J1mmcdonc, could you give us more information on the conflict?00:02
* philiKON is trying it out now00:02
J1mIt wouldn't expect a conflict w/o an upper bound.00:02
mcdoncJ1m: you can download the pickle, it's a pickled list of dictionaries, one dict per package, with detailed console output in each00:03
J1mUh huh.  Can't you just tell me what the conflict was? :)00:03
* mcdonc unpickles00:03
J1measy_install will always grab the most recent version in can find.00:03
philiKONInstalled /Users/philipp/TEST/lib/python2.4/site-packages/
philiKONerror: Installed distribution zope.traversing 3.4.0 conflicts with requirement zope.traversing>
philiKONi suppose 3.5.0a1 isn't on the cheeseshop00:04
philiKONit's in our find-links location00:04
philiKONso it didn't find it00:04
philiKONat first00:04
J1mat first?00:04
philiKONwell, when it first installed zope.traversing00:04
philiKONit found 3.4.0 as the max version00:04
philiKONso it installed it00:05
philiKONbut then something else wanted a higher version00:05
J1mBut it wouldn't find it later either.00:05
philiKONit doesn't have to00:05
philiKONwhen an egg is already installed, setuptools won't try to find a newer version00:05
philiKONit'll just check the version requirements00:05
philiKONand here they fail00:05
J1mso the thing that wanted the higher version should have listed in its dependency links.00:05
J1mphiliKON, you are mistaken.00:06
J1mohm maybe not00:06
philiKONhow so?00:06
philiKONi might very well be :)00:06
J1mwell, I would expect setuptools to look for a newer version if it has a requirement that is unmet.00:07
J1mBut maybe it's dummer than I expect. :(00:07
philiKONi think it's dummer than we all expect00:07
*** junkafarian has quit IRC00:07
J1mI can immagine it being that dumb.00:07
J1mphiliKON, did you try adding the necessary find link on the command line?00:08
philiKONlemme try again00:08
philiKONthat's -f right?00:08
J1mmaaaaybe :)00:08
TresEquisfirst, who released an egg which depends on a zope.traversing version which was not even properly released?00:09
TresEquisthe .dev-r#### eggs must *not* be released into the wild00:09
philiKONwell, this one wasn't00:09
philiKONand becuase of that it's causing problems00:09
TresEquis*somebody* uploaded zope.traversing-3.5dev-r78730 somewhere00:10
TresEquisor Chris' script wouldn't have had a conflict00:10
*** gidas has quit IRC00:11
TresEquisand *somebody* released something requiring zope.traversing>=3.5.0a1 without ensuring that there was a properly-released egg satisfying that requirement00:11
philiKONapparently install doesn't take -f or --find-links00:11
philiKONTresEquis: yup, i agree, that was a mistake00:11
TresEquisthis stuff infected grok, too00:11
philiKONi know. :(00:11
J1mphiliKON, hm. That's annoying.00:12
srichterTresEquis: pointing fingers does not help at all; we trying to figure out how to handle the situation00:12
TresEquishonestly, we should yank all the .dev-r###' eggs from, as well as the cheeseshop, and replace with with real releases00:12
philiKONJ1m: sure is00:12
philiKONTresEquis: last time i did that with just one egg, people got real mad00:12
mcdoncphiliKON: no it doesnt and that sucks, but you can change the's dependency_links=00:12
philiKONmcdonc: ok, will do00:12
TresEquisand we have to point fingers so that people understand why "releasing" such an egg is a *terrrible* mistake00:13
* mcdonc still attempts to figure out the conflict flailingly00:13
TresEquisif your package needs a feature which is not yet available in a properly-released version of your dependency, YOU CAN'T RELEASE YET00:13
J1mmcdonc, it doesn't sound like a conflict.00:13
mcdoncwell it says... Installed distribution zope.traversing 3.4.0 conflicts with requirement zope.traversing>
J1mIt sounds like an unmet requirement that just plays a conflict in TV.00:14
TresEquissomething requires the 3.5+ egg00:14
TresEquiswhich is not available in the universe Chris was pulling from00:14
mcdoncsetuptools does a horrible job of telling you where the requirement came from00:14
TresEquis $ find lib/python/site-packages/ -name "requires.txt" | xargs grep "\.dev"00:15
J1mmcdonc, I tried hard to provide good diagnistics in buildout. :)  If you run it in verbose mode, it tells you why it looks for each requirement.00:15
*** Jell-O-Fishi has joined #zope3-dev00:15
* philiKON installs again with find_links in setup.py00:16
*** afd_ has quit IRC00:18
philiKONthat doesn't seem to help, though00:19
J1mparts = myegg00:20
J1meggs = myegg00:20
J1mif you run that with -v and substituting the package in question for myegg, then you should get some helpful info.00:21
J1myou probably also want: eggs-directory = eggs00:21
J1min the buildout section if you have a shared eggs for00:21
J1mthere ya go00:21
TresEquis $ svn blame  $ZSVN/
mcdoncJ1m: thanks...00:23
TresEquissays it was 'zagy' who added the faux-released packages00:23
mcdoncwhile we're at it...00:23
TresEquisHis checkin comment says: "Added a `layer` attribute to `xmlrpc:view`. This works just like layers for00:24
TresEquis`browser:view` etc. but uses the `IXMLRPCSkinType`.00:24
J1mmcdonc, that one shouldn't be a problem, as a matching relrase is in the cheeseshop.00:24
J1mThe easy "fix" for is to add to the dependeoncy_links.00:25
J1mand make a new release.00:25
TresEquisthe root directory there is completely polluted with faux releases00:26
J1mDude, this is only an alpha release.00:26 contains a lot of nasty packages; making it part of your dep links has bad side effects00:26
J1mThis is an alpha release of
mcdoncwell, note that this was from an install of "zope.tales"00:27
mcdoncwhich is its own insanity00:27
J1malternatively, someone could copy the needed zope.traversing link to the cheeseshop.00:27
J1malternatively, someone could copy the needed zope.traversing release to the cheeseshop.00:27
TresEquisWe need to tell people to quit depending on being able to download faux releases00:28
TresEquisif they need such a thing, they should maintain their own link server00:28
TresEquisand then we need to wipe every one of them from the "public" locations00:29
TresEquisreleasing stuff prematurely is going to kill our story00:29
J1mexcept that removing links has its own problems.00:30
J1mI think we all agree,00:30
TresEquisWe give people warning00:30
TresEquisand then remvoe the stuff00:30
J1mOTOH, I think there is a better test than what mcdonc did.00:30
J1mI suggest only non-final releases that are actually in the cheeseshop should be tested.00:30
TresEquisWhy 'non-final'?00:31
J1mWe shouldn't worry that much about whether a releases alpha works, much less an unreleased subversion trunk.00:31
J1mI meant final.00:31
mcdoncJ1m: i offered to do that in my mail00:31
J1mOK, I accept. :)00:31
mcdoncI need to figure out how to know what "the final releases" are00:32
J1mThat should be pretty easy.00:32
TresEquiswe could do a search using trove classifiers00:32
J1mYou can figure this put by looking at the versions.00:32
J1mThis is a faaaairly straighforward pkg_resources script to write.00:33
mcdoncnote though that it doesnt really matter for the "traversing" dependency error... easy_install zope.pagetemplate, you'll get the same error.00:33
J1mEspecially in comibination with the pypi xml-rpc interface.00:33
philiKONalas, the trove classifiers aren't set automatically based on the a,b,c suffix00:33
J1mI wouldn't use the trovr classifiers at all.00:33
TresEquiswe have to re-release packages which have already "escaped"00:33
*** srichter has quit IRC00:34
J1mTresEquis, yes00:34
*** srichter has joined #zope3-dev00:34
*** pbugni has joined #zope3-dev00:34
mcdonci didnt know there was a pypi xmlrpc interface00:34
TresEquisand fix packages which have goofy requirements00:34
*** ChanServ sets mode: +o srichter00:34
mcdonck, well, glad thats solved ;-)00:35
mcdoncvery cool00:35
J1mOf course, it's made trickier by the fact that there's no prefer-final option for easy install.00:35
J1mso even if you easy_install a final release, it may pick up non-final releases of it's dependencies.00:36
* TresEquis is planning to hack in a multi-index feature00:36
J1mI'm rather worried about how screwed up the setuptools development process is.00:36
*** RaFromBRC has quit IRC00:37
J1mIn fact, the lack of a prefer-final option in easy_install is a pretty good argument for *never* uploading non-final releases to the cheeseshop.00:37
*** RaFromBRC has joined #zope3-dev00:38
J1mI wonder if it would make sense to create a mirror that filters out non-final releases.00:38
J1mOf course, it would folter out setuptools and buildout.00:39
srichterwell, KGS can do this, a bit manually00:41
J1mWhat we really need is a prefer-final option in easy_install.00:42
J1mow we need to clean up the cheeseshop. :)00:42
philiKON+1 to clean up00:42
*** strichter has joined #zope3-dev00:45
mcdonc+1 on cleaning up the cheeseshop or not using it00:46
ccombit's the same kind of problematics than the debian repositories00:47
J1mwell, for what you are trying to do, I don't think not using it is an option.00:47
*** alecm has quit IRC00:49
mcdoncwe aren't currently using it for our egg releases, although it's likely we will eventually... but its not useful as an index from which to retrieve a known good set (obviously).. just as a place to poke around and see what's new00:49
TresEquisI think that having some kind of "egglint" thing we use to check packaging sanity before release would be good00:50
TresEquismaybe it just runs ' egg_info' and then grubbing in the '.egg-info' directory might be enough00:52
*** benji has quit IRC00:52
TresEquisit should be possible to do ' develop' on a checkout inside an empty virtualenv, using only the metadata in the setup.py00:53
TresEquisif that doesn't work, then the package isn't ready for the cheeseshop00:53
TresEquisor even for ''00:53
TresEquissrichter: why did you mirror the whole cheeseshop into the KGS for Zope 3.4?00:54
TresEquisNo way is all that stuff "known good"00:54
*** jsadjohnson has quit IRC00:54
*** dobee has quit IRC00:54
*** spython has quit IRC00:56
*** srichter has quit IRC00:59
*** re|potatopancake has left #zope3-dev01:02
*** hazmat has quit IRC01:03
*** MJ has quit IRC01:06
strichterTresEquis: No, only the controlled packages are known good01:08
strichterTresEquis: setuptools only supports one index01:08
TresEquisbut if I use the KGS, I don't want to fall back to cheeseshop packages01:10
TresEquisI want to fail, and let the user override the index to pick  the cheeseshop01:10
TresEquisotherwise the KGS appears to contain a random bunch of unreviewed packages01:10
strichterthe KGS is controlled-packages.cfg01:14
TresEquisIf I look at is a *package index* generated from it01:15
TresEquisI see wild west, not known good01:15
strichterwell, you can have only *one* index01:15
TresEquisI don't understand why packages which are not in the KGS are in the index01:15
TresEquisbut I don't *want* wild west packges included in the index which is supposed to represent "known good" stuff01:16
strichterthen look at
TresEquisthat isn't an index01:16
strichterok, I give up01:16
TresEquisI want an index which contains *only* packages which are in the KGS01:16
strichterbut you can't!01:16
TresEquisif I want to use the cheeseshop, I won't point at the KGS-derived indeex01:16
TresEquisI'll point at the cheeseshop01:16
TresEquisyes you can01:17
TresEquisjust not in the same run01:17
TresEquis $ bin/easy_install --index-url= zope.interface01:17
strichterwell, that's not a use case I ahve01:17
strichterI just want it to work in buildout01:18
strichteryou can use the controlled-packages.cfg file to generate a different output01:18
TresEquisIf your index contains packages which you haven't reviewed, then you have unintended consequences01:18
TresEquispeople who rely on the "known goodness" of your index may get burned01:18
strichterlook, the usage has been well explained and documented on the list several time and in the release notes01:19
TresEquisit should be possible to install *any* egg from the "known good" set without going to the cheeseshop01:19
TresEquisright now, if somebody gets an egg into the KGS which has dependencies which can only be satisfied from the cheeseshop mirror, that would be a Bad Thing, no?01:20
TresEquisso how can you tell that such a Bad Thing occurs, if you mirror the cheeseshop into the index?01:21
TresEquisif people want to write buildouts which install eggs from other indexes (like the cheeseshop), then they should override the index_url for those eggs (assuming that the default is the KGS index)01:21
strichterno, that's not a bad thing01:27
strichtermy project might depend on package foo01:27
strichterbut our KGS does not control foo01:28
*** lucielejard has quit IRC01:30
TresEquisso your project should not be using the KGS-derived index when trying to fetch foo01:31
TresEquisit should use the cheeseshop, or some other index which knows about foo01:31
strichterI totally disagree01:33
TresEquiswhy do you imagine that an index needs to know about every package in the world?01:35
TresEquisthat is insane01:35
TresEquisit should know only about packages^wprojects whose metadata it tracks (and, for sanity, whose distributions it controls)01:36
strichterbecause we can only have one01:36
TresEquisnot so01:36
strichterthat's not my limitation01:36
TresEquisbut it can be reconfigured01:36
TresEquisif a package needs to specify the KGS index, but also depend on finding packages not in that index, it can add find-links01:37
*** harobed has quit IRC01:38
J1mTresEquis, the KGS that strichter and I devised is a mirror of the cheesehop except that it controls some packages.01:39
J1mThat satisfies a reasonable set of use cases.01:39
J1mOne could certianly create a KGS that excluded anything it didn't control.01:40
J1mThat latter would be inconvenient for lots of packages.01:40
TresEquispackage quality will suffer if people don't use a restricted index by default01:40
TresEquisprojects which want a broader set should be able to "federate" indexes, perhaps01:41
J1mUnfortunately, find-links don't get you very far because a valid index isn't necessarily a valid source of find links.01:41
J1mNote that dfor development, you could live with some instability.01:42
TresEquisif we had a "kgs" index, and then another which federated it with the cheeseshop, I would use the first one exclusively01:42
J1mNote that for development, you could live with some instability.01:42
J1mI/we could set such a thing up.01:42
TresEquisbut I'd want to choose it explicitly01:42
TresEquisrather than getting it foisted on me by a packaging error01:43
J1mIt would go a loooong way if easy_install grew a perfer-final option.01:43
TresEquiswhen *doing* packaging, the restricted index should be the "testbed"01:43
mcdoncsrichter: maybe you didnt mean this, but if you did, i dont understand why you'd want to upload releases to the cheeseshop if they cant be installed via easy_install... if they're only useful for buildout-based installs, shouldnt they just go into a private repo for people to not trip over when they go try one out with setuptools stuff?01:43
TresEquisdownstream, folks who want to mix the KGS packages with cheeseshop should have easy ways to do that01:44
TresEquisbut the folks who are maintaining the zope.* and related packages should test *without* the cheeseshop01:44
mcdoncstrichter: that's in response to "I just want it to work in buildout", BTW.01:45
*** dunny has quit IRC01:46
*** philiKON_ has joined #zope3-dev01:47
* strichter points out that he did not sleep last night and is a bit cranky, unconcentrated and innprecise today01:48
*** dunny has joined #zope3-dev01:48
strichtermcdonc: I think I misundestood your question a bit before...01:48
TresEquisJ1m: I don't think prefer-final is enough01:49
TresEquisI think that we need to think of the constraint solving problem as:01:50
strichterTresEquis: it is very trivial to create a restricted index from the cotnrolled-packages.cfg file01:50
TresEquis - packages express minimal constraints (just names where possible, '>=' where absolutely necessary)01:50
strichterzope.release has already several things that can help01:50
TresEquis - indexes track a set of candidates which can be used to satisfy those requirements, plus those supplied by the client (buildout, easy install)01:51
strichteralso note that controlled-packages.cfg supports an "extends" feature like buildout, so you can extend the controlled set based on another one (or more)01:51
*** pcardune has joined #zope3-dev01:51
TresEquis - the clients should be flexible enough to handle multiple indexes, and to impose additional constratints01:51
TresEquis - indexes should be able to expose their dependency maps so that clients can do resolution without downloading01:52
* J1m is stll wondering why prefer-final isn't enough01:52
strichteryeah, but that requires a lot of new software accross several communities; it's a good long term goal, but I won't hold my breath01:52
TresEquisIf the indexes are narrow enough, you don't actually need much in the way of software01:53
strichterok, anyways; I am out of here; I am too much of a zombie01:53
TresEquisthe model here is like Debian / Ubuntu01:54
*** philiKON has quit IRC01:54
TresEquisfeisty has main, restricted, universe, etc01:54
TresEquisthose are frozen when feisty was released01:54
TresEquisthen feisty-security has the same sets01:54
TresEquisbut gets modified as new security fixes are released01:55
TresEquisfeisty-updates has non-security bugfixes01:55
TresEquisfesity-backports has "feature" releases which have had some testing against the feisty baseline01:55
TresEquisthe apt client knows how to solve against the packages it finds from each index01:56
TresEquisbut it doesn't need to filter against the "unstable" stuff (whatever adjective the heron has now)01:56
TresEquisbecause those packages aren't included in the index set01:56
J1mI suppose that if the indexes were flat, you could use find-links and if they were all clean, you'd have what you want.01:57
J1mPersonally, I'd be happy to see a variation on the existing KGS that only included known packages.01:57
J1mI expect that it would be a small change to the existing setup.01:58
*** nathany has quit IRC02:01
*** pcardune has quit IRC02:01
*** elro has quit IRC02:02
*** J1m has quit IRC02:02
*** TresEquis has quit IRC02:13
*** gstratton_ has quit IRC02:14
*** pyqwer has quit IRC02:22
*** b52lap has quit IRC02:35
*** pbugni has quit IRC02:38
*** niemeyer has quit IRC02:39
*** rmarianski has quit IRC02:41
*** hazmat has joined #zope3-dev02:57
*** Ariel_Calzada has quit IRC02:58
*** alecm has joined #zope3-dev03:01
*** timte has quit IRC03:10
*** RaFromBRC has quit IRC03:21
*** rmarianski has joined #zope3-dev03:35
*** rmarianski has quit IRC03:41
*** Ariel_Calzada has joined #zope3-dev03:59
*** dabu has quit IRC04:05
*** rmarianski has joined #zope3-dev04:11
*** greenman has quit IRC04:16
*** ktwilight_ has joined #zope3-dev04:19
*** stub has quit IRC04:25
*** aaltepet has joined #zope3-dev04:27
aaltepetgreets.  I'm looking for a way to get all interfaces an object implements.04:28
aaltepetI need both direct and indirect (inherited) interfaces.04:29
aaltepetI know instancedOfObjectImplements will get me the direct ones04:29
aaltepetBut what api call will give me the others?04:29
*** ktwilight has quit IRC04:32
*** RaFromBRC has joined #zope3-dev04:34
*** rcrafton has quit IRC04:53
*** pcardune has joined #zope3-dev04:57
*** RaFromBRC has quit IRC05:00
*** Ariel_Calzada has quit IRC05:07
*** hazmat has quit IRC05:07
*** hazmat has joined #zope3-dev05:09
*** Ariel_Calzada has joined #zope3-dev05:26
*** aaltepet has quit IRC05:34
*** rmarianski has quit IRC05:41
*** Ariel_Calzada has quit IRC05:44
*** lucielejard has joined #zope3-dev06:14
*** mcdonc has quit IRC06:43
*** mcdonc has joined #zope3-dev06:44
*** cursor has joined #zope3-dev06:53
*** reco has quit IRC07:04
*** lucielejard has quit IRC07:16
*** TresEquis has joined #zope3-dev07:23
*** alecm has quit IRC07:32
*** TresEquis has quit IRC07:45
*** afd_ has joined #zope3-dev08:20
*** hdima has joined #zope3-dev08:36
*** cursor has quit IRC08:42
*** __mac__ has joined #zope3-dev08:52
*** jodok has joined #zope3-dev08:53
*** __mac__ has quit IRC08:54
*** jodok has quit IRC08:54
*** zagy has joined #zope3-dev08:55
*** jodok has joined #zope3-dev08:56
*** __mac__ has joined #zope3-dev08:56
*** sorin has joined #zope3-dev09:15
*** sorin is now known as sorindregan09:16
*** projekt01 has joined #zope3-dev09:18
*** schwendinger has joined #zope3-dev09:22
*** afd__ has joined #zope3-dev09:28
*** dobee has joined #zope3-dev09:30
*** stub has joined #zope3-dev09:33
*** afd_ has quit IRC09:41
*** gstratton has joined #zope3-dev09:45
*** cursor has joined #zope3-dev09:47
*** goschtl has joined #zope3-dev09:54
*** xenru has quit IRC09:56
*** quodt has joined #zope3-dev09:59
*** timte has joined #zope3-dev10:12
*** pyqwer has joined #zope3-dev10:15
*** pelle_ has joined #zope3-dev10:17
*** b52laptop has joined #zope3-dev10:24
*** pcardune has quit IRC10:33
*** regebro has joined #zope3-dev10:33
*** elro has joined #zope3-dev10:39
*** elro has quit IRC10:40
*** _srichter has joined #zope3-dev10:44
*** strichter has quit IRC11:00
*** harobed has joined #zope3-dev11:05
*** elro has joined #zope3-dev11:08
*** dunny_ has joined #zope3-dev11:08
*** elro has left #zope3-dev11:11
*** dunny has quit IRC11:19
*** cursor_ has joined #zope3-dev11:28
*** salfield has quit IRC11:31
*** dunny has joined #zope3-dev11:36
*** cursor has quit IRC11:36
*** MJ has joined #zope3-dev11:41
*** thruflo has joined #zope3-dev11:41
*** dunny_ has quit IRC11:49
*** Ariel_Calzada has joined #zope3-dev12:08
*** dunny has quit IRC12:11
*** dunny has joined #zope3-dev12:18
*** huajie has joined #zope3-dev12:22
*** thruflo has quit IRC12:31
*** philiKON_ is now known as philiKON12:34
*** gidas has joined #zope3-dev12:39
*** niemeyer has joined #zope3-dev12:44
*** dunny has quit IRC12:46
*** schwendinger has quit IRC12:53
*** cursor has joined #zope3-dev12:55
*** schwendinger has joined #zope3-dev12:58
*** _srichter is now known as srichter12:59
*** ChanServ sets mode: +o srichter12:59
*** mgedmin has joined #zope3-dev13:00
*** cursor_ has quit IRC13:03
*** pelle_ has quit IRC13:23
*** ignas has joined #zope3-dev13:27
*** malthe has joined #zope3-dev13:32
*** pelle_ has joined #zope3-dev13:32
*** malthe has quit IRC13:33
*** salfield has joined #zope3-dev13:34
*** baijum has joined #zope3-dev13:36
*** yvl has joined #zope3-dev13:38
*** jsadjohnson has joined #zope3-dev13:40
*** malthe has joined #zope3-dev13:41
*** junkafarian has joined #zope3-dev13:49
*** afd__ has quit IRC13:52
*** MJ has quit IRC13:53
*** Ariel_Calzada has quit IRC14:05
*** __mac__ has quit IRC14:07
*** fcorrea has joined #zope3-dev14:14
*** mgedmin_ has joined #zope3-dev14:15
*** mgedmin has quit IRC14:16
*** hazmat has quit IRC14:22
*** rcrafton has joined #zope3-dev14:28
*** thruflo has joined #zope3-dev14:29
*** Ariel_Calzada has joined #zope3-dev14:31
*** afd_ has joined #zope3-dev14:37
*** grahal has joined #zope3-dev14:42
*** ghendi has joined #zope3-dev14:50
*** afd_ has quit IRC14:56
*** afd_ has joined #zope3-dev14:56
*** Ariel_Calzada has quit IRC15:04
*** pelle_ has quit IRC15:06
*** pan_ has joined #zope3-dev15:11
*** ktwilight_ is now known as ktwilight15:11
*** pelle_ has joined #zope3-dev15:13
*** ccomb has quit IRC15:17
*** benji has joined #zope3-dev15:18
*** projekt01 has quit IRC15:19
*** ccomb has joined #zope3-dev15:24
*** jfroche has joined #zope3-dev15:25
*** pan has quit IRC15:28
*** MJ has joined #zope3-dev15:35
*** huajie has quit IRC15:35
*** projekt01 has joined #zope3-dev15:36
gstrattonIf I have a field which should be a choice of contained objects, what's the best way of implementing this?15:40
*** rcrafton has quit IRC15:42
goschtlgstratton: you can have a look at this Ok this is plone but it use the formlib and should give you an idea what to do...15:42
*** yvl has quit IRC15:45
*** alga has joined #zope3-dev15:56
*** hazmat has joined #zope3-dev15:58
gstrattonI don't think that example really fits my usecase. I guess I basically want a TextLine field, but with a select widget which is populated with a dynamically generated list.15:58
*** lucielejard has joined #zope3-dev16:08
*** lucielejard has quit IRC16:08
*** cursor has quit IRC16:10
ccombgstratton you can use a vocabulary16:11
gstrattonccomb: Yup, just got there, thanks!16:11
*** __mac__ has joined #zope3-dev16:21
dennis__ccomb ?16:23
dennis__your mail to the list:   if you search for value in root['btree'].values() this is just a sequential full-tree scan? are you sure you don't want to do something like   value in root['btree']  ?16:23
dennis__if you need to search for values rather than keys (or both), i would set up a reverse index16:25
*** lucielejard has joined #zope3-dev16:30
*** __mac__ has quit IRC16:56
*** __mac__ has joined #zope3-dev16:58
*** yvl has joined #zope3-dev16:58
*** mkerrin has joined #zope3-dev17:04
*** hdima has quit IRC17:04
*** sorindregan has quit IRC17:06
*** ghendi has quit IRC17:08
*** thruflo has quit IRC17:12
*** cursor has joined #zope3-dev17:16
*** MJ has quit IRC17:29
*** MJ has joined #zope3-dev17:31
ccombdennis__ : I need to loop over a large amount of data to build means or alerts on event detection17:32
ccombWhen I change the threshold of the alerts, I need to loop again a rebuild a graph17:33
*** yvl has left #zope3-dev17:33
dennis__ccomb so you do really need to touch every single object in your btree ?17:34
ccombyes to get the actual values17:34
ccomball this in a single request.17:35
*** baijum has quit IRC17:35
ccombbut I don't understand why looping implies to raise everything in the memory17:36
dennis__i think a btree should do it piece-wise, which if you are going through the whole thing means everything, but it should be able to garbage collect the pieces you have visited (depends a bit on the distribution of keys over the buckets i guess)17:37
dennis__how do you measure the memory use actually?17:39
ccombI'm looking at a dynamic chart or at 'top'17:39
*** rmarianski has joined #zope3-dev17:40
dennis__what happens if you import gc, and every few hundred objects call gc.collect()  ?17:41
ccombThere is something really weird, because now I'm just doing root.keys(), and I see the memory eating 400MB at once17:42
ccombit should just list the keys of the root Folder17:42
dennis__how much is in there?17:42
ccomb>>> list(root.keys())  =>   [u'b', u'epidemio', u'l']17:43
ccomb3 items17:43
ccombb is the btree, l is the list17:43
ccomb(they are just for test, the real data is in a subfolder of u'epidemio')17:45
*** dobee has quit IRC17:46
ccombI removed the list 'l', and now root.keys() takes nothing17:47
dennis__well the list needs to be loaded completely before being able to jump to any values. the btree is much better and loads on demand (so all this is only still about the actual references to the objects being contained)17:49
ccombbut why is the list completely loaded when I just get the keys of the container?17:50
*** jfroche has quit IRC17:50
dennis__i don't know why it wants to wake up the list object17:51
dennis__try:   it = iter( root['b'].values() )17:52
dennis__for i in it: pass17:52
ccombI've already tried this :  for i in root['b'].itervalues() : pass17:53
ccombit eats17:53
ccombtrying with iter( values() )17:53
*** whitmo has joined #zope3-dev17:53
dennis__ah sorry17:54
*** whitmo_ has joined #zope3-dev17:55
*** nathany has joined #zope3-dev17:57
*** nathany has quit IRC18:00
*** nathany has joined #zope3-dev18:00
*** whit has quit IRC18:02
*** whitmo_ is now known as whit18:02
*** goschtl has quit IRC18:07
ccombWorking with slice works in the following manner :18:08
lisppaste6ccomb pasted "loop with slice" at
ccombthis way it eats only the memory for 1000 tuples. But it's slower18:08
ccombI'm sure there is some tweak I could use with transactions18:09
dennis__ugly solution that :)18:11
ccombugly is better than lobotomized :)18:11
*** whitmo has quit IRC18:11
ccomb(that's one of my favourites)18:11
dennis__it's a very positive thought18:12
*** natea_ is now known as natea18:17
*** hazmat has quit IRC18:18
*** hazmat has joined #zope3-dev18:19
*** ChanServ sets mode: +o hazmat18:20
*** ktwilight has quit IRC18:23
*** ktwilight has joined #zope3-dev18:24
mcdoncccomb: i think you may want something like btree._p_jar.cacheMinimize()18:34
mcdonc(or actually anypersistentobject._p_jar.cacheMinimize())18:35
ccombmcdonc: that looks interesting18:36
*** jodok has quit IRC18:37
mcdoncccomb: you can "deactivate" a single persistent object using anypersistentobject._p_deactivate()18:37
*** jodok has joined #zope3-dev18:37
* ccomb will just try this18:38
mcdoncccomb: which, if the object state is loaded into memory, turns in back into a "ghost" object without only "stub" state in memory (it will need to be retrieved from the database or the cache again)18:38
*** ktwilight has quit IRC18:39
*** ktwilight has joined #zope3-dev18:40
dennis__setting cacheSize doesn't prevent the memory usage from growing, but calling cacheMinimize() every few thousand objects does help18:41
dennis__ccomb of course what is also a problem is that your objects (tuples) are not themselves persistent18:42
*** pyqwer has quit IRC18:42
dennis__if you put classes there which derive from Persistent, they can be deactivated18:42
*** mgedmin_ is now known as baloon_slayer18:42
ccombdennis__ I was thinking about it, that could be the core of the problem18:42
ccombthe funny thing is that I've chosen tuples for memory efficiency?18:43
*** gstratton_ has joined #zope3-dev18:44
*** __mac__ has quit IRC18:45
*** alga has quit IRC18:49
dennis__ccomb my guess is that because the tuples are not persistent, their size all becomes part of the btree, which counts itself only as relatively few objects in the cache (certainly not 700,000)18:50
dennis__i am trying that theory but ran out of memory :P18:50
*** alecm has joined #zope3-dev18:52
ccombok. I confirm that   _p_jar.cacheMinimize()   works very well if I run it every N loops18:53
dennis__ccomb if I use persistent.list instead of tuples everything works exactly right18:53
ccomba first thanks for this solution18:53
dennis__without the cacheMinimize hack18:54
ccombdennis__ I will also try with persistent objects18:54
ccombthen summarize the story on the list18:54
*** harobed has quit IRC18:54
dennis__with tuples: [('BTrees.OOBTree.OOBTree', 4)]   (in the cache after iterating)18:54
dennis__with persistent lists: [('BTrees.OOBTree.OOBTree', 56),('persistent.list.PersistentList', 100000)]18:55
ccombhow do you query the cache?18:55
dennis__and all those lists can be deactivated18:55
*** bung_vijay_ has joined #zope3-dev18:55
dennis__please post your findings though after using persistent objects, I'm curious how it works out :)18:56
*** gstratton__ has joined #zope3-dev18:57
*** bung_vijay_ is now known as bung_vijay18:57
ccombyes I will also mention it, since this is the real correct solution18:57
ccombpersistency need some care :)18:57
dennis__it all makes sense though, and then it works exactly as you'd expect18:57
ccombI'm glad this was not a dead-end18:58
dennis__learned something again, thanks for bringing it up :)18:58
ccombthanks to you and mcdonc too18:58
mcdoncccomb: no problem ;-)  i'm sort of curious why putting nonpersistent objects in the btree would cause them all to be loaded in memory at once while iterating over it. though...19:00
*** gstratton has quit IRC19:00
ccombis it worth reporting a bug at least to investigate?19:01
mcdoncif tim peters were here, i'm sure he'd tell us in detail, absolutely correctly, and make us laugh while he was doing it ;-)19:01
mcdoncccomb: i think putting up a bug in the zope collector (with the topic "zodb" selected from the topic dropdown) might not be a bad idea, as long as you attach a repeatable test script19:02
*** gstratton has joined #zope3-dev19:03
*** MJ has quit IRC19:03
philiKONthere's a zodb collector19:04
philiKONplease file it there19:04
philiKONit's also on launchpad19:04
mcdoncphiliKON: .. had no idea19:06
philiKONtheuni did some real cleanup last weekend19:07
Theunimcdonc: *g19:09
mcdoncTheuni, philiKON: please stop breaking the curve ;-)19:09
*** malthe is now known as malthe|away19:11
*** gstratton_ has quit IRC19:12
*** jfroche has joined #zope3-dev19:12
*** jfroche_ has joined #zope3-dev19:15
*** gstratton__ has quit IRC19:18
*** bung_vijay has quit IRC19:19
*** bung_vijay has joined #zope3-dev19:20
*** schwendinger has quit IRC19:26
*** jodok has quit IRC19:33
*** quodt has quit IRC19:35
*** projekt01 has quit IRC19:43
*** bung_vijay has quit IRC19:49
*** bung_vijay has joined #zope3-dev19:49
*** Ariel_Calzada has joined #zope3-dev19:50
*** pbugni has joined #zope3-dev20:14
*** quodt has joined #zope3-dev20:31
*** jfroche has quit IRC20:33
*** RaFromBRC has joined #zope3-dev20:41
*** baloon_slayer is now known as balloon_slayer20:42
*** thruflo has joined #zope3-dev20:43
*** pcardune has joined #zope3-dev20:51
*** balloon_slayer has quit IRC20:53
*** schwendinger has joined #zope3-dev20:54
*** ignas has quit IRC20:56
*** schwendinger_ has joined #zope3-dev20:59
*** rcrafton has joined #zope3-dev21:00
*** jodok has joined #zope3-dev21:05
*** mkerrin has quit IRC21:10
*** thruflo has quit IRC21:10
*** schwendinger has quit IRC21:12
*** schwendinger has joined #zope3-dev21:13
*** projekt01 has joined #zope3-dev21:23
*** pcardune has quit IRC21:23
*** schwendinger has quit IRC21:24
*** gidas has quit IRC21:27
*** schwendinger_ has quit IRC21:31
*** pelle_ has quit IRC21:33
*** grahal has quit IRC21:38
*** pcardune has joined #zope3-dev21:40
*** markusleist has quit IRC21:41
*** Ariel_Calzada has quit IRC21:44
*** Ariel_Calzada has joined #zope3-dev21:46
*** dunny has joined #zope3-dev21:47
*** projet01_ has joined #zope3-dev21:47
*** projekt01 has quit IRC21:51
*** MJ has joined #zope3-dev21:51
*** MJ has quit IRC21:52
*** MJ has joined #zope3-dev21:54
mcdoncphiliKON, Theuni: any ideas/opinions on my motivation in
*** faassen has joined #zope3-dev22:14
*** norro has joined #zope3-dev22:15
*** malthe|away is now known as malthe22:15
*** Ariel_Calzada has quit IRC22:16
*** Ariel_Calzada has joined #zope3-dev22:18
*** Ariel_Calzada has quit IRC22:18
*** projet01_ has quit IRC22:25
*** jodok has quit IRC22:25
*** cursor has quit IRC22:29
*** jodok has joined #zope3-dev22:31
*** xenru has joined #zope3-dev22:33
xenruphiliKON: are you here?22:34
xenruphiliKON: I need to ask some questions about book22:34
*** pcardune has quit IRC22:44
*** whit has quit IRC22:48
*** jodok has quit IRC22:57
*** bung_vijay has quit IRC23:06
*** fcorrea has quit IRC23:09
*** MJ has quit IRC23:15
*** norro has quit IRC23:33
*** whit has joined #zope3-dev23:35
*** jodok has joined #zope3-dev23:38
*** junkafarian_ has joined #zope3-dev23:39
*** junkafarian_ has quit IRC23:39
*** afd_ has quit IRC23:45

Generated by 2.15.1 by Marius Gedminas - find it at!