| * J1m looks at the transacript above more closely. | 00:00 | |
| *** norro has quit IRC | 00:00 | |
| * philiKON tries to install one of the failing packages | 00:00 | |
| srichter | J1m: could we make buildout ignore the version specification of setup.py? | 00:01 |
|---|---|---|
| srichter | (if so, I would not mind the versions in setup.py that much anymore) | 00:01 |
| J1m | srichter, no | 00:01 |
| J1m | philiKON's explanation above doesn't sound right, | 00:01 |
| J1m | mcdonc, could you give us more information on the conflict? | 00:02 |
| * philiKON is trying it out now | 00:02 | |
| J1m | It wouldn't expect a conflict w/o an upper bound. | 00:02 |
| mcdonc | J1m: you can download the pickle, it's a pickled list of dictionaries, one dict per package, with detailed console output in each | 00:03 |
| J1m | Uh huh. Can't you just tell me what the conflict was? :) | 00:03 |
| * mcdonc unpickles | 00:03 | |
| J1m | easy_install will always grab the most recent version in can find. | 00:03 |
| philiKON | Installed /Users/philipp/TEST/lib/python2.4/site-packages/zope.app.interface-3.4.0-py2.4.egg | 00:04 |
| philiKON | error: Installed distribution zope.traversing 3.4.0 conflicts with requirement zope.traversing>=3.5.0a1.dev-r78730 | 00:04 |
| philiKON | i suppose 3.5.0a1 isn't on the cheeseshop | 00:04 |
| philiKON | it's in our find-links location | 00:04 |
| philiKON | so it didn't find it | 00:04 |
| J1m | ah | 00:04 |
| philiKON | at first | 00:04 |
| J1m | at first? | 00:04 |
| philiKON | well, when it first installed zope.traversing | 00:04 |
| philiKON | it found 3.4.0 as the max version | 00:04 |
| philiKON | so it installed it | 00:05 |
| philiKON | but then something else wanted a higher version | 00:05 |
| J1m | But it wouldn't find it later either. | 00:05 |
| philiKON | right | 00:05 |
| philiKON | it doesn't have to | 00:05 |
| philiKON | when an egg is already installed, setuptools won't try to find a newer version | 00:05 |
| philiKON | it'll just check the version requirements | 00:05 |
| philiKON | and here they fail | 00:05 |
| J1m | so the thing that wanted the higher version should have listed download.zope.org in its dependency links. | 00:05 |
| philiKON | apparently | 00:05 |
| J1m | philiKON, you are mistaken. | 00:06 |
| J1m | ohm maybe not | 00:06 |
| philiKON | how so? | 00:06 |
| J1m | sorry | 00:06 |
| philiKON | i might very well be :) | 00:06 |
| J1m | well, I would expect setuptools to look for a newer version if it has a requirement that is unmet. | 00:07 |
| J1m | But maybe it's dummer than I expect. :( | 00:07 |
| philiKON | i think it's dummer than we all expect | 00:07 |
| *** junkafarian has quit IRC | 00:07 | |
| J1m | I can immagine it being that dumb. | 00:07 |
| J1m | philiKON, did you try adding the necessary find link on the command line? | 00:08 |
| philiKON | lemme try again | 00:08 |
| philiKON | that's -f right? | 00:08 |
| J1m | maaaaybe :) | 00:08 |
| J1m | yup | 00:09 |
| TresEquis | first, who released an egg which depends on a zope.traversing version which was not even properly released? | 00:09 |
| TresEquis | the .dev-r#### eggs must *not* be released into the wild | 00:09 |
| philiKON | well, this one wasn't | 00:09 |
| philiKON | and becuase of that it's causing problems | 00:09 |
| TresEquis | *somebody* uploaded zope.traversing-3.5dev-r78730 somewhere | 00:10 |
| TresEquis | or Chris' script wouldn't have had a conflict | 00:10 |
| *** gidas has quit IRC | 00:11 | |
| TresEquis | and *somebody* released something requiring zope.traversing>=3.5.0a1 without ensuring that there was a properly-released egg satisfying that requirement | 00:11 |
| philiKON | apparently setup.py install doesn't take -f or --find-links | 00:11 |
| philiKON | TresEquis: yup, i agree, that was a mistake | 00:11 |
| TresEquis | this stuff infected grok, too | 00:11 |
| philiKON | i know. :( | 00:11 |
| J1m | philiKON, hm. That's annoying. | 00:12 |
| srichter | TresEquis: pointing fingers does not help at all; we trying to figure out how to handle the situation | 00:12 |
| TresEquis | honestly, we should yank all the .dev-r###' eggs from download.zope.org, as well as the cheeseshop, and replace with with real releases | 00:12 |
| philiKON | J1m: sure is | 00:12 |
| philiKON | TresEquis: last time i did that with just one egg, people got real mad | 00:12 |
| mcdonc | philiKON: no it doesnt and that sucks, but you can change the setup.py's dependency_links= | 00:12 |
| philiKON | mcdonc: ok, will do | 00:12 |
| TresEquis | and we have to point fingers so that people understand why "releasing" such an egg is a *terrrible* mistake | 00:13 |
| * mcdonc still attempts to figure out the conflict flailingly | 00:13 | |
| TresEquis | if your package needs a feature which is not yet available in a properly-released version of your dependency, YOU CAN'T RELEASE YET | 00:13 |
| J1m | mcdonc, it doesn't sound like a conflict. | 00:13 |
| mcdonc | well it says... Installed distribution zope.traversing 3.4.0 conflicts with requirement zope.traversing>=3.5.0a1.dev-r78730 | 00:14 |
| J1m | It sounds like an unmet requirement that just plays a conflict in TV. | 00:14 |
| mcdonc | ha! | 00:14 |
| philiKON | right | 00:14 |
| TresEquis | something requires the 3.5+ egg | 00:14 |
| TresEquis | which is not available in the universe Chris was pulling from | 00:14 |
| mcdonc | setuptools does a horrible job of telling you where the requirement came from | 00:14 |
| TresEquis | $ find lib/python/site-packages/ -name "requires.txt" | xargs grep "\.dev" | 00:15 |
| J1m | mcdonc, I tried hard to provide good diagnistics in buildout. :) If you run it in verbose mode, it tells you why it looks for each requirement. | 00:15 |
| *** Jell-O-Fishi has joined #zope3-dev | 00:15 | |
| * philiKON installs again with find_links in setup.py | 00:16 | |
| *** afd_ has quit IRC | 00:18 | |
| philiKON | that doesn't seem to help, though | 00:19 |
| J1m | [buildout] | 00:20 |
| J1m | parts = myegg | 00:20 |
| J1m | [myegg] | 00:20 |
| J1m | recipe=zc.recipe.egg | 00:20 |
| J1m | eggs = myegg | 00:20 |
| J1m | if you run that with -v and substituting the package in question for myegg, then you should get some helpful info. | 00:21 |
| mcdonc | lib/python2.4/site-packages//zope.app.publisher-3.5.0a2-py2.4.egg/EGG-INFO/requires.txt:zope.traversing>=3.5.0a1.dev-r78730 | 00:21 |
| J1m | you probably also want: eggs-directory = eggs | 00:21 |
| J1m | in the buildout section if you have a shared eggs for | 00:21 |
| J1m | there ya go | 00:21 |
| TresEquis | $ svn blame $ZSVN/zope.app.publisher/tags/3.5.0a2/setup.py | 00:22 |
| mcdonc | J1m: thanks... | 00:23 |
| TresEquis | says it was 'zagy' who added the faux-released packages | 00:23 |
| mcdonc | while we're at it... | 00:23 |
| mcdonc | lib/python2.4/site-packages//zope.annotation-3.4.0-py2.4.egg/EGG-INFO/requires.txt:zope.location>=3.4.0b1.dev-r78903 | 00:23 |
| TresEquis | His checkin comment says: "Added a `layer` attribute to `xmlrpc:view`. This works just like layers for | 00:24 |
| TresEquis | `browser:view` etc. but uses the `IXMLRPCSkinType`. | 00:24 |
| J1m | mcdonc, that one shouldn't be a problem, as a matching relrase is in the cheeseshop. | 00:24 |
| J1m | The easy "fix" for zope.app.publisher is to add download.zope.org to the dependeoncy_links. | 00:25 |
| J1m | and make a new release. | 00:25 |
| TresEquis | the root directory there is completely polluted with faux releases | 00:26 |
| J1m | Dude, this is only an alpha release. | 00:26 |
| srichter | download.zope.org/distribution contains a lot of nasty packages; making it part of your dep links has bad side effects | 00:26 |
| J1m | This is an alpha release of zope.app.publisher. | 00:27 |
| mcdonc | well, note that this was from an install of "zope.tales" | 00:27 |
| mcdonc | which is its own insanity | 00:27 |
| mcdonc | but... | 00:27 |
| J1m | alternatively, someone could copy the needed zope.traversing link to the cheeseshop. | 00:27 |
| J1m | alternatively, someone could copy the needed zope.traversing release to the cheeseshop. | 00:27 |
| TresEquis | We need to tell people to quit depending on being able to download faux releases | 00:28 |
| TresEquis | if they need such a thing, they should maintain their own link server | 00:28 |
| TresEquis | and then we need to wipe every one of them from the "public" locations | 00:29 |
| TresEquis | releasing stuff prematurely is going to kill our story | 00:29 |
| J1m | except that removing links has its own problems. | 00:30 |
| J1m | I think we all agree, | 00:30 |
| TresEquis | We give people warning | 00:30 |
| TresEquis | and then remvoe the stuff | 00:30 |
| J1m | OTOH, I think there is a better test than what mcdonc did. | 00:30 |
| J1m | I suggest only non-final releases that are actually in the cheeseshop should be tested. | 00:30 |
| TresEquis | Why 'non-final'? | 00:31 |
| J1m | We shouldn't worry that much about whether a releases alpha works, much less an unreleased subversion trunk. | 00:31 |
| J1m | sorry | 00:31 |
| J1m | I meant final. | 00:31 |
| J1m | :) | 00:31 |
| mcdonc | J1m: i offered to do that in my mail | 00:31 |
| J1m | OK, I accept. :) | 00:31 |
| mcdonc | I need to figure out how to know what "the final releases" are | 00:32 |
| J1m | That should be pretty easy. | 00:32 |
| TresEquis | we could do a search using trove classifiers | 00:32 |
| J1m | You can figure this put by looking at the versions. | 00:32 |
| J1m | This is a faaaairly straighforward pkg_resources script to write. | 00:33 |
| mcdonc | note though that it doesnt really matter for the "traversing" dependency error... easy_install zope.pagetemplate, you'll get the same error. | 00:33 |
| J1m | Especially in comibination with the pypi xml-rpc interface. | 00:33 |
| philiKON | alas, the trove classifiers aren't set automatically based on the a,b,c suffix | 00:33 |
| J1m | I wouldn't use the trovr classifiers at all. | 00:33 |
| TresEquis | we have to re-release packages which have already "escaped" | 00:33 |
| *** srichter has quit IRC | 00:34 | |
| J1m | TresEquis, yes | 00:34 |
| *** srichter has joined #zope3-dev | 00:34 | |
| *** pbugni has joined #zope3-dev | 00:34 | |
| mcdonc | i didnt know there was a pypi xmlrpc interface | 00:34 |
| TresEquis | and fix packages which have goofy requirements | 00:34 |
| J1m | Yes | 00:34 |
| *** ChanServ sets mode: +o srichter | 00:34 | |
| mcdonc | k, well, glad thats solved ;-) | 00:35 |
| J1m | http://wiki.python.org/moin/CheeseShopXmlRpc | 00:35 |
| mcdonc | very cool | 00:35 |
| J1m | Of course, it's made trickier by the fact that there's no prefer-final option for easy install. | 00:35 |
| J1m | so even if you easy_install a final release, it may pick up non-final releases of it's dependencies. | 00:36 |
| * TresEquis is planning to hack in a multi-index feature | 00:36 | |
| J1m | I'm rather worried about how screwed up the setuptools development process is. | 00:36 |
| *** RaFromBRC has quit IRC | 00:37 | |
| J1m | In fact, the lack of a prefer-final option in easy_install is a pretty good argument for *never* uploading non-final releases to the cheeseshop. | 00:37 |
| *** RaFromBRC has joined #zope3-dev | 00:38 | |
| J1m | I wonder if it would make sense to create a mirror that filters out non-final releases. | 00:38 |
| J1m | Of course, it would folter out setuptools and buildout. | 00:39 |
| J1m | filter | 00:39 |
| srichter | well, KGS can do this, a bit manually | 00:41 |
| J1m | What we really need is a prefer-final option in easy_install. | 00:42 |
| J1m | ow we need to clean up the cheeseshop. :) | 00:42 |
| J1m | or | 00:42 |
| philiKON | +1 to clean up | 00:42 |
| TresEquis | +1 | 00:42 |
| *** strichter has joined #zope3-dev | 00:45 | |
| mcdonc | +1 on cleaning up the cheeseshop or not using it | 00:46 |
| ccomb | it's the same kind of problematics than the debian repositories | 00:47 |
| J1m | well, for what you are trying to do, I don't think not using it is an option. | 00:47 |
| *** alecm has quit IRC | 00:49 | |
| mcdonc | we aren't currently using it for our egg releases, although it's likely we will eventually... but its not useful as an index from which to retrieve a known good set (obviously).. just as a place to poke around and see what's new | 00:49 |
| TresEquis | I think that having some kind of "egglint" thing we use to check packaging sanity before release would be good | 00:50 |
| TresEquis | maybe it just runs 'setup.py egg_info' and then grubbing in the '.egg-info' directory might be enough | 00:52 |
| *** benji has quit IRC | 00:52 | |
| TresEquis | it should be possible to do 'setup.py develop' on a checkout inside an empty virtualenv, using only the metadata in the setup.py | 00:53 |
| TresEquis | if that doesn't work, then the package isn't ready for the cheeseshop | 00:53 |
| TresEquis | or even for 'download.zope.org' | 00:53 |
| TresEquis | srichter: why did you mirror the whole cheeseshop into the KGS for Zope 3.4? | 00:54 |
| TresEquis | No way is all that stuff "known good" | 00:54 |
| *** jsadjohnson has quit IRC | 00:54 | |
| *** dobee has quit IRC | 00:54 | |
| *** spython has quit IRC | 00:56 | |
| *** srichter has quit IRC | 00:59 | |
| *** re|potatopancake has left #zope3-dev | 01:02 | |
| *** hazmat has quit IRC | 01:03 | |
| *** MJ has quit IRC | 01:06 | |
| strichter | TresEquis: No, only the controlled packages are known good | 01:08 |
| strichter | TresEquis: setuptools only supports one index | 01:08 |
| TresEquis | but if I use the KGS, I don't want to fall back to cheeseshop packages | 01:10 |
| TresEquis | I want to fail, and let the user override the index to pick the cheeseshop | 01:10 |
| TresEquis | otherwise the KGS appears to contain a random bunch of unreviewed packages | 01:10 |
| strichter | huh? | 01:14 |
| strichter | the KGS is controlled-packages.cfg | 01:14 |
| TresEquis | If I look at http://download.zope.org/zope3.4 | 01:15 |
| strichter | download.zoep.org/zope3.4 is a *package index* generated from it | 01:15 |
| TresEquis | I see wild west, not known good | 01:15 |
| strichter | well, you can have only *one* index | 01:15 |
| TresEquis | I don't understand why packages which are not in the KGS are in the index | 01:15 |
| TresEquis | right | 01:15 |
| TresEquis | but I don't *want* wild west packges included in the index which is supposed to represent "known good" stuff | 01:16 |
| strichter | then look at http://download.zope.org/zope3.4/controlled-packages.cfg | 01:16 |
| TresEquis | that isn't an index | 01:16 |
| strichter | ok, I give up | 01:16 |
| TresEquis | I want an index which contains *only* packages which are in the KGS | 01:16 |
| strichter | but you can't! | 01:16 |
| TresEquis | if I want to use the cheeseshop, I won't point at the KGS-derived indeex | 01:16 |
| TresEquis | I'll point at the cheeseshop | 01:16 |
| TresEquis | yes you can | 01:17 |
| TresEquis | just not in the same run | 01:17 |
| TresEquis | $ bin/easy_install --index-url=http://download.zope.org/3.4 zope.interface | 01:17 |
| strichter | well, that's not a use case I ahve | 01:17 |
| strichter | I just want it to work in buildout | 01:18 |
| strichter | you can use the controlled-packages.cfg file to generate a different output | 01:18 |
| TresEquis | If your index contains packages which you haven't reviewed, then you have unintended consequences | 01:18 |
| TresEquis | people who rely on the "known goodness" of your index may get burned | 01:18 |
| strichter | look, the usage has been well explained and documented on the list several time and in the release notes | 01:19 |
| TresEquis | it should be possible to install *any* egg from the "known good" set without going to the cheeseshop | 01:19 |
| TresEquis | right now, if somebody gets an egg into the KGS which has dependencies which can only be satisfied from the cheeseshop mirror, that would be a Bad Thing, no? | 01:20 |
| TresEquis | so how can you tell that such a Bad Thing occurs, if you mirror the cheeseshop into the index? | 01:21 |
| TresEquis | if people want to write buildouts which install eggs from other indexes (like the cheeseshop), then they should override the index_url for those eggs (assuming that the default is the KGS index) | 01:21 |
| strichter | no, that's not a bad thing | 01:27 |
| strichter | my project might depend on package foo | 01:27 |
| strichter | but our KGS does not control foo | 01:28 |
| *** lucielejard has quit IRC | 01:30 | |
| TresEquis | so your project should not be using the KGS-derived index when trying to fetch foo | 01:31 |
| TresEquis | it should use the cheeseshop, or some other index which knows about foo | 01:31 |
| strichter | I totally disagree | 01:33 |
| TresEquis | why do you imagine that an index needs to know about every package in the world? | 01:35 |
| TresEquis | that is insane | 01:35 |
| TresEquis | it should know only about packages^wprojects whose metadata it tracks (and, for sanity, whose distributions it controls) | 01:36 |
| strichter | because we can only have one | 01:36 |
| TresEquis | not so | 01:36 |
| strichter | that's not my limitation | 01:36 |
| TresEquis | one-at-a-time | 01:36 |
| TresEquis | but it can be reconfigured | 01:36 |
| TresEquis | if a package needs to specify the KGS index, but also depend on finding packages not in that index, it can add find-links | 01:37 |
| *** harobed has quit IRC | 01:38 | |
| J1m | TresEquis, the KGS that strichter and I devised is a mirror of the cheesehop except that it controls some packages. | 01:39 |
| J1m | That satisfies a reasonable set of use cases. | 01:39 |
| J1m | One could certianly create a KGS that excluded anything it didn't control. | 01:40 |
| J1m | That latter would be inconvenient for lots of packages. | 01:40 |
| TresEquis | package quality will suffer if people don't use a restricted index by default | 01:40 |
| TresEquis | projects which want a broader set should be able to "federate" indexes, perhaps | 01:41 |
| J1m | Unfortunately, find-links don't get you very far because a valid index isn't necessarily a valid source of find links. | 01:41 |
| J1m | Perhaps. | 01:41 |
| J1m | Note that dfor development, you could live with some instability. | 01:42 |
| TresEquis | if we had a "kgs" index, and then another which federated it with the cheeseshop, I would use the first one exclusively | 01:42 |
| J1m | Note that for development, you could live with some instability. | 01:42 |
| TresEquis | Sure | 01:42 |
| J1m | I/we could set such a thing up. | 01:42 |
| TresEquis | but I'd want to choose it explicitly | 01:42 |
| TresEquis | rather than getting it foisted on me by a packaging error | 01:43 |
| J1m | It would go a loooong way if easy_install grew a perfer-final option. | 01:43 |
| TresEquis | when *doing* packaging, the restricted index should be the "testbed" | 01:43 |
| mcdonc | srichter: maybe you didnt mean this, but if you did, i dont understand why you'd want to upload releases to the cheeseshop if they cant be installed via easy_install... if they're only useful for buildout-based installs, shouldnt they just go into a private repo for people to not trip over when they go try one out with setuptools stuff? | 01:43 |
| TresEquis | downstream, folks who want to mix the KGS packages with cheeseshop should have easy ways to do that | 01:44 |
| TresEquis | but the folks who are maintaining the zope.* and related packages should test *without* the cheeseshop | 01:44 |
| mcdonc | strichter: that's in response to "I just want it to work in buildout", BTW. | 01:45 |
| *** dunny has quit IRC | 01:46 | |
| *** philiKON_ has joined #zope3-dev | 01:47 | |
| * strichter points out that he did not sleep last night and is a bit cranky, unconcentrated and innprecise today | 01:48 | |
| *** dunny has joined #zope3-dev | 01:48 | |
| strichter | mcdonc: I think I misundestood your question a bit before... | 01:48 |
| TresEquis | J1m: I don't think prefer-final is enough | 01:49 |
| TresEquis | I think that we need to think of the constraint solving problem as: | 01:50 |
| strichter | TresEquis: it is very trivial to create a restricted index from the cotnrolled-packages.cfg file | 01:50 |
| TresEquis | - packages express minimal constraints (just names where possible, '>=' where absolutely necessary) | 01:50 |
| strichter | zope.release has already several things that can help | 01:50 |
| TresEquis | - indexes track a set of candidates which can be used to satisfy those requirements, plus those supplied by the client (buildout, easy install) | 01:51 |
| strichter | also note that controlled-packages.cfg supports an "extends" feature like buildout, so you can extend the controlled set based on another one (or more) | 01:51 |
| *** pcardune has joined #zope3-dev | 01:51 | |
| TresEquis | - the clients should be flexible enough to handle multiple indexes, and to impose additional constratints | 01:51 |
| TresEquis | - indexes should be able to expose their dependency maps so that clients can do resolution without downloading | 01:52 |
| * J1m is stll wondering why prefer-final isn't enough | 01:52 | |
| strichter | yeah, but that requires a lot of new software accross several communities; it's a good long term goal, but I won't hold my breath | 01:52 |
| TresEquis | If the indexes are narrow enough, you don't actually need much in the way of software | 01:53 |
| strichter | ok, anyways; I am out of here; I am too much of a zombie | 01:53 |
| TresEquis | the model here is like Debian / Ubuntu | 01:54 |
| *** philiKON has quit IRC | 01:54 | |
| TresEquis | feisty has main, restricted, universe, etc | 01:54 |
| TresEquis | those are frozen when feisty was released | 01:54 |
| TresEquis | then feisty-security has the same sets | 01:54 |
| TresEquis | but gets modified as new security fixes are released | 01:55 |
| TresEquis | feisty-updates has non-security bugfixes | 01:55 |
| TresEquis | fesity-backports has "feature" releases which have had some testing against the feisty baseline | 01:55 |
| TresEquis | the apt client knows how to solve against the packages it finds from each index | 01:56 |
| TresEquis | but it doesn't need to filter against the "unstable" stuff (whatever adjective the heron has now) | 01:56 |
| TresEquis | because those packages aren't included in the index set | 01:56 |
| J1m | I suppose that if the indexes were flat, you could use find-links and if they were all clean, you'd have what you want. | 01:57 |
| J1m | Personally, I'd be happy to see a variation on the existing KGS that only included known packages. | 01:57 |
| J1m | I expect that it would be a small change to the existing setup. | 01:58 |
| *** nathany has quit IRC | 02:01 | |
| *** pcardune has quit IRC | 02:01 | |
| *** elro has quit IRC | 02:02 | |
| *** J1m has quit IRC | 02:02 | |
| *** TresEquis has quit IRC | 02:13 | |
| *** gstratton_ has quit IRC | 02:14 | |
| *** pyqwer has quit IRC | 02:22 | |
| *** b52lap has quit IRC | 02:35 | |
| *** pbugni has quit IRC | 02:38 | |
| *** niemeyer has quit IRC | 02:39 | |
| *** rmarianski has quit IRC | 02:41 | |
| *** hazmat has joined #zope3-dev | 02:57 | |
| *** Ariel_Calzada has quit IRC | 02:58 | |
| *** alecm has joined #zope3-dev | 03:01 | |
| *** timte has quit IRC | 03:10 | |
| *** RaFromBRC has quit IRC | 03:21 | |
| *** rmarianski has joined #zope3-dev | 03:35 | |
| *** rmarianski has quit IRC | 03:41 | |
| *** Ariel_Calzada has joined #zope3-dev | 03:59 | |
| *** dabu has quit IRC | 04:05 | |
| *** rmarianski has joined #zope3-dev | 04:11 | |
| *** greenman has quit IRC | 04:16 | |
| *** ktwilight_ has joined #zope3-dev | 04:19 | |
| *** stub has quit IRC | 04:25 | |
| *** aaltepet has joined #zope3-dev | 04:27 | |
| aaltepet | greets. I'm looking for a way to get all interfaces an object implements. | 04:28 |
| aaltepet | I need both direct and indirect (inherited) interfaces. | 04:29 |
| aaltepet | I know instancedOfObjectImplements will get me the direct ones | 04:29 |
| aaltepet | But what api call will give me the others? | 04:29 |
| *** ktwilight has quit IRC | 04:32 | |
| *** RaFromBRC has joined #zope3-dev | 04:34 | |
| *** rcrafton has quit IRC | 04:53 | |
| *** pcardune has joined #zope3-dev | 04:57 | |
| *** RaFromBRC has quit IRC | 05:00 | |
| *** Ariel_Calzada has quit IRC | 05:07 | |
| *** hazmat has quit IRC | 05:07 | |
| *** hazmat has joined #zope3-dev | 05:09 | |
| *** Ariel_Calzada has joined #zope3-dev | 05:26 | |
| *** aaltepet has quit IRC | 05:34 | |
| *** rmarianski has quit IRC | 05:41 | |
| *** Ariel_Calzada has quit IRC | 05:44 | |
| *** lucielejard has joined #zope3-dev | 06:14 | |
| *** mcdonc has quit IRC | 06:43 | |
| *** mcdonc has joined #zope3-dev | 06:44 | |
| *** cursor has joined #zope3-dev | 06:53 | |
| *** reco has quit IRC | 07:04 | |
| *** lucielejard has quit IRC | 07:16 | |
| *** TresEquis has joined #zope3-dev | 07:23 | |
| *** alecm has quit IRC | 07:32 | |
| *** TresEquis has quit IRC | 07:45 | |
| *** afd_ has joined #zope3-dev | 08:20 | |
| *** hdima has joined #zope3-dev | 08:36 | |
| *** cursor has quit IRC | 08:42 | |
| *** __mac__ has joined #zope3-dev | 08:52 | |
| *** jodok has joined #zope3-dev | 08:53 | |
| *** __mac__ has quit IRC | 08:54 | |
| *** jodok has quit IRC | 08:54 | |
| *** zagy has joined #zope3-dev | 08:55 | |
| *** jodok has joined #zope3-dev | 08:56 | |
| *** __mac__ has joined #zope3-dev | 08:56 | |
| *** sorin has joined #zope3-dev | 09:15 | |
| *** sorin is now known as sorindregan | 09:16 | |
| *** projekt01 has joined #zope3-dev | 09:18 | |
| *** schwendinger has joined #zope3-dev | 09:22 | |
| *** afd__ has joined #zope3-dev | 09:28 | |
| *** dobee has joined #zope3-dev | 09:30 | |
| *** stub has joined #zope3-dev | 09:33 | |
| *** afd_ has quit IRC | 09:41 | |
| *** gstratton has joined #zope3-dev | 09:45 | |
| *** cursor has joined #zope3-dev | 09:47 | |
| *** goschtl has joined #zope3-dev | 09:54 | |
| *** xenru has quit IRC | 09:56 | |
| *** quodt has joined #zope3-dev | 09:59 | |
| *** timte has joined #zope3-dev | 10:12 | |
| *** pyqwer has joined #zope3-dev | 10:15 | |
| *** pelle_ has joined #zope3-dev | 10:17 | |
| *** b52laptop has joined #zope3-dev | 10:24 | |
| *** pcardune has quit IRC | 10:33 | |
| *** regebro has joined #zope3-dev | 10:33 | |
| *** elro has joined #zope3-dev | 10:39 | |
| *** elro has quit IRC | 10:40 | |
| *** _srichter has joined #zope3-dev | 10:44 | |
| *** strichter has quit IRC | 11:00 | |
| *** harobed has joined #zope3-dev | 11:05 | |
| *** elro has joined #zope3-dev | 11:08 | |
| *** dunny_ has joined #zope3-dev | 11:08 | |
| *** elro has left #zope3-dev | 11:11 | |
| *** dunny has quit IRC | 11:19 | |
| *** cursor_ has joined #zope3-dev | 11:28 | |
| *** salfield has quit IRC | 11:31 | |
| *** dunny has joined #zope3-dev | 11:36 | |
| *** cursor has quit IRC | 11:36 | |
| *** MJ has joined #zope3-dev | 11:41 | |
| *** thruflo has joined #zope3-dev | 11:41 | |
| *** dunny_ has quit IRC | 11:49 | |
| *** Ariel_Calzada has joined #zope3-dev | 12:08 | |
| *** dunny has quit IRC | 12:11 | |
| *** dunny has joined #zope3-dev | 12:18 | |
| *** huajie has joined #zope3-dev | 12:22 | |
| *** thruflo has quit IRC | 12:31 | |
| *** philiKON_ is now known as philiKON | 12:34 | |
| *** gidas has joined #zope3-dev | 12:39 | |
| *** niemeyer has joined #zope3-dev | 12:44 | |
| *** dunny has quit IRC | 12:46 | |
| *** schwendinger has quit IRC | 12:53 | |
| *** cursor has joined #zope3-dev | 12:55 | |
| *** schwendinger has joined #zope3-dev | 12:58 | |
| *** _srichter is now known as srichter | 12:59 | |
| *** ChanServ sets mode: +o srichter | 12:59 | |
| *** mgedmin has joined #zope3-dev | 13:00 | |
| *** cursor_ has quit IRC | 13:03 | |
| *** pelle_ has quit IRC | 13:23 | |
| *** ignas has joined #zope3-dev | 13:27 | |
| *** malthe has joined #zope3-dev | 13:32 | |
| *** pelle_ has joined #zope3-dev | 13:32 | |
| *** malthe has quit IRC | 13:33 | |
| *** salfield has joined #zope3-dev | 13:34 | |
| *** baijum has joined #zope3-dev | 13:36 | |
| *** yvl has joined #zope3-dev | 13:38 | |
| *** jsadjohnson has joined #zope3-dev | 13:40 | |
| *** malthe has joined #zope3-dev | 13:41 | |
| *** junkafarian has joined #zope3-dev | 13:49 | |
| *** afd__ has quit IRC | 13:52 | |
| *** MJ has quit IRC | 13:53 | |
| *** Ariel_Calzada has quit IRC | 14:05 | |
| *** __mac__ has quit IRC | 14:07 | |
| *** fcorrea has joined #zope3-dev | 14:14 | |
| *** mgedmin_ has joined #zope3-dev | 14:15 | |
| *** mgedmin has quit IRC | 14:16 | |
| *** hazmat has quit IRC | 14:22 | |
| *** rcrafton has joined #zope3-dev | 14:28 | |
| *** thruflo has joined #zope3-dev | 14:29 | |
| *** Ariel_Calzada has joined #zope3-dev | 14:31 | |
| *** afd_ has joined #zope3-dev | 14:37 | |
| *** grahal has joined #zope3-dev | 14:42 | |
| *** ghendi has joined #zope3-dev | 14:50 | |
| *** afd_ has quit IRC | 14:56 | |
| *** afd_ has joined #zope3-dev | 14:56 | |
| *** Ariel_Calzada has quit IRC | 15:04 | |
| *** pelle_ has quit IRC | 15:06 | |
| *** pan_ has joined #zope3-dev | 15:11 | |
| *** ktwilight_ is now known as ktwilight | 15:11 | |
| *** pelle_ has joined #zope3-dev | 15:13 | |
| *** ccomb has quit IRC | 15:17 | |
| *** benji has joined #zope3-dev | 15:18 | |
| *** projekt01 has quit IRC | 15:19 | |
| *** ccomb has joined #zope3-dev | 15:24 | |
| *** jfroche has joined #zope3-dev | 15:25 | |
| *** pan has quit IRC | 15:28 | |
| *** MJ has joined #zope3-dev | 15:35 | |
| *** huajie has quit IRC | 15:35 | |
| *** projekt01 has joined #zope3-dev | 15:36 | |
| gstratton | If I have a field which should be a choice of contained objects, what's the best way of implementing this? | 15:40 |
| *** rcrafton has quit IRC | 15:42 | |
| goschtl | gstratton: you can have a look at this https://svn.plone.org/svn/plone/plone.app.controlpanel/trunk/plone/app/controlpanel/filter.py. Ok this is plone but it use the formlib and should give you an idea what to do... | 15:42 |
| *** yvl has quit IRC | 15:45 | |
| *** alga has joined #zope3-dev | 15:56 | |
| *** hazmat has joined #zope3-dev | 15:58 | |
| gstratton | I don't think that example really fits my usecase. I guess I basically want a TextLine field, but with a select widget which is populated with a dynamically generated list. | 15:58 |
| *** lucielejard has joined #zope3-dev | 16:08 | |
| *** lucielejard has quit IRC | 16:08 | |
| *** cursor has quit IRC | 16:10 | |
| ccomb | gstratton you can use a vocabulary | 16:11 |
| gstratton | ccomb: Yup, just got there, thanks! | 16:11 |
| *** __mac__ has joined #zope3-dev | 16:21 | |
| dennis__ | ccomb ? | 16:23 |
| dennis__ | your mail to the list: if you search for value in root['btree'].values() this is just a sequential full-tree scan? are you sure you don't want to do something like value in root['btree'] ? | 16:23 |
| dennis__ | if you need to search for values rather than keys (or both), i would set up a reverse index | 16:25 |
| *** lucielejard has joined #zope3-dev | 16:30 | |
| *** __mac__ has quit IRC | 16:56 | |
| *** __mac__ has joined #zope3-dev | 16:58 | |
| *** yvl has joined #zope3-dev | 16:58 | |
| *** mkerrin has joined #zope3-dev | 17:04 | |
| *** hdima has quit IRC | 17:04 | |
| *** sorindregan has quit IRC | 17:06 | |
| *** ghendi has quit IRC | 17:08 | |
| *** thruflo has quit IRC | 17:12 | |
| *** cursor has joined #zope3-dev | 17:16 | |
| *** MJ has quit IRC | 17:29 | |
| *** MJ has joined #zope3-dev | 17:31 | |
| ccomb | dennis__ : I need to loop over a large amount of data to build means or alerts on event detection | 17:32 |
| ccomb | When I change the threshold of the alerts, I need to loop again a rebuild a graph | 17:33 |
| *** yvl has left #zope3-dev | 17:33 | |
| dennis__ | ccomb so you do really need to touch every single object in your btree ? | 17:34 |
| ccomb | yes to get the actual values | 17:34 |
| ccomb | all this in a single request. | 17:35 |
| *** baijum has quit IRC | 17:35 | |
| ccomb | but I don't understand why looping implies to raise everything in the memory | 17:36 |
| dennis__ | i think a btree should do it piece-wise, which if you are going through the whole thing means everything, but it should be able to garbage collect the pieces you have visited (depends a bit on the distribution of keys over the buckets i guess) | 17:37 |
| dennis__ | how do you measure the memory use actually? | 17:39 |
| ccomb | I'm looking at a dynamic chart or at 'top' | 17:39 |
| *** rmarianski has joined #zope3-dev | 17:40 | |
| dennis__ | what happens if you import gc, and every few hundred objects call gc.collect() ? | 17:41 |
| ccomb | There is something really weird, because now I'm just doing root.keys(), and I see the memory eating 400MB at once | 17:42 |
| ccomb | it should just list the keys of the root Folder | 17:42 |
| dennis__ | how much is in there? | 17:42 |
| ccomb | >>> list(root.keys()) => [u'b', u'epidemio', u'l'] | 17:43 |
| ccomb | 3 items | 17:43 |
| ccomb | b is the btree, l is the list | 17:43 |
| ccomb | (they are just for test, the real data is in a subfolder of u'epidemio') | 17:45 |
| *** dobee has quit IRC | 17:46 | |
| ccomb | I removed the list 'l', and now root.keys() takes nothing | 17:47 |
| dennis__ | well the list needs to be loaded completely before being able to jump to any values. the btree is much better and loads on demand (so all this is only still about the actual references to the objects being contained) | 17:49 |
| ccomb | but why is the list completely loaded when I just get the keys of the container? | 17:50 |
| *** jfroche has quit IRC | 17:50 | |
| dennis__ | i don't know why it wants to wake up the list object | 17:51 |
| dennis__ | try: it = iter( root['b'].values() ) | 17:52 |
| dennis__ | for i in it: pass | 17:52 |
| ccomb | I've already tried this : for i in root['b'].itervalues() : pass | 17:53 |
| ccomb | it eats | 17:53 |
| ccomb | trying with iter( values() ) | 17:53 |
| *** whitmo has joined #zope3-dev | 17:53 | |
| dennis__ | ah sorry | 17:54 |
| *** whitmo_ has joined #zope3-dev | 17:55 | |
| *** nathany has joined #zope3-dev | 17:57 | |
| *** nathany has quit IRC | 18:00 | |
| *** nathany has joined #zope3-dev | 18:00 | |
| *** whit has quit IRC | 18:02 | |
| *** whitmo_ is now known as whit | 18:02 | |
| *** goschtl has quit IRC | 18:07 | |
| ccomb | Working with slice works in the following manner : | 18:08 |
| lisppaste6 | ccomb pasted "loop with slice" at http://paste.lisp.org/display/50823 | 18:08 |
| ccomb | this way it eats only the memory for 1000 tuples. But it's slower | 18:08 |
| ccomb | I'm sure there is some tweak I could use with transactions | 18:09 |
| dennis__ | ugly solution that :) | 18:11 |
| ccomb | ugly is better than lobotomized :) | 18:11 |
| *** whitmo has quit IRC | 18:11 | |
| ccomb | (that's one of my favourites) | 18:11 |
| dennis__ | it's a very positive thought | 18:12 |
| *** natea_ is now known as natea | 18:17 | |
| *** hazmat has quit IRC | 18:18 | |
| *** hazmat has joined #zope3-dev | 18:19 | |
| *** ChanServ sets mode: +o hazmat | 18:20 | |
| *** ktwilight has quit IRC | 18:23 | |
| *** ktwilight has joined #zope3-dev | 18:24 | |
| mcdonc | ccomb: i think you may want something like btree._p_jar.cacheMinimize() | 18:34 |
| mcdonc | (or actually anypersistentobject._p_jar.cacheMinimize()) | 18:35 |
| ccomb | mcdonc: that looks interesting | 18:36 |
| *** jodok has quit IRC | 18:37 | |
| mcdonc | ccomb: you can "deactivate" a single persistent object using anypersistentobject._p_deactivate() | 18:37 |
| *** jodok has joined #zope3-dev | 18:37 | |
| * ccomb will just try this | 18:38 | |
| mcdonc | ccomb: which, if the object state is loaded into memory, turns in back into a "ghost" object without only "stub" state in memory (it will need to be retrieved from the database or the cache again) | 18:38 |
| *** ktwilight has quit IRC | 18:39 | |
| *** ktwilight has joined #zope3-dev | 18:40 | |
| dennis__ | setting cacheSize doesn't prevent the memory usage from growing, but calling cacheMinimize() every few thousand objects does help | 18:41 |
| dennis__ | ccomb of course what is also a problem is that your objects (tuples) are not themselves persistent | 18:42 |
| *** pyqwer has quit IRC | 18:42 | |
| dennis__ | if you put classes there which derive from Persistent, they can be deactivated | 18:42 |
| *** mgedmin_ is now known as baloon_slayer | 18:42 | |
| ccomb | dennis__ I was thinking about it, that could be the core of the problem | 18:42 |
| ccomb | the funny thing is that I've chosen tuples for memory efficiency? | 18:43 |
| *** gstratton_ has joined #zope3-dev | 18:44 | |
| *** __mac__ has quit IRC | 18:45 | |
| *** alga has quit IRC | 18:49 | |
| dennis__ | ccomb my guess is that because the tuples are not persistent, their size all becomes part of the btree, which counts itself only as relatively few objects in the cache (certainly not 700,000) | 18:50 |
| dennis__ | i am trying that theory but ran out of memory :P | 18:50 |
| *** alecm has joined #zope3-dev | 18:52 | |
| ccomb | ok. I confirm that _p_jar.cacheMinimize() works very well if I run it every N loops | 18:53 |
| dennis__ | ccomb if I use persistent.list instead of tuples everything works exactly right | 18:53 |
| ccomb | a first thanks for this solution | 18:53 |
| dennis__ | without the cacheMinimize hack | 18:54 |
| ccomb | dennis__ I will also try with persistent objects | 18:54 |
| ccomb | then summarize the story on the list | 18:54 |
| *** harobed has quit IRC | 18:54 | |
| dennis__ | with tuples: [('BTrees.OOBTree.OOBTree', 4)] (in the cache after iterating) | 18:54 |
| dennis__ | with persistent lists: [('BTrees.OOBTree.OOBTree', 56),('persistent.list.PersistentList', 100000)] | 18:55 |
| ccomb | how do you query the cache? | 18:55 |
| dennis__ | and all those lists can be deactivated | 18:55 |
| *** bung_vijay_ has joined #zope3-dev | 18:55 | |
| dennis__ | .cacheDetail() | 18:56 |
| dennis__ | please post your findings though after using persistent objects, I'm curious how it works out :) | 18:56 |
| *** gstratton__ has joined #zope3-dev | 18:57 | |
| *** bung_vijay_ is now known as bung_vijay | 18:57 | |
| ccomb | yes I will also mention it, since this is the real correct solution | 18:57 |
| ccomb | persistency need some care :) | 18:57 |
| dennis__ | it all makes sense though, and then it works exactly as you'd expect | 18:57 |
| ccomb | I'm glad this was not a dead-end | 18:58 |
| dennis__ | learned something again, thanks for bringing it up :) | 18:58 |
| ccomb | thanks to you and mcdonc too | 18:58 |
| mcdonc | ccomb: no problem ;-) i'm sort of curious why putting nonpersistent objects in the btree would cause them all to be loaded in memory at once while iterating over it. though... | 19:00 |
| *** gstratton has quit IRC | 19:00 | |
| ccomb | is it worth reporting a bug at least to investigate? | 19:01 |
| mcdonc | if tim peters were here, i'm sure he'd tell us in detail, absolutely correctly, and make us laugh while he was doing it ;-) | 19:01 |
| ccomb | :) | 19:01 |
| mcdonc | ccomb: i think putting up a bug in the zope collector (with the topic "zodb" selected from the topic dropdown) might not be a bad idea, as long as you attach a repeatable test script | 19:02 |
| *** gstratton has joined #zope3-dev | 19:03 | |
| *** MJ has quit IRC | 19:03 | |
| philiKON | there's a zodb collector | 19:04 |
| philiKON | please file it there | 19:04 |
| philiKON | it's also on launchpad | 19:04 |
| ccomb | ok | 19:04 |
| mcdonc | philiKON: https://bugs.launchpad.net/zodb/ .. had no idea | 19:06 |
| philiKON | :) | 19:06 |
| philiKON | theuni did some real cleanup last weekend | 19:07 |
| Theuni | mcdonc: *g | 19:09 |
| mcdonc | Theuni, philiKON: please stop breaking the curve ;-) | 19:09 |
| *** malthe is now known as malthe|away | 19:11 | |
| *** gstratton_ has quit IRC | 19:12 | |
| *** jfroche has joined #zope3-dev | 19:12 | |
| *** jfroche_ has joined #zope3-dev | 19:15 | |
| *** gstratton__ has quit IRC | 19:18 | |
| *** bung_vijay has quit IRC | 19:19 | |
| *** bung_vijay has joined #zope3-dev | 19:20 | |
| *** schwendinger has quit IRC | 19:26 | |
| *** jodok has quit IRC | 19:33 | |
| *** quodt has quit IRC | 19:35 | |
| *** projekt01 has quit IRC | 19:43 | |
| *** bung_vijay has quit IRC | 19:49 | |
| *** bung_vijay has joined #zope3-dev | 19:49 | |
| *** Ariel_Calzada has joined #zope3-dev | 19:50 | |
| *** pbugni has joined #zope3-dev | 20:14 | |
| *** quodt has joined #zope3-dev | 20:31 | |
| *** jfroche has quit IRC | 20:33 | |
| *** RaFromBRC has joined #zope3-dev | 20:41 | |
| *** baloon_slayer is now known as balloon_slayer | 20:42 | |
| *** thruflo has joined #zope3-dev | 20:43 | |
| *** pcardune has joined #zope3-dev | 20:51 | |
| *** balloon_slayer has quit IRC | 20:53 | |
| *** schwendinger has joined #zope3-dev | 20:54 | |
| *** ignas has quit IRC | 20:56 | |
| *** schwendinger_ has joined #zope3-dev | 20:59 | |
| *** rcrafton has joined #zope3-dev | 21:00 | |
| *** jodok has joined #zope3-dev | 21:05 | |
| *** mkerrin has quit IRC | 21:10 | |
| *** thruflo has quit IRC | 21:10 | |
| *** schwendinger has quit IRC | 21:12 | |
| *** schwendinger has joined #zope3-dev | 21:13 | |
| *** projekt01 has joined #zope3-dev | 21:23 | |
| *** pcardune has quit IRC | 21:23 | |
| *** schwendinger has quit IRC | 21:24 | |
| *** gidas has quit IRC | 21:27 | |
| *** schwendinger_ has quit IRC | 21:31 | |
| *** pelle_ has quit IRC | 21:33 | |
| *** grahal has quit IRC | 21:38 | |
| *** pcardune has joined #zope3-dev | 21:40 | |
| *** markusleist has quit IRC | 21:41 | |
| *** Ariel_Calzada has quit IRC | 21:44 | |
| *** Ariel_Calzada has joined #zope3-dev | 21:46 | |
| *** dunny has joined #zope3-dev | 21:47 | |
| *** projet01_ has joined #zope3-dev | 21:47 | |
| *** projekt01 has quit IRC | 21:51 | |
| *** MJ has joined #zope3-dev | 21:51 | |
| *** MJ has quit IRC | 21:52 | |
| *** MJ has joined #zope3-dev | 21:54 | |
| mcdonc | philiKON, Theuni: any ideas/opinions on my motivation in http://mail.zope.org/pipermail/zope-dev/2007-November/030281.html | 22:13 |
| *** faassen has joined #zope3-dev | 22:14 | |
| *** norro has joined #zope3-dev | 22:15 | |
| *** malthe|away is now known as malthe | 22:15 | |
| *** Ariel_Calzada has quit IRC | 22:16 | |
| *** Ariel_Calzada has joined #zope3-dev | 22:18 | |
| *** Ariel_Calzada has quit IRC | 22:18 | |
| *** projet01_ has quit IRC | 22:25 | |
| *** jodok has quit IRC | 22:25 | |
| *** cursor has quit IRC | 22:29 | |
| *** jodok has joined #zope3-dev | 22:31 | |
| *** xenru has joined #zope3-dev | 22:33 | |
| xenru | Hi! | 22:33 |
| xenru | philiKON: are you here? | 22:34 |
| xenru | philiKON: I need to ask some questions about book | 22:34 |
| *** pcardune has quit IRC | 22:44 | |
| *** whit has quit IRC | 22:48 | |
| *** jodok has quit IRC | 22:57 | |
| *** bung_vijay has quit IRC | 23:06 | |
| *** fcorrea has quit IRC | 23:09 | |
| *** MJ has quit IRC | 23:15 | |
| *** norro has quit IRC | 23:33 | |
| *** whit has joined #zope3-dev | 23:35 | |
| *** jodok has joined #zope3-dev | 23:38 | |
| *** junkafarian_ has joined #zope3-dev | 23:39 | |
| *** junkafarian_ has quit IRC | 23:39 | |
| *** afd_ has quit IRC | 23:45 | |
Generated by irclog2html.py 2.15.1 by Marius Gedminas - find it at mg.pov.lt!