*** b52laptop has joined #zope3-dev | 00:00 | |
*** norro_ has quit IRC | 00:04 | |
*** ktwilight has quit IRC | 00:05 | |
*** d2m has quit IRC | 00:27 | |
*** whitmo has joined #zope3-dev | 00:30 | |
*** whit has quit IRC | 00:33 | |
*** whitmo is now known as whit | 00:33 | |
*** lucielejard has quit IRC | 00:33 | |
malthe | what's required to run z3 on python 2.5? | 00:35 |
---|---|---|
*** J1m has quit IRC | 00:39 | |
whit | is there a buildout equivalent of "dependency_links"? | 00:52 |
* whit realize the answer to his own question | 00:52 | |
*** reco has quit IRC | 00:53 | |
*** greenman has joined #zope3-dev | 00:59 | |
*** benji has quit IRC | 01:00 | |
*** norro has quit IRC | 01:01 | |
*** nathany has quit IRC | 01:10 | |
*** tarek_ has joined #zope3-dev | 01:11 | |
*** tarek has quit IRC | 01:18 | |
*** salfield has quit IRC | 01:25 | |
*** tarek_ has quit IRC | 01:26 | |
*** tarek_ has joined #zope3-dev | 01:26 | |
*** rmarianski has quit IRC | 01:32 | |
*** greenman has quit IRC | 01:38 | |
*** _tarek_ has joined #zope3-dev | 01:51 | |
*** tarek_ has quit IRC | 02:00 | |
hazmat | malthe, as far as i know, nothing special, just worked for me... | 02:07 |
*** malthe has quit IRC | 02:09 | |
*** sm has quit IRC | 02:15 | |
*** alecm has quit IRC | 02:35 | |
*** bigkevmcd has quit IRC | 02:42 | |
*** hazmat has left #zope3-dev | 02:43 | |
*** rcrafton has quit IRC | 02:45 | |
*** djohnson has joined #zope3-dev | 03:10 | |
*** jsadjohnson has quit IRC | 03:16 | |
*** salfield has joined #zope3-dev | 03:21 | |
*** niemeyer has quit IRC | 03:32 | |
*** djohnson has quit IRC | 03:53 | |
*** salfield has quit IRC | 04:01 | |
*** pcardune has quit IRC | 04:07 | |
*** rcrafton has joined #zope3-dev | 04:33 | |
*** whit has quit IRC | 05:08 | |
*** redir has joined #zope3-dev | 05:43 | |
*** sm has joined #zope3-dev | 06:22 | |
*** sm has quit IRC | 06:51 | |
*** whit has joined #zope3-dev | 06:58 | |
*** rcrafton has quit IRC | 07:02 | |
*** afd_ has joined #zope3-dev | 07:04 | |
*** whit has quit IRC | 07:15 | |
*** timte has quit IRC | 07:33 | |
*** timte has joined #zope3-dev | 08:20 | |
*** timte has quit IRC | 08:21 | |
*** timte has joined #zope3-dev | 08:21 | |
*** jukart has joined #zope3-dev | 08:27 | |
*** menesis has joined #zope3-dev | 08:31 | |
*** d2m has joined #zope3-dev | 08:38 | |
*** afd_ has quit IRC | 08:39 | |
*** jodok has joined #zope3-dev | 08:41 | |
*** pelle_ has quit IRC | 08:53 | |
*** sorin has joined #zope3-dev | 09:05 | |
*** sorin is now known as sorindregan | 09:05 | |
*** stub has joined #zope3-dev | 09:15 | |
*** yvl has joined #zope3-dev | 09:18 | |
*** goschtl has joined #zope3-dev | 09:24 | |
*** agroszer has joined #zope3-dev | 09:34 | |
*** b52laptop has quit IRC | 09:34 | |
*** afd_ has joined #zope3-dev | 09:40 | |
*** quodt has joined #zope3-dev | 09:41 | |
*** flox has joined #zope3-dev | 09:42 | |
*** dobee has quit IRC | 09:44 | |
*** stu1 has joined #zope3-dev | 09:56 | |
*** stub has quit IRC | 09:56 | |
*** stu1 is now known as stub | 09:56 | |
*** dobee has joined #zope3-dev | 09:57 | |
*** toko^ has joined #zope3-dev | 09:57 | |
*** toko has joined #zope3-dev | 09:59 | |
*** jpcw2002 has joined #zope3-dev | 10:03 | |
*** pelle_ has joined #zope3-dev | 10:07 | |
*** markusleist has joined #zope3-dev | 10:22 | |
*** harobed has joined #zope3-dev | 10:33 | |
*** MJ has joined #zope3-dev | 10:34 | |
*** zagy has joined #zope3-dev | 10:38 | |
*** _tarek_ has quit IRC | 10:39 | |
*** tarek has joined #zope3-dev | 10:50 | |
*** alecm has joined #zope3-dev | 10:52 | |
*** mkerrin has joined #zope3-dev | 11:01 | |
*** greenman has joined #zope3-dev | 11:08 | |
*** thruflo has joined #zope3-dev | 11:10 | |
*** norro has joined #zope3-dev | 11:13 | |
*** malthe has joined #zope3-dev | 11:23 | |
*** thruflo has quit IRC | 11:26 | |
*** thruflo has joined #zope3-dev | 11:26 | |
*** b52laptop has joined #zope3-dev | 11:31 | |
*** maurits has joined #zope3-dev | 11:32 | |
*** kursor has joined #zope3-dev | 11:44 | |
*** faassen has joined #zope3-dev | 11:58 | |
*** dunny has quit IRC | 12:24 | |
*** projekt01 has joined #zope3-dev | 12:24 | |
*** pelle_ has quit IRC | 12:33 | |
*** pelle_ has joined #zope3-dev | 12:33 | |
*** salfield has joined #zope3-dev | 12:41 | |
*** pelle_ has quit IRC | 12:49 | |
*** pelle_ has joined #zope3-dev | 12:49 | |
*** yvl has left #zope3-dev | 12:53 | |
*** lisppaste6 has quit IRC | 13:07 | |
*** malthe is now known as malthe|away | 13:18 | |
*** lisppaste6 has joined #zope3-dev | 13:20 | |
*** greenman has quit IRC | 13:21 | |
*** pelle_ has quit IRC | 13:21 | |
*** pelle_ has joined #zope3-dev | 13:22 | |
*** ct_ has joined #zope3-dev | 13:36 | |
*** MJ is now known as MJ|lunch | 13:44 | |
*** malthe|away is now known as malthe | 13:47 | |
*** afd_ has quit IRC | 13:48 | |
*** ignas has joined #zope3-dev | 14:02 | |
*** tesdal has joined #zope3-dev | 14:06 | |
malthe | tesdal: too much noise | 14:06 |
tesdal | what's up d4wgs | 14:07 |
malthe | so collective.indexing is a bit Plone-specific | 14:07 |
*** jpcw2002 has quit IRC | 14:07 | |
tesdal | malthe: a bit I guess | 14:07 |
malthe | somewhat without reason, right? | 14:07 |
tesdal | malthe: we'd love to make it completely independent of Plone | 14:07 |
malthe | right---put the integration stuff in, say, plone.search | 14:07 |
tesdal | search is extraction | 14:08 |
tesdal | indexing is indexing | 14:08 |
tesdal | but yes | 14:08 |
malthe | ok gotcha | 14:08 |
tesdal | it makes sense to use a plone namespace for plone specifics | 14:08 |
malthe | at any rate, the indexing abstractions and implementation should probably be zope-only. | 14:08 |
tesdal | guess that makes sense as well | 14:09 |
malthe | hmm how to proceed | 14:09 |
tesdal | we put it in collective.indexing first because it was useful at this stage | 14:09 |
malthe | right | 14:09 |
tesdal | but it would be great if parts went into Zope itself or a zope namespace, and plone specifics in a plone namespace | 14:10 |
malthe | so on one hand, you have ore.xapian, which is an abstraction *and* an implementation | 14:10 |
tesdal | and we could plip and include in plone 4 | 14:10 |
tesdal | I think ore.xapian is the same, it solves a problem and it made sense to do it like that when it was implemented | 14:10 |
malthe | right | 14:11 |
tesdal | but I think the concepts and abstractions are quite similar for all of the indexing solutions | 14:11 |
malthe | yes | 14:11 |
tesdal | malthe: are you at a sprint? | 14:13 |
malthe | tesdal: yes | 14:13 |
tesdal | malthe: are you working together with someone on indexing? | 14:14 |
*** niemeyer has joined #zope3-dev | 14:14 | |
malthe | tesdal: yes, sylvain from infrae | 14:15 |
tesdal | ok | 14:15 |
*** benji has joined #zope3-dev | 14:15 | |
tesdal | malthe: we'd love some feedback on collective.indexing even if you choose not to use it in any way | 14:16 |
tesdal | malthe: whether the ideas and components are sane, and if you're able to understand what's going on in there etc | 14:16 |
*** goschtl has quit IRC | 14:18 | |
tesdal | malthe: I've got a blog posting draft available, http://www.jarn.com/blog/plone-indexing-performance | 14:18 |
*** toutpt has joined #zope3-dev | 14:22 | |
*** jpcw2002 has joined #zope3-dev | 14:27 | |
*** alecm has quit IRC | 14:28 | |
*** sm has joined #zope3-dev | 14:31 | |
*** djohnson has joined #zope3-dev | 14:32 | |
*** witsch has joined #zope3-dev | 14:40 | |
witsch | hey malthe, tesdal :) | 14:40 |
tesdal | malthe: witsch has done most of the coding on collective.indexing | 14:40 |
witsch | malthe: yeah, i guess after winning our code contest i got to do the rest as well... %) | 14:43 |
*** goschtl has joined #zope3-dev | 14:44 | |
tesdal | :) | 14:47 |
*** djohnson has quit IRC | 14:48 | |
witsch | goschtl: let's discuss things over here, though... malthe and tesdal have already been talking about collective.indexing | 14:50 |
*** mgedmin has joined #zope3-dev | 14:51 | |
goschtl | witsch: ok this channel | 14:56 |
witsch | :) | 14:57 |
tesdal | goschtl: any comments so far? | 14:57 |
*** rcrafton has joined #zope3-dev | 15:00 | |
goschtl | tesdal: no i think i understand what happens. looks very clean to me. | 15:01 |
tesdal | goschtl: excellent :) | 15:02 |
goschtl | what are your plans with other indexes date, path, extendedpath ... | 15:02 |
goschtl | for collective.solr? | 15:02 |
tesdal | goschtl: we don't have any plans for extendedpath in collective.solr | 15:03 |
tesdal | goschtl: but you can replicate a lot of the functionality by /query/by/path/* and /explicity/path/query | 15:04 |
tesdal | goschtl: I think there is a date index already? | 15:04 |
tesdal | it might be possible to make a tokenizer/analyzer in solr that replicates at least pathindex | 15:09 |
*** kursor has quit IRC | 15:09 | |
witsch | tesdal: uh, java? :) | 15:10 |
tesdal | if needed | 15:10 |
tesdal | but it's just about how to split stuff | 15:10 |
tesdal | split in / and add a level indicator or whatever | 15:10 |
* tesdal thinks string and asterisk will work | 15:11 | |
goschtl | tesdal: this means you will have a plonesite with indexes in solr and idexes in zcatalog, no replace of zcatalog? | 15:11 |
tesdal | goschtl: yes | 15:12 |
tesdal | goschtl: but that's not a problem | 15:12 |
*** djohnson has joined #zope3-dev | 15:13 | |
witsch | goschtl: you can always unregister the catalog-index-processor, effectively turning off indexing in zcatalog | 15:15 |
witsch | goschtl: that would be just one call to the ca | 15:15 |
witsch | tesdal: perhaps we're gonna need some ui after all ;) | 15:16 |
tesdal | :) | 15:17 |
tesdal | the button | 15:17 |
*** alecm has joined #zope3-dev | 15:19 | |
*** whit has joined #zope3-dev | 15:19 | |
*** johbo has joined #zope3-dev | 15:19 | |
tesdal | the exotic navtree builder code in EPI is going to be hard to replicate I think | 15:19 |
*** lucielejard has joined #zope3-dev | 15:24 | |
witsch | tesdal: the hard part will be to find somebody who's willing to do this... :) | 15:25 |
tesdal | maybe it can be replicated with a different data structure, or maybe we can build the navigation differently | 15:25 |
witsch | tesdal: but perhaps plone should adopt the new explorer portlet and dump the concept of using the catalog for navigation all together | 15:25 |
tesdal | witsch: something like that | 15:26 |
witsch | tesdal: it looks like plone.folder should actually make it possible to not even need the catalog anymore | 15:26 |
tesdal | sounds good | 15:26 |
witsch | we'll see :) | 15:27 |
*** alga has joined #zope3-dev | 15:29 | |
*** pelle_ has quit IRC | 15:29 | |
*** pelle_ has joined #zope3-dev | 15:29 | |
*** jpcw2002_ has joined #zope3-dev | 15:30 | |
*** J1m has joined #zope3-dev | 15:34 | |
*** chtp_ has joined #zope3-dev | 15:36 | |
*** jpcw2002 has quit IRC | 15:38 | |
*** jpcw2002_ has left #zope3-dev | 15:40 | |
*** jpcw2002 has joined #zope3-dev | 15:42 | |
tesdal | hemul in #lucene suggests storing multiple terms in the path field | 15:43 |
tesdal | like /folder, /folder/with, folder/with/contents | 15:43 |
tesdal | and possibly some kind of marker at the end | 15:43 |
*** MJ|lunch is now known as MJ | 15:44 | |
tesdal | witsch: "i would implement a tokenfilter that split up the aoslte path in parts, and store the absolute term untokenized in another field." | 15:46 |
tesdal | in fact we might be able to replicate EPI with some clever tokenization there... | 15:48 |
*** chtp has joined #zope3-dev | 15:48 | |
goschtl | tesdal: or is it possible to make plone-navigation plone breadcrumbs without epi. :) | 15:49 |
*** ct_ has quit IRC | 15:49 | |
tesdal | goschtl: of course it is | 15:49 |
tesdal | this means we can easily implement regular pathindex | 15:50 |
tesdal | and EPI implementation would be an interesting challenge, although not strictly necessary | 15:50 |
*** pelle_ has quit IRC | 15:50 | |
*** pelle_ has joined #zope3-dev | 15:50 | |
goschtl | iirc regebro has some ideas for a plone-navigation tool based on btrees . | 15:52 |
witsch | tesdal: yep, sounds doable | 15:53 |
tesdal | goschtl: we're interested | 15:53 |
tesdal | the current navigation solved a problem at the time, but is not the perfect way of doing things | 15:53 |
*** jayaraj has joined #zope3-dev | 15:55 | |
*** chtp_ has quit IRC | 15:57 | |
whit | anybody have any recipes for dealing with eggs in mercurial? | 15:57 |
projekt01 | goschtl, there is z3c.jsontree a JSON-RPC based IContainer tree builder | 15:58 |
*** pelle_ has quit IRC | 16:00 | |
*** pelle_ has joined #zope3-dev | 16:00 | |
*** pelle_ has quit IRC | 16:01 | |
*** pelle_ has joined #zope3-dev | 16:01 | |
*** hazmat has joined #zope3-dev | 16:02 | |
*** ChanServ sets mode: +o hazmat | 16:02 | |
whit | is it possible to define dependency links directly in a buildout.cfg? | 16:05 |
*** jayaraj has quit IRC | 16:09 | |
*** witsch has quit IRC | 16:10 | |
*** witsch_ has joined #zope3-dev | 16:10 | |
hazmat | whit, you mean find links? | 16:12 |
hazmat | to look for dependencies | 16:12 |
whit | dependency_links ala setup.py | 16:12 |
whit | like "dependency_links=['http://some.dist.tar.gz#egg=MyEgg-dev']" | 16:13 |
hazmat | whit, yeah.. i use find links with the same spec | 16:14 |
whit | hazmat: hmmm... I'll try that. in setuptools they mean something different I think | 16:14 |
*** witsch_ is now known as witsch | 16:14 | |
goschtl | tesdal: you can see regebros ideas here: http://www.openplans.org/projects/enterprising/project-home. | 16:15 |
hazmat | whit, it still works the way you want i afaics.. ie pass a tarball as a findlink, attach an egg spec and use it to grab the spec'd egg | 16:15 |
whit | cool... | 16:16 |
whit | hey hazmat, do you know if there is a way to run a buildout without being inside a folder with a setup.py? | 16:19 |
* whit is trying to modify an existing buildout and is not sure what is a zc.buildout constraint and what is a constraint of the build he is modifying | 16:19 | |
*** witsch has quit IRC | 16:22 | |
*** djohnson has quit IRC | 16:25 | |
*** stub has quit IRC | 16:25 | |
*** mweichert has joined #zope3-dev | 16:28 | |
*** rmarianski has joined #zope3-dev | 16:35 | |
hazmat | whit, buildout doesn't need a setup.py anywhere | 16:37 |
hazmat | whit, except for development eggs | 16:37 |
hazmat | and you can specify the path to those | 16:37 |
whit | hazmat: would that be "develop = . " maybe? | 16:38 |
whit | ok... this makes more sense now | 16:38 |
malthe | tesdal: I'm not sure what the best strategy is; definitely I strongly believe collective.indexing has been implemented at the wrong layer. | 16:39 |
hazmat | whit, yup, thats it | 16:39 |
tesdal | malthe: wrong layer? | 16:39 |
malthe | yes, as in Zope 2 and even with Plone dependencies. | 16:40 |
tesdal | malthe: right | 16:40 |
malthe | so this is an acute problem | 16:40 |
tesdal | malthe: is it clean enough for you to split it up and refactor? | 16:40 |
malthe | I'll need to look at it some more; I also think there's a lot of duplicate effort between collective.indexing and ore.xapian | 16:41 |
tesdal | malthe: or just reuse parts of it if you want to | 16:41 |
hazmat | malthe, there is also enfold.xapian | 16:41 |
malthe | yeah i'm a bit allergic to enfold.* | 16:42 |
malthe | seems proprietary | 16:42 |
* tesdal doesn't recall how much overlap there is between ore.xapian and collective.indexing | 16:44 | |
hazmat | fair enough.. its probably bitrotted at this point as well.. but its available on a public repo.. http://code.google.com/p/pyxapian/ | 16:46 |
*** sm is now known as sm-run | 16:47 | |
tesdal | malthe: there isn't that much duplicate effort, is there? | 16:47 |
malthe | no I think you're right | 16:47 |
tesdal | there are similar semantics with add, update and delete | 16:48 |
tesdal | both have queues, but one is within transaction and one is async | 16:48 |
tesdal | there are subscribers of course because that's how you add stuff to the queue | 16:48 |
malthe | right the patterns are necessarily similar | 16:48 |
tesdal | I like the resolver as adapter | 16:48 |
tesdal | to enable resolving of other stuff as well | 16:48 |
tesdal | if I remember correctly we're making assumptions about the content that goes into the queue in collective.indexing | 16:49 |
tesdal | that it has a physicalpath or something | 16:49 |
tesdal | malthe: if you have the time to extract the zope level stuff from the existing indexing solutions I'm happy to either ditch collective.indexing and put the remains into plone.indexing or just refactor collective.indexing and making it more lightweight | 16:50 |
malthe | I should have; we're for a couple of days more. | 16:52 |
tesdal | :) | 16:52 |
malthe | hazmat: what's the reason for ore.xapian over enfold.xapian? | 16:53 |
tesdal | would be really neat if this all just worked with grok, vudo, plone etc | 16:53 |
malthe | seem to have been developed at the same time | 16:53 |
malthe | ditto xappy vs. pyxapian | 16:54 |
*** dobee has quit IRC | 16:55 | |
tesdal | I think xappy and flaxcode is the new stuff | 16:55 |
*** dobee has joined #zope3-dev | 16:55 | |
malthe | " We recommend use of Xappy instead of PyXapian for all new projects" | 16:55 |
hazmat | malthe, xappy is actively developed from the prototype done in pyxapian.. i just decided to do a new implementation against it | 16:55 |
malthe | cool | 16:55 |
hazmat | that would fit the use cases i had for indexing external content.. rdb.. svn.. etc | 16:56 |
malthe | gotcha | 16:56 |
malthe | perhaps ``z3c.indexing.queue`` could be the framework part of the project | 16:57 |
malthe | then ``z3c.indexing.xapian`` etc. | 16:57 |
malthe | to keep these efforts together | 16:57 |
malthe | any other preferences, ideas? | 16:58 |
tesdal | what would the queue do | 16:58 |
tesdal | and would it be a zope transaction queue? | 16:59 |
tesdal | or async | 16:59 |
* tesdal would like to see both | 16:59 | |
malthe | right | 16:59 |
tesdal | maybe the async as a transaction queue consumer | 16:59 |
malthe | yes | 16:59 |
malthe | definitely | 16:59 |
tesdal | if queue is the transaction queue | 16:59 |
malthe | yes that was the thought | 17:00 |
tesdal | you'd have a que with add, update and delete semantics | 17:00 |
tesdal | interface for queue reducer | 17:00 |
tesdal | and processing | 17:00 |
malthe | and optionally, an async way to process that queue | 17:00 |
tesdal | and transaction manager or similar there | 17:00 |
tesdal | the async queue processor would be a consumer just like xapian or solr could be consumers | 17:00 |
tesdal | if only async queue processor was registered as transaction queue consumer you'd have queuecatalog | 17:01 |
malthe | ok | 17:01 |
tesdal | but it makes sense to be able to config it | 17:01 |
malthe | but the async queue consumer should be able to talk to the other consumers as well | 17:01 |
tesdal | if we can find a way that xapian and solr etc wouldn't care whether they consume from async queue or transaction queue | 17:01 |
tesdal | exactly | 17:01 |
tesdal | async queue is just an intermediate and optional queue | 17:02 |
malthe | failing to configure correctly would open up for a nice infinite consumption | 17:02 |
tesdal | extension of the transaction request queue | 17:02 |
malthe | which is like the real world | 17:02 |
tesdal | if the async queue processes the async queue? :) | 17:02 |
malthe | yes | 17:03 |
tesdal | sure | 17:03 |
malthe | at any rate, i'll get to work on this; name? | 17:03 |
tesdal | what does z3c mean? | 17:03 |
tesdal | zope 3 component? | 17:04 |
malthe | no zope 3 community | 17:04 |
tesdal | ah | 17:04 |
tesdal | I think z3c.indexing.queue sounds good then | 17:04 |
malthe | it's a bit verbose | 17:04 |
malthe | but at least is somewhat representative | 17:05 |
tesdal | verbose isn't a problem | 17:05 |
tesdal | we'll probably have z3c.indexing.asyncqueue, solr, xapian and possibly fast, autonomy and gsa | 17:05 |
*** b52laptop has quit IRC | 17:05 | |
malthe | tesdal: exactly | 17:06 |
*** b52laptop has joined #zope3-dev | 17:06 | |
*** witsch has joined #zope3-dev | 17:07 | |
tesdal | witsch: malte suggested the name z3c.indexing.queue for the basic zope stuff | 17:08 |
tesdal | the queue and transaction manager basically | 17:08 |
* tesdal thinks it makes sense | 17:08 | |
witsch | tesdal: z3c.indexing is taken already, or too generic? | 17:08 |
tesdal | witsch: is it? | 17:09 |
malthe | it's free | 17:09 |
witsch | tesdal: no, i'm asking | 17:09 |
tesdal | witsch: we'd have z3c.indexing.solr and z3c.indexing.xapian | 17:09 |
malthe | this will be quite generic | 17:09 |
witsch | malthe: hey :) | 17:09 |
malthe | hey andi | 17:09 |
witsch | and z3c.indexing.zcatalog? :) | 17:10 |
witsch | but yeah, if we're gonna have other stuff under z3c.indexing., then yes | 17:10 |
tesdal | I think we'll have other stuff there | 17:11 |
witsch | but imho we should first have a release of collective.indexing and let things settle for a little while | 17:11 |
witsch | refactoring right now would cause even more confusion | 17:11 |
malthe | witsch: well it's counter to zope 3 development efforts alas | 17:11 |
tesdal | depends on whether they need it right now | 17:11 |
tesdal | it's not a problem to change collective.indexing | 17:11 |
tesdal | to import interfaces and queue manager from z3c.indexing for example | 17:11 |
malthe | naturally, I'll look to all existing efforts | 17:12 |
tesdal | as long as this is all adapters etc, it's not data structure | 17:12 |
witsch | nope, it's not | 17:12 |
witsch | malthe: what are the zope3 efforts? | 17:12 |
witsch | malthe: or did you mean the collective namespace is? | 17:12 |
malthe | witsch: ore.xapian | 17:12 |
malthe | and a zodb-counterpart we've developed here at the sprint in sorrento | 17:13 |
witsch | what's the ore namespace? | 17:13 |
malthe | objectrealms | 17:13 |
witsch | ah | 17:13 |
witsch | i guess the transaction manager stuff would just be zodb-related really, right? | 17:14 |
witsch | so this and the queue could be used with zope2 and/or zope3 | 17:15 |
witsch | is zope3 using anything from the collective namespace anyway? | 17:15 |
malthe | not to my knowledge | 17:15 |
malthe | transactions make sense outside of zodb | 17:16 |
tesdal | as z3c is zope 3 collective they're basically the same but z3c is more future buzz | 17:16 |
witsch | cause z3c. would just make it the other way round, i.e. sort of imply zope3 | 17:16 |
malthe | yes, but let's not be afraid of that :-) | 17:16 |
* tesdal is ok with implying zope 3 | 17:16 | |
malthe | pun intended | 17:16 |
witsch | malthe: well sure, a generic interface and an adapter for zodb would make sense of course | 17:16 |
malthe | witsch: right and it'll be more transparent I think | 17:17 |
tesdal | because plone is using both zope2 and zope3 | 17:17 |
witsch | right | 17:17 |
malthe | and collective.indexing monkeypatches things | 17:17 |
tesdal | and grok and zope3 geared projects would be more reluctant to adopting anything in collective | 17:17 |
witsch | sure | 17:17 |
tesdal | the monkeypatch part would not be part of anything zope | 17:17 |
tesdal | that is purely plone/archetypes layer | 17:17 |
*** johbo has quit IRC | 17:18 | |
tesdal | collective.indexing or plone.indexing or something | 17:18 |
malthe | right | 17:18 |
witsch | ok, but what's the timeframe you're proposing? asap? | 17:18 |
tesdal | collective.indexing can't just be renamed | 17:18 |
malthe | the ZODB uses transactions, not the other way around | 17:18 |
tesdal | ah, lib/python/transaction | 17:19 |
witsch | and where should this live then? zope repos? | 17:19 |
*** sunew has joined #zope3-dev | 17:19 | |
* tesdal thought it was in zodb | 17:19 | |
malthe | tesdal: no because it's also valid for, say, sqlalchemy | 17:19 |
witsch | tesdal: kind of, transaction comes with zodb | 17:19 |
tesdal | sure | 17:19 |
* malthe tries to get an overview | 17:20 | |
witsch | tesdal: how about making an alpha nowish and add these plans to the readme, blog entry, todo list etc? | 17:21 |
witsch | i'd kinda like to get it out asap, not refactor everything again beforehand | 17:21 |
malthe | witsch: the good thing is that there's no persistency involved | 17:21 |
witsch | i mean, i'm +1 for refactoring, but first we should let people try it and all | 17:21 |
malthe | so migration will be smooth sailing | 17:21 |
malthe | i.e. they can use the alpha, then just move over to whatever comes next | 17:22 |
witsch | malthe: there are some local utilities | 17:22 |
malthe | ugh | 17:22 |
tesdal | witsch: sure | 17:22 |
tesdal | witsch: collective.indexing is kind of a testbed | 17:23 |
tesdal | making it available right now | 17:23 |
*** witsch_ has joined #zope3-dev | 17:24 | |
witsch_ | malthe: shouldn't be a problem, though | 17:24 |
witsch_ | sorry, my connection sucks today | 17:24 |
malthe | sure | 17:24 |
witsch_ | malthe: the utilities just need to be persistent to be registered, they don't store any data | 17:25 |
tesdal | and because this isn't about data structure, it's not a problem to move interfaces and classes around for a beta of collective.indexing | 17:25 |
witsch_ | so migration shouldn't be too difficult | 17:25 |
witsch_ | tesdal: right | 17:25 |
*** witsch has quit IRC | 17:25 | |
*** witsch_ is now known as witsch | 17:25 | |
witsch | but again, where do you think this should live then? svn.zope.org? | 17:26 |
witsch | well, i guess, tons of z3c stuff there already | 17:26 |
malthe | sure | 17:26 |
* tesdal has to get svn.zope.org access sometime | 17:27 | |
witsch | tesdal: :) | 17:27 |
malthe | tesdal: send a fax | 17:27 |
* tesdal is going home | 17:27 | |
malthe | hehe | 17:27 |
witsch | tesdal: you're in tbg? | 17:27 |
tesdal | witsch: no | 17:27 |
*** tesdal has quit IRC | 17:27 | |
witsch | malthe: well, sounds like a plan to me... let us add it to the readme and blog entry etc and make a first release in the collective namespace (since it's ready & i was gonna do that today anyway) and then start refactoring things soonish | 17:29 |
malthe | witsch: sure; I'll work parallel in a local repository | 17:29 |
witsch | malthe: you're doing the xapian stuff on zope3 only, right? | 17:29 |
malthe | since it's a sprint | 17:29 |
malthe | witsch: yes | 17:29 |
malthe | but I'd like to get to Plone as well | 17:29 |
malthe | but I can hang on a bit with that | 17:29 |
witsch | right | 17:30 |
malthe | it's more important to get it correctly implemented on zope 3 | 17:30 |
witsch | but are you actually replicating the same functionality at the sprint? | 17:30 |
malthe | yes I'm going to copy over your stuff | 17:30 |
malthe | and mix it with ore.xapian | 17:30 |
witsch | ah, okay | 17:30 |
witsch | sounds good | 17:30 |
malthe | and take all credit | 17:30 |
malthe | haha | 17:30 |
witsch | heh | 17:30 |
malthe | just kidding | 17:30 |
malthe | we'll all be famous | 17:31 |
witsch | lol | 17:31 |
witsch | we should be careful in how we announce and move things etc, though | 17:31 |
*** nathany has joined #zope3-dev | 17:31 | |
witsch | to avoid further confusion | 17:31 |
*** menesis has quit IRC | 17:31 | |
witsch | perhaps it makes sense if we start the packages on svn.zope.org, for example... so that people see it's the same effort and not another person doing the same stuff. know what i mean? | 17:32 |
witsch | malthe: ah well, i guess you haven't been following the discussion on the enterprise list, have you? | 17:32 |
malthe | no | 17:33 |
malthe | it's not problem; I'll just add an AUTHORS.txt | 17:33 |
malthe | do you have a problem with relicensing to ZPL btw? | 17:34 |
witsch | malthe: right, good idea... but what i meant is let's coordinate so helge or i actually create the package dirs and then you can take over | 17:34 |
witsch | i don't, but i don't know too much about licenses... better ask helge | 17:35 |
malthe | basically ZPL is required for svn.zope.org | 17:35 |
witsch | yeah, i know | 17:36 |
witsch | phone, hang on | 17:36 |
*** michwill has joined #zope3-dev | 17:41 | |
witsch | malthe: ok, so how about if i try to make that release tonight (incl mention of the plan we just discussed) and then create the z3c packages right afterwards? that way you could commit your code at the end of sorrento | 17:42 |
*** MJ has quit IRC | 17:43 | |
malthe | witsch: sure---in the interim I'll commit the code to the vudo git repository then | 17:43 |
malthe | we use it as a playground for such things | 17:43 |
*** chtp is now known as ctp | 17:43 | |
michwill | Hi all.. Does anybody know how to delete a persistent object from memory? Once I have loaded one, it can't be removed (may be because it can't be introspected) and so it occurs something like memory leakage | 17:44 |
witsch | malthe: right, i'll pull it tomorrow and have a look then | 17:44 |
malthe | cool | 17:44 |
malthe | btw: there's no code yet | 17:45 |
witsch | malthe: that's what i thought :) | 17:45 |
witsch | sounded like you're evaluating atm | 17:45 |
malthe | witsch: definitely | 17:45 |
witsch | malthe: but have you started on the xapian integration at all? | 17:46 |
malthe | witsch: yes, that part's done | 17:46 |
witsch | ah, cool | 17:46 |
malthe | but it integrates with ore.xapian; so there's some working sorting out all the connections | 17:46 |
witsch | but without the queuing part then, right? | 17:46 |
malthe | but the API helge has laid down is good | 17:46 |
malthe | i.e. the idea of a chain of consumers | 17:46 |
malthe | the async consumer being a middle man for asynchronous processing | 17:47 |
malthe | yes, without the queuing part | 17:47 |
witsch | malthe: the one from collective.indexing you mean? | 17:47 |
witsch | the API | 17:47 |
malthe | also, but also the ideas in this chat | 17:47 |
*** timte has quit IRC | 17:47 | |
witsch | ah okay, i guess i've missed those then | 17:47 |
malthe | I'll use the collective.indexing stuff as the transaction and queue-foundation | 17:47 |
malthe | (read: carbon copy) | 17:48 |
witsch | right | 17:48 |
malthe | it's good stuff | 17:48 |
mgedmin | michwill: ? | 17:49 |
*** sm-run is now known as sm | 17:49 | |
witsch | the whole async queueing idea is from enfold.indexing/solr btw, we just extended it with the dispatching part | 17:49 |
mgedmin | michwill: clear the zodb cache, perhaps | 17:49 |
witsch | malthe: anyway, i need to get some other stuff done, so i can go back to work on this soon enough :) | 17:50 |
malthe | sweet | 17:50 |
witsch | so ttyl and have fun at the sprint :) | 17:50 |
*** witsch has left #zope3-dev | 17:50 | |
michwill | mgedmin, zodb is in remote server.. | 17:51 |
mgedmin | the client keeps a cache | 17:51 |
mgedmin | actually, every thread keeps a cache | 17:51 |
mgedmin | a certain max number of objects | 17:51 |
Theuni | aaaaah | 17:52 |
Theuni | It looks like REPORT_NDIFF conflicts with testrunners -1 option for doctests | 17:52 |
michwill | mgedmin, hm.. and.. where is the size of such a cache and how to clean it? Memory "leaks" in sycle like for i i n self.context.values(): del(i) for a persistent object | 17:53 |
mgedmin | Theuni: doctest's global option flags mechanism is BROKEN horribly | 17:53 |
mgedmin | you specify any of the REPORT_foo, and ALL global flags WILL be ignored | 17:54 |
mgedmin | which breaks testrunner's -1 | 17:54 |
*** alga_ has joined #zope3-dev | 17:54 | |
mgedmin | michwill: but it won't leak any more memory if you repeat that loop more than once | 17:54 |
Theuni | hmm | 17:55 |
* mgedmin just passes REPORT_ONLY_FIRST_FAILURE wherever he passes REPORT_NDIFF etc. | 17:55 | |
michwill | mgedmin, of course, it won't, I see. So it isn't actually a memory leak | 17:56 |
Theuni | i wish zope.testing's doctest factory would have a better default | 17:56 |
mgedmin | michwill: nope | 17:56 |
Theuni | first failure, ellipsis, ndiff etc. should IMHO be enabled by default | 17:56 |
mgedmin | ellipsis is tricky | 17:56 |
mgedmin | it sometimes bites | 17:56 |
mgedmin | but first failure, yes | 17:56 |
mgedmin | and ndiff for any bit of output longer than, say, 2 lines | 17:56 |
Theuni | normalize whitespace is pretty useful most of the time too | 17:57 |
mgedmin | also, one thing I dislike, when you add debugging prints to your code and then get an exception, doctest only shows you the traceback but not what your debugging prints printed to stdout | 17:57 |
Theuni | hmm | 17:57 |
Theuni | true | 17:57 |
*** sorindregan has quit IRC | 17:59 | |
Theuni | the test runner probably shouldn't set global options but inspect the tests it has for whether they are doctests and modify their flags accordingly | 18:01 |
Theuni | (if that is possible) | 18:01 |
*** chtp has joined #zope3-dev | 18:01 | |
*** chtp has quit IRC | 18:02 | |
*** alga__ has joined #zope3-dev | 18:05 | |
*** michwill has quit IRC | 18:05 | |
*** alga has quit IRC | 18:06 | |
*** goschtl has quit IRC | 18:06 | |
*** goschtl has joined #zope3-dev | 18:09 | |
*** goschtl has quit IRC | 18:10 | |
*** ctp has quit IRC | 18:13 | |
*** pelle_ has quit IRC | 18:21 | |
*** pcardune has joined #zope3-dev | 18:22 | |
*** grahal has joined #zope3-dev | 18:22 | |
*** grahal has left #zope3-dev | 18:22 | |
*** jodok has quit IRC | 18:24 | |
*** alga_ has quit IRC | 18:25 | |
*** b52laptop has quit IRC | 18:32 | |
mgedmin | Theuni: that's an idea! | 18:53 |
pcardune | Theuni: gocept.registration is pretty darn cool | 18:54 |
pcardune | (as long as we're saying things to Theuni, I thought I would throw that in | 18:54 |
*** agroszer_ has joined #zope3-dev | 18:55 | |
Theuni | pcardune: heh :) | 18:55 |
Theuni | thanks | 18:55 |
Theuni | srichter worked on it too | 18:55 |
Theuni | it's a pretty good example for framework by extraction, i think | 18:56 |
pcardune | yep | 18:56 |
pcardune | Theuni: are you going to make a pypi release by chance? | 18:56 |
Theuni | oh, sure. i haven't used it in a project yet myself, so feel free to tell me if it's actually working and i'll just cut a release | 18:59 |
Theuni | OTOH | 18:59 |
Theuni | why don't you just make one and give me ownership as well? | 19:00 |
*** afd_ has joined #zope3-dev | 19:00 | |
*** toko has quit IRC | 19:01 | |
*** projekt01 has quit IRC | 19:03 | |
*** flox has quit IRC | 19:04 | |
pcardune | Theuni: ok, that sounds good (and yes, it is working :) | 19:04 |
*** markusleist has quit IRC | 19:07 | |
*** agroszer has quit IRC | 19:07 | |
*** agroszer_ is now known as agroszer | 19:08 | |
*** harobed has quit IRC | 19:08 | |
*** toutpt has quit IRC | 19:33 | |
*** jukart has quit IRC | 19:37 | |
*** sunew has quit IRC | 19:38 | |
*** romanofski has quit IRC | 19:40 | |
*** sm has quit IRC | 19:42 | |
rocky | srichter: ping | 19:51 |
*** dobee has quit IRC | 19:52 | |
rocky | ok, let me mention this general ... | 19:52 |
*** quodt has quit IRC | 19:53 | |
rocky | is anyone aware of the issue where marking a persistent object with a marker interface and then removing that interface from sys.path so it can't be imported anymore and then trying to load the object... you get a nasty unpickling error | 19:53 |
*** zenwryly has joined #zope3-dev | 19:53 | |
faassen | rocky: this sounds like the normal issue of persisting references to things that don't exist anymore. | 19:55 |
rocky | faassen: yeah, i think it can be broken down that way, but the thing is it happens *alot* for me in plone setting where people install apps and then uninstall apps | 19:55 |
rocky | and very specifically to interfaces | 19:55 |
rocky | so could it be simply that the use case is quite rarer in a zope3 setting where people don't often uninstall py libs? | 19:56 |
faassen | rocky: apparently references to the interfaces are left in the ZODB then.. shouldn't the uninstaller remove those? | 19:56 |
faassen | rocky: well, it's just the general case. if you uninstall a library that defines something that got persisted, your ZODB won't be able to load that object. | 19:56 |
*** stub has joined #zope3-dev | 19:57 | |
rocky | well yeah, that's sort of my fix... but it's quite messy in a large plone site... where i have 2 options, query the catalog for my objects (speedily) and risk missing some objects ... or crawling the zodb hierarchy for the plone site | 19:57 |
faassen | rocky: that is, if you remove a class of which instances are persisted, or you remove the marker interface definition while you make some instances providei t. | 19:57 |
*** pcardune has quit IRC | 19:57 | |
*** jodok has joined #zope3-dev | 19:58 | |
rocky | and then of course there's the use case where what if someone reinstalls the app and would like for their persistent objects to retain knowledge of the marker interfaces... in the solution you and i describe, that's not possible | 19:59 |
rocky | i understand this isn't much of a use case for pure zope 3 apps... but it happens alot in the plone world | 19:59 |
faassen | well, a pure zope 3 app would have the same issue. | 20:00 |
faassen | but I imagine we don't have people uninstalling things a lot yet. | 20:00 |
rocky | yeah of course it has the same issue, but that scenario doesn't present itself very often i suspect in the pure zope3 setting | 20:00 |
rocky | right | 20:00 |
rocky | that's what i'm trying to say ;) | 20:00 |
faassen | right. any system that was like plone and allowed you to install plugins would have similar issues though. | 20:00 |
*** ignas has quit IRC | 20:03 | |
*** norro_ has joined #zope3-dev | 20:14 | |
*** norro__ has joined #zope3-dev | 20:15 | |
*** pcardune has joined #zope3-dev | 20:19 | |
*** jodok has quit IRC | 20:19 | |
*** chtp_ has joined #zope3-dev | 20:19 | |
*** jsadjohnson has joined #zope3-dev | 20:20 | |
*** jodok has joined #zope3-dev | 20:20 | |
*** jodok has quit IRC | 20:21 | |
*** thruflo has quit IRC | 20:26 | |
*** alex_smith_ has quit IRC | 20:26 | |
*** chtp_ has quit IRC | 20:26 | |
*** alex_smith has joined #zope3-dev | 20:27 | |
*** mkerrin has quit IRC | 20:28 | |
*** dobee has joined #zope3-dev | 20:31 | |
*** norro_ has quit IRC | 20:32 | |
*** norro__ has quit IRC | 20:32 | |
*** reco has joined #zope3-dev | 20:33 | |
*** malthe has quit IRC | 20:48 | |
*** maurits has quit IRC | 20:50 | |
*** b52laptop has joined #zope3-dev | 20:51 | |
*** quodt has joined #zope3-dev | 20:59 | |
*** tarek has quit IRC | 21:06 | |
*** mgedmin has quit IRC | 21:09 | |
*** harobed has joined #zope3-dev | 21:14 | |
*** agroszer has quit IRC | 21:14 | |
*** stub has quit IRC | 21:14 | |
*** alga__ has quit IRC | 21:17 | |
*** afd_ has quit IRC | 21:18 | |
*** reco has quit IRC | 21:23 | |
*** reco has joined #zope3-dev | 21:23 | |
*** reco has quit IRC | 21:29 | |
*** reco has joined #zope3-dev | 21:30 | |
*** jpcw2002 has left #zope3-dev | 21:49 | |
*** redir has joined #zope3-dev | 21:50 | |
*** salfield has quit IRC | 21:53 | |
*** markusleist has joined #zope3-dev | 21:56 | |
*** alex_smith has quit IRC | 22:14 | |
*** alex_smith has joined #zope3-dev | 22:14 | |
*** malthe has joined #zope3-dev | 22:16 | |
*** djohnson has joined #zope3-dev | 22:19 | |
*** faassen has quit IRC | 22:20 | |
*** MiUlEr_ has joined #zope3-dev | 22:23 | |
*** tarek has joined #zope3-dev | 22:24 | |
*** reco has quit IRC | 22:26 | |
*** reco has joined #zope3-dev | 22:26 | |
*** MiUlEr_ has quit IRC | 22:27 | |
*** d2m has quit IRC | 22:31 | |
*** djohnson has quit IRC | 22:40 | |
*** sunew has joined #zope3-dev | 22:43 | |
*** dobee has quit IRC | 22:46 | |
*** alga has joined #zope3-dev | 22:57 | |
*** tesdal has joined #zope3-dev | 22:59 | |
malthe | tesdal: work is progressing, but I've felt the urge to refactor and restructure some of the components. | 23:00 |
tesdal | malthe: sure, is it in svn somewhere? | 23:00 |
malthe | tesdal: no, we're using the vudo repository as a temporary workspace | 23:00 |
tesdal | ok | 23:00 |
malthe | I expect something interesting to be there tomorrow afternoon | 23:00 |
tesdal | excellent | 23:00 |
malthe | the index queue has been replaced by "transactional dispatcher" | 23:01 |
tesdal | malthe: is that a name change or something more? | 23:01 |
*** jpcw2002 has joined #zope3-dev | 23:02 | |
malthe | it's an effort to make the indexing flow abstraction more sane | 23:02 |
malthe | the transactional indexing dispatcher is now the key entry point | 23:02 |
malthe | you can't get around it | 23:02 |
malthe | (which changes nothing) | 23:03 |
tesdal | ok | 23:03 |
malthe | the motivation will be clearer in the README.txt | 23:03 |
* tesdal is looking forward to seeing it :) | 23:03 | |
malthe | the main reasoning is that there's no-longer a queue processor. | 23:04 |
malthe | it'll all about dispatching indexing operations | 23:04 |
malthe | transactions beyond the main dispatcher don't make much sense (we feel). | 23:05 |
malthe | although both xapian and solr support them | 23:05 |
malthe | we simply don't have a use for a doubly transactional contract. | 23:05 |
tesdal | hmm | 23:05 |
tesdal | is there a queue? | 23:06 |
tesdal | what do you mean by queue processor? | 23:06 |
malthe | that was in the original package | 23:06 |
*** zagy has quit IRC | 23:06 | |
tesdal | malthe: the zcatalog consumer? | 23:06 |
malthe | solr would be a queue processor in the original package | 23:06 |
tesdal | right | 23:06 |
malthe | yes | 23:06 |
malthe | we've gotten ridden of that concept | 23:07 |
tesdal | np | 23:07 |
tesdal | as long as we can reduce duplicates and dispatch to both zcatalog and externals it's all good :) | 23:07 |
malthe | yes we'll be able to do that | 23:07 |
malthe | you can code rewrite it | 23:08 |
malthe | that would be great | 23:08 |
malthe | err | 23:08 |
tesdal | ? | 23:08 |
malthe | review it hehe | 23:08 |
tesdal | :) | 23:08 |
tesdal | did you rename the package from indexing.queue to something else? | 23:08 |
*** sunew has quit IRC | 23:08 | |
malthe | yes z3c.indexing.dispatch | 23:08 |
tesdal | ok | 23:08 |
malthe | it might seem odd, but it'll grow on you | 23:08 |
* tesdal likes it | 23:09 | |
tesdal | it still has a queue, but it's clearer that it's about dispatching | 23:09 |
tesdal | the queue is an implementation details | 23:10 |
malthe | exactly! | 23:10 |
malthe | and --- transactional dispatching | 23:10 |
malthe | that's really the focus | 23:10 |
tesdal | yes | 23:12 |
tesdal | malthe: is it you and sylvain working on it? | 23:13 |
malthe | yep | 23:14 |
malthe | mostly me right now; sylvain will join tomorrow on actual implementation | 23:14 |
tesdal | ok | 23:14 |
malthe | we've mostly discussed things together until now | 23:14 |
tesdal | discussing is important | 23:15 |
malthe | yeah very much so; sylvain is good | 23:15 |
tesdal | getting more input on naming and how to set up components | 23:15 |
malthe | yeah and just if the overall idea is sane | 23:15 |
tesdal | yup | 23:15 |
*** quodt has quit IRC | 23:31 | |
*** junkafarian has joined #zope3-dev | 23:34 | |
junkafarian | hey, is it possible to create 2 objects (one parent container and a contained object) from a single add form in the create method? | 23:37 |
junkafarian | the main problem im running into is the inability to add the child object before the container is placed | 23:38 |
junkafarian | im not sure what the best approach is though | 23:39 |
*** faassen has joined #zope3-dev | 23:47 | |
*** lucielejard has quit IRC | 23:52 | |
*** jsadjohnson has quit IRC | 23:53 | |
malthe | tesdal: git clone http://repo.or.cz/w/voodoo.git | 23:58 |
Generated by irclog2html.py 2.15.1 by Marius Gedminas - find it at mg.pov.lt!