IRC log of #zodb for Monday, 2016-12-05

*** kuetrzi has quit IRC08:24
*** kuetrzi has joined #zodb08:31
*** kuetrzi has quit IRC08:50
*** kuetrzi has joined #zodb09:02
*** jensens has joined #zodb12:23
*** kuetrzi has quit IRC13:05
*** J1m has joined #zodb18:55
*** jensens has quit IRC20:04
*** J1m has quit IRC20:28
*** kuetrzi has joined #zodb21:05
*** kuetrzi has quit IRC21:28
*** J1m has joined #zodb21:30
*** kuetrzi has joined #zodb21:37
*** kuetrzi has quit IRC22:10
*** jmuchemb has joined #zodb22:30
jmuchembhello Jim22:30
J1mHi jmuchemb22:31
jmuchembI wonder if there's any object for a new release of zodbpickle, because it adds support for Python 3.522:32
jmuchembthat would be easier to package for Debian22:32
J1mIDK, btw, can you remind me why we need it? :)22:32
J1mOK, I can look into making a release.22:33
J1mI don't remember exactly why we need it,22:33
jmuchembthe need to package to Debian ? I do it on my free time. I want update the zodb package of Debian, which is still 3.10.722:34
jmuchembI think it's good for the popularity of ZODB :)22:34
J1mNo, the need to zodbpickle22:35
J1mThis (zodbpickle) was done when I wasn't really paying attention.22:35
jmuchemb"the need to zodbpickle" > ah, the README gives the impression it's useful, but I didn't whether it's true22:36
J1mI think there was some hope that this would make it possible to share pickles between Python 2 and Python 3, but I don't think that happened.22:38
jmuchembat least, for Python 2 it's nice to have the new protocol22:38
jmuchembfor Python 3, it add support for the ``noload`` operations used by ZODB: it seems required by ZODB and py3 does not have it22:40
jmuchemb > I wonder how James managed to use the builtin pickle22:41
jmuchemb... on Python 322:41
J1mjason btw22:41
J1mone could write gc to not use noload.22:42
J1mnoload is an optimation.22:42
J1mMy guess is that it matters less for external garbage collectors.22:43
J1mDoes NEO use an external GC?22:43
jmuchembit should22:43
jmuchembwe haven't found the time to test22:43
J1mON a slightly related note, I've thought of a new way of thinking about ZRS:22:45
J1mtransaction pub/sum with buffering.22:45
jmuchemboh, you don't have the right to upload zodbpickle on PyPI22:45
J1mNo, I don't. I need to find someone who does.22:45
J1mpub/sub above22:46
J1mwait, are you looking for a binary release of zodbpickle?22:47
jmuchembDebian folks would prefer (I wish they stop working only with tarballs)22:48
J1mdo you have reason to believe the source release won't work with Python 3.5, because it works fine for me.22:48
jmuchembthere was a MR for that22:49
jmuchemb(s/MR/PR/ in GitHub term)22:49
J1mI don't see any binary linux releases for zodbpickle, but I guess this is an issue for you because zodbpickle only now matters.22:50
jmuchembbut I didn't check since I don't use 3.5 yet on my machine22:50
jmuchemblet me clarify22:50
jmuchemb"wait, are you looking for a binary release of zodbpickle?" > actually no: currently, for python modules, Debian prefers source tarballs from PyPI22:51
* J1m waits for more clarity...22:53
jmuchemb"do you have reason to believe the source release won't work with Python 3.5, because it works fine for me" > I just checked does not change code for Python 3.5, so I could live with the current release22:53
jmuchembyes :) (I won't have to wait: the Debian freeze is coming to quickly)22:55
jmuchembso for ZRS ? (no sure I can follow because I know almost nothing)22:56
J1mOh, I see, 0.7.0 was never released.  I guess they only certified it.22:56
J1mWell, an idea I've been rambling about for a while now is using ZRS to feed things other than replicas, like external indexers.22:57
J1mAnother possible application is garbage collection (in the style of zc.zodbdgc).22:58
J1mI realized recently that what makes this interesting is the ability to publish and subscribe to transaction data.22:58
jmuchemban invalidation is a kind of pub22:59
J1mFor example, I could imagine publishing transaction data using something like kafka or AWS kinesis to allow *asynchronous* processing of committed data.23:00
jmuchembkafka ?23:00
jmuchemb(ah i missed the 2nd result on google)23:01
J1mZRS provides this with more or less infinite buffering, but it would be interesting in cases like relstorage history-free storages if you could get by with limited buffering.23:03
J1mThis is a pattern I'd like to make useful accross NEO, ZEO and RelStorage someday.23:03
jmuchembfor a full replication of the DB, I think NEO already has the ultimate solution (immediate start of replication upon a new transaction, and parallel transfer from M storages to N storages)23:10
jmuchembbut as you write, if you find other usage to ZRS, that could benefit to NEO23:10
J1mIn this context, I'm relly thinking of applications other than replication.23:11
J1mLike updating external indexes.23:11
J1mOr building other sorts of representations of the data.23:12
jmuchembI wonder what kind of criteria you're thinking about for pub/sub23:12
jmuchembthe ZODB data model is quite limited23:13
jmuchembat least the data part is an obscure key->value storage23:14
jmuchembthere's a feature in NEO I am very impatient to work on is a partial pack23:15
jmuchembmaybe something relying on categorizing oids in some way23:16
J1mI'm thinking pickles :)23:16
J1mAnd then applying transformations to them.23:16
J1mFor example,
J1mI'm currently playing with that to create json representations of pickles in Postgres jsonb columns to suppliment RelStorage pickles.23:17
J1mAnd then indexing and searching that data in Postgres.23:18
jmuchembif json was not more verbose (no ?), maybe zobdpickle could output json directly23:19
J1mThe problem with json is that it's not as expresssive as pickle.23:20
J1mAnd adding additional bits to it to make it richer (in a compatible way) would make it hard to use as json.23:21
J1mThis is why I'm looking at supplimenting pickle data with it.23:21
J1mBut if you has a pickle subscriber, you could easily start by converting to some more generic format and then applying additional filtering/transformation to that.23:22
*** kuetrzi has joined #zodb23:22
jmuchembI see23:23
J1mAn earlier experiment years ago was xmlpickle
J1mIt isn't lossy, but would probably be impractical to use for this sort of application.23:25
jmuchembxml pickles is what we do for erp5 business templates23:25
J1mPickle tries very hard to be DRY, so if you have a repeated object, you only have one copy and references.23:26
J1mBut references are a pain to deal with outside of the context of pickling, so I bet your xmlpickles don't do that.23:27
jmuchembI don't remember23:28
*** J1m has quit IRC23:52
*** jmuchemb has quit IRC23:53

Generated by 2.15.1 by Marius Gedminas - find it at!