| *** kuetrzi has quit IRC | 08:24 | |
| *** kuetrzi has joined #zodb | 08:31 | |
| *** kuetrzi has quit IRC | 08:50 | |
| *** kuetrzi has joined #zodb | 09:02 | |
| *** jensens has joined #zodb | 12:23 | |
| *** kuetrzi has quit IRC | 13:05 | |
| *** J1m has joined #zodb | 18:55 | |
| *** jensens has quit IRC | 20:04 | |
| *** J1m has quit IRC | 20:28 | |
| *** kuetrzi has joined #zodb | 21:05 | |
| *** kuetrzi has quit IRC | 21:28 | |
| *** J1m has joined #zodb | 21:30 | |
| *** kuetrzi has joined #zodb | 21:37 | |
| *** kuetrzi has quit IRC | 22:10 | |
| *** jmuchemb has joined #zodb | 22:30 | |
| jmuchemb | hello Jim | 22:30 |
|---|---|---|
| J1m | Hi jmuchemb | 22:31 |
| jmuchemb | I wonder if there's any object for a new release of zodbpickle, because it adds support for Python 3.5 | 22:32 |
| jmuchemb | that would be easier to package for Debian | 22:32 |
| J1m | IDK, btw, can you remind me why we need it? :) | 22:32 |
| J1m | OK, I can look into making a release. | 22:33 |
| J1m | I don't remember exactly why we need it, | 22:33 |
| jmuchemb | the need to package to Debian ? I do it on my free time. I want update the zodb package of Debian, which is still 3.10.7 | 22:34 |
| jmuchemb | I think it's good for the popularity of ZODB :) | 22:34 |
| J1m | No, the need to zodbpickle | 22:35 |
| jmuchemb | btw, https://lists.debian.org/debian-python/2016/11/msg00002.html | 22:35 |
| J1m | This (zodbpickle) was done when I wasn't really paying attention. | 22:35 |
| jmuchemb | "the need to zodbpickle" > ah, the README gives the impression it's useful, but I didn't whether it's true | 22:36 |
| J1m | heh | 22:37 |
| J1m | I think there was some hope that this would make it possible to share pickles between Python 2 and Python 3, but I don't think that happened. | 22:38 |
| jmuchemb | at least, for Python 2 it's nice to have the new protocol | 22:38 |
| jmuchemb | for Python 3, it add support for the ``noload`` operations used by ZODB: it seems required by ZODB and py3 does not have it | 22:40 |
| J1m | Yeah. | 22:41 |
| jmuchemb | https://github.com/zopefoundation/zodbpickle/issues/21 > I wonder how James managed to use the builtin pickle | 22:41 |
| jmuchemb | ... on Python 3 | 22:41 |
| J1m | jason btw | 22:41 |
| jmuchemb | oops | 22:41 |
| J1m | Interesting | 22:42 |
| J1m | one could write gc to not use noload. | 22:42 |
| J1m | noload is an optimation. | 22:42 |
| J1m | My guess is that it matters less for external garbage collectors. | 22:43 |
| jmuchemb | ok | 22:43 |
| J1m | Does NEO use an external GC? | 22:43 |
| jmuchemb | it should | 22:43 |
| jmuchemb | we haven't found the time to test | 22:43 |
| J1m | ON a slightly related note, I've thought of a new way of thinking about ZRS: | 22:45 |
| J1m | transaction pub/sum with buffering. | 22:45 |
| jmuchemb | oh, you don't have the right to upload zodbpickle on PyPI | 22:45 |
| J1m | No, I don't. I need to find someone who does. | 22:45 |
| J1m | pub/sub above | 22:46 |
| J1m | wait, are you looking for a binary release of zodbpickle? | 22:47 |
| jmuchemb | Debian folks would prefer (I wish they stop working only with tarballs) | 22:48 |
| J1m | do you have reason to believe the source release won't work with Python 3.5, because it works fine for me. | 22:48 |
| jmuchemb | there was a MR for that | 22:49 |
| jmuchemb | (s/MR/PR/ in GitHub term) | 22:49 |
| J1m | I don't see any binary linux releases for zodbpickle, but I guess this is an issue for you because zodbpickle only now matters. | 22:50 |
| jmuchemb | but I didn't check since I don't use 3.5 yet on my machine | 22:50 |
| jmuchemb | let me clarify | 22:50 |
| jmuchemb | "wait, are you looking for a binary release of zodbpickle?" > actually no: currently, for python modules, Debian prefers source tarballs from PyPI | 22:51 |
| * J1m waits for more clarity... | 22:53 | |
| jmuchemb | "do you have reason to believe the source release won't work with Python 3.5, because it works fine for me" > I just checked https://github.com/zopefoundation/zodbpickle/pull/19 does not change code for Python 3.5, so I could live with the current release | 22:53 |
| J1m | cool | 22:53 |
| jmuchemb | yes :) (I won't have to wait: the Debian freeze is coming to quickly) | 22:55 |
| jmuchemb | so for ZRS ? (no sure I can follow because I know almost nothing) | 22:56 |
| J1m | Oh, I see, 0.7.0 was never released. I guess they only certified it. | 22:56 |
| J1m | Well, an idea I've been rambling about for a while now is using ZRS to feed things other than replicas, like external indexers. | 22:57 |
| J1m | Another possible application is garbage collection (in the style of zc.zodbdgc). | 22:58 |
| J1m | I realized recently that what makes this interesting is the ability to publish and subscribe to transaction data. | 22:58 |
| jmuchemb | an invalidation is a kind of pub | 22:59 |
| J1m | For example, I could imagine publishing transaction data using something like kafka or AWS kinesis to allow *asynchronous* processing of committed data. | 23:00 |
| J1m | Yuo | 23:00 |
| J1m | Yup | 23:00 |
| jmuchemb | kafka ? | 23:00 |
| jmuchemb | (ah i missed the 2nd result on google) | 23:01 |
| J1m | https://kafka.apache.org/ | 23:01 |
| J1m | ZRS provides this with more or less infinite buffering, but it would be interesting in cases like relstorage history-free storages if you could get by with limited buffering. | 23:03 |
| J1m | This is a pattern I'd like to make useful accross NEO, ZEO and RelStorage someday. | 23:03 |
| jmuchemb | for a full replication of the DB, I think NEO already has the ultimate solution (immediate start of replication upon a new transaction, and parallel transfer from M storages to N storages) | 23:10 |
| jmuchemb | but as you write, if you find other usage to ZRS, that could benefit to NEO | 23:10 |
| J1m | In this context, I'm relly thinking of applications other than replication. | 23:11 |
| J1m | Like updating external indexes. | 23:11 |
| jmuchemb | yes | 23:12 |
| J1m | Or building other sorts of representations of the data. | 23:12 |
| jmuchemb | I wonder what kind of criteria you're thinking about for pub/sub | 23:12 |
| jmuchemb | the ZODB data model is quite limited | 23:13 |
| jmuchemb | at least the data part is an obscure key->value storage | 23:14 |
| jmuchemb | there's a feature in NEO I am very impatient to work on is a partial pack | 23:15 |
| jmuchemb | maybe something relying on categorizing oids in some way | 23:16 |
| J1m | I'm thinking pickles :) | 23:16 |
| J1m | And then applying transformations to them. | 23:16 |
| J1m | For example, https://github.com/jimfulton/xpickle | 23:17 |
| J1m | I'm currently playing with that to create json representations of pickles in Postgres jsonb columns to suppliment RelStorage pickles. | 23:17 |
| J1m | And then indexing and searching that data in Postgres. | 23:18 |
| jmuchemb | :) | 23:18 |
| jmuchemb | if json was not more verbose (no ?), maybe zobdpickle could output json directly | 23:19 |
| J1m | The problem with json is that it's not as expresssive as pickle. | 23:20 |
| jmuchemb | yes | 23:20 |
| J1m | And adding additional bits to it to make it richer (in a compatible way) would make it hard to use as json. | 23:21 |
| J1m | This is why I'm looking at supplimenting pickle data with it. | 23:21 |
| J1m | But if you has a pickle subscriber, you could easily start by converting to some more generic format and then applying additional filtering/transformation to that. | 23:22 |
| *** kuetrzi has joined #zodb | 23:22 | |
| J1m | s/has/had/ | 23:22 |
| jmuchemb | I see | 23:23 |
| J1m | An earlier experiment years ago was xmlpickle https://github.com/zopefoundation/zope.xmlpickle | 23:24 |
| J1m | It isn't lossy, but would probably be impractical to use for this sort of application. | 23:25 |
| jmuchemb | xml pickles is what we do for erp5 business templates | 23:25 |
| J1m | Pickle tries very hard to be DRY, so if you have a repeated object, you only have one copy and references. | 23:26 |
| jmuchemb | yes | 23:27 |
| J1m | But references are a pain to deal with outside of the context of pickling, so I bet your xmlpickles don't do that. | 23:27 |
| jmuchemb | I don't remember | 23:28 |
| *** J1m has quit IRC | 23:52 | |
| *** jmuchemb has quit IRC | 23:53 | |
Generated by irclog2html.py 2.15.1 by Marius Gedminas - find it at mg.pov.lt!