| *** kosh_ has joined #zope | 00:40 | |
| *** kosh has quit IRC | 00:43 | |
| *** mup has quit IRC | 01:09 | |
| *** mup_ has joined #zope | 01:09 | |
| *** mup_ is now known as mup | 01:09 | |
| *** stereo_ has quit IRC | 01:26 | |
| *** kosh_ has quit IRC | 01:49 | |
| *** supton has joined #zope | 02:31 | |
| *** supton has quit IRC | 02:45 | |
| *** Bahman has joined #zope | 02:47 | |
| *** Bahman has quit IRC | 02:50 | |
| *** menesis has quit IRC | 05:59 | |
| *** alecm has quit IRC | 06:11 | |
| *** Arfrever has joined #zope | 06:32 | |
| *** alecm has joined #zope | 07:19 | |
| *** alecm has joined #zope | 07:19 | |
| *** giacomos has joined #zope | 07:45 | |
| *** supton has joined #zope | 08:10 | |
| *** supton has quit IRC | 08:57 | |
| *** __mac__ has joined #zope | 09:04 | |
| *** tisto has joined #zope | 09:11 | |
| *** JaRoel|4D has quit IRC | 09:40 | |
| *** alecm has quit IRC | 09:47 | |
| *** alecm has joined #zope | 09:47 | |
| *** JaRoel|4D has joined #zope | 10:04 | |
| *** PeterZ1 has joined #zope | 10:07 | |
| *** PeterZ1 has left #zope | 10:07 | |
| *** kosh has joined #zope | 10:20 | |
| *** sylvain has joined #zope | 10:35 | |
| sylvain | Hello mgedmin | 10:36 |
|---|---|---|
| mgedmin | hi sylvain | 10:38 |
| sylvain | you saw my large pull request | 10:38 |
| sylvain | I will fix all the silly mistakes about format and english | 10:38 |
| sylvain | but I wanted to explain why a couple of things are like this | 10:39 |
| sylvain | I will put some comments on github as well | 10:39 |
| mgedmin | that sounds like a good idea | 10:39 |
| sylvain | in fact | 10:39 |
| sylvain | we used Relstorage history free | 10:39 |
| sylvain | and did a pack using the official pack script | 10:39 |
| sylvain | on a write heavy database | 10:39 |
| sylvain | that toke a long time and created all kind of POSKey error | 10:40 |
| mgedmin | I actually found some of my questions answered in the README | 10:40 |
| sylvain | so we wanted to browser the database to find the objects pointing to the missing one | 10:40 |
| sylvain | yes | 10:40 |
| mgedmin | it's a shame github doesn't let me review my comments and questions before publishing them | 10:40 |
| sylvain | well I am a bit confused how to answer all of them, but I will try to figure it out | 10:41 |
| sylvain | we added the zodbcheck to find them and start browsing the objects from the missing ones | 10:41 |
| sylvain | the main issue is | 10:42 |
| sylvain | most of the code only works for Relstorage history free | 10:42 |
| sylvain | if you have multiple versions of objects | 10:42 |
| sylvain | all the algorithms used stop working :/ | 10:42 |
| mgedmin | oh! I see! | 10:42 |
| sylvain | and we did that way | 10:43 |
| sylvain | because we have a large database | 10:43 |
| sylvain | and did not wait weeks to do things | 10:43 |
| sylvain | that's why we precompute the index with the zodbcheck | 10:43 |
| sylvain | and while doing the check we check for poskey errors | 10:43 |
| sylvain | we do that every day on the database | 10:44 |
| mgedmin | how large is your database? | 10:44 |
| sylvain | and if we find any, we have the index and open in zodbbrowser | 10:44 |
| sylvain | 22 millions of objects | 10:44 |
| sylvain | those are small, so it is only 6 GB if packed | 10:45 |
| sylvain | same thing for the search, that's in a script because it takes some time to go through the database | 10:46 |
| mgedmin | that's fine | 10:46 |
| sylvain | and the pack script, well, we can definitely move it outside of this package | 10:46 |
| sylvain | but we tested some other solution, like relstorage_packer | 10:46 |
| mgedmin | it's just that longterm I want this to be in the web app, but done in a background task | 10:46 |
| sylvain | I can understand that | 10:46 |
| sylvain | that would be a nice feature | 10:46 |
| mgedmin | (in fact I want some of the current history crawling code to be done in the background) | 10:46 |
| sylvain | but without index, it does take some time | 10:47 |
| sylvain | so the pack, we tested some other script, that toke 4 or 5 days to run | 10:47 |
| sylvain | and after we thought we had most of the information in the zodbcheck database | 10:47 |
| sylvain | and wrote this SQL query, that takes 15 minutes to find out that to do delete on a fast computer | 10:48 |
| sylvain | that's why it is there | 10:48 |
| mgedmin | nice | 10:48 |
| sylvain | but again | 10:48 |
| sylvain | all of this makes sense only for history free, like in relstorage | 10:49 |
| sylvain | I am not sure if there is any other backend that have this option | 10:49 |
| mgedmin | MemoryStorage? :) | 10:49 |
| sylvain | we wondered if we should make a pull request for a long time | 10:49 |
| sylvain | because it is kind of specific | 10:49 |
| mgedmin | I would love for ZODB to have an API to iterate over all the objects (not transactions) | 10:49 |
| sylvain | there is one | 10:50 |
| sylvain | but that is not implemented by all storage | 10:50 |
| mgedmin | what is it? | 10:50 |
| sylvain | you can do this on FileStorage I believe | 10:50 |
| sylvain | let me look of it | 10:50 |
| sylvain | I think that's the record_iternext | 10:52 |
| sylvain | like I used in zodbupdate | 10:52 |
| sylvain | yes | 10:53 |
| sylvain | I guess it would be possible to make a custom solution to iterate of FileStorage for instance | 10:53 |
| sylvain | to have the zodbcheck script works there | 10:53 |
| mgedmin | ah, I see | 10:54 |
| mgedmin | it's part of an interface: https://github.com/zopefoundation/ZODB/blob/master/src/ZODB/interfaces.py#L1127 | 10:55 |
| sylvain | yes | 10:55 |
| sylvain | but only implemented by FileStorage | 10:55 |
| sylvain | Relstorage does not have this | 10:55 |
| sylvain | it only have with transactions | 10:55 |
| mgedmin | nor ClientStorage :/ | 10:55 |
| mgedmin | and the API is not exactly beautiful | 10:56 |
| mgedmin | oh well | 10:56 |
| mgedmin | maybe someone will create a ZODB PR for this ;) | 10:56 |
| sylvain | yes but if you use ZEO, you most probably have a FileStorage and could run the zodbcheck directly on it | 10:56 |
| mgedmin | so, I'm very happy that you submitted the PR | 10:56 |
| sylvain | I will cleaned it up | 10:56 |
| sylvain | we wrote most of this code in a hurry trying to solve problems | 10:57 |
| mgedmin | I planned to build an in-memory index for backward references | 10:57 |
| mgedmin | but an on-disk index is maybe even better | 10:57 |
| mgedmin | say rather "this code was battle-tested" ;) | 10:57 |
| sylvain | we have an in-memory one we used in tests | 10:57 |
| mgedmin | also, I'd rather have multiple PRs for little bits of this than this one cleaned up | 10:57 |
| sylvain | to check that if we delete an object, there are no references left to it | 10:57 |
| sylvain | yes, I can maybe investigate how to do this multiple PR, but I am not sure I know how to do ti | 10:58 |
| sylvain | it* | 10:58 |
| mgedmin | create new branch, cherry-pick the relevant commits, maybe git rebase -i to make it pretty and/or squash related bits/fixups | 10:59 |
| mgedmin | repeat for other PRs | 10:59 |
| sylvain | ok | 10:59 |
| mgedmin | I like to start with smallest changes first, because those are easiest | 10:59 |
| sylvain | I will first make a bunch of commit to fix all the comments you already put there | 11:00 |
| sylvain | because most of them makes sense | 11:00 |
| mgedmin | hm, actually I'm not sure what the best approach is | 11:00 |
| mgedmin | maybe fix up typos etc first? | 11:00 |
| mgedmin | maybe instead of cherry-picking use a merge tool to pick specific changes? | 11:00 |
| mgedmin | then when a standalone PR is merged, maybe github will manage to cleanly update this one to show only remaining changes? | 11:01 |
| sylvain | well I should already fix all the typo and format issues, because I will still to do it anyway | 11:01 |
| sylvain | and after I will see to make nicer pull requests | 11:02 |
| mgedmin | and then there's always the approach of creating a new branch and reimplementing the feature from scratch, using the existing branch as a prototype (and liberally copy-pasting code from it) | 11:02 |
| mgedmin | this requires practically no git-fu | 11:02 |
| sylvain | I will see | 11:02 |
| mgedmin | pick whichever's easiest for you | 11:03 |
| sylvain | I will fix some issues today, but I have some other work to do as well | 11:03 |
| * mgedmin nods | 11:03 | |
| sylvain | so it will takes some time | 11:03 |
| mgedmin | I have some ideas how to make the reference DB building work with multiple object versions | 11:03 |
| mgedmin | I'll comment on the PR | 11:04 |
| mgedmin | https://github.com/mgedmin/zodbbrowser/pull/19/files#r33017986 | 11:06 |
| *** kosh has quit IRC | 11:07 | |
| *** agroszer has joined #zope | 11:08 | |
| *** MrTango has joined #zope | 11:10 | |
| *** kosh has joined #zope | 11:17 | |
| *** Arfrever has quit IRC | 11:20 | |
| *** projekt01 has joined #zope | 11:23 | |
| *** kosh has quit IRC | 11:28 | |
| *** menesis has joined #zope | 12:27 | |
| *** Pumukel has joined #zope | 12:35 | |
| *** maurits has joined #zope | 12:48 | |
| *** fredvd has joined #zope | 12:58 | |
| *** projekt01 has quit IRC | 13:24 | |
| *** maurits has quit IRC | 13:37 | |
| *** maurits has joined #zope | 14:42 | |
| *** yvl has joined #zope | 15:08 | |
| *** menesis has quit IRC | 15:22 | |
| *** giacomos has quit IRC | 15:24 | |
| *** menesis has joined #zope | 15:34 | |
| *** giacomos has joined #zope | 15:35 | |
| *** yvl has quit IRC | 16:14 | |
| *** sm has quit IRC | 16:37 | |
| *** menesis has quit IRC | 16:43 | |
| *** menesis has joined #zope | 16:51 | |
| *** sm has joined #zope | 16:59 | |
| *** projekt01 has joined #zope | 17:08 | |
| *** supton has joined #zope | 17:51 | |
| *** MrTango has quit IRC | 18:06 | |
| *** __mac__ has quit IRC | 18:21 | |
| *** agroszer has quit IRC | 18:31 | |
| *** sylvain has quit IRC | 18:47 | |
| *** fredvd has quit IRC | 18:48 | |
| *** JaRoel|4D has quit IRC | 18:54 | |
| *** daMaestro has joined #zope | 19:01 | |
| *** tisto has quit IRC | 19:09 | |
| *** projekt01 has quit IRC | 19:18 | |
| *** JaRoel|4D has joined #zope | 19:18 | |
| *** kevin7kal_ has joined #zope | 19:27 | |
| *** kevin7kal has quit IRC | 19:29 | |
| *** Pumukel has quit IRC | 19:30 | |
| *** maurits has quit IRC | 19:41 | |
| *** giacomos has quit IRC | 19:50 | |
| *** kosh has joined #zope | 20:17 | |
| *** __mac__ has joined #zope | 20:27 | |
| *** __mac__ has quit IRC | 20:39 | |
| *** __mac__ has joined #zope | 20:40 | |
| *** menesis has quit IRC | 20:41 | |
| *** menesis has joined #zope | 21:25 | |
| *** __mac__ has joined #zope | 21:27 | |
| *** __mac__ has quit IRC | 22:25 | |
| *** Arfrever has joined #zope | 22:54 | |
| *** betabug has quit IRC | 23:29 | |
| *** betabug has joined #zope | 23:30 | |
| *** kosh has quit IRC | 23:53 | |
Generated by irclog2html.py 2.15.1 by Marius Gedminas - find it at mg.pov.lt!