IRC log of #zope for Tuesday, 2015-06-23

*** kosh_ has joined #zope00:40
*** kosh has quit IRC00:43
*** mup has quit IRC01:09
*** mup_ has joined #zope01:09
*** mup_ is now known as mup01:09
*** stereo_ has quit IRC01:26
*** kosh_ has quit IRC01:49
*** supton has joined #zope02:31
*** supton has quit IRC02:45
*** Bahman has joined #zope02:47
*** Bahman has quit IRC02:50
*** menesis has quit IRC05:59
*** alecm has quit IRC06:11
*** Arfrever has joined #zope06:32
*** alecm has joined #zope07:19
*** alecm has joined #zope07:19
*** giacomos has joined #zope07:45
*** supton has joined #zope08:10
*** supton has quit IRC08:57
*** __mac__ has joined #zope09:04
*** tisto has joined #zope09:11
*** JaRoel|4D has quit IRC09:40
*** alecm has quit IRC09:47
*** alecm has joined #zope09:47
*** JaRoel|4D has joined #zope10:04
*** PeterZ1 has joined #zope10:07
*** PeterZ1 has left #zope10:07
*** kosh has joined #zope10:20
*** sylvain has joined #zope10:35
sylvainHello mgedmin10:36
mgedminhi sylvain10:38
sylvainyou saw my large pull request10:38
sylvainI will fix all the silly mistakes about format and english10:38
sylvainbut I wanted to explain why a couple of things are like this10:39
sylvainI will put some comments on github as well10:39
mgedminthat sounds like a good idea10:39
sylvainin fact10:39
sylvainwe used Relstorage history free10:39
sylvainand did a pack using the official pack script10:39
sylvainon a write heavy database10:39
sylvainthat toke a long time and created all kind of POSKey error10:40
mgedminI actually found some of my questions answered in the README10:40
sylvainso we wanted to browser the database to find the objects pointing to the missing one10:40
mgedminit's a shame github doesn't let me review my comments and questions before publishing them10:40
sylvainwell I am a bit confused how to answer all of them, but I will try to figure it out10:41
sylvainwe added the zodbcheck to find them and start browsing the objects from the missing ones10:41
sylvainthe main issue is10:42
sylvainmost of the code only works for Relstorage history free10:42
sylvainif you have multiple versions of objects10:42
sylvainall the algorithms used stop working :/10:42
mgedminoh! I see!10:42
sylvainand we did that way10:43
sylvainbecause we have a large database10:43
sylvainand did not wait weeks to do things10:43
sylvainthat's why we precompute the index with the zodbcheck10:43
sylvainand while doing the check we check for poskey errors10:43
sylvainwe do that every day on the database10:44
mgedminhow large is your database?10:44
sylvainand if we find any, we have the index and open in zodbbrowser10:44
sylvain22 millions of objects10:44
sylvainthose are small, so it is only 6 GB if packed10:45
sylvainsame thing for the search, that's in a script because it takes some time to go through the database10:46
mgedminthat's fine10:46
sylvainand the pack script, well, we can definitely move it outside of this package10:46
sylvainbut we tested some other solution, like relstorage_packer10:46
mgedminit's just that longterm I want this to be in the web app, but done in a background task10:46
sylvainI can understand that10:46
sylvainthat would be a nice feature10:46
mgedmin(in fact I want some of the current history crawling code to be done in the background)10:46
sylvainbut without index, it does take some time10:47
sylvainso the pack, we tested some other script, that toke 4 or 5 days to run10:47
sylvainand after we thought we had most of the information in the zodbcheck database10:47
sylvainand wrote this SQL query, that takes 15 minutes to find out that to do delete on a fast computer10:48
sylvainthat's why it is there10:48
sylvainbut again10:48
sylvainall of this makes sense only for history free, like in relstorage10:49
sylvainI am not sure if there is any other backend that have this option10:49
mgedminMemoryStorage? :)10:49
sylvainwe wondered if we should make a pull request for a long time10:49
sylvainbecause it is kind of specific10:49
mgedminI would love for ZODB to have an API to iterate over all the objects (not transactions)10:49
sylvainthere is one10:50
sylvainbut that is not implemented by all storage10:50
mgedminwhat is it?10:50
sylvainyou can do this on FileStorage I believe10:50
sylvainlet me look of it10:50
sylvainI think that's the record_iternext10:52
sylvainlike I used in zodbupdate10:52
sylvainI guess it would be possible to make a custom solution to iterate of FileStorage for instance10:53
sylvainto have the zodbcheck script works there10:53
mgedminah, I see10:54
mgedminit's part of an interface:
sylvainbut only implemented by FileStorage10:55
sylvainRelstorage does not have this10:55
sylvainit only have with transactions10:55
mgedminnor ClientStorage :/10:55
mgedminand the API is not exactly beautiful10:56
mgedminoh well10:56
mgedminmaybe someone will create a ZODB PR for this ;)10:56
sylvainyes but if you use ZEO, you most probably have a FileStorage and could run the zodbcheck directly on it10:56
mgedminso, I'm very happy that you submitted the PR10:56
sylvainI will cleaned it up10:56
sylvainwe wrote most of this code in a hurry trying to solve problems10:57
mgedminI planned to build an in-memory index for backward references10:57
mgedminbut an on-disk index is maybe even better10:57
mgedminsay rather "this code was battle-tested" ;)10:57
sylvainwe have an in-memory one we used in tests10:57
mgedminalso, I'd rather have multiple PRs for little bits of this than this one cleaned up10:57
sylvainto check that if we delete an object, there are no references left to it10:57
sylvainyes, I can maybe investigate how to do this multiple PR, but I am not sure I know how to do ti10:58
mgedmincreate new branch, cherry-pick the relevant commits, maybe git rebase -i to make it pretty and/or squash related bits/fixups10:59
mgedminrepeat for other PRs10:59
mgedminI like to start with smallest changes first, because those are easiest10:59
sylvainI will first make a bunch of commit to fix all the comments you already put there11:00
sylvainbecause most of them makes sense11:00
mgedminhm, actually I'm not sure what the best approach is11:00
mgedminmaybe fix up typos etc first?11:00
mgedminmaybe instead of cherry-picking use a merge tool to pick specific changes?11:00
mgedminthen when a standalone PR is merged, maybe github will manage to cleanly update this one to show only remaining changes?11:01
sylvainwell I should already fix all the typo and format issues, because I will still to do it anyway11:01
sylvainand after I will see to make nicer pull requests11:02
mgedminand then there's always the approach of creating a new branch and reimplementing the feature from scratch, using the existing branch as a prototype (and liberally copy-pasting code from it)11:02
mgedminthis requires practically no git-fu11:02
sylvainI will see11:02
mgedminpick whichever's easiest for you11:03
sylvainI will fix some issues today, but I have some other work to do as well11:03
* mgedmin nods11:03
sylvainso it will takes some time11:03
mgedminI have some ideas how to make the reference DB building work with multiple object versions11:03
mgedminI'll comment on the PR11:04
*** kosh has quit IRC11:07
*** agroszer has joined #zope11:08
*** MrTango has joined #zope11:10
*** kosh has joined #zope11:17
*** Arfrever has quit IRC11:20
*** projekt01 has joined #zope11:23
*** kosh has quit IRC11:28
*** menesis has joined #zope12:27
*** Pumukel has joined #zope12:35
*** maurits has joined #zope12:48
*** fredvd has joined #zope12:58
*** projekt01 has quit IRC13:24
*** maurits has quit IRC13:37
*** maurits has joined #zope14:42
*** yvl has joined #zope15:08
*** menesis has quit IRC15:22
*** giacomos has quit IRC15:24
*** menesis has joined #zope15:34
*** giacomos has joined #zope15:35
*** yvl has quit IRC16:14
*** sm has quit IRC16:37
*** menesis has quit IRC16:43
*** menesis has joined #zope16:51
*** sm has joined #zope16:59
*** projekt01 has joined #zope17:08
*** supton has joined #zope17:51
*** MrTango has quit IRC18:06
*** __mac__ has quit IRC18:21
*** agroszer has quit IRC18:31
*** sylvain has quit IRC18:47
*** fredvd has quit IRC18:48
*** JaRoel|4D has quit IRC18:54
*** daMaestro has joined #zope19:01
*** tisto has quit IRC19:09
*** projekt01 has quit IRC19:18
*** JaRoel|4D has joined #zope19:18
*** kevin7kal_ has joined #zope19:27
*** kevin7kal has quit IRC19:29
*** Pumukel has quit IRC19:30
*** maurits has quit IRC19:41
*** giacomos has quit IRC19:50
*** kosh has joined #zope20:17
*** __mac__ has joined #zope20:27
*** __mac__ has quit IRC20:39
*** __mac__ has joined #zope20:40
*** menesis has quit IRC20:41
*** menesis has joined #zope21:25
*** __mac__ has joined #zope21:27
*** __mac__ has quit IRC22:25
*** Arfrever has joined #zope22:54
*** betabug has quit IRC23:29
*** betabug has joined #zope23:30
*** kosh has quit IRC23:53

Generated by 2.15.1 by Marius Gedminas - find it at!