IRC log of #zope for Tuesday, 2014-08-26

*** stereo_ has joined #zope00:03
*** threesome has joined #zope00:07
*** daMaestro has joined #zope00:13
*** menesis has quit IRC00:30
*** MrTango has joined #zope00:36
*** menesis has joined #zope00:44
*** Arfrever has quit IRC00:45
*** J1m has joined #zope00:59
*** MrTango has quit IRC01:00
*** J1m_ has quit IRC01:01
*** motto has joined #zope01:07
*** Gogo|tty has quit IRC01:10
*** alecm has quit IRC01:11
*** m8 has quit IRC01:11
*** Gogo|tty has joined #zope01:11
*** threesome has quit IRC01:14
*** supton has quit IRC01:30
*** menesis has quit IRC01:38
*** HyperJohnGraham has joined #zope01:39
*** HyperJohnGraham has quit IRC01:42
*** J1m has quit IRC01:46
*** J1m has joined #zope01:48
*** motto has quit IRC01:54
*** jodok has quit IRC01:56
*** jodok has joined #zope02:01
*** jodok has joined #zope02:01
*** J1m has quit IRC02:06
*** kosh has joined #zope02:09
*** RiverRat has quit IRC02:23
*** RiverRat has joined #zope02:24
*** tiwula has quit IRC02:30
*** stereo_ has quit IRC02:42
*** jodok has quit IRC02:43
*** daMaestro has quit IRC03:20
*** do3cc has quit IRC03:42
*** do3cc has joined #zope03:54
*** rbanffy_ has quit IRC05:01
*** mup has quit IRC06:09
*** mup has joined #zope06:09
*** Arfrever has joined #zope08:09
*** kosh has quit IRC08:36
*** yvl has joined #zope08:40
*** threesome has joined #zope09:05
*** jodok has joined #zope09:10
*** avoinea has joined #zope09:21
*** agroszer has joined #zope09:25
*** jodok has quit IRC09:26
*** jodok has joined #zope09:28
*** menesis has joined #zope09:39
*** jodok has quit IRC09:40
*** menesis has quit IRC09:42
*** menesis has joined #zope09:43
*** jodok has joined #zope09:44
*** jodok has joined #zope09:44
*** menesis has quit IRC09:45
*** menesis has joined #zope10:11
*** jodok has quit IRC10:13
*** jodok has joined #zope10:14
*** giacomos has joined #zope10:16
*** jodok has quit IRC10:22
*** jodok has joined #zope10:23
*** Pumukel has joined #zope10:25
*** Arfrever has quit IRC10:33
*** avoinea has quit IRC10:48
*** avoinea has joined #zope10:48
*** maurits has joined #zope10:57
*** avoinea has quit IRC11:07
*** avoinea has joined #zope11:08
*** jodok has quit IRC11:19
*** jodok has joined #zope11:24
*** jodok has quit IRC11:25
*** jodok has joined #zope11:32
*** jodok has joined #zope11:32
*** Arfrever has joined #zope11:37
*** eperez has joined #zope11:58
*** MrTango has joined #zope11:59
*** jodok has quit IRC12:04
*** jodok has joined #zope12:24
*** jodok has quit IRC12:26
*** avoinea has quit IRC12:28
*** jodok has joined #zope12:31
*** avoinea has joined #zope12:32
*** menesis has quit IRC12:57
*** fredvd has joined #zope13:01
*** J1m has joined #zope13:33
*** menesis has joined #zope13:39
*** jodok has quit IRC14:05
*** jodok has joined #zope14:07
*** jodok has joined #zope14:07
*** jodok has quit IRC14:09
*** jodok has joined #zope14:12
*** jodok has quit IRC14:13
*** do3cc has quit IRC15:02
*** do3cc has joined #zope15:05
*** jakke has joined #zope15:43
jakkehi guys, is there any known, documented way to loop over objects in the ZODB?15:43
jakkeI started with looping over the transactions, unpickle the objects and get my information that way15:44
do3ccjakke, if you want, you can treat a containerish object like a dictionary15:44
jakkebut given a few big plone sites with blobstorage in ZODB, it takes way too much hours to get the information I need15:44
do3ccso [x for x in folder.values()]15:44
jakkedo3cc, yeah, but I'd still need access to the zope root and write some kind of tree-walking code15:45
jakkeand do the above for containerish objects15:45
jakkestill would need a solution for all the other objects15:46
do3ccin Plone, I'd use the catalog15:46
jakkeproblem is 2 fold15:46
jakke1. Data.fs is huge, and we have no indication why, blobs are stored externally15:46
jakke2. we have broken portlets, presumably the id's got lost somehow, and broken pages without id's15:47
jakkebut there as well, I have no idea about the scope of the problem15:47
do3ccdid you migrate the site from plone3 to four? maybe some datastructures exist as blob and in fs now15:47
jakkewe did so, year or 2 ago15:47
jakkebasically when P4 came out15:47
do3ccmost of the time, broken portlets mean that you removed a product and the code is missing15:48
jakkeit's not consequent15:48
jakkemost portlets are ok, some aren't15:48
jakkeand it includes static-text-portlets15:48
do3ccDo you have a traceback?15:48
jakkeguess you can't make much easier portlets15:48
jakkeno, no traceback, cause zope/plone doesn't crash on it15:49
do3cceasy and portlets are two different things15:49
jakkeand no indications what happened either on user side15:49
jakkedo3cc: LOL :)15:49
do3ccI think I remember that there is some catch all to hide problems15:49
jakkeso I'm really in the dark here15:49
do3ccDo you see no log entry or no log entry with tracebacks?15:49
do3ccAnd what is huge? Do you use versions?15:50
jakkenot completely, if you have a navigation portlet that tries to access a page without an idea, portlets crash, and consequential plone as well15:50
jakke16Gb Data.fs, 30Gb blobstorage, yes, we use revisions15:50
do3ccCrash most of the time means that a portlet is not being displayed, but apparently you are seeing different things15:51
jakkebut cleaning these (in a reasonable way) does not reduce the Data.fs in a sensible way15:51
do3ccIt is not too easy to clean versions15:51
jakkeif it has no id (static portlet) it's not shown, we've seen that15:51
do3ccI'd create a copy of the database and try to delete the three tools related to versioning. Then pack and see what is left15:52
jakkebut the issue about the navigation portlet I've just mentioned is present and reproducable ...15:52
do3ccYou said consequential plone as well, what do you mean by that?15:52
jakkeif navigation portlet crashes on an no-id page, the whole page errors-out, not just the portlet15:53
do3ccwhat is a no-id page?15:53
jakkeI'll try to get rid of the history, if only in an attempt to make the datafs manageble15:53
jakkesorry, shortcut for page without an id15:53
jakkeit has an id attribute, but the value got lost somehow15:53
do3ccthis is not good15:56
do3ccI am really at a loss for suggestions here. I wonder how you might have lost your document ids.15:57
*** eperez has quit IRC15:58
do3ccI mean, you could always retrieve them from the parent folder again, but first I'd try to find out how I managed to loose ids...15:58
jakkefor now I don't care how I lost them, I need the service back up16:03
jakkebut yes, I need to know how we lost them16:04
jakkewe did some packing 2-3 weeks ago16:04
*** Bahman has joined #zope16:05
do3ccI'd just readd the id everywhere then16:05
jakkeobjects are in a bad state too, so I  need to re-create them and recover from a backup16:06
jakkebut I need to know which ones are broken16:06
jakkeWe just bump into one or two every now and then16:06
do3ccare your objects broken in other ways than just missing ids?16:07
J1mjakke, to iterate over all (current) objects in a file storage, you can use record_iternext.16:10
jakkeyes, they are in a strange state16:10
jakkeJ1m, ok, I'll google how to use that one16:11
J1mor look in interfaces.py16:11
J1mfor ZODB16:11
jakkethat looks like a better approach then looping over the transactions16:12
J1mwell, if what you want is current objects...16:12
jakkeand a storage is a FileStorage object I suppose?16:12
jakkeor could be16:12
J1mgiven the oids, you can then fetch the objects from a connection.16:13
*** J1m has quit IRC16:14
*** jodok has joined #zope16:14
*** J1m has joined #zope16:14
J1mHm, I got disconnected...16:14
J1mgiven the oids, you can then fetch the objects from a connection.16:15
jakkeJ1m: thx!16:15
jakkeand how is the performance of that call?16:15
jakkecause for looping over the transactions, I had to call the gc explicitly16:15
jakkeprocess got killed after consuming 24G of ram ;-)16:15
J1mThe same issue will apply16:15
jakkeok, good to know upfront, saves me an iteration16:16
J1mIf this database is being used in production, then of course, you'll want to work on a copy on a separate machine.16:17
jakkethat's already covered of course :)16:17
*** J1m has quit IRC16:18
*** J1m has joined #zope16:18
jakkeJ1m, I suppose I'm in for some of this _p_64 or what was it again exactly to feth the object from the connection, based on the oid?16:25
*** kosh has joined #zope16:27
jakkeright, I like16:28
jakkeis there any way to find out how many current objects there are in the ZODB?16:28
jakkeexcept looping over and counting16:28
agroszerjakke, for iterating and eating all RAM:16:38
agroszerthat should free up some RAM16:39
agroszerof course do that only after X iterations16:40
jakkeagroszer, great, I've just found 3 of the few people who now this much about ZODB (at least to my impression), THX!16:41
*** rbanffy has joined #zope16:44
J1mjakke, agroszer's suggestion only makes sense if you're modifying data as you go.16:45
J1mare you?16:46
jakkethe transaction optimistic savepoint, I figured that bit out yes :)16:46
J1motherwise, just call cacheGC as you go.16:46
jakkeno, first find out what exactly is going on16:46
J1mso no point in savepoints.16:46
jakkefigured that out, but thanks for pointing it out16:46
J1malso, you can do len(storage)16:47
jakkeand that's the amount of objects?16:47
jakke14 million + ...16:47
*** agroszer has quit IRC16:48
jakkeI think it's going to take a bit time to process all that16:48
*** jodok has quit IRC17:50
*** maurits has quit IRC18:16
*** maurits has joined #zope18:16
*** jodok has joined #zope18:18
*** jodok has joined #zope18:18
*** daMaestro has joined #zope18:22
*** fredvd has quit IRC18:45
*** maurits has quit IRC19:07
*** jodok has quit IRC19:23
*** do3cc has quit IRC19:24
*** jodok has joined #zope19:27
*** do3cc has joined #zope19:27
*** kosh has quit IRC19:28
*** do3cc has quit IRC19:28
*** giacomos has quit IRC19:41
*** MrTango has quit IRC19:50
*** threesome has quit IRC20:12
*** rbanffy has quit IRC20:55
*** jodok has quit IRC21:17
*** Pumukel has quit IRC21:25
*** menesis has quit IRC21:27
*** J1m has quit IRC21:55
*** J1m has joined #zope21:56
*** alecm has joined #zope22:02
*** alecm has joined #zope22:02
*** avoinea has quit IRC22:03
*** m8 has joined #zope22:04
*** Bahman has quit IRC22:07
*** J1m_ has joined #zope22:17
*** J1m has quit IRC22:17
*** jodok has joined #zope22:18
*** jodok has joined #zope22:18
*** River_Rat has joined #zope23:27
*** RiverRat has quit IRC23:29
*** kosh has joined #zope23:53

Generated by 2.15.1 by Marius Gedminas - find it at!