| *** stereo_ has joined #zope | 00:03 | |
| *** threesome has joined #zope | 00:07 | |
| *** daMaestro has joined #zope | 00:13 | |
| *** menesis has quit IRC | 00:30 | |
| *** MrTango has joined #zope | 00:36 | |
| *** menesis has joined #zope | 00:44 | |
| *** Arfrever has quit IRC | 00:45 | |
| *** J1m has joined #zope | 00:59 | |
| *** MrTango has quit IRC | 01:00 | |
| *** J1m_ has quit IRC | 01:01 | |
| *** motto has joined #zope | 01:07 | |
| *** Gogo|tty has quit IRC | 01:10 | |
| *** alecm has quit IRC | 01:11 | |
| *** m8 has quit IRC | 01:11 | |
| *** Gogo|tty has joined #zope | 01:11 | |
| *** threesome has quit IRC | 01:14 | |
| *** supton has quit IRC | 01:30 | |
| *** menesis has quit IRC | 01:38 | |
| *** HyperJohnGraham has joined #zope | 01:39 | |
| *** HyperJohnGraham has quit IRC | 01:42 | |
| *** J1m has quit IRC | 01:46 | |
| *** J1m has joined #zope | 01:48 | |
| *** motto has quit IRC | 01:54 | |
| *** jodok has quit IRC | 01:56 | |
| *** jodok has joined #zope | 02:01 | |
| *** jodok has joined #zope | 02:01 | |
| *** J1m has quit IRC | 02:06 | |
| *** kosh has joined #zope | 02:09 | |
| *** RiverRat has quit IRC | 02:23 | |
| *** RiverRat has joined #zope | 02:24 | |
| *** tiwula has quit IRC | 02:30 | |
| *** stereo_ has quit IRC | 02:42 | |
| *** jodok has quit IRC | 02:43 | |
| *** daMaestro has quit IRC | 03:20 | |
| *** do3cc has quit IRC | 03:42 | |
| *** do3cc has joined #zope | 03:54 | |
| *** rbanffy_ has quit IRC | 05:01 | |
| *** mup has quit IRC | 06:09 | |
| *** mup has joined #zope | 06:09 | |
| *** Arfrever has joined #zope | 08:09 | |
| *** kosh has quit IRC | 08:36 | |
| *** yvl has joined #zope | 08:40 | |
| *** threesome has joined #zope | 09:05 | |
| *** jodok has joined #zope | 09:10 | |
| *** avoinea has joined #zope | 09:21 | |
| *** agroszer has joined #zope | 09:25 | |
| *** jodok has quit IRC | 09:26 | |
| *** jodok has joined #zope | 09:28 | |
| *** menesis has joined #zope | 09:39 | |
| *** jodok has quit IRC | 09:40 | |
| *** menesis has quit IRC | 09:42 | |
| *** menesis has joined #zope | 09:43 | |
| *** jodok has joined #zope | 09:44 | |
| *** jodok has joined #zope | 09:44 | |
| *** menesis has quit IRC | 09:45 | |
| *** menesis has joined #zope | 10:11 | |
| *** jodok has quit IRC | 10:13 | |
| *** jodok has joined #zope | 10:14 | |
| *** giacomos has joined #zope | 10:16 | |
| *** jodok has quit IRC | 10:22 | |
| *** jodok has joined #zope | 10:23 | |
| *** Pumukel has joined #zope | 10:25 | |
| *** Arfrever has quit IRC | 10:33 | |
| *** avoinea has quit IRC | 10:48 | |
| *** avoinea has joined #zope | 10:48 | |
| *** maurits has joined #zope | 10:57 | |
| *** avoinea has quit IRC | 11:07 | |
| *** avoinea has joined #zope | 11:08 | |
| *** jodok has quit IRC | 11:19 | |
| *** jodok has joined #zope | 11:24 | |
| *** jodok has quit IRC | 11:25 | |
| *** jodok has joined #zope | 11:32 | |
| *** jodok has joined #zope | 11:32 | |
| *** Arfrever has joined #zope | 11:37 | |
| *** eperez has joined #zope | 11:58 | |
| *** MrTango has joined #zope | 11:59 | |
| *** jodok has quit IRC | 12:04 | |
| *** jodok has joined #zope | 12:24 | |
| *** jodok has quit IRC | 12:26 | |
| *** avoinea has quit IRC | 12:28 | |
| *** jodok has joined #zope | 12:31 | |
| *** avoinea has joined #zope | 12:32 | |
| *** menesis has quit IRC | 12:57 | |
| *** fredvd has joined #zope | 13:01 | |
| *** J1m has joined #zope | 13:33 | |
| *** menesis has joined #zope | 13:39 | |
| *** jodok has quit IRC | 14:05 | |
| *** jodok has joined #zope | 14:07 | |
| *** jodok has joined #zope | 14:07 | |
| *** jodok has quit IRC | 14:09 | |
| *** jodok has joined #zope | 14:12 | |
| *** jodok has quit IRC | 14:13 | |
| *** do3cc has quit IRC | 15:02 | |
| *** do3cc has joined #zope | 15:05 | |
| *** jakke has joined #zope | 15:43 | |
| jakke | hi guys, is there any known, documented way to loop over objects in the ZODB? | 15:43 |
|---|---|---|
| jakke | I started with looping over the transactions, unpickle the objects and get my information that way | 15:44 |
| do3cc | jakke, if you want, you can treat a containerish object like a dictionary | 15:44 |
| jakke | but given a few big plone sites with blobstorage in ZODB, it takes way too much hours to get the information I need | 15:44 |
| do3cc | so [x for x in folder.values()] | 15:44 |
| jakke | do3cc, yeah, but I'd still need access to the zope root and write some kind of tree-walking code | 15:45 |
| jakke | and do the above for containerish objects | 15:45 |
| jakke | still would need a solution for all the other objects | 15:46 |
| do3cc | in Plone, I'd use the catalog | 15:46 |
| jakke | problem is 2 fold | 15:46 |
| jakke | 1. Data.fs is huge, and we have no indication why, blobs are stored externally | 15:46 |
| jakke | 2. we have broken portlets, presumably the id's got lost somehow, and broken pages without id's | 15:47 |
| jakke | but there as well, I have no idea about the scope of the problem | 15:47 |
| do3cc | did you migrate the site from plone3 to four? maybe some datastructures exist as blob and in fs now | 15:47 |
| jakke | we did so, year or 2 ago | 15:47 |
| jakke | basically when P4 came out | 15:47 |
| do3cc | most of the time, broken portlets mean that you removed a product and the code is missing | 15:48 |
| jakke | it's not consequent | 15:48 |
| jakke | most portlets are ok, some aren't | 15:48 |
| jakke | and it includes static-text-portlets | 15:48 |
| do3cc | Do you have a traceback? | 15:48 |
| jakke | guess you can't make much easier portlets | 15:48 |
| jakke | no, no traceback, cause zope/plone doesn't crash on it | 15:49 |
| do3cc | easy and portlets are two different things | 15:49 |
| jakke | and no indications what happened either on user side | 15:49 |
| jakke | do3cc: LOL :) | 15:49 |
| do3cc | I think I remember that there is some catch all to hide problems | 15:49 |
| jakke | so I'm really in the dark here | 15:49 |
| do3cc | Do you see no log entry or no log entry with tracebacks? | 15:49 |
| do3cc | And what is huge? Do you use versions? | 15:50 |
| jakke | not completely, if you have a navigation portlet that tries to access a page without an idea, portlets crash, and consequential plone as well | 15:50 |
| jakke | 16Gb Data.fs, 30Gb blobstorage, yes, we use revisions | 15:50 |
| do3cc | Crash most of the time means that a portlet is not being displayed, but apparently you are seeing different things | 15:51 |
| jakke | but cleaning these (in a reasonable way) does not reduce the Data.fs in a sensible way | 15:51 |
| do3cc | It is not too easy to clean versions | 15:51 |
| jakke | if it has no id (static portlet) it's not shown, we've seen that | 15:51 |
| do3cc | I'd create a copy of the database and try to delete the three tools related to versioning. Then pack and see what is left | 15:52 |
| jakke | but the issue about the navigation portlet I've just mentioned is present and reproducable ... | 15:52 |
| do3cc | You said consequential plone as well, what do you mean by that? | 15:52 |
| jakke | if navigation portlet crashes on an no-id page, the whole page errors-out, not just the portlet | 15:53 |
| do3cc | what is a no-id page? | 15:53 |
| jakke | I'll try to get rid of the history, if only in an attempt to make the datafs manageble | 15:53 |
| jakke | sorry, shortcut for page without an id | 15:53 |
| jakke | it has an id attribute, but the value got lost somehow | 15:53 |
| do3cc | this is not good | 15:56 |
| do3cc | I am really at a loss for suggestions here. I wonder how you might have lost your document ids. | 15:57 |
| *** eperez has quit IRC | 15:58 | |
| do3cc | I mean, you could always retrieve them from the parent folder again, but first I'd try to find out how I managed to loose ids... | 15:58 |
| jakke | for now I don't care how I lost them, I need the service back up | 16:03 |
| jakke | but yes, I need to know how we lost them | 16:04 |
| jakke | we did some packing 2-3 weeks ago | 16:04 |
| *** Bahman has joined #zope | 16:05 | |
| do3cc | I'd just readd the id everywhere then | 16:05 |
| jakke | objects are in a bad state too, so I need to re-create them and recover from a backup | 16:06 |
| jakke | but I need to know which ones are broken | 16:06 |
| jakke | We just bump into one or two every now and then | 16:06 |
| do3cc | are your objects broken in other ways than just missing ids? | 16:07 |
| J1m | jakke, to iterate over all (current) objects in a file storage, you can use record_iternext. | 16:10 |
| jakke | yes, they are in a strange state | 16:10 |
| jakke | J1m, ok, I'll google how to use that one | 16:11 |
| J1m | or look in interfaces.py | 16:11 |
| J1m | for ZODB | 16:11 |
| jakke | that looks like a better approach then looping over the transactions | 16:12 |
| J1m | well, if what you want is current objects... | 16:12 |
| jakke | and a storage is a FileStorage object I suppose? | 16:12 |
| jakke | or could be | 16:12 |
| J1m | Yes. | 16:13 |
| J1m | given the oids, you can then fetch the objects from a connection. | 16:13 |
| *** J1m has quit IRC | 16:14 | |
| *** jodok has joined #zope | 16:14 | |
| *** J1m has joined #zope | 16:14 | |
| J1m | Hm, I got disconnected... | 16:14 |
| J1m | Yes. | 16:15 |
| J1m | given the oids, you can then fetch the objects from a connection. | 16:15 |
| jakke | J1m: thx! | 16:15 |
| jakke | and how is the performance of that call? | 16:15 |
| jakke | cause for looping over the transactions, I had to call the gc explicitly | 16:15 |
| jakke | process got killed after consuming 24G of ram ;-) | 16:15 |
| J1m | The same issue will apply | 16:15 |
| jakke | ok, good to know upfront, saves me an iteration | 16:16 |
| J1m | If this database is being used in production, then of course, you'll want to work on a copy on a separate machine. | 16:17 |
| jakke | that's already covered of course :) | 16:17 |
| *** J1m has quit IRC | 16:18 | |
| *** J1m has joined #zope | 16:18 | |
| jakke | J1m, I suppose I'm in for some of this _p_64 or what was it again exactly to feth the object from the connection, based on the oid? | 16:25 |
| *** kosh has joined #zope | 16:27 | |
| jakke | right, I like | 16:28 |
| jakke | is there any way to find out how many current objects there are in the ZODB? | 16:28 |
| jakke | except looping over and counting | 16:28 |
| agroszer | jakke, for iterating and eating all RAM: | 16:38 |
| agroszer | transaction.savepoint(optimistic=True) | 16:38 |
| agroszer | conn.cacheGC() | 16:38 |
| agroszer | that should free up some RAM | 16:39 |
| agroszer | of course do that only after X iterations | 16:40 |
| jakke | agroszer, great, I've just found 3 of the few people who now this much about ZODB (at least to my impression), THX! | 16:41 |
| *** rbanffy has joined #zope | 16:44 | |
| J1m | jakke, agroszer's suggestion only makes sense if you're modifying data as you go. | 16:45 |
| J1m | are you? | 16:46 |
| jakke | the transaction optimistic savepoint, I figured that bit out yes :) | 16:46 |
| J1m | otherwise, just call cacheGC as you go. | 16:46 |
| jakke | no, first find out what exactly is going on | 16:46 |
| J1m | so no point in savepoints. | 16:46 |
| jakke | figured that out, but thanks for pointing it out | 16:46 |
| J1m | also, you can do len(storage) | 16:47 |
| jakke | and that's the amount of objects? | 16:47 |
| J1m | yes | 16:47 |
| jakke | 14 million + ... | 16:47 |
| *** agroszer has quit IRC | 16:48 | |
| jakke | I think it's going to take a bit time to process all that | 16:48 |
| *** jodok has quit IRC | 17:50 | |
| *** maurits has quit IRC | 18:16 | |
| *** maurits has joined #zope | 18:16 | |
| *** jodok has joined #zope | 18:18 | |
| *** jodok has joined #zope | 18:18 | |
| *** daMaestro has joined #zope | 18:22 | |
| *** fredvd has quit IRC | 18:45 | |
| *** maurits has quit IRC | 19:07 | |
| *** jodok has quit IRC | 19:23 | |
| *** do3cc has quit IRC | 19:24 | |
| *** jodok has joined #zope | 19:27 | |
| *** do3cc has joined #zope | 19:27 | |
| *** kosh has quit IRC | 19:28 | |
| *** do3cc has quit IRC | 19:28 | |
| *** giacomos has quit IRC | 19:41 | |
| *** MrTango has quit IRC | 19:50 | |
| *** threesome has quit IRC | 20:12 | |
| *** rbanffy has quit IRC | 20:55 | |
| *** jodok has quit IRC | 21:17 | |
| *** Pumukel has quit IRC | 21:25 | |
| *** menesis has quit IRC | 21:27 | |
| *** J1m has quit IRC | 21:55 | |
| *** J1m has joined #zope | 21:56 | |
| *** alecm has joined #zope | 22:02 | |
| *** alecm has joined #zope | 22:02 | |
| *** avoinea has quit IRC | 22:03 | |
| *** m8 has joined #zope | 22:04 | |
| *** Bahman has quit IRC | 22:07 | |
| *** J1m_ has joined #zope | 22:17 | |
| *** J1m has quit IRC | 22:17 | |
| *** jodok has joined #zope | 22:18 | |
| *** jodok has joined #zope | 22:18 | |
| *** River_Rat has joined #zope | 23:27 | |
| *** RiverRat has quit IRC | 23:29 | |
| *** kosh has joined #zope | 23:53 | |
Generated by irclog2html.py 2.15.1 by Marius Gedminas - find it at mg.pov.lt!