| fgs | it looks to me that once you hit target the situation gets bad. _check_blob_cache_size() will clean just enough, as soon as a new file gets in there it will check again | 00:02 |
|---|---|---|
| fgs | if the size of your files are more or less similar you'll have the thread running continuously | 00:03 |
| fgs | and that will cause continuous locks | 00:03 |
| J1m | I didn't say there might not be contention. If there wasn't contention, you almost wouldn't need the lock. | 00:03 |
| J1m | No, not really. | 00:04 |
| J1m | In any case, it's the directories that matter, not the files. | 00:04 |
| J1m | An improvement might be to release a directory lock while downloading from the server. You'd have to get it again before adding the downloaded file. | 00:05 |
| fgs | the check removes enough files so size <= target | 00:07 |
| J1m | The blob cache uses 997 buckets. You'd have to have quite a bit of activity to get contention on that many buckets. | 00:07 |
| J1m | Yes, but it only locks a directory while it's removing a file from it. | 00:08 |
| fgs | crawlers getting all the files | 00:08 |
| fgs | no | 00:08 |
| J1m | yes | 00:08 |
| fgs | it also does it when it's downloading it | 00:08 |
| fgs | look at _lock_blob | 00:08 |
| fgs | both _lock_blob (used in 2 other methods) | 00:08 |
| fgs | and _check_blob_size_cache | 00:08 |
| fgs | lock on the directory | 00:08 |
| J1m | yes, it also gets locked while downloading | 00:09 |
| J1m | but the crawler only locks one directory at a time. | 00:09 |
| J1m | and you'd have to be downloading hundreds of blobs simultaneously to get contention due to downloads. | 00:09 |
| fgs | where is that assumption coming from? | 00:09 |
| J1m | what assumption? | 00:10 |
| fgs | the crawler is requesting several files simultaneously | 00:10 |
| fgs | so it's locking several directories at the same time | 00:10 |
| fgs | and the cache size is also kicking it | 00:10 |
| J1m | no, look at the code | 00:10 |
| J1m | It decides which blobs to reap without getting a lock. | 00:11 |
| J1m | Then it iterates over each one and for each one it locks it's directory while it's removing it. | 00:11 |
| fgs | i must be blind. it gets the list, iterates over it and tries to lock the directory for each blob | 00:12 |
| *** mwilkes|away is now known as MatthewWilkes | 00:20 | |
| fgs | oh well, i will re-read the code tomorrow when i have some sleep. | 00:23 |
| fgs | thanks for replying | 00:23 |
| *** sp0cksbeard has quit IRC | 00:43 | |
| *** Spanktar has quit IRC | 00:56 | |
| *** Spanktar has joined #zope | 00:59 | |
| *** MatthewWilkes is now known as mwilkes|away | 01:02 | |
| *** Arfrever has quit IRC | 01:10 | |
| *** mwilkes|away is now known as MatthewWilkes | 01:13 | |
| *** MatthewWilkes is now known as mwilkes|away | 01:34 | |
| *** Spanktar has quit IRC | 01:43 | |
| *** kosh has quit IRC | 01:52 | |
| *** fdrake has quit IRC | 01:54 | |
| *** J1m has quit IRC | 01:54 | |
| *** _mup_ has quit IRC | 02:00 | |
| *** _mup_ has joined #zope | 02:00 | |
| *** Spanktar has joined #zope | 02:14 | |
| *** aodag has joined #zope | 02:40 | |
| *** aodag_ has joined #zope | 02:47 | |
| *** aodag has quit IRC | 02:49 | |
| *** daMaestro has quit IRC | 02:50 | |
| *** tiwula has quit IRC | 02:52 | |
| *** Spanktar has quit IRC | 03:06 | |
| *** polysix has quit IRC | 03:17 | |
| *** polysix has joined #zope | 03:18 | |
| *** mr_jolly has joined #zope | 03:18 | |
| *** thetet has quit IRC | 03:29 | |
| *** mcdonc has joined #zope | 03:33 | |
| *** mr_jolly has quit IRC | 03:36 | |
| *** mr_jolly has joined #zope | 03:38 | |
| *** mr_jolly has quit IRC | 03:46 | |
| *** mr_jolly has joined #zope | 03:51 | |
| *** kosh has joined #zope | 03:57 | |
| *** mcdonc has quit IRC | 04:14 | |
| *** nueces has joined #zope | 04:22 | |
| *** fdrake has joined #zope | 04:53 | |
| *** RichyB has quit IRC | 05:14 | |
| *** tiwula has joined #zope | 05:17 | |
| *** aodag_ has quit IRC | 06:04 | |
| *** mr_jolly has quit IRC | 06:11 | |
| *** jham_ has joined #zope | 06:19 | |
| *** jham has quit IRC | 06:19 | |
| *** aodag has joined #zope | 06:45 | |
| *** __mac__ has joined #zope | 06:50 | |
| *** yvl has quit IRC | 06:54 | |
| *** __mac__ has quit IRC | 07:02 | |
| *** __mac__ has joined #zope | 07:07 | |
| *** __mac__ has quit IRC | 07:26 | |
| *** nueces has quit IRC | 07:48 | |
| *** kosh has quit IRC | 07:50 | |
| *** __mac__ has joined #zope | 07:54 | |
| *** __mac__ has quit IRC | 07:56 | |
| *** zenwryly has quit IRC | 08:00 | |
| *** aodag_ has joined #zope | 09:03 | |
| *** aodag has quit IRC | 09:03 | |
| *** tiwula has quit IRC | 09:11 | |
| *** miano has joined #zope | 09:16 | |
| *** miano_ has quit IRC | 09:17 | |
| *** aodag_ has quit IRC | 09:27 | |
| *** aodag has joined #zope | 09:28 | |
| *** agroszer has joined #zope | 09:36 | |
| *** menesis has quit IRC | 09:45 | |
| *** tisto has joined #zope | 09:56 | |
| *** giacomos has joined #zope | 10:12 | |
| *** thetet has joined #zope | 10:12 | |
| *** tisto has quit IRC | 10:17 | |
| *** tisto has joined #zope | 10:18 | |
| *** goschtl has joined #zope | 10:36 | |
| *** eperez has joined #zope | 10:46 | |
| *** __mac__ has joined #zope | 10:58 | |
| *** avoinea has joined #zope | 10:59 | |
| *** sunew has joined #zope | 11:00 | |
| *** MrTango has joined #zope | 11:08 | |
| *** __mac__ has quit IRC | 11:10 | |
| *** aodag_ has joined #zope | 11:15 | |
| *** aodag has quit IRC | 11:16 | |
| *** aodag_ has quit IRC | 11:30 | |
| *** giacomos has quit IRC | 11:53 | |
| *** Gogo|gui has joined #zope | 11:53 | |
| *** AdamDangoor has joined #zope | 12:02 | |
| *** evilbungle has joined #zope | 12:08 | |
| *** giacomos has joined #zope | 12:08 | |
| *** mr_jolly has joined #zope | 12:09 | |
| *** giacomos has quit IRC | 12:10 | |
| *** goschtl has quit IRC | 12:10 | |
| *** thetet has quit IRC | 12:10 | |
| *** _mup_ has quit IRC | 12:10 | |
| *** kiorky has quit IRC | 12:10 | |
| *** _mup_ has joined #zope | 12:10 | |
| *** jakke1 has joined #zope | 12:11 | |
| *** jakke1 has left #zope | 12:11 | |
| *** jakke has joined #zope | 12:12 | |
| *** giacomos has joined #zope | 12:13 | |
| *** goschtl has joined #zope | 12:13 | |
| *** thetet has joined #zope | 12:13 | |
| *** kiorky has joined #zope | 12:13 | |
| *** fgs has quit IRC | 12:13 | |
| jakke | Hi guys, I'm looking for a way to register an adapter from zope.schema interface to a python basetype (dict, list, string, ...) | 12:14 |
| *** AdamDangoor has quit IRC | 12:14 | |
| jakke | where can I find the interfaces for those basic python types? because I have to register the adapter for a specific interface | 12:14 |
| *** AdamDangoor has joined #zope | 12:14 | |
| *** fgs has joined #zope | 12:15 | |
| *** mr_jolly has quit IRC | 12:18 | |
| *** mr_jolly_ has joined #zope | 12:21 | |
| *** mr_jolly_ is now known as mr_jolly | 12:21 | |
| *** mwilkes|away is now known as MatthewWilkes | 12:22 | |
| *** mr_jolly has left #zope | 12:23 | |
| *** RichyB has joined #zope | 12:33 | |
| *** AdamDangoor has quit IRC | 12:37 | |
| *** AdamDangoor has joined #zope | 12:37 | |
| *** maurits has joined #zope | 12:44 | |
| *** JT has quit IRC | 12:46 | |
| *** JT has joined #zope | 12:47 | |
| *** zagy has quit IRC | 13:09 | |
| *** J1m has joined #zope | 13:13 | |
| *** zagy has joined #zope | 13:21 | |
| *** zagy has quit IRC | 13:27 | |
| *** fdrake has quit IRC | 13:33 | |
| *** AdamDangoor has quit IRC | 13:34 | |
| *** AdamDangoor has joined #zope | 13:34 | |
| *** zagy has joined #zope | 13:36 | |
| *** mr_jolly has joined #zope | 14:02 | |
| *** LeoRochael has joined #zope | 14:07 | |
| *** crazydiamond has joined #zope | 14:08 | |
| *** yvl has joined #zope | 14:10 | |
| *** maurits has quit IRC | 14:18 | |
| *** AdamDangoor has quit IRC | 14:19 | |
| *** AdamDangoor has joined #zope | 14:19 | |
| *** giacomos has quit IRC | 14:23 | |
| *** giacomos has joined #zope | 14:23 | |
| *** maurits has joined #zope | 14:28 | |
| *** giacomos has quit IRC | 14:28 | |
| *** giacomos has joined #zope | 14:28 | |
| *** goschtl has quit IRC | 14:42 | |
| *** giacomos has quit IRC | 15:03 | |
| *** giacomos has joined #zope | 15:19 | |
| *** J1m has quit IRC | 15:21 | |
| *** supton has quit IRC | 15:23 | |
| *** thetet has quit IRC | 15:30 | |
| *** mr_jolly has quit IRC | 15:31 | |
| *** fdrake has joined #zope | 15:42 | |
| *** AdamDangoor has quit IRC | 15:53 | |
| *** AdamDangoor has joined #zope | 15:53 | |
| *** menesis has joined #zope | 15:59 | |
| *** J1m has joined #zope | 16:13 | |
| fgs | so i've looked at the blob locking stuff today | 16:14 |
| fgs | first, there is an issue with zc.lockfile. it's logging the exception and raising | 16:14 |
| fgs | so while the raise is caught the logger.exception is filling my event log | 16:15 |
| *** tisto has quit IRC | 16:16 | |
| fgs | the locking contention is only happening when the blob cache size is being cleaned | 16:16 |
| *** eperez has quit IRC | 16:19 | |
| fgs | i'm waiting for the crawler to kick again to have better numbers but judging by the number of entries in the event log it's kinda bad | 16:21 |
| *** supton has joined #zope | 16:22 | |
| *** sp0cksbeard has joined #zope | 16:25 | |
| *** crazydiamond has quit IRC | 16:26 | |
| *** fredvd has joined #zope | 16:33 | |
| moo-_- | fgs: you might want to try your luck on stackoverflow.com | 16:35 |
| moo-_- | here it's quiet like in a ghost town | 16:35 |
| fgs | i was talking to J1m last night | 16:35 |
| fgs | that was mostly directed to him | 16:35 |
| fgs | and it was not a question | 16:36 |
| moo-_- | fgs: aha :) | 16:37 |
| moo-_- | fgs: my backlog is so full of join and parts entries so missed the earlier conversation, sorry :( | 16:37 |
| fgs | no worries :) | 16:37 |
| fgs | i guess in my case the problem is the crawler is fetching things in the same order over and over again | 16:40 |
| fgs | the last object being crawled causes the oldest item(s) to be cleaned while the crawler goes back to the first one | 16:41 |
| fgs | (oldest one) but that directory is locked by the cleaner thread | 16:42 |
| fgs | perhaps in my case 10% is not enough | 16:42 |
| fgs | but i will see if that's what's happening once my blob cache hits 1G | 16:43 |
| *** miano has quit IRC | 16:46 | |
| *** kosh has joined #zope | 16:53 | |
| *** AdamDangoor has quit IRC | 16:57 | |
| *** AdamDangoor has joined #zope | 16:58 | |
| *** Gogo|gui has quit IRC | 17:02 | |
| *** crazydiamond has joined #zope | 17:08 | |
| *** crazydiamond has quit IRC | 17:10 | |
| *** crazydiamond has joined #zope | 17:11 | |
| *** mr_jolly has joined #zope | 17:14 | |
| *** crazydiamond has quit IRC | 17:20 | |
| *** menesis has quit IRC | 17:23 | |
| *** menesis has joined #zope | 17:30 | |
| *** zagy has quit IRC | 17:32 | |
| *** sunew has quit IRC | 17:40 | |
| supton | given a list of transaction records for ZODB, and contained data records, how can I tell if an oid is deleted in the most recent transaction? Does Connection.get() raise a KeyError on an oid that is in some old transaction, but no longer existing in the most current? | 17:49 |
| benji | supton: I don't remember the exact details, but something like that. Trying it out to see should be easy. | 17:59 |
| J1m | supton, for the first part of your question, delete data records have a data length of 0. | 18:00 |
| J1m | for the second part, yes | 18:00 |
| J1m | of course, afaik, the only way to get delete records is to use zc.zodbdgc. | 18:01 |
| J1m | also, ftr, I hand in #zodb, although I never tols anyone before and it's kinda lonely there :) | 18:02 |
| supton | J1m: good to know, thanks | 18:02 |
| *** AdamDangoor has quit IRC | 18:11 | |
| *** AdamDangoor has joined #zope | 18:11 | |
| *** jakke has left #zope | 18:19 | |
| *** crazydiamond has joined #zope | 18:19 | |
| *** kosh has quit IRC | 18:24 | |
| *** supersheep has quit IRC | 18:35 | |
| *** agroszer has quit IRC | 18:39 | |
| *** tiwula has joined #zope | 19:01 | |
| *** giacomos has quit IRC | 19:03 | |
| *** goschtl has joined #zope | 19:05 | |
| *** Gogo|gui has joined #zope | 19:07 | |
| *** fredvd has quit IRC | 19:09 | |
| *** Spanktar has joined #zope | 19:24 | |
| *** maurits has quit IRC | 19:27 | |
| *** Gogo|gui has quit IRC | 19:31 | |
| *** Gogo|gui1 has joined #zope | 19:31 | |
| *** Gogo|gui1 has quit IRC | 19:33 | |
| *** Gogo|gui has joined #zope | 19:33 | |
| *** Gogo|gui has quit IRC | 19:35 | |
| *** Gogo|gui has joined #zope | 19:37 | |
| *** nande has joined #zope | 19:47 | |
| *** Gogo|gui has quit IRC | 19:48 | |
| *** AdamDangoor has quit IRC | 19:52 | |
| *** AdamDangoor has joined #zope | 19:52 | |
| *** dayne has joined #zope | 20:00 | |
| *** nueces has joined #zope | 20:10 | |
| *** Gogo|gui has joined #zope | 20:18 | |
| *** evilbungle has quit IRC | 20:20 | |
| *** AdamDangoor_ has joined #zope | 20:24 | |
| *** AdamDangoor has quit IRC | 20:26 | |
| *** Gogo|gui has quit IRC | 20:28 | |
| *** MatthewWilkes is now known as mwilkes|away | 20:31 | |
| *** mr_jolly has quit IRC | 20:43 | |
| *** MrTango has quit IRC | 20:48 | |
| *** mr_jolly has joined #zope | 20:49 | |
| *** davisagli has joined #zope | 21:00 | |
| *** davisagli has left #zope | 21:00 | |
| *** mr_jolly has quit IRC | 21:02 | |
| *** KageSenshi has quit IRC | 21:06 | |
| *** avn has quit IRC | 21:06 | |
| *** KageSenshi has joined #zope | 21:07 | |
| *** Gogo|gui has joined #zope | 21:09 | |
| *** Gogo|gui1 has joined #zope | 21:22 | |
| *** Gogo|gui has quit IRC | 21:24 | |
| *** zagy has joined #zope | 21:37 | |
| *** maurits has joined #zope | 21:39 | |
| *** avoinea has quit IRC | 21:57 | |
| *** nueces has quit IRC | 21:59 | |
| *** Gogo|gui1 has quit IRC | 21:59 | |
| *** nueces has joined #zope | 21:59 | |
| *** nueces has quit IRC | 22:08 | |
| *** nueces has joined #zope | 22:08 | |
| *** roq_ has left #zope | 22:32 | |
| *** LeoRochael has quit IRC | 22:37 | |
| mgedmin | J1m, could I get a copy of authors.txt for svn->git conversion? | 22:39 |
| mgedmin | I promise not to use it for evil | 22:39 |
| J1m | heh, of course | 22:39 |
| J1m | but it'll be just a snapshot | 22:40 |
| *** AdamDangoor has joined #zope | 22:42 | |
| *** AdamDangoor has quit IRC | 22:45 | |
| *** goschtl has quit IRC | 22:49 | |
| *** daMaestro has joined #zope | 22:52 | |
| *** Gogo|gui has joined #zope | 22:59 | |
| *** Gogo|gui has quit IRC | 23:04 | |
| *** supton_ has joined #zope | 23:13 | |
| *** supton_ has quit IRC | 23:14 | |
| *** supton has quit IRC | 23:17 | |
| *** m8 has joined #zope | 23:30 | |
| *** avn has joined #zope | 23:40 | |
Generated by irclog2html.py 2.15.1 by Marius Gedminas - find it at mg.pov.lt!