| *** JaRoel|4d has joined #zodb | 00:13 | |
| *** Ilja has joined #zodb | 00:42 | |
| Ilja | hi, is this chat alive? | 00:42 |
|---|---|---|
| J1m_ | yes | 00:59 |
| Ilja | I know it's most probably Sunday and people are off - but is it generally a populated chat? I'm looking for help/consultant with good understanding of ZODB to help fix issue with the way we've integrated twisted and ZODB | 01:10 |
| Ilja | getting hit by consistency errors and not sure "whom to blame" | 01:10 |
| J1m_ | If you're using ZODB with twisted, then you almost certainly don't want to use the default thread-based tranasction manager. | 01:11 |
| Ilja | hm, can you hint me with more keywords? we are using threadpool, actually | 01:12 |
| J1m_ | as far as a consultant, you might contact agendaless and see if they offer any ZODB consulting. | 01:12 |
| J1m_ | well, I know nothing of your architecture, so I was just guessing. | 01:13 |
| Ilja | to put it short: https://github.com/opennode/opennode-management/blob/master/opennode/oms/zodb/db.py#L165 | 01:13 |
| J1m_ | if you're only accessing ZODB from workers in your thread pool, then I can't think of any reason you'd run into trouble. | 01:13 |
| Ilja | methods that access zodb are decorated with @transact | 01:13 |
| Ilja | it feels that it might be related to MVCC | 01:14 |
| Ilja | and connections not being 'cleaned up' properly | 01:14 |
| J1m_ | I don't see any connection management in that code. | 01:16 |
| J1m_ | No offence, but I'm not up for a deep dive in your code. :) | 01:16 |
| J1m_ | if you can think of any questions ... | 01:17 |
| Ilja | yes, sorry | 01:17 |
| J1m_ | np | 01:18 |
| Ilja | so, connection is essentially created once: https://github.com/opennode/opennode-management/blob/master/opennode/oms/zodb/db.py#L142 | 01:18 |
| J1m_ | That's your problem. | 01:19 |
| Ilja | ok, let me rephrase, it's a bit late here and my ordering of thoughts is a bit out of order :) | 01:19 |
| J1m_ | Each thread needs its own connection. | 01:19 |
| Ilja | hm | 01:19 |
| J1m_ | You should open the connection in your transact decorator. | 01:20 |
| J1m_ | and close it. | 01:20 |
| Ilja | how expensive is this on average? we have quite a lot of functions yielded, each one in its own transact | 01:21 |
| Ilja | decorator | 01:21 |
| J1m_ | Better yet, use the db.transaction context manager. | 01:21 |
| J1m_ | It's not expensive. Connections are pooled. | 01:22 |
| J1m_ | Just keep in mind that each connection has it's own partial mirror of the database. | 01:22 |
| J1m_ | so if you're accessing a lot of data, you'll want to limit the number of threads/connections. | 01:22 |
| J1m_ | This is essentially the same as software transactional memory, except that the memory is also saved persistently. | 01:23 |
| Ilja | hm, actually, we already have a single connection per thread, sorry for confusion. | 01:24 |
| Ilja | i.e. connection is thread-local | 01:24 |
| Ilja | our internal discussion suggests that the warning in an old post / http://twistedmatrix.com/pipermail/twisted-python/2005-November/011967.html - might still be relevant | 01:25 |
| Ilja | I don't want to pollute this chat with technical details, which might be specific to our design - (though imho a correct twisted + zodb integration example would be very-very useful) | 01:27 |
| Ilja | so will ping agendaless, thanks! | 01:27 |
| J1m_ | k, good luck. | 01:29 |
| *** Ilja has quit IRC | 01:44 | |
| *** J1m_ has quit IRC | 01:59 | |
| RixiM | I am using gevent and I get this error: http://pastebin.com/Ln5JyWzb anyone know a fix? | 02:16 |
| RixiM | should I be closing my connections or something? | 02:16 |
| *** J1m_ has joined #zodb | 02:43 | |
| *** JaRoel|4d has quit IRC | 02:47 | |
| *** srichter__ is now known as srichter | 03:13 | |
| srichter | RixiM: why do you think this is due to the ZODB? | 03:13 |
| RixiM | because that's the error in the paste. | 03:14 |
| RixiM | i mean... it's not zodb per se. | 03:14 |
| RixiM | it's how i was forking the workers. | 03:14 |
| RixiM | but I don't know enough to even make intelligent statements about it. | 03:14 |
| RixiM | :/ | 03:14 |
| srichter | right | 03:14 |
| srichter | I am not sure you can fork processes with open ZEO conenctions | 03:14 |
| RixiM | i think that is exactly what the problem was. when i stopped forking, i stopped getting that error. | 03:15 |
| srichter | right | 03:15 |
| *** JaRoel|4d has joined #zodb | 03:18 | |
| *** JaRoel|4d has quit IRC | 03:26 | |
| *** J1m_ has quit IRC | 03:57 | |
| *** Ilja has joined #zodb | 07:45 | |
| *** Ilja has quit IRC | 07:56 | |
| *** Ilja has joined #zodb | 08:08 | |
| *** Ilja has quit IRC | 08:11 | |
| *** RixiM has left #zodb | 08:32 | |
| *** agroszer has joined #zodb | 09:43 | |
| *** agroszer has quit IRC | 10:01 | |
| *** agroszer has joined #zodb | 10:01 | |
| *** JaRoel|4d has joined #zodb | 10:13 | |
| *** agroszer has quit IRC | 10:23 | |
| *** agroszer has joined #zodb | 10:23 | |
| *** JaRoel has joined #zodb | 11:41 | |
| *** JaRoel|4d has quit IRC | 11:44 | |
| *** agroszer has quit IRC | 12:48 | |
| *** agroszer has joined #zodb | 12:49 | |
| *** fdrake has quit IRC | 13:31 | |
| *** srichter has quit IRC | 13:36 | |
| *** srichter has joined #zodb | 13:41 | |
| *** srichter has quit IRC | 14:37 | |
| *** J1m_ has joined #zodb | 14:44 | |
| *** J1m_ has quit IRC | 14:56 | |
| *** J1m_ has joined #zodb | 14:59 | |
| *** J1m_ has quit IRC | 15:04 | |
| *** fdrake has joined #zodb | 15:19 | |
| *** Ilja has joined #zodb | 16:10 | |
| *** J1m_ has joined #zodb | 16:14 | |
| *** J1m has joined #zodb | 16:17 | |
| *** Ilja has quit IRC | 17:22 | |
| *** agroszer has quit IRC | 17:24 | |
| *** Ilja has joined #zodb | 18:01 | |
| *** srichter has joined #zodb | 18:46 | |
| *** JaRoel has quit IRC | 18:51 | |
| *** Ilja has quit IRC | 19:25 | |
| *** Ilja has joined #zodb | 19:30 | |
| *** agroszer has joined #zodb | 20:06 | |
| *** Ilja has quit IRC | 20:49 | |
| *** agroszer has quit IRC | 20:51 | |
| *** Ilja has joined #zodb | 21:08 | |
| *** Ilja has quit IRC | 23:28 | |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!