| *** sunew has joined #zope | 00:08 | |
| *** menesis has quit IRC | 00:22 | |
| *** Arfrever has quit IRC | 00:35 | |
| *** dayne1 has joined #zope | 00:46 | |
| *** dayne has quit IRC | 00:46 | |
| *** mcdonc has joined #zope | 01:06 | |
| *** mcdonc has quit IRC | 01:06 | |
| *** sunew has quit IRC | 01:31 | |
| *** evilbungle has quit IRC | 02:47 | |
| *** _mup_ has quit IRC | 03:00 | |
| *** _mup_ has joined #zope | 03:00 | |
| *** mr_jolly has left #zope | 03:25 | |
| *** thetet has quit IRC | 04:04 | |
| *** dayne1 has quit IRC | 06:35 | |
| *** agroszer has joined #zope | 07:17 | |
| *** agroszer has quit IRC | 07:29 | |
| *** River_Rat has joined #zope | 08:12 | |
| *** RiverRat has quit IRC | 08:14 | |
| *** __mac__ has joined #zope | 09:11 | |
| *** River_Rat is now known as RiverRat | 09:36 | |
| *** mitchell`off is now known as mitchell` | 09:48 | |
| *** polysix has quit IRC | 09:52 | |
| *** polysix has joined #zope | 09:52 | |
| *** betabug_ has joined #zope | 10:10 | |
| *** rodgort has quit IRC | 10:11 | |
| *** betabug has quit IRC | 10:11 | |
| *** rodgort has joined #zope | 10:16 | |
| *** menesis has joined #zope | 10:29 | |
| *** ccomb has joined #zope | 10:42 | |
| *** avoinea has joined #zope | 11:09 | |
| *** mr_jolly has joined #zope | 11:26 | |
| *** goschtl has joined #zope | 11:31 | |
| *** thetet has joined #zope | 11:36 | |
| *** eperez has joined #zope | 11:41 | |
| *** eperez has quit IRC | 11:45 | |
| *** eperez has joined #zope | 11:46 | |
| *** MrTango has joined #zope | 12:01 | |
| *** evilbungle has joined #zope | 12:04 | |
| *** do3cc has joined #zope | 12:16 | |
| *** maurits has joined #zope | 12:22 | |
| *** betabug_ is now known as betabug | 12:37 | |
| *** _mup_ has quit IRC | 13:07 | |
| *** _mup_ has joined #zope | 13:07 | |
| *** agroszer has joined #zope | 13:33 | |
| *** moo-_- has quit IRC | 14:03 | |
| *** moo-_- has joined #zope | 14:11 | |
| *** agroszer has quit IRC | 14:17 | |
| *** tisto has joined #zope | 14:18 | |
| *** agroszer has joined #zope | 14:18 | |
| *** tisto has quit IRC | 14:31 | |
| *** menesis has quit IRC | 14:39 | |
| *** zsh has quit IRC | 14:40 | |
| *** zsh has joined #zope | 14:53 | |
| *** menesis has joined #zope | 15:45 | |
| *** menesis has quit IRC | 15:47 | |
| *** menesis has joined #zope | 15:47 | |
| *** running_clam has joined #zope | 15:49 | |
| *** ericof has joined #zope | 16:17 | |
| *** ericof has left #zope | 16:19 | |
| *** running_clam has left #zope | 16:27 | |
| *** tisto has joined #zope | 16:44 | |
| *** J1m has joined #zope | 16:46 | |
| *** dayne has joined #zope | 17:10 | |
| *** thetet has quit IRC | 17:18 | |
| mgedmin | anyone ever heard of mysteriously dying zope.server threads? | 17:18 |
|---|---|---|
| mgedmin | not a peep in the logs, but three out of four threads are now gone | 17:18 |
| *** goschtl has quit IRC | 17:31 | |
| mgedmin | where's the bug tracker for zc.zservertracelog? | 17:44 |
| *** dayne has quit IRC | 18:10 | |
| kosh | it is not something I have ever seen | 18:25 |
| kosh | not in 10+ years of using zope | 18:25 |
| betabug | mgedmin: gone? or deadlocked maybe? | 18:26 |
| kosh | betabug: so how goes the fall of civilization? | 18:26 |
| betabug | endless, how else | 18:26 |
| mgedmin | gone: pstree shows only 4 running threads there (main thread, zeo client thread, mail delivery thread, one worker thread) | 18:26 |
| mgedmin | the staging instance on the same server with very similar config has 7 threads that show up in pstree -- and also in sys._current_frames() | 18:27 |
| mgedmin | I've collectd graphs showing me when the thread count goes down | 18:27 |
| mgedmin | but I haven't been able to pin the cause yet | 18:27 |
| mgedmin | nothing suspicious in the logs (z3.log, trace.log, access.log)... | 18:27 |
| mgedmin | well, I saw four exceptions about a short socket read (client closed connection before the entire response was sent) | 18:28 |
| mgedmin | if I'd seen three, I could believe they might've shut the three threads down somehow | 18:28 |
| mgedmin | shame that there's no thread ID in trace.log | 18:29 |
| *** daMaestro has joined #zope | 18:31 | |
| mgedmin | oh, hey, actually | 18:36 |
| mgedmin | three out of those four "ERROR zope.server.taskthreads Exception during task" messages correlate rather neatly with the thread count going down by one in the collectd graph | 18:36 |
| mgedmin | the exceptions are here: http://pastie.org/3989489 | 18:39 |
| mgedmin | I see no differences between the first and the rest | 18:39 |
| mgedmin | and here's the thread # graph: http://imgur.com/8FYyZ | 18:41 |
| mgedmin | sunday night the thread # went down to zero live worker threads, which alerted me to the issue | 18:42 |
| kosh | geeze that sounds so bizarre I wonder what is in your code or that your code is using that would cause that problem | 18:43 |
| mgedmin | sure, blame my code for bugs in zope.server :) | 18:44 |
| *** agroszer has quit IRC | 18:44 | |
| kosh | it just seems if it was a general bug in zope.server more people would have the problem | 18:45 |
| mgedmin | I agree | 18:45 |
| kosh | I just suspected some kind of c extension involved that is causing the thread to die | 18:45 |
| mgedmin | I've been running this configuration for a long time, only now started noticing strangeness | 18:45 |
| kosh | hardware or library update problem? | 18:45 |
| kosh | although I have no idea how that would be involved | 18:45 |
| mgedmin | looking at the yearly graph, thread # started going down in mid-April | 18:46 |
| mgedmin | no significant OS-level upgrades near that time | 18:47 |
| mgedmin | there were app version updates, I'll have to check if they changed any package versions | 18:47 |
| * mgedmin needs a break | 18:48 | |
| kosh | I can get a vacation in a little over 2 years it looks like | 18:49 |
| mgedmin | well, good luck with that :) | 18:50 |
| kosh | just the unfortunate reality of running a business and going back to school with other issues having come up | 18:57 |
| betabug | I don't need a vacation, I just go to work on an island | 18:57 |
| *** sp0cksbeard has joined #zope | 19:24 | |
| *** runyaga has joined #zope | 19:30 | |
| benbangert | J1m: gevent + real os threads is such a blast.... | 19:32 |
| J1m | I like both of those. :) | 19:33 |
| J1m | The combination can be explosive. :) | 19:33 |
| benbangert | https://github.com/python-zk/kazoo/blob/master/kazoo/handlers/gevent.py seems to work safely | 19:33 |
| benbangert | almost got the base client bit setup, then I'll mix in the testing stuff from zc.zk | 19:33 |
| benbangert | and add the higher level children/properties API's from zc.zk | 19:34 |
| J1m | very cool | 19:34 |
| benbangert | that pipe trick feels kind of weak, but oh well | 19:35 |
| benbangert | thats the main thing when using the coordination objects, also, even weird, waking the async object from a separete real os thread can apparently cause issues, which is why the actual set/set_exception is curryed over to the gevent thread to run | 19:35 |
| benbangert | the side effect of the 3 greenlets/threads at the moment means its possible for session events to fire at the same time as a watcher executes... which doesn't immediately strike me as problematic, does that raise any warning bells in your head? | 19:37 |
| benbangert | at the very least, it makes it easier to use ZK calls in a watcher without worrying about blocking session re-establishment | 19:37 |
| benbangert | (normally you'd need to spawn your watcher to another thread to ensure you don't block those anyways) | 19:37 |
| *** tisto has quit IRC | 19:41 | |
| *** maurits has quit IRC | 19:44 | |
| *** menesis has quit IRC | 19:48 | |
| *** tiwula has joined #zope | 20:00 | |
| *** __mac__ has quit IRC | 20:01 | |
| *** avoinea has quit IRC | 20:02 | |
| *** eperez has quit IRC | 20:03 | |
| *** sm has quit IRC | 20:15 | |
| *** ccomb has quit IRC | 20:19 | |
| *** sm has joined #zope | 20:23 | |
| *** menesis has joined #zope | 20:27 | |
| *** moo-_- has quit IRC | 20:34 | |
| *** tiwula has quit IRC | 20:38 | |
| *** runyaga has quit IRC | 20:45 | |
| *** moo-_- has joined #zope | 20:49 | |
| *** tiwula has joined #zope | 21:01 | |
| *** __mac__ has joined #zope | 21:07 | |
| *** __mac__ has quit IRC | 21:12 | |
| *** do3cc has quit IRC | 21:13 | |
| *** mitchell` is now known as mitchell`off | 21:41 | |
| *** MrTango has quit IRC | 22:04 | |
| *** Arfrever has joined #zope | 22:47 | |
| *** thetet has joined #zope | 22:47 | |
| *** rob0t7_ has joined #zope | 22:51 | |
| rob0t7_ | what is the big difference between zope.deprecation and using python's warnings lib? | 22:51 |
| *** rob0t7_ has quit IRC | 23:05 | |
| *** do3cc has joined #zope | 23:07 | |
| mgedmin | actually, the zope.server-dying-thread problem is probably my fault | 23:55 |
| mgedmin | or, rather, the fault of zilch: | 23:56 |
| mgedmin | https://github.com/bbangert/zilch/issues/5 | 23:56 |
| mgedmin | if this happens in zope.server's handlerThread method, in log.exception(), that terminates the entire thread | 23:56 |
| mgedmin | http://pastie.org/3991149 | 23:56 |
| mgedmin | is the code from zope.server | 23:56 |
Generated by irclog2html.py 2.15.1 by Marius Gedminas - find it at mg.pov.lt!