IRC log of #zope for Wednesday, 2018-03-07

*** regebro has quit IRC00:13
*** srichter has joined #zope00:15
*** srichter has quit IRC02:17
*** srichter has joined #zope03:18
*** benji has quit IRC06:25
*** __mac__ has joined #zope08:19
*** __mac__ has quit IRC08:19
*** __mac__ has joined #zope08:23
*** nilo has joined #zope09:56
*** sylvain has joined #zope10:13
*** MrTango has joined #zope10:13
*** regebro has joined #zope11:31
*** MrTango has quit IRC11:48
*** srichter has quit IRC13:11
*** MrTango has joined #zope13:29
*** srichter has joined #zope13:38
*** nilo has quit IRC13:40
*** nilo has joined #zope13:52
*** __mac__ has quit IRC14:09
*** __mac__ has joined #zope14:10
*** benji has joined #zope14:18
*** __mac__1 has joined #zope14:47
*** __mac__ has quit IRC14:47
*** MrTango has quit IRC15:03
*** noodlepie has joined #zope15:18
*** projekt01 has joined #zope15:28
*** nilo has quit IRC16:16
*** MrTango has joined #zope16:42
*** MrTango has quit IRC16:57
*** danielblackburn_ has quit IRC17:21
*** danielblackburn has joined #zope17:21
*** danielbl_ has joined #zope17:24
*** danielblackburn has quit IRC17:28
*** flipmcf has joined #zope17:44
*** flipmcf has joined #zope17:44
flipmcfI'm having an issue making my Jenkins CI realize that a build is bad.17:44
flipmcfwhen I do ./bin/instance fg on my box, I get an exception: (conflicting zcml definitions)  Easy enough to fix.17:45
flipmcfbut when I do ./bin/instance start - the daemon doesn't log the exception, and just keeps kicking off plone in a tight loop.  I never get feedback that things are broken17:46
flipmcfother than port 8080 doesn't respond correctly.17:46
mgedminwhat does your job do, other than start a background daemon?17:48
flipmcfmy jenkins final test is to do a './bin/instance status' and check the output.  If status says it's running, all is good.   Well, it's running (zdaemon) but the plone instance is crashing and restarting.  This is a fail17:48
flipmcftrying to get the actual command out of jenkins, but now that box is down.  sigh.17:49
*** projekt01 has quit IRC17:50
flipmcfsh "bin/instance status | grep 'program running' > /dev/null 2>&1"17:51
flipmcfJenkins runs that.  grep returns non-zero if 'program running' isn't found, and that's what I'm relying on to trigger a failure.17:51
mgedminah, ok, so you use jenkins to (re)start daemons?17:52
mgedminhm, this is tricky17:52
mgedmintwo approaches come to mind: (1) test actual http connectivity in a loop, or (2) grep zdaemon's log files for messages17:52
flipmcfyeah.   I threw it together with little knowlege - this is the first time I'm using Jenkins for CI17:52
flipmcf#2 won't work, because the exception isn't logged.  ?!17:53
mgedminCI -- to me -- would mean you start it up, run a bunch of tests, then shut it down17:53
flipmcfmgedmin, yep.  I get that.17:53
mgedminbut that _would_ notice problems if the start up fails the way it does here17:53
flipmcfwe're really early in this build.  We're far from production or code complete, so building Jenkins is happening now.17:54
flipmcfwe have no tests yet.17:54
flipmcfbut maybe, if I try to test the entire plone package (yuck) I'd see it.17:54
flipmcfbut we don't have tests in our product yet.17:54
mgedminfrom what I remember there may be two (or three) interesting log files17:54
flipmcfthere are two17:54
mgedminzdaemon has one, zdaemon can redirect the program's stdout to another, and the program itself may have its own logfile17:55
mgedminone of the three should show any errors that prevent zope from starting up17:55
flipmcfit's like stderr went somewhere else! good thinking17:55
mgedminyou may need to tweak zdaemon's configs to enable all the logs17:55
flipmcftweaking zdaemon's configs is what I feel is right too.   But never done that :)17:56
mgedminone log file is <runner> ... transcript ... </runner> in zdaemon.conf (the stdout/stderr of the app goes there)17:56
mgedminanother is <eventlog> in zdaemon.conf (zdaemon's own logs go there)17:57
mgedminand zope has its own logs17:57
mgedminTBH if you're thinking about going into production, do consider if you want to use zdaemon17:57
flipmcfgood thread to pull on.  thanks a lot!17:57
mgedminit's not very maintained17:57
mgedminbetter options might be supervisor or systemd17:57
flipmcfI'm hoping supervisord will be used instead.17:58
mgedmin(I have my own doubts about the maintenance status of supervisor, but somebody told me they saw recent commits in git, IIRC)17:58
flipmcfthen there is this other one suggested who'se name escapes me.17:58
mgedminthe other thing is, the supervisor process (zdaemon/supervisord/systemctl) can launch the zope process17:58
flipmcfI've always worked alongside a sysadmin who can figure this out for me.17:58
mgedminbut it cannot see when the zope process is ready to serve requests17:58
mgedminwhich makes detecting this kind of startup failure difficult17:59
mgedminmaybe it would make sense for the supervisor to detect a crashed process in the first 60 seconds (configurable?) and then consider it failed, and stop restarting17:59
mgedminbut maybe not17:59
mgedmin(what if startup failed because of a transient error, e.g. the network being down temporarily?)18:00
mgedminI would rely on a separate monitoring system that watches the HTTP port18:00
flipmcfwhen I showed this to my sysadmin (we're running a plone 4 instance in production) he expected it to fail in 2 minutes18:00
flipmcfand it didn't18:00
mgedmin(*cough* a cron script because I hate nagios *cough*)18:00
flipmcfno.  I want to check in a bad zcml config and make jenkins yell at me.18:01
flipmcfthat's it.18:01
mgedminit would be nice if it were possible to distinguish transient errors from configuration errors in a reliable, automated way18:01
mgedmincurrently there isn't18:01
flipmcfit would also be nice if jenkins launched the instance so selenium can come by and poke around on the rendered templates18:01
mgedmin(zope 2 itself is not very well maintained and I would hesitate to put it into production... except the plone people are keeping it alive somehow, so maybe it's not as bad as I make it sound)18:02
flipmcfthese sound related.18:02
mgedminso, my suggestion: 1. bin/instance start, 2. wait in a loop for port 8080 to start responding to requests, with a timeout18:02
flipmcf'not well maintained' may be a  'not broken don't fix it' in discuise  which is where I see zope today.18:03
mgedminoh hey, I've a project here that does the wait loop (for a non-zope web app) with a single wget command18:03
mgedminwget -q --retry-connrefused -T 60 -O /dev/null && ...18:03
flipmcfhow about launch the instance without wrapping it in zdaemon?18:03
flipmcfI'm not testing zdaemon, so why it it part of my CI?18:03
mgedminor - hey - can you maybe disable the autorestart-on-crash logic in zdaemon.cfg?18:04
flipmcfthat's where the easy answer is.18:04
mgedminthat will not help with the wait-until-it-comes-up bit18:04
flipmcfyep, I can wait for the log file to say something about 'ready to handle requests'18:05
mgedminin fact, if you don't want to wait 60 seconds, it'd be better to have an actuall shell loop that checks instance status OR port 8080 being open18:05
*** sylvain has quit IRC18:05
mgedminanyway good luck!18:06
flipmcfsleep(x) is a great way to create race conditions.  I don't want to be woken up at 1am because the CI threw a build error because of a race condition.18:06
flipmcfeven worse, I don't want to wake someone else up because I wrote a race condition.18:06
flipmcfjesus said something about that.18:07
mgedmindo-not-disturb mode on my phone between 9 pm and 10 am PROBLEM SOLVED :D18:07
flipmcfthe person I hire to respond to Jenkins noise shal not have DND on their phone.18:07
flipmcfproblem solved18:08
flipmcfor created.... depends who you are.18:08
*** __mac__1 has quit IRC18:34
flipmcfI cannot find these configuration files.18:36
flipmcfI'm going to spelunk in pdb.  hopefully someone can chime in before I learn too much stuff I don't need to know.18:37
*** MrTango has joined #zope20:37
*** __mac__ has joined #zope20:43
*** noodlepie has quit IRC20:43
*** __mac__ has quit IRC20:43
*** noodlepie has joined #zope20:44
*** __mac__ has joined #zope21:00
*** lregebro has joined #zope21:58
*** regebro has quit IRC22:00
*** __mac__ has quit IRC22:09
*** phizzy has joined #zope22:51
*** noodlepie has quit IRC22:55
*** MrTango has quit IRC23:06

Generated by 2.15.1 by Marius Gedminas - find it at!