IRC log of #zodb for Tuesday, 2016-06-14

*** jamadden has quit IRC08:09
*** jensens has joined #zodb10:53
*** jamadden has joined #zodb13:25
*** aclark has joined #zodb14:29
*** aclark has quit IRC14:31
*** aclark has joined #zodb16:02
*** aclark70 has joined #zodb16:03
*** aclark has quit IRC16:03
*** J1m has joined #zodb16:33
*** aclark70 has quit IRC17:04
J1mWe had this behavior that if invalidateCache() was called on a connection, then the connection would get read-conflict errors until the next transaction boudary. This makes no sense to me.  I'm getting id of it while adding the IMVCCStorage adapter.17:45
jamaddenDoes that increase the odds of working with inconsistent data in that transaction? Seems like invalidateCache() would be called if something changed outside the normal course of events and there's a reasonable chance that in-flight transactions are probably invalid now. (Dunno, I've never called invalidateCache() in a live app.)17:51
J1mI think incalidateCache is motivated by ZEO.  It gets called when ZEO has been disconnected too long to replay invalidations.  So we don't know how to make sure either the ZEO or object cache is valid.17:53
J1mRelStorage has a similar concept. If poll_invalidations returns None, then the object cache needs to be entirely invalidated.17:53
J1mThe thing is that mid-txn, we're loading data at a point in time and at the beginning of the transaction, we deemed the cache valid wrt that time, so I see now reason to poison it.17:54
J1mIt seems to ne that invalidateCache should affect the next txn.17:55
jamaddenfair enough, I guess. worst case, the in-flight transaction goes through a bunch of conflict resolutions that either work or don't17:56
J1mRight. It doesn't mean that we know the existing objects should be invalidated, it just means we don't know they shouldn't.17:57
jamaddenso are you finding some time to work on IMVCC adapter coding? I was beginning to feel more confident that I might be able to help with it once the low-hanging RelStorage fruit is knocked out, but I'm more than happy to leave it to the expert :)18:00
J1mYeah, I decided to work on it because it would make the zodb5 release less exciting.18:03
J1mIt was pretty straightforward, but I have ~30 failing tests remaining that I'm slogging my way through.18:03
J1mBut this a behavioral change that I wanted to air.18:04
J1mUnfortunately, lots of the ZODB tests are pretty white-box, so API changes tend to hurt a lot.18:07
jamaddenindeed. plus, doctests are just hard to debug, at least for me18:07
J1m<shrug> I don't find doctests any harder to debug than any other kind of test.18:09
jamaddengiven that I can't use pdb or even add extra print statements without introducing extra failure noise, I find it to be a lot more guess-and-check than usual18:10
J1mWhy can't you use pdb? I use pdb with doctests all the time.18:10
jamaddenI once asked Tres for tips on how to debug doctests and he said to rewrite them as unittest tests :)18:10
J1mYeah, he hates doctests. He wasn't a good one to ask.18:11
jamaddenreally? I wish somebody (Tres!) had told me. How do you step through with all output being captured?18:11
J1mIt's also somewhat of a function of your tests runner.18:11
J1mnose and py.test make doctests very difficult and I think mostly get pdb wrong.18:12
J1mAlthough I got pdb working with doctest and nose maybe half the time in my last job.18:12
jamaddenI've never used py.test and I started transitioning all my projects away from nose once I discovered layers. we're split between nose2 and zope.testrunner now18:13
J1mI hate nose. Deeply. I think I hate nose more than Tres hates doctest, which is a lot.18:13
J1mWRT doctest, I don't champion it as strongly as a used to.  It has pros and cons. The pros outweight the controversy for me so I've been using doctest for new tests less.18:15
J1mI still like it for testable documentation, like the bobo and zc.ngi docs, but that also involves manuel.18:16
J1mIf you use the zope test runner, then pdb should just work with doctests.18:16
J1mThe only limitiation is that you can't step across examples.18:17
jamaddenI guess I quit trying after being told it didn't work. I'll try again next time I'm in doctestland18:17
J1mBut I useually use pdb.set_trace as a prefix in an example and then step into.18:17
jamaddenif I was looking for the last committed transaction ID in a FileStorage, would FileStorage.lastInvalidations(1)[0] give it to me?19:49
jamaddenthis relates to the —incremental option of RelStorage's zodbconvert19:49
J1mNo, just lastTransaction()19:51
J1mDo you use a mac?19:51
jamaddenThere is no lastTransaction method on FileStorage19:52
J1mIt's in IStorage...19:52
J1mIt's inherited from BaseStorage19:53
jamaddenok, i see it now. thanks. I assumed there wasn't something that easy because the RS comment says there's not a portable way…must be old comment. (The two methods do seem to result in the same answer but the lastTransaction() is cached and uses the index and is presumably portable.)19:56
J1mwhen I try to run relstorage tests, it runs tests for lots of other packages because it's dependencies are listed in the testrunner recipe's eggs.20:04
J1mI'm fixing that.20:05
J1mBut... what databases do I need to run the tests?20:05
jamaddenif you say 'python -m relstorage.tests.alltests' it will only attempt to run the tests for the databases you have drivers installed for, so MySQL-python implies a running mysql, psycopg implies Postgres, cx_oracle implies Oracle.20:06
jamaddenThere are scripts to set up such a DB and install the appropriate dependency in the .travis/ directory20:06
jamaddentox is also set up to run isolated tests, though you still have to have the DB running20:07
J1mwhat a pain20:07
jamaddenI don't know what kind of shape the buildout.cfg is in, I use a virtualenv like Travis setup20:07
jamaddenI know. Hence sqlite support :)20:07
J1merror in RelStorage setup command: Invalid environment marker: platform_python_implementation=="CPython"20:24
jamaddenthat can happen with very old setuptools versions20:25
J1mThis is when trying to run buidout after pulling recent changes.20:25
J1mso a way you can help with the imvcc adapter is to review a pr.20:27
J1mwould you rather review a pr against the no-more-load branch?20:27
J1mor would you prefer I merge the change into the no-more-load branch and ask you to review that?20:28
jamaddenum, the isolated change of a PR against the no-more-load branch might be a little smaller and easier to review20:28
jamaddensigh, RelStorage implements lastTransaction() entirely in memory, per-RelStorage-object. it has a completely different custom mechanism to get the globally persistent last transaction.20:37
J1mYes. I changed that in my PR that we reverted.20:46
J1mNote that ZODB5 won't care about that any more soon.20:47
jamaddenthe functionality is still important for zodbconvert20:47
J1mwell, the fix is lurking in that PR. It took me a while to work out.20:48
J1mI really don't like the way relstorage clones itself. Very hackish and leads to weirdness like this.20:55
J1mBut fixing it probably requires rationalizing this API more and I don't want to spend the time now. :)20:56
*** aclark has joined #zodb21:44
J1mjamadden, I'm fighting a relstorage test failure against my branch, and I think it's a bug.22:18
J1mRelstorage's reconnect logic makes a lot of assumptions about the order things are called in.22:18
jamaddenThat does not surprise me22:21
J1mIn checkAutoReconnect, the test closes a relstorage's connections.  But then, when a ZODB connection is opened, ZODB calls sync(False) and sync errors when it tries to roll back the connection.22:21
jamaddencalling sync(False) on connection open is new behaviour in your branch?22:23
jamaddenyeah, I see it. I think we can just ignore closed/disconnected exceptions there…it happens anyway22:25
J1mOK, I'm going t ignore this test failure for now then.22:26
J1mOf course, relstorage will also need to stop monkey patching zodb22:28
J1mI've already commented that out in my test env.22:28
jamaddenI think it should be possible to write a test in RS independent of your branch that demonstrates the problem and then fix it (for both tests). I can do that after PR #44 is done…which I think it is22:28
*** aclark has quit IRC22:34
*** jamadden is now known as jamadden_away22:37
*** jamadden_away is now known as jamadden23:15
*** J1m has quit IRC23:45
*** J1m has joined #zodb23:45
*** J1m has quit IRC23:48

Generated by 2.15.1 by Marius Gedminas - find it at!