| *** jamadden has quit IRC | 08:09 | |
| *** jensens has joined #zodb | 10:53 | |
| *** jamadden has joined #zodb | 13:25 | |
| *** aclark has joined #zodb | 14:29 | |
| *** aclark has quit IRC | 14:31 | |
| *** aclark has joined #zodb | 16:02 | |
| *** aclark70 has joined #zodb | 16:03 | |
| *** aclark has quit IRC | 16:03 | |
| *** J1m has joined #zodb | 16:33 | |
| *** aclark70 has quit IRC | 17:04 | |
| J1m | We had this behavior that if invalidateCache() was called on a connection, then the connection would get read-conflict errors until the next transaction boudary. This makes no sense to me. I'm getting id of it while adding the IMVCCStorage adapter. | 17:45 |
|---|---|---|
| jamadden | Does that increase the odds of working with inconsistent data in that transaction? Seems like invalidateCache() would be called if something changed outside the normal course of events and there's a reasonable chance that in-flight transactions are probably invalid now. (Dunno, I've never called invalidateCache() in a live app.) | 17:51 |
| J1m | I think incalidateCache is motivated by ZEO. It gets called when ZEO has been disconnected too long to replay invalidations. So we don't know how to make sure either the ZEO or object cache is valid. | 17:53 |
| J1m | RelStorage has a similar concept. If poll_invalidations returns None, then the object cache needs to be entirely invalidated. | 17:53 |
| J1m | The thing is that mid-txn, we're loading data at a point in time and at the beginning of the transaction, we deemed the cache valid wrt that time, so I see now reason to poison it. | 17:54 |
| J1m | It seems to ne that invalidateCache should affect the next txn. | 17:55 |
| jamadden | fair enough, I guess. worst case, the in-flight transaction goes through a bunch of conflict resolutions that either work or don't | 17:56 |
| J1m | Right. It doesn't mean that we know the existing objects should be invalidated, it just means we don't know they shouldn't. | 17:57 |
| jamadden | so are you finding some time to work on IMVCC adapter coding? I was beginning to feel more confident that I might be able to help with it once the low-hanging RelStorage fruit is knocked out, but I'm more than happy to leave it to the expert :) | 18:00 |
| J1m | Yeah, I decided to work on it because it would make the zodb5 release less exciting. | 18:03 |
| J1m | It was pretty straightforward, but I have ~30 failing tests remaining that I'm slogging my way through. | 18:03 |
| J1m | But this a behavioral change that I wanted to air. | 18:04 |
| J1m | Unfortunately, lots of the ZODB tests are pretty white-box, so API changes tend to hurt a lot. | 18:07 |
| jamadden | indeed. plus, doctests are just hard to debug, at least for me | 18:07 |
| J1m | <shrug> I don't find doctests any harder to debug than any other kind of test. | 18:09 |
| jamadden | given that I can't use pdb or even add extra print statements without introducing extra failure noise, I find it to be a lot more guess-and-check than usual | 18:10 |
| J1m | Why can't you use pdb? I use pdb with doctests all the time. | 18:10 |
| jamadden | I once asked Tres for tips on how to debug doctests and he said to rewrite them as unittest tests :) | 18:10 |
| J1m | Yeah, he hates doctests. He wasn't a good one to ask. | 18:11 |
| jamadden | really? I wish somebody (Tres!) had told me. How do you step through with all output being captured? | 18:11 |
| J1m | It's also somewhat of a function of your tests runner. | 18:11 |
| J1m | nose and py.test make doctests very difficult and I think mostly get pdb wrong. | 18:12 |
| J1m | Although I got pdb working with doctest and nose maybe half the time in my last job. | 18:12 |
| jamadden | I've never used py.test and I started transitioning all my projects away from nose once I discovered layers. we're split between nose2 and zope.testrunner now | 18:13 |
| J1m | I hate nose. Deeply. I think I hate nose more than Tres hates doctest, which is a lot. | 18:13 |
| jamadden | ouch | 18:13 |
| J1m | WRT doctest, I don't champion it as strongly as a used to. It has pros and cons. The pros outweight the controversy for me so I've been using doctest for new tests less. | 18:15 |
| J1m | I still like it for testable documentation, like the bobo and zc.ngi docs, but that also involves manuel. | 18:16 |
| J1m | If you use the zope test runner, then pdb should just work with doctests. | 18:16 |
| J1m | The only limitiation is that you can't step across examples. | 18:17 |
| jamadden | I guess I quit trying after being told it didn't work. I'll try again next time I'm in doctestland | 18:17 |
| J1m | But I useually use pdb.set_trace as a prefix in an example and then step into. | 18:17 |
| jamadden | if I was looking for the last committed transaction ID in a FileStorage, would FileStorage.lastInvalidations(1)[0] give it to me? | 19:49 |
| jamadden | this relates to the —incremental option of RelStorage's zodbconvert | 19:49 |
| J1m | No, just lastTransaction() | 19:51 |
| J1m | Do you use a mac? | 19:51 |
| jamadden | There is no lastTransaction method on FileStorage | 19:52 |
| jamadden | yes | 19:52 |
| J1m | It's in IStorage... | 19:52 |
| J1m | It's inherited from BaseStorage | 19:53 |
| jamadden | ok, i see it now. thanks. I assumed there wasn't something that easy because the RS comment says there's not a portable way…must be old comment. (The two methods do seem to result in the same answer but the lastTransaction() is cached and uses the index and is presumably portable.) | 19:56 |
| J1m | when I try to run relstorage tests, it runs tests for lots of other packages because it's dependencies are listed in the testrunner recipe's eggs. | 20:04 |
| J1m | I'm fixing that. | 20:05 |
| J1m | But... what databases do I need to run the tests? | 20:05 |
| jamadden | if you say 'python -m relstorage.tests.alltests' it will only attempt to run the tests for the databases you have drivers installed for, so MySQL-python implies a running mysql, psycopg implies Postgres, cx_oracle implies Oracle. | 20:06 |
| jamadden | There are scripts to set up such a DB and install the appropriate dependency in the .travis/ directory | 20:06 |
| jamadden | tox is also set up to run isolated tests, though you still have to have the DB running | 20:07 |
| J1m | what a pain | 20:07 |
| jamadden | I don't know what kind of shape the buildout.cfg is in, I use a virtualenv like Travis setup | 20:07 |
| jamadden | I know. Hence sqlite support :) | 20:07 |
| J1m | error in RelStorage setup command: Invalid environment marker: platform_python_implementation=="CPython" | 20:24 |
| J1m | ? | 20:24 |
| jamadden | that can happen with very old setuptools versions | 20:25 |
| J1m | This is when trying to run buidout after pulling recent changes. | 20:25 |
| J1m | k | 20:25 |
| J1m | so a way you can help with the imvcc adapter is to review a pr. | 20:27 |
| J1m | would you rather review a pr against the no-more-load branch? | 20:27 |
| J1m | or would you prefer I merge the change into the no-more-load branch and ask you to review that? | 20:28 |
| jamadden | um, the isolated change of a PR against the no-more-load branch might be a little smaller and easier to review | 20:28 |
| J1m | k | 20:29 |
| jamadden | sigh, RelStorage implements lastTransaction() entirely in memory, per-RelStorage-object. it has a completely different custom mechanism to get the globally persistent last transaction. | 20:37 |
| J1m | Yes. I changed that in my PR that we reverted. | 20:46 |
| J1m | Note that ZODB5 won't care about that any more soon. | 20:47 |
| jamadden | the functionality is still important for zodbconvert | 20:47 |
| J1m | well, the fix is lurking in that PR. It took me a while to work out. | 20:48 |
| J1m | I really don't like the way relstorage clones itself. Very hackish and leads to weirdness like this. | 20:55 |
| J1m | But fixing it probably requires rationalizing this API more and I don't want to spend the time now. :) | 20:56 |
| *** aclark has joined #zodb | 21:44 | |
| J1m | jamadden, I'm fighting a relstorage test failure against my branch, and I think it's a bug. | 22:18 |
| J1m | Relstorage's reconnect logic makes a lot of assumptions about the order things are called in. | 22:18 |
| jamadden | That does not surprise me | 22:21 |
| J1m | In checkAutoReconnect, the test closes a relstorage's connections. But then, when a ZODB connection is opened, ZODB calls sync(False) and sync errors when it tries to roll back the connection. | 22:21 |
| jamadden | calling sync(False) on connection open is new behaviour in your branch? | 22:23 |
| J1m | yes | 22:23 |
| jamadden | yeah, I see it. I think we can just ignore closed/disconnected exceptions there…it happens anyway | 22:25 |
| J1m | OK, I'm going t ignore this test failure for now then. | 22:26 |
| J1m | Of course, relstorage will also need to stop monkey patching zodb | 22:28 |
| J1m | I've already commented that out in my test env. | 22:28 |
| jamadden | I think it should be possible to write a test in RS independent of your branch that demonstrates the problem and then fix it (for both tests). I can do that after PR #44 is done…which I think it is | 22:28 |
| J1m | cool | 22:29 |
| *** aclark has quit IRC | 22:34 | |
| *** jamadden is now known as jamadden_away | 22:37 | |
| *** jamadden_away is now known as jamadden | 23:15 | |
| *** J1m has quit IRC | 23:45 | |
| *** J1m has joined #zodb | 23:45 | |
| *** J1m has quit IRC | 23:48 | |
Generated by irclog2html.py 2.15.1 by Marius Gedminas - find it at mg.pov.lt!