IRC log of #zodb for Monday, 2016-08-01

*** povbot has joined #zodb01:16
*** aclark has quit IRC02:38
*** J1m has joined #zodb02:56
*** J1m has quit IRC02:58
*** J1m has joined #zodb04:00
*** J1m has quit IRC04:39
*** jensens has joined #zodb13:39
*** J1m has joined #zodb15:43
bloodbareping J1m17:58
J1mbloodbare, hi17:58
bloodbareFinally the timezones are hard !17:58
J1m:) Where are you?17:58
bloodbaresorry I don’t want to annoy you if we say somethings that are completlly wrong17:58
* J1m likes Spain17:59
bloodbareplone.server its an experiment and I’ve not been a zodb/zope guru, more a plone core :P17:59
* bloodbare prefers Catalonia :P17:59
J1m:) I had a feeling you'd say there. I didn't make it to that part of Spain when I visited my son there.17:59
J1msay thar18:00
bloodbareyou must visit catalonia, a nice country close to spain :)18:00
J1msorry, to Catalonia :)18:00
bloodbareji ji ji no problem, we will get there18:00
J1mYes, I must.18:00
bloodbaredo you have some minutes to talk about this plone.server idea ?18:01
bloodbarethe idea came from asko and me that we tried to create a REST API only backend connected to ZODB stack, and we wanted to test to run it on a asyncloop18:02
bloodbarefrom asyncio18:02
bloodbareso we saw that transactions are tied to threads in order to do its magic18:03
J1mNo, they aren't.18:03
J1mThat's just a convenience for sane applications.18:03
J1mBut it's easy to divorce them from threads.18:04
J1mFor example, the TM that DB uses to create the root object isn't tied to a thread in any way.18:04
J1mAnd zc.zodbwsgi has an option to use a TM that's not tied to a thread and of course, connections a connection to a request.18:05
bloodbarewe saw that, in that case each request has a connection no ?18:05
J1mNote that you can close the connection, for example, to deal with long polling, websockets, etc.18:06
bloodbareas we have now we are doing maybe somethings wrong :18:07
bloodbare+ in order to register an object we need to look for the request, because we reuse the connection18:07
bloodbareand is the request who has the transaction and the data manager18:08
J1mThat looks insane to me.18:08
bloodbareI can see now, it was mostly for lack of knowledge on how ZODB works18:08
bloodbareyou don’t see any good point about reusing the connection from one request to the other ?18:09
J1mThat happens automatically.18:10
J1mWhen you close a connection, it gets added back to a pool that behaves like a stack.18:10
J1mSo that the most used connections stay near the top of the stack.18:10
bloodbareok, and the cache ?18:11
J1mEach connection has it's own cache.18:11
bloodbareand when its open again is cleaned ?18:11
J1mNo. When you reopen a connection, the cache is retained.18:12
J1mThe cache drains slowly for unused connections.18:12
bloodbareour idea, maybe wrong, is that if you have 700 concurrent requests, we will need to open 700 connection with its cache for each one, and the memory may be increased18:12
bloodbarethat’s why we thought that reusing one connection for all them may be smaller in memory18:13
J1mI don't know how you could make that work correctly, and if I don't know...18:13
bloodbareji ji ji ji ji18:14
J1mGenerally having 700 open database connections is going to be a bad idea regardless of the database.18:14
bloodbareexactly, that’s why we thought about reusing one connection18:14
J1mDid you also try pixie dust? That might help too. ;)18:15
J1mAre these long-running connections?18:15
bloodbareja ja ja don’t get the pixie dust idea but i feel it18:16
J1mAre these long-running connections?18:16
bloodbareits for a Content management REST API, so mostly are short calls18:17
J1m700 open connections seems rather optimiistic then.18:17
J1mThe classic way we've done this in Zope is to use an async library to handle IO and to have a thread pool with a small number of threads (often 1) to do computations.18:18
J1mIn that model, you only need ad many connections as there are threads, typically 1.18:18
J1mI say 1, because I try to arrange that the working set fits in ram so you don't spend much time waiting for the storage server.18:19
J1mIf you're compute bound, you won't be able to benefit from additional threads because the GIL.18:20
bloodbareon my specific use case where I’m experimenting this idea, its a REST API with websockets and realtime content editting18:20
J1mOK, so long running requests.18:21
bloodbarebut yes, also long running requests18:21
J1mIn that case, I'd disconnect requests from connections and open requests only when you're ready to do something with the database.18:21
J1mGenerally, tou want the database interaction to be pretty short lived.18:22
bloodbareI see , good idea, but how much expensive is openning a ZODB connection ?18:23
bloodbareyou would say to cache on the request websocket the operations and transaction to DB once each time openning the connection and closing18:23
J1mwhen you have a mix, you can let connections get opened for requests, and close them for long-running requests, as I do here
J1mopening ZODB connections (not to be confused with ZEO connections) is cheap.18:24
J1mI'm not sure I understand that last question, but in the context of a websocket app (or long poll), I'd only access the database when you need to.18:25
bloodbareok , I think I’ve a clearer idea about this subject18:25
bloodbareimagine you have a col·laborative writing text editor18:26
bloodbarethat lots of people write at the same time18:26
bloodbareand websocket receives frames of edit operations18:26
J1mIn my experience with websokets and longpoll, I typically need to push data to a bunch of clients when some client makes a change.  In that case, to push data, I often don't need to look at the database to push data.18:26
bloodbareexactly, we need to commit to DB the changes only once every minut18:27
J1mSo, someone makes a change.  That action opens a connection, makes some database operation and computes a change to push to other clients.18:27
J1mThe data to be sent to the other clients can be computed in the initial request and to push it out doesn't really require any additional database access.18:28
J1mSo the clients being pushed to don't need a database connection to get the push.18:28
bloodbareexactly, only once every minut we need to write to the DB18:28
J1mso no need for a shared connection.18:29
bloodbareso I’ll try to compare the performance numbers from what we have now using the connection as transaction holder and connection tied to request18:30
bloodbarereally thanks a lot I hope we will meet at Boston in October :) and really sorry if I annoyied with my comments :P18:32
J1mI look forward to seeing you then.18:32
*** J1m has quit IRC21:00
*** J1m has joined #zodb21:03
*** aclark has joined #zodb21:05
*** aclark has quit IRC21:05
*** aclark has joined #zodb21:07
*** jensens has quit IRC21:16
*** aclark has quit IRC22:19
*** jensens has joined #zodb22:34
bloodbareJ1m: Just some tests we did, opening and closing the connection its just 115% slower than reusing a connection :) on 50 concurrent writing operations22:40
bloodbare( but we are async locking the resource to avoid collisions )22:41
*** aclark has joined #zodb23:46

Generated by 2.15.1 by Marius Gedminas - find it at!