| *** povbot has joined #zodb | 01:16 | |
| *** aclark has quit IRC | 02:38 | |
| *** J1m has joined #zodb | 02:56 | |
| *** J1m has quit IRC | 02:58 | |
| *** J1m has joined #zodb | 04:00 | |
| *** J1m has quit IRC | 04:39 | |
| *** jensens has joined #zodb | 13:39 | |
| *** J1m has joined #zodb | 15:43 | |
| bloodbare | ping J1m | 17:58 |
|---|---|---|
| J1m | bloodbare, hi | 17:58 |
| bloodbare | Finally the timezones are hard ! | 17:58 |
| J1m | :) Where are you? | 17:58 |
| bloodbare | sorry I don’t want to annoy you if we say somethings that are completlly wrong | 17:58 |
| bloodbare | barcelona | 17:58 |
| J1m | ah | 17:58 |
| * J1m likes Spain | 17:59 | |
| bloodbare | plone.server its an experiment and I’ve not been a zodb/zope guru, more a plone core :P | 17:59 |
| * bloodbare prefers Catalonia :P | 17:59 | |
| J1m | :) I had a feeling you'd say there. I didn't make it to that part of Spain when I visited my son there. | 17:59 |
| J1m | say thar | 18:00 |
| J1m | that | 18:00 |
| bloodbare | you must visit catalonia, a nice country close to spain :) | 18:00 |
| J1m | sorry, to Catalonia :) | 18:00 |
| bloodbare | ji ji ji no problem, we will get there | 18:00 |
| J1m | Yes, I must. | 18:00 |
| bloodbare | do you have some minutes to talk about this plone.server idea ? | 18:01 |
| J1m | sure | 18:01 |
| bloodbare | the idea came from asko and me that we tried to create a REST API only backend connected to ZODB stack, and we wanted to test to run it on a asyncloop | 18:02 |
| bloodbare | from asyncio | 18:02 |
| bloodbare | so we saw that transactions are tied to threads in order to do its magic | 18:03 |
| J1m | No, they aren't. | 18:03 |
| J1m | That's just a convenience for sane applications. | 18:03 |
| J1m | But it's easy to divorce them from threads. | 18:04 |
| J1m | For example, the TM that DB uses to create the root object isn't tied to a thread in any way. | 18:04 |
| J1m | And zc.zodbwsgi has an option to use a TM that's not tied to a thread and of course, connections a connection to a request. | 18:05 |
| bloodbare | we saw that, in that case each request has a connection no ? | 18:05 |
| J1m | Yes. | 18:05 |
| J1m | Note that you can close the connection, for example, to deal with long polling, websockets, etc. | 18:06 |
| bloodbare | as we have now we are doing maybe somethings wrong : | 18:07 |
| bloodbare | + https://github.com/plone/plone.server/blob/master/src/plone.server/plone/server/transactions.py#L132 in order to register an object we need to look for the request, because we reuse the connection | 18:07 |
| bloodbare | and is the request who has the transaction and the data manager | 18:08 |
| J1m | That looks insane to me. | 18:08 |
| bloodbare | I can see now, it was mostly for lack of knowledge on how ZODB works | 18:08 |
| bloodbare | zodb/transactions/persist | 18:08 |
| bloodbare | you don’t see any good point about reusing the connection from one request to the other ? | 18:09 |
| J1m | That happens automatically. | 18:10 |
| J1m | When you close a connection, it gets added back to a pool that behaves like a stack. | 18:10 |
| J1m | So that the most used connections stay near the top of the stack. | 18:10 |
| bloodbare | ok, and the cache ? | 18:11 |
| J1m | Each connection has it's own cache. | 18:11 |
| bloodbare | and when its open again is cleaned ? | 18:11 |
| J1m | No. When you reopen a connection, the cache is retained. | 18:12 |
| J1m | The cache drains slowly for unused connections. | 18:12 |
| bloodbare | our idea, maybe wrong, is that if you have 700 concurrent requests, we will need to open 700 connection with its cache for each one, and the memory may be increased | 18:12 |
| bloodbare | that’s why we thought that reusing one connection for all them may be smaller in memory | 18:13 |
| J1m | I don't know how you could make that work correctly, and if I don't know... | 18:13 |
| bloodbare | ji ji ji ji ji | 18:14 |
| J1m | Generally having 700 open database connections is going to be a bad idea regardless of the database. | 18:14 |
| bloodbare | exactly, that’s why we thought about reusing one connection | 18:14 |
| J1m | Did you also try pixie dust? That might help too. ;) | 18:15 |
| J1m | Are these long-running connections? | 18:15 |
| bloodbare | ja ja ja don’t get the pixie dust idea but i feel it | 18:16 |
| J1m | Are these long-running connections? | 18:16 |
| bloodbare | its for a Content management REST API, so mostly are short calls | 18:17 |
| J1m | 700 open connections seems rather optimiistic then. | 18:17 |
| J1m | The classic way we've done this in Zope is to use an async library to handle IO and to have a thread pool with a small number of threads (often 1) to do computations. | 18:18 |
| J1m | In that model, you only need ad many connections as there are threads, typically 1. | 18:18 |
| J1m | I say 1, because I try to arrange that the working set fits in ram so you don't spend much time waiting for the storage server. | 18:19 |
| J1m | If you're compute bound, you won't be able to benefit from additional threads because the GIL. | 18:20 |
| bloodbare | on my specific use case where I’m experimenting this idea, its a REST API with websockets and realtime content editting | 18:20 |
| J1m | OK, so long running requests. | 18:21 |
| bloodbare | both | 18:21 |
| bloodbare | but yes, also long running requests | 18:21 |
| J1m | In that case, I'd disconnect requests from connections and open requests only when you're ready to do something with the database. | 18:21 |
| J1m | Generally, tou want the database interaction to be pretty short lived. | 18:22 |
| J1m | s/tou/you/ | 18:22 |
| bloodbare | I see , good idea, but how much expensive is openning a ZODB connection ? | 18:23 |
| bloodbare | you would say to cache on the request websocket the operations and transaction to DB once each time openning the connection and closing | 18:23 |
| J1m | when you have a mix, you can let connections get opened for requests, and close them for long-running requests, as I do here https://github.com/zc/twotieredkanban/blob/master/server/zc/twotieredkanban/apiboard.py#L54 | 18:23 |
| J1m | opening ZODB connections (not to be confused with ZEO connections) is cheap. | 18:24 |
| J1m | I'm not sure I understand that last question, but in the context of a websocket app (or long poll), I'd only access the database when you need to. | 18:25 |
| bloodbare | ok , I think I’ve a clearer idea about this subject | 18:25 |
| bloodbare | imagine you have a col·laborative writing text editor | 18:26 |
| bloodbare | that lots of people write at the same time | 18:26 |
| bloodbare | and websocket receives frames of edit operations | 18:26 |
| J1m | In my experience with websokets and longpoll, I typically need to push data to a bunch of clients when some client makes a change. In that case, to push data, I often don't need to look at the database to push data. | 18:26 |
| bloodbare | exactly, we need to commit to DB the changes only once every minut | 18:27 |
| J1m | So, someone makes a change. That action opens a connection, makes some database operation and computes a change to push to other clients. | 18:27 |
| J1m | The data to be sent to the other clients can be computed in the initial request and to push it out doesn't really require any additional database access. | 18:28 |
| J1m | So the clients being pushed to don't need a database connection to get the push. | 18:28 |
| bloodbare | exactly, only once every minut we need to write to the DB | 18:28 |
| J1m | so no need for a shared connection. | 18:29 |
| bloodbare | no | 18:29 |
| bloodbare | so I’ll try to compare the performance numbers from what we have now using the connection as transaction holder and connection tied to request | 18:30 |
| bloodbare | really thanks a lot I hope we will meet at Boston in October :) and really sorry if I annoyied with my comments :P | 18:32 |
| J1m | I look forward to seeing you then. | 18:32 |
| *** J1m has quit IRC | 21:00 | |
| *** J1m has joined #zodb | 21:03 | |
| *** aclark has joined #zodb | 21:05 | |
| *** aclark has quit IRC | 21:05 | |
| *** aclark has joined #zodb | 21:07 | |
| *** jensens has quit IRC | 21:16 | |
| *** aclark has quit IRC | 22:19 | |
| *** jensens has joined #zodb | 22:34 | |
| bloodbare | J1m: Just some tests we did, opening and closing the connection its just 115% slower than reusing a connection :) on 50 concurrent writing operations | 22:40 |
| bloodbare | ( but we are async locking the resource to avoid collisions ) | 22:41 |
| *** aclark has joined #zodb | 23:46 | |
Generated by irclog2html.py 2.15.1 by Marius Gedminas - find it at mg.pov.lt!