IRC log of #zope3-dev for Saturday, 2005-04-02

srichterJ1m: I want to write a small preferences setup where the user can choose the skin s/he wants to use...00:01
srichterwhat would be the best package name?
srichterLater we probably want to expand the preferences00:01
*** nathany has left #zope3-dev00:06
*** srichter has quit IRC00:14
*** admp has joined #zope3-dev00:24
*** admp has quit IRC00:26
*** srichter has joined #zope3-dev00:36
*** ChanServ sets mode: +o srichter00:38
tarek_srichter: I have written down what would be a global utility in charge of executing code asynchronously,00:51
tarek_i would like to make a proposal on this00:52
*** tonico has quit IRC00:52
srichtertarek_: ok, do it :-)00:53
srichteror are you not a power user yet and cannot modify proposals?00:53
srichteryou shoulod always have a bookmark to:
tarek_ok i have it00:54
srichterclick on Proposals and add it there00:54
tarek_ok i'll do it now00:54
*** admp has joined #zope3-dev01:00
tarek_oh well, there is a very similar Proposal in progress : TaskSchedule01:00
*** admp is now known as admp|sleep01:00
srichtertarek_: it has not been worked on for a long time (like all proposals)01:01
srichterso if you are willing to do a new proposal and implement it, feel free to change the existing one to your heart's content01:02
tarek_srichter:  ok, it's a bit differen, i will add a new one and make links on it01:02
srichtertarek_: btw, add new proposals to the top01:08
tarek_ok, i'll move it up01:09
tarek_I hope it won't look ridiculous, as my knowledge of Z3  is very poor at this time :)01:12
srichtertarek_: the worst that can happen is that people correct your proposal and comment on open issues01:13
srichterI think the tough issue here is transaction handling with the ZODB01:13
tarek_yes indeed01:14
srichterwhich is not a Z3-pure issue01:14
*** J1m has quit IRC01:31
*** admp|sleep has quit IRC01:35
*** SteveA has quit IRC01:39
*** SteveA has joined #zope3-dev01:39
*** nisha_cgx has joined #zope3-dev01:47
*** FarcePest has quit IRC01:55
*** Arnia has quit IRC01:56
*** tarek_ is now known as tarek_away02:12
*** bradb has quit IRC02:25
projekt01srichter, I will look at preferences tomorrow.02:37
*** bradb has joined #zope3-dev02:54
srichterprojekt01: ok, I am writing a demo right now03:12
*** BjornT has quit IRC03:20
*** eaon has quit IRC03:30
*** bskahan has joined #zope3-dev04:27
*** RaFromBRC is now known as RaFromBRC|afk04:36
*** _projekt01 has joined #zope3-dev04:47
*** projekt01 has quit IRC04:47
*** BjornT has joined #zope3-dev04:59
*** bskahan has quit IRC05:12
*** _projekt01 has left #zope3-dev05:30
*** nisha_cgx has quit IRC05:30
*** `anthony has quit IRC05:40
*** `anthony has joined #zope3-dev05:41
*** RaFromBRC|afk is now known as RaFromBRC06:07
*** `anthony has quit IRC06:27
*** `anthony has joined #zope3-dev06:39
*** RaFromBRC is now known as RaFromBRC|out06:46
*** hazmat has quit IRC06:51
*** th1a has quit IRC07:51
*** admp has joined #zope3-dev08:36
*** BjornT has quit IRC09:57
*** d2m has joined #zope3-dev10:33
*** RaFromBRC|out is now known as RaFromBRC10:43
*** htd has joined #zope3-dev11:26
*** tarek_away has quit IRC11:42
*** tarek_away has joined #zope3-dev11:49
*** mexiKON has joined #zope3-dev12:11
*** gintas has joined #zope3-dev12:24
*** philiKON has quit IRC12:34
*** tarek_away has quit IRC13:02
*** viyyer has joined #zope3-dev13:43
*** projekt01 has joined #zope3-dev14:06
*** gintas has quit IRC14:14
*** htd has left #zope3-dev14:25
*** stub has quit IRC14:46
*** admp has joined #zope3-dev14:52
*** J1m has joined #zope3-dev15:04
*** tarek_ has joined #zope3-dev15:15
*** projekt01 has left #zope3-dev15:23
srichtergood morning everyine15:23
tarek_does anyone knows where i can find documentation about ZODB transactions ?15:24
tarek_also, good morning  :)15:24
srichterJ1m: I got the preferences system done yesterday; I am pretty happy with it; as demo you can change the skin you are using by simply changing the setting :-)15:25
srichtertarek_: I think you do not need to lock the ZODB explicitely; somehow the code should be able to handle ConflictErrors15:26
tarek_in fact, i get weird errors in the thread15:27
tarek_so i have tried to do a commit / begin in it but it's not the solution15:27
srichterdo you properly start and end the transaction?15:27
J1mtarek_, what is the problem?15:28
srichtermmh, maybe we can reuse some of the publisher framework15:28
tarek_ok, i am launching a thread from zope that does indexing stuff15:29
J1mwhy are you launching a thread?15:29
tarek_the goal is described in the proposal i have made15:29
tarek_let me get you the url /15:29
tarek_in two words i need to run aysnchronous things15:31
J1mDo you mean, you want, from a web request to start processing that the web request doesn't wait for?15:32
tarek_to give back the hand to the user15:33
tarek_like does for sending mails,15:33
J1mI don't know how works.15:34
tarek_it's a threaded queue that sends mail asynchronously15:34
J1mso thre is a separate thread that handles the queue?15:35
tarek_the extra thread is created once and get jobs15:35
J1mAnd app code adds things to the queue, by writing data to a directory?15:35
J1mThis doesn't sound like what you want.15:36
tarek_not exactly15:36
tarek_since I want to execute methods15:36
tarek_but i do want a queue that launches those methods15:36
tarek_and die after the work is done15:37
J1msounds cmplicated15:37
J1msounds complicated15:37
J1mLet's get back to the original use case15:37
tarek_ok, in other words15:37
tarek_the use case :15:37
tarek_i have a webmail15:37
tarek_that shows mails15:37
*** efge has joined #zope3-dev15:38
tarek_when a mail is shown, there's a few things that has to be sent to imap server15:38
tarek_those thingdoes not interfer with the render15:38
tarek_but has to be done then15:38
tarek_one example :15:38
tarek_setting a flag to "read"15:38
efgeyo tarek_15:39
tarek_hello efge15:39
tarek_this lags sometimes, or worse, makes my web request timout15:39
tarek_if it's async it should be ok15:39
tarek_but for serialization15:39
tarek_the best way would be to have a queue per session id15:40
J1mThis particular example doesn't need access to ZODB, does it?15:40
J1mwhy is the "best way to have a queue per session id"?15:40
tarek_no you are right, but as a matter of fact, it's calling some imap handling utility15:40
tarek_because one session id = one mailbox15:41
J1mwhat is a session id?15:41
tarek_what i call a session id, is a user connection15:41
J1mto imap?15:41
tarek_to the web app15:41
*** projekt01 has joined #zope3-dev15:41
J1mSo having a queue per web session sounds especially complicated15:42
J1mwhy do you need that?15:42
J1mwhy a queue?15:42
tarek_to be sure that all calls to one imap mailbox are serialized15:43
*** SteveA has quit IRC15:43
J1mThis keeps getting more and more complicated and now fairly imap specific15:43
tarek_there must be more elegant and simpler way to do it for sure :)15:45
tarek_do you want me to erase that proposal ?15:45
J1mI don't want something in the core to try to solve your specific problem.15:46
srichterno, proposals don't get ereased; the worst would be to document why this proposal was not accepted, reject it and move it to OldProposals15:46
J1mWhy do you need to serialize requests to imap?15:47
tarek_i don't think it's imap specific15:47
tarek_it's just a web specific thing imho :15:47
J1mYou have this weird serialization requirement that seems to be imap specific15:47
tarek_they can be stuff done to render a page, and work done background15:48
J1mYou don't need a queue for that15:48
J1mWhy do you need to serialize requests to imap?15:48
efgeSome imap operations cannot be reordered15:49
tarek_here's the two reasons that made me thaught about it :15:49
tarek_1. orders15:49
tarek_2. if there are too much calls there goigne to be threads all over the place15:49
tarek_the order one is evry imap specific you are right15:52
tarek_but imho the 2. is not15:52
J1mIt's also unclear what the ordering requirement is.15:52
*** mexiKON has quit IRC15:53
srichterJ1m: why? what comes in first must be done first.15:54
srichterJ1m: what's not clear about that?15:54
J1mHow do you decide what comes in first?  Especially in a multi-threaded or multi-process (ZEO) application?15:55
J1mRequest start time?15:55
J1mRequest fincish time?15:55
J1mTransaction cmmit order?15:55
tarek_it's up to the system15:55
J1mTransaction commit order?15:55
J1mThat's not an answer15:56
tarek_i mean, anyone would fit15:56
srichterhowever, I think we do not need to overthink the problem15:56
J1mIf any policy would fit, then I'm skeptical of the ordering requirement15:57
srichterbasically we want to ensure that a certain step A is done before B15:57
tarek_it has to be fixed somewhere within the process15:57
J1mIs it just important that operations performed in a single transaction are ordered?15:57
srichtera single user would always send A before B15:57
J1msrichter, if they are sending them in separate web requests, the order isn't clear15:58
srichterwhy not?15:58
J1mare we really only talking about operations within the same transaction?15:58
srichterIn the UI, the user does first request A then B15:58
J1mBecause the browser may send the requests in paralel15:58
srichterthat's true, but not practical for a single user15:58
J1mIt happens all the time, depending on the app15:59
srichterI cannot delete a mail before I fetched it, for example15:59
J1mwhy not?15:59
srichterbecause I do not know I have it in the first place15:59
J1mso, you are saying that the user will view the result of one op before sending another16:00
J1mbut we seem to be talking about ops that the user doesn't wait for and thus doesn't see16:00
tarek_or see aftewards16:01
srichter:-) good point; so I should think about something else16:01
srichterlet's say a message gets moved to folder X, but then I delete the message from folder X16:01
J1mIn that case, it's not so clear what the order is16:01
J1mAre these done in 2 web requests?16:02
J1mThen you are toast16:02
tarek_let's say : i am looking to my folder,16:02
tarek_fresh mails there16:02
J1mSo stop a second16:02
tarek_i read them and the thread tells map i have read them16:02
tarek_next tile i get to the folder the icon "new mail" changed16:03
J1mThe only way to make this work is if you commit the request to a queue that is transactional16:03
J1mYou don't return from the first request until the op request has been written to a transactional queue16:03
tarek_would'nt that be the threaded queue ?16:04
J1mI don't know what you mean by a threaded queue16:04
tarek_a transactinonal queue that is not running in the same thread than the first request16:05
tarek_that's what i need yeah :)16:05
J1mBut it needs to be able to work with multiple processes16:05
J1mThere queue needs to be in a transactional store, like ZODB16:06
tarek_that's the part wich is way over my head16:06
J1mTherefore, the objects put in the queue cannot be simple functions.16:06
tarek_i've thaught about adapters16:06
J1mThis is the sort of problem that the tesk schedule was designed for16:07
tarek_that could wrap the method somehow16:07
J1mYou need objects16:07
tarek_are your talking about the Task ScheduleProposal?16:08
J1mFor example, to mark a message read, you create a MessageReadTask which has necessary data16:08
J1mYou add the message to the task schedule16:08
J1m(or a task queue)16:08
J1mIn fact, forget the task schedule16:09
J1mI would recommend a local task queue utility16:10
tarek_sounds good16:10
J1mIt has to be local because, for now, only local utilities can store data in zodb16:11
tarek_oh ok16:11
J1mThe tricky part is going to be avoiding write conflicts.16:11
tarek_yes indeed16:11
J1mYopu would then need to define a separate thread or process to process the queue16:12
J1mYou would then need to define a separate thread or process to process the queue16:12
J1mThis separate thread or process would have to run continuously, polling and processing the queue16:12
J1m(obviously with sleep when there's nothing to do)16:13
tarek_the hardest part is the write protection16:13
J1mI guess that write conflicts can be handled using conflict resolution16:14
J1mBut it will require some cleverness.16:14
tarek_and a good load of zodb knowledge :-)16:14
J1mNote that the tasks should be picklable, but not persistent.16:15
J1mYou don't want to consume database records for them.16:15
tarek_ok , right16:15
J1mNot so much zodb knowledge, but you will need a clever conflict resolution algorithm.16:15
*** SteveA has joined #zope3-dev16:17
J1mI'm still skeptical that you will get the ordering you want on top of a stateless protocol.16:17
J1mI think people tend to expect too much from HTTP16:18
J1mIf people use a web-mail system rather than a real imap client, they should expect less, IMO16:18
J1mHTTP is overused, IMO16:18
tarek_I agree, but Zope is so powerful, we want to do extensive things with it16:19
* tarek_ is not sure of the word "extensive"16:19
J1mGreat, but there are inherent limits in HTTP. Of course, Zope isn't limited to HTTP.16:20
tarek_do you think it could be interesting to modify the proposal with all this points, or is this more likely to be something out of the proposals ?16:20
J1mI think it's something that should be prototyped as a third-party app.16:20
J1mI'm not clear how generally useful this will be.16:20
J1mIt's very specific to the queuing desire.16:21
J1mThere are much simpler solutions if queuing isn't necessary.16:21
J1mAnd the price to pay is pretty high.16:22
J1mManaging the queue is a pain16:22
J1mFor some apps, the pain may be worth it16:22
J1mHere's an example that illustrates the weakness of HTTP.16:22
J1mThe user views their folder.16:23
J1mThe see an unread message.16:23
J1mThe read the message.16:23
J1mWhen the message is read, an operation is queued to update imap16:23
J1mThe user then decides to view the folder again.16:23
J1mYou can't queue the folder display because you need the result to show the user,16:24
J1mThe queue is backed up for whatever reason and the request to update imap hasn't been processed.16:24
*** projekt01 has left #zope3-dev16:24
J1mso the user sees the folder listing with the message shown as unread.16:25
J1mThis is unavoidable because HTTP is stateless.16:25
J1mIf a stateful protocol was used, then the second folder display would be held until the read operations was written to the imap server.16:26
srichterUnless you locally already mark the message as read and just send the op to IMAP for preocessing16:26
tarek_yes that's what i do16:26
J1mwell, that's a totally different architecture.16:27
tarek_it's just a matter of syncing imap with local operations16:27
J1mIn that case, why bother with the imap server in the first place16:27
J1mSo, you are going to duplicate the state of the imap server on the web server so you have an up to date representation.16:27
J1mAnd then sync over time?16:28
srichterwell, is that not exactely what we try toi do?16:28
srichterwe record all the changes and have it sync over time16:28
srichterexcept that we clear the queue as fast as possible instead of waiting a while16:29
J1mNo, we don't simply record the changes, we also keep a local representation of the mailbox.16:29
tarek_a partial representation16:29
J1mThe mailbox folder listing doesn't use imap at all, it uses the local copy16:29
J1mI think it needs to be a full representation.16:30
J1mor pretty full16:30
tarek_yes but imap keeps bodies and atatched files16:30
tarek_zodb stores headers infos and flags in my case16:30
J1mIf it's not full, then you have to be even more clever about keeping track of changes.16:30
tarek_bodies and attached file do not change16:31
J1mEssentially, you are building a mozilla mail client into your mail server.16:31
J1mEssentially, you are building an imap mail client into your mail server.16:31
tarek_99% of operations on a mailbox can be done without calling an imap server if there is a local copy of folder structure and header infos16:33
J1mwhy use an imap server at all?16:33
tarek_to avoid having bodies and attached files16:34
efgeand because that's what's already there and what customers want to use16:35
J1mThat seems like *lot* of complexity *just* to avoid that,16:35
J1m(Especially given the recent blob work.)16:35
efgeany IMAP client keeps a local dumbed down representation of the content on IMAP...16:35
srichtertarek_: are you sure? All ops I can think of require a change on the IMAP server16:35
tarek_srichter: most of the time you are browsing folder and make searches16:36
* J1m is pissed that gnome now overrides my keyboard setup16:36
srichterJ1m: get a real desktop and use KDE :-)16:36
* J1m is tempted16:37
srichterKDE works nicely together with xkb (or however it is called)16:37
J1mI'm not familiar with xkb16:37
efgeI just read the past messages and I agree that a Task list stored in ZODB is the way to go for async16:37
J1mI'm using xmodmap16:38
tarek_efge: yes indeed16:38
srichteranyways, I my CAPS LOCAK works great as TAB16:38
J1mefge, except that, in this case, a straight-ahead queue is probably more apropriate16:38
efgeI'm just wondering about what J1m said: "Note that the tasks should be picklable, but not persistent. You don't want to consume database records for them."16:38
J1mThey are never mutable so they don't need to subclass Persistent16:39
efgeto me that means that whenever something is changed in the queue, everything is re-written, that's a lot of traffic. I'd have done small persistent objects. That also helps with conflict resolution16:39
efgechanged = task added or removed16:40
efgesrichter: doh, CAPS LOCK should be CTRL not TAB !! :)16:40
srichterthat's what I meant16:41
J1mNo, caps lock should be function 35, so I can remap it to something useful.  (I already have a control key.)16:41
efgeJim you use emacs without caps-lock as control! must be hard on the little finger...16:42
tarek_ok thanks for the talk all, it was very interesting | have to run16:43
*** tarek_ is now known as tarek_away16:43
efgeJ1m: would Z2's QueueCatalog architecture be close to what you're suggesting here ?16:45
J1mI don't think so.16:48
J1mIt ios far more specialized.16:48
J1mIt is far more specialized.16:48
J1mIt's related, of course.16:54
J1m(It's conflict resolution code has someproblems though)16:54
J1mefge, wrt using or not using persistent objects, it would not necessarily affect conflict resolution.17:00
efgeJ1m: my feeling was that it'd be simpler to add to the queue concurrently with removing something if a system of buckets or the like was used17:01
efgebut that's only part of the potential conflicts17:01
J1mbuckets of some sort could still be helpful17:02
J1mThose would be persistent.17:03
*** bskahan has joined #zope3-dev17:16
*** J1m has quit IRC17:18
*** J1m has joined #zope3-dev17:22
*** kaczordek has joined #zope3-dev17:30
*** kaczordek has quit IRC17:33
*** admp has quit IRC17:35
*** admp has joined #zope3-dev17:36
*** admp has quit IRC17:40
*** bskahan has quit IRC17:43
*** BjornT has joined #zope3-dev17:46
*** gintas has joined #zope3-dev17:49
*** tonico has joined #zope3-dev17:51
*** J1m has quit IRC17:56
*** admp has joined #zope3-dev18:01
*** SteveA has quit IRC18:08
*** SteveA has joined #zope3-dev18:22
*** bskahan has joined #zope3-dev18:33
*** admp has joined #zope3-dev18:40
*** admp has quit IRC18:42
*** bskahan has quit IRC18:57
*** bskahan has joined #zope3-dev19:02
*** admp has joined #zope3-dev19:02
*** tonico has quit IRC19:02
*** admp has quit IRC19:35
*** palmTree has joined #zope3-dev19:49
*** admp has joined #zope3-dev19:52
*** admp has joined #zope3-dev20:02
*** admp has quit IRC20:02
*** projekt01 has joined #zope3-dev20:08
*** tvon has joined #zope3-dev20:26
*** bskahan has quit IRC20:35
*** hazmat has joined #zope3-dev21:06
*** bskahan has joined #zope3-dev21:07
*** philiKON has joined #zope3-dev21:19
*** hazmat has left #zope3-dev21:52
*** tarek_away is now known as tarek21:53
*** RaFromBRC has quit IRC21:54
*** admp has joined #zope3-dev22:07
*** projekt01 has quit IRC22:10
*** palmTree has quit IRC22:19
*** bskahan has quit IRC22:20
*** tarek has quit IRC22:26
*** deo has quit IRC22:28
*** tarek has joined #zope3-dev22:34
*** palmTree has joined #zope3-dev22:40
*** admp has quit IRC22:42
*** projekt01 has joined #zope3-dev23:11
*** tvon has quit IRC23:32

Generated by 2.15.1 by Marius Gedminas - find it at!