[Cubicweb] Service API implementation with ømq

Pierre-Yves David pierre-yves.david at logilab.fr
Wed Feb 29 12:35:51 CET 2012


On Wed, Feb 29, 2012 at 11:54:57AM +0100, Aurélien Campéas wrote:
> Le 28/02/2012 17:05, Pierre-Yves David a écrit :
> >
> >The Introduction
> >-----------------
> >
> >This proposal is based on the last sprint conclusion that we badly need
> >improved multi-processing capability in cubicweb. It also take ømq as our best
> >tool to move in this direction. So we should use less mono lyrical block and more ømq.
> >
> >
> >The Case
> >---------
> >
> >Cubicweb 3.15.0 is introducing a new "Service" feature[1].
> >
> >This feature allows to call synchronous and asynchronous "Service" from both
> >repo and web part. This a very useful feature that will help lighten several
> >part of cubicweb.
> >
> >However I'm not satisfy with the current repository implementation::
> >
> >     def call_service(self, sessionid, regid, async, **kwargs):
> >        """
> >        See :class:`cubicweb.dbapi.Connection.call_service`
> >        and :class:`cubicweb.server.Service`
> >        """
> >        def task():
> >            session = self._get_session(sessionid)
> >            service = session.vreg['services'].select(regid, session, **kwargs)
> >            return service.call(session, **kwargs)
> >        if async:
> >            self.info('calling service %s asynchronously', regid)
> >            self.threaded_task(task)
> >        else:
> >            self.info('calling service %s synchronously', regid)
> >            return task()
> >
> 
> FWIW here I sure don't like the look it has, esp. the
> return/no-return (function/procedure) depending on the async bool.
> These aspects should be independant of each other.
> 
> Is this in cw already ? or still a patch ?

It is un cw already

> 
> >Issues
> >---------
> >
> >With this implementation, we are:
> >
> >1. using standard python call passing standard python reference,
> 
> You mean ... object ?
> I have no pb with that ...
> 
> >2. inside the main repository code and thread,
> >3. handling thread spawn by hand at the same place than the remaining logic.
> >
> >
> >This goes in the wrong direction:
> >
> >:using standard python call passing standard python reference:
> >
> >     Not restricting usage of this API means that people will use it
> >     unrestricted. If people use it unrestricted it'll be painful to move toward
> >     more distributed approach.
> 
> I using zmq entails restricting ourselves to ints and strings, then
> the zmq way has a problem (maybe).
> 
> But also I can imagine a `zmq_(un)serialize` function (and its
> implementations __zmq_(un)serialize__)

I think that automatic serialiszation/deserialization will have to appear one day or another but:

* I'm not sure this is required right now (can be implememented later)

* People have to be aware that it has a cost and try to avoid it when possible.

> >:inside the main repository code and thread:
> >
> >     This code does not belong here:
> >
> >     Adding any logic code to repository is a bad idea. The logic we are adding
> >     here is related to "a server processing tasks" not "data repository".
> >
> >     For the same reason, repository have no reason to handle Thread spawning
> >
> >:handling Thread spawn by hand at the same place than the remaining logic:
> >
> >     Dispatching tasks and running tasks are two different things. We'll
> >     probably use efficient third party for the dispatching part one day. Not
> >     entangling the two logics now is a later win.
> >
> >
> >Solution
> >---------
> >
> >We are not ready for multi process instances today, but nothing prevent use of
> >ømq to communicate within the same process. The idea here is to have every call
> >to service be processed through a ømq socket. Both end of the socket will live
> >in the same process and handle the same object. But will will be sure that the
> >code is ready to migrate to multi-process instance without more work both in
> >core mechanism and user.
> >
> >
> >In practice, I advocate moving most of the logic out the repository class. Only
> >keeping such code in the repository:
> >
> >     def call_service(self, sessionid, regid, async, **kwargs):
> >        """
> >        See :class:`cubicweb.dbapi.Connection.call_service`
> >        and :class:`cubicweb.server.Service`
> >        """
> >        # the zmq code is garanted wrong
> >        msg = [sessionid, regid, kwargs]
> >        if async:
> >            zmq_push_push.send([sessionid, regid, kwargs])
> >         else:
> >            return zmq_req_response.send([sessionid, regid, kwargs])
> 
> Do we agree that this should be an appobject's method and thus be
> replaceable on a whim ?

An appobject methods? A session one maybe?

> >At another end of the ømq socket are the code handling the services call.
> >
> >     def zmq_callback(msg):
> >         sessionid, regid, kwargs = msg
> >         session = self._get_session(sessionid)
> >         service = session.vreg['services'].select(regid, session, **kwargs)
> >         return service.call(session, **kwargs)
> >
> >
> >This proposal only change *server side* implementation and does not impact the
> >dbapi. However having ømz server side will help future version of the
> >dbapi base on ømq too.
> 
> The "dbapi" term used in cw is quite confusing (allegedly I'm easily
> confused ...). Shouldn't we talk here about a "repo api" instead ?

It stand for "I'm accessing to a repository from client code" in opposition to "I'm in a hook/repo side"

> >Conclusion
> >-----------
> >
> >Let's not release anything that will later need to be changed to scope with our
> >current goal. Let's enforce ømq limitation **now** so we do not need to handle
> >backward compatibility later. Let's open the new ømq way on a new feature with
> >no user yet. Let's do it right the first time so we do not need to fix it
> >later.
> >
> 
> 
> Let's build the thing with pieces that work _right_now_ and are made
> to be easily updated/replaced.

IF we do not implement this with ømq _right_know_ we expose ourseld to
something hard to update and replace. We also cut ourself from a perfect
experimentation ground.

-- 
Pierre-Yves David

http://www.logilab.fr/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature
URL: <http://lists.cubicweb.org/pipermail/cubicweb/attachments/20120229/f3dfc66d/attachment-0212.sig>


More information about the Cubicweb mailing list