[Cubicweb] Service API implementation with ømq

Aurélien Campéas aurelien.campeas at logilab.fr
Wed Feb 29 11:54:57 CET 2012


Le 28/02/2012 17:05, Pierre-Yves David a écrit :
>
> The Introduction
> -----------------
>
> This proposal is based on the last sprint conclusion that we badly need
> improved multi-processing capability in cubicweb. It also take ømq as our best
> tool to move in this direction. So we should use less mono lyrical block and more ømq.
>
>
> The Case
> ---------
>
> Cubicweb 3.15.0 is introducing a new "Service" feature[1].
>
> This feature allows to call synchronous and asynchronous "Service" from both
> repo and web part. This a very useful feature that will help lighten several
> part of cubicweb.
>
> However I'm not satisfy with the current repository implementation::
>
>      def call_service(self, sessionid, regid, async, **kwargs):
>         """
>         See :class:`cubicweb.dbapi.Connection.call_service`
>         and :class:`cubicweb.server.Service`
>         """
>         def task():
>             session = self._get_session(sessionid)
>             service = session.vreg['services'].select(regid, session, **kwargs)
>             return service.call(session, **kwargs)
>         if async:
>             self.info('calling service %s asynchronously', regid)
>             self.threaded_task(task)
>         else:
>             self.info('calling service %s synchronously', regid)
>             return task()
>

FWIW here I sure don't like the look it has, esp. the return/no-return 
(function/procedure) depending on the async bool. These aspects should 
be independant of each other.

Is this in cw already ? or still a patch ?

> Issues
> ---------
>
> With this implementation, we are:
>
> 1. using standard python call passing standard python reference,

You mean ... object ?
I have no pb with that ...

> 2. inside the main repository code and thread,
> 3. handling thread spawn by hand at the same place than the remaining logic.
>
>
> This goes in the wrong direction:
>
> :using standard python call passing standard python reference:
>
>      Not restricting usage of this API means that people will use it
>      unrestricted. If people use it unrestricted it'll be painful to move toward
>      more distributed approach.

I using zmq entails restricting ourselves to ints and strings, then the 
zmq way has a problem (maybe).

But also I can imagine a `zmq_(un)serialize` function (and its 
implementations __zmq_(un)serialize__)

>
> :inside the main repository code and thread:
>
>      This code does not belong here:
>
>      Adding any logic code to repository is a bad idea. The logic we are adding
>      here is related to "a server processing tasks" not "data repository".
>
>      For the same reason, repository have no reason to handle Thread spawning
>
> :handling Thread spawn by hand at the same place than the remaining logic:
>
>      Dispatching tasks and running tasks are two different things. We'll
>      probably use efficient third party for the dispatching part one day. Not
>      entangling the two logics now is a later win.
>
>
> Solution
> ---------
>
> We are not ready for multi process instances today, but nothing prevent use of
> ømq to communicate within the same process. The idea here is to have every call
> to service be processed through a ømq socket. Both end of the socket will live
> in the same process and handle the same object. But will will be sure that the
> code is ready to migrate to multi-process instance without more work both in
> core mechanism and user.
>
>
> In practice, I advocate moving most of the logic out the repository class. Only
> keeping such code in the repository:
>
>      def call_service(self, sessionid, regid, async, **kwargs):
>         """
>         See :class:`cubicweb.dbapi.Connection.call_service`
>         and :class:`cubicweb.server.Service`
>         """
>         # the zmq code is garanted wrong
>         msg = [sessionid, regid, kwargs]
>         if async:
>             zmq_push_push.send([sessionid, regid, kwargs])
>          else:
>             return zmq_req_response.send([sessionid, regid, kwargs])

Do we agree that this should be an appobject's method and thus be 
replaceable on a whim ?

>
> At another end of the ømq socket are the code handling the services call.
>
>      def zmq_callback(msg):
>          sessionid, regid, kwargs = msg
>          session = self._get_session(sessionid)
>          service = session.vreg['services'].select(regid, session, **kwargs)
>          return service.call(session, **kwargs)
>
>
> This proposal only change *server side* implementation and does not impact the
> dbapi. However having ømz server side will help future version of the
> dbapi base on ømq too.

The "dbapi" term used in cw is quite confusing (allegedly I'm easily 
confused ...). Shouldn't we talk here about a "repo api" instead ?

>
>
> Conclusion
> -----------
>
> Let's not release anything that will later need to be changed to scope with our
> current goal. Let's enforce ømq limitation **now** so we do not need to handle
> backward compatibility later. Let's open the new ømq way on a new feature with
> no user yet. Let's do it right the first time so we do not need to fix it
> later.
>


Let's build the thing with pieces that work _right_now_ and are made to 
be easily updated/replaced.




More information about the Cubicweb mailing list