[Cubicweb] Service API implementation with ømq

Vincent Michel vincent.michel at logilab.fr
Thu Mar 1 17:05:52 CET 2012


On Tuesday 28 February 2012 17:05:37 Pierre-Yves David wrote:
> The Introduction
> -----------------
> 
> This proposal is based on the last sprint conclusion that we badly need
> improved multi-processing capability in cubicweb. It also take ømq as our
> best tool to move in this direction. So we should use less mono lyrical
> block and more ømq.
> 
> 
> The Case
> ---------
> 
> Cubicweb 3.15.0 is introducing a new "Service" feature[1].
> 
> This feature allows to call synchronous and asynchronous "Service" from
> both repo and web part. This a very useful feature that will help lighten
> several part of cubicweb.
> 
> However I'm not satisfy with the current repository implementation::
> 
>     def call_service(self, sessionid, regid, async, **kwargs):
>        """
>        See :class:`cubicweb.dbapi.Connection.call_service`
>        and :class:`cubicweb.server.Service`
>        """
>        def task():
>            session = self._get_session(sessionid)
>            service = session.vreg['services'].select(regid, session,
> **kwargs) return service.call(session, **kwargs)
>        if async:
>            self.info('calling service %s asynchronously', regid)
>            self.threaded_task(task)
>        else:
>            self.info('calling service %s synchronously', regid)
>            return task()
> 
> Issues
> ---------
> 
> With this implementation, we are:
> 
> 1. using standard python call passing standard python reference,
> 2. inside the main repository code and thread,
> 3. handling thread spawn by hand at the same place than the remaining
> logic.
> 
> This goes in the wrong direction:
> :using standard python call passing standard python reference:
>     Not restricting usage of this API means that people will use it
>     unrestricted. If people use it unrestricted it'll be painful to move
> toward more distributed approach.
> 
> :inside the main repository code and thread:
>     This code does not belong here:
> 
>     Adding any logic code to repository is a bad idea. The logic we are
> adding here is related to "a server processing tasks" not "data
> repository".
> 
>     For the same reason, repository have no reason to handle Thread
> spawning
> 
> :handling Thread spawn by hand at the same place than the remaining logic:
>     Dispatching tasks and running tasks are two different things. We'll
>     probably use efficient third party for the dispatching part one day.
> Not entangling the two logics now is a later win.
> 
> 
> Solution
> ---------
> 
> We are not ready for multi process instances today, but nothing prevent use
> of ømq to communicate within the same process. The idea here is to have
> every call to service be processed through a ømq socket. Both end of the
> socket will live in the same process and handle the same object. But will
> will be sure that the code is ready to migrate to multi-process instance
> without more work both in core mechanism and user.
> 
> 
> In practice, I advocate moving most of the logic out the repository class.
> Only keeping such code in the repository:
> 
>     def call_service(self, sessionid, regid, async, **kwargs):
>        """
>        See :class:`cubicweb.dbapi.Connection.call_service`
>        and :class:`cubicweb.server.Service`
>        """
>        # the zmq code is garanted wrong
>        msg = [sessionid, regid, kwargs]
>        if async:
>            zmq_push_push.send([sessionid, regid, kwargs])
>         else:
>            return zmq_req_response.send([sessionid, regid, kwargs])
> 
> At another end of the ømq socket are the code handling the services call.
> 
>     def zmq_callback(msg):
>         sessionid, regid, kwargs = msg
>         session = self._get_session(sessionid)
>         service = session.vreg['services'].select(regid, session, **kwargs)
>         return service.call(session, **kwargs)
> 
> 
> This proposal only change *server side* implementation and does not impact
> the dbapi. However having ømz server side will help future version of the
> dbapi base on ømq too.
> 
> 
> Conclusion
> -----------
> 
> Let's not release anything that will later need to be changed to scope with
> our current goal. Let's enforce ømq limitation **now** so we do not need
> to handle backward compatibility later. Let's open the new ømq way on a
> new feature with no user yet. Let's do it right the first time so we do
> not need to fix it later.

JFYI, from what I've seen during the sprint, the debian package of pyzmq seems 
a bit old (some annoying issues exist). Of course, in order to avoid these 
bugs, it is still possible download pyzmq from the Git repo.

My two cents,

Vincent




More information about the Cubicweb mailing list