[Cubicweb] work on dataimport stores

aurélien campéas aurelien.campeas at gmail.com
Thu Feb 26 15:11:44 CET 2015


2015-02-26 13:33 GMT+01:00 Sylvain Thénault <sylvain.thenault at logilab.fr>:

> On 26 février 12:38, aurélien campéas wrote:
> > If we want to compare comparable things, we should have at least a run
> > with hooks explicitly disabled. That's what I do in the first joint diff.
>
> IMO we should include fastimport and RQLObjectStore both with and without
> hooks activated. In any case, we should ensure that standard metadatas are
> properly set and we could tweak hooks so that e.g. only integrity hooks
> are run.
>

Yes, both.
In any case, fastimport provides metadata (even "incorrectly" e.g. if the
metadata
category is disabled).

"activeintegrity" hooks may have to be special cased also ...


>
> > Note that fastimport may appear faster than it should because it is
> missing
> > symmetric relation handling (and I havent quantified its impact, but it
> > should
> > not be huge).
>
> Probably not.
>
> > With hooks activated (followup diff) the run time explodes. It shows
> > how ruthlessly inefficient the cubicweb hook system is currently... but
> > nothing that we didn't knew before hand.
>

I did a mistake that made the hooks run twice, so it is a little less bad
than it really
looks ...


>
> I though the point of fastimport was to provide a custom hooks manager
> implementation. And that's why I thought we should run the store with hooks
> activated (and now that's why we should also add the RQLObjectStore with
> hooks
> activated for fair comparison).
>

Yes, that's the point. But it's important to have a benchmark that compares
baseline
behaviours of the current stores, without the hooks overhead, just showing
the other
run time aspects overhead.

I'll leave it to you to schedule a night-run of the RQLObject store with
hooks :)


>
> > Note that fastimport precisely gives tools to fine-tune the runnable
> hooks
> > (decide which to skip or which to defer in later transactions) to work
> > around
> > this general issue.
>
> OK. This should be demonstrated or at least described a bit in the (yet to
> be
> written) "strategy" section.
>

I'll start with the (yet to be written) fastimport documentation ...
Not that the code is unreadable, but some simple high-level description of
the api
should be provided.


Regards,
Aurélien.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.cubicweb.org/pipermail/cubicweb/attachments/20150226/9ed5f342/attachment-0127.html>


More information about the Cubicweb mailing list