[Cubicweb] work on dataimport stores

aurélien campéas aurelien.campeas at gmail.com
Thu Feb 26 09:10:59 CET 2015

2015-02-25 23:30 GMT+01:00 Sylvain Thénault <sylvain.thenault at logilab.fr>:

> On 25 février 18:30, aurélien campéas wrote:
> > I pushed a patch on fastimport to provide a store.
> nice :)
> > Using this the dibench code can shrink a bit.
> > I reran the tests.
> >
> > You can use the joint diff on dibench.
> I've integrated all the code related part. I get an error on the
> fastimport test
> after that though:
>   UnknownEid: No entity with eid -1 in the repository

Ah, that's probably because I provide a fake eid on return on .create_entity
and the importer wants to use it.

> This is probably related to the fact that I've updated the overall
> benchmark
> code to use the latest skos cube, there was a severe issue before that:
> hooks
> we actually not activated!

Yes indeed. Wasn't it by design ? That's what I thought anyway.

> > Note that to make the test pass I had to de-inline the pref_label
> relation.
> >
> > I'm seriously considering adding inlined relation support to
> > .insert_relations.
> If not, this deserves a note in the results.txt file. Notice that iirc
> other stores
> rely on inlined relations being set through insert_entity or similar.

Maybe but that relation is reported missing in the check after the
fastimport test, even though it now uses the standard skos import code.

> Once we get all that fixed, we should add some kind of final measurements.
> We
> should probably also add the RQLObjectStore btw.

Yes but it takes so much time and I have so much patience ... :-p

> > +auc: can't the massive object store also slow down considerably
> concurrent transactions while it runs ?
> Why more than another? That's possible, though I would think the
> performance
> gain is worth it...

IIRC dataio drops indexes while performing
the import, and because of this, some index scans are probably transformed
in linear scan for concurrent transaction. Unless postgres is awesomer than
believe it is !

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.cubicweb.org/pipermail/cubicweb/attachments/20150226/5aeccfc3/attachment-0186.html>

More information about the Cubicweb mailing list