[Cubicweb] work on dataimport stores

[aurélien campéas]

2015-02-25 23:30 GMT+01:00 Sylvain Thénault <sylvain.thenault at logilab.fr>:

> On 25 février 18:30, aurélien campéas wrote:
> > I pushed a patch on fastimport to provide a store.
>
> nice :)
>
> > Using this the dibench code can shrink a bit.
> > I reran the tests.
> >
> > You can use the joint diff on dibench.
>
> I've integrated all the code related part. I get an error on the
> fastimport test
> after that though:
>
>   UnknownEid: No entity with eid -1 in the repository
>

Ah, that's probably because I provide a fake eid on return on .create_entity
and the importer wants to use it.


>
> This is probably related to the fact that I've updated the overall
> benchmark
> code to use the latest skos cube, there was a severe issue before that:
> hooks
> we actually not activated!
>

Yes indeed. Wasn't it by design ? That's what I thought anyway.


>
> > Note that to make the test pass I had to de-inline the pref_label
> relation.
> >
> > I'm seriously considering adding inlined relation support to
> > .insert_relations.
>
> If not, this deserves a note in the results.txt file. Notice that iirc
> other stores
> rely on inlined relations being set through insert_entity or similar.
>

Maybe but that relation is reported missing in the check after the
fastimport test, even though it now uses the standard skos import code.



>
> Once we get all that fixed, we should add some kind of final measurements.
> We
> should probably also add the RQLObjectStore btw.
>

Yes but it takes so much time and I have so much patience ... :-p


>
> > +auc: can't the massive object store also slow down considerably
> concurrent transactions while it runs ?
>
> Why more than another? That's possible, though I would think the
> performance
> gain is worth it...
>

IIRC dataio drops indexes while performing
the import, and because of this, some index scans are probably transformed
in linear scan for concurrent transaction. Unless postgres is awesomer than
I
believe it is !


Cheers,
Aurélien.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.cubicweb.org/pipermail/cubicweb/attachments/20150226/5aeccfc3/attachment-0186.html>