[Cubicweb] work on dataimport stores
aurélien campéas
aurelien.campeas at gmail.com
Thu Feb 26 09:10:59 CET 2015
2015-02-25 23:30 GMT+01:00 Sylvain Thénault <sylvain.thenault at logilab.fr>:
> On 25 février 18:30, aurélien campéas wrote:
> > I pushed a patch on fastimport to provide a store.
>
> nice :)
>
> > Using this the dibench code can shrink a bit.
> > I reran the tests.
> >
> > You can use the joint diff on dibench.
>
> I've integrated all the code related part. I get an error on the
> fastimport test
> after that though:
>
> UnknownEid: No entity with eid -1 in the repository
>
Ah, that's probably because I provide a fake eid on return on .create_entity
and the importer wants to use it.
>
> This is probably related to the fact that I've updated the overall
> benchmark
> code to use the latest skos cube, there was a severe issue before that:
> hooks
> we actually not activated!
>
Yes indeed. Wasn't it by design ? That's what I thought anyway.
>
> > Note that to make the test pass I had to de-inline the pref_label
> relation.
> >
> > I'm seriously considering adding inlined relation support to
> > .insert_relations.
>
> If not, this deserves a note in the results.txt file. Notice that iirc
> other stores
> rely on inlined relations being set through insert_entity or similar.
>
Maybe but that relation is reported missing in the check after the
fastimport test, even though it now uses the standard skos import code.
>
> Once we get all that fixed, we should add some kind of final measurements.
> We
> should probably also add the RQLObjectStore btw.
>
Yes but it takes so much time and I have so much patience ... :-p
>
> > +auc: can't the massive object store also slow down considerably
> concurrent transactions while it runs ?
>
> Why more than another? That's possible, though I would think the
> performance
> gain is worth it...
>
IIRC dataio drops indexes while performing
the import, and because of this, some index scans are probably transformed
in linear scan for concurrent transaction. Unless postgres is awesomer than
I
believe it is !
Cheers,
Aurélien.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.cubicweb.org/pipermail/cubicweb/attachments/20150226/5aeccfc3/attachment-0186.html>
More information about the Cubicweb
mailing list