[Cubicweb] work on dataimport stores

Sylvain Thénault sylvain.thenault at logilab.fr
Thu Feb 26 09:38:33 CET 2015


On 26 février 09:10, aurélien campéas wrote:
> 2015-02-25 23:30 GMT+01:00 Sylvain Thénault <sylvain.thenault at logilab.fr>:
> > This is probably related to the fact that I've updated the overall
> > benchmark
> > code to use the latest skos cube, there was a severe issue before that:
> > hooks
> > we actually not activated!
> >
> 
> Yes indeed. Wasn't it by design ? That's what I thought anyway.

no that was because of a misunderstanding. If we want to compare the store
performance vs pro and cons of each one, we should imo run the test with hooks
activated.


> > > Note that to make the test pass I had to de-inline the pref_label
> > relation.
> > >
> > > I'm seriously considering adding inlined relation support to
> > > .insert_relations.
> >
> > If not, this deserves a note in the results.txt file. Notice that iirc
> > other stores
> > rely on inlined relations being set through insert_entity or similar.
> 
> Maybe but that relation is reported missing in the check after the
> fastimport test, even though it now uses the standard skos import code.

Though this used to work on my first implementation using the fastimport cube.
It's unclear to me what's changed that may cause that.
 
> > Once we get all that fixed, we should add some kind of final measurements.
> > We
> > should probably also add the RQLObjectStore btw.
> 
> Yes but it takes so much time and I have so much patience ... :-p

:) 


> > > +auc: can't the massive object store also slow down considerably
> > concurrent transactions while it runs ?
> >
> > Why more than another? That's possible, though I would think the
> > performance
> > gain is worth it...
> >
> 
> IIRC dataio drops indexes while performing
> the import, and because of this, some index scans are probably transformed
> in linear scan for concurrent transaction. Unless postgres is awesomer than
> I
> believe it is !

You're probably true here. That should be explained in the missing section of
the results.txt file explaining each store strategy, pro and cons.

-- 
Sylvain Thénault, LOGILAB, Paris (01.45.32.03.12) - Toulouse (05.62.17.16.42)
Formations Python, Debian, Méth. Agiles: http://www.logilab.fr/formations
Développement logiciel sur mesure:       http://www.logilab.fr/services
CubicWeb, the semantic web framework:    http://www.cubicweb.org



More information about the Cubicweb mailing list