[Cubicweb] work on dataimport stores

Sylvain Thénault sylvain.thenault at logilab.fr
Wed Feb 25 08:10:50 CET 2015

On 24 février 17:43, aurélien campéas wrote:
> Hi Sylvain,
Hi Aurélien, 

> > That would be great if some people could take a look at it and may be take
> > some
> > time to champion one store or another, as I suppose that the current
> > "generic"
> > implementation could be optimized depending on each store particularities.
> >
> > You'll find setup instructions in the README file. The results.txt file
> > states
> > the overall goals I'm looking for, and provides some results and
> > discussion. You
> > should probably at least take a look at this if you're interested:
> > http://hg.logilab.org/users/sthenault/dibench/file/tip/results.txt
> I am interested in the memory consumption issue of fastimport.
> However I stop it at around 8mins run time (without seeing a memory
> spike, but with Christophe fix applied).
> Upon investigation I suspect the FastExtEntitiesImporter is suboptimal,
> though I have yet to understand what it does...
> I will dig into this.
Great. As I said, it's much likely that the FastExtEntitiesImporter could be
optimized for the need of one store or another. You're welcome to provide an
implementation tweaked for the fastimport store. We should probably do that
before starting discussing of a common API.

> > You're much welcome to comment about any point on the list or to provide
> > some
> >
> >
> Small note: the patch to handle only eids in fastimport is moot provided a
> small patch
> on dibench that sends entities instead of eids.
> I don't think it will affect performance (though I should test it) since at
> relation insertion
> time the entity is actually cached on the cnx anyway.

Though imo the patch is not that intrusive and makes the fastimport store more
similar to other stores (which may not use entities at all, and so without
anything in the cnx cache). We'll probably want this at some point.

Sylvain Thénault, LOGILAB, Paris ( - Toulouse (
Formations Python, Debian, Méth. Agiles: http://www.logilab.fr/formations
Développement logiciel sur mesure:       http://www.logilab.fr/services
CubicWeb, the semantic web framework:    http://www.cubicweb.org

More information about the Cubicweb mailing list