[Cubicweb] work on dataimport stores

aurélien campéas aurelien.campeas at gmail.com
Wed Feb 25 13:08:30 CET 2015


Right now, fastimport does not provide a store api, but I plan to do so (if
we agree
to call a "store" the thing that lives in cw/dataimport).

see http://www.cubicweb.org/ticket/4985775

This will reduce the need to write complicated code to use it and provide a
drop-in replacement for other stores.

2015-02-25 8:10 GMT+01:00 Sylvain Thénault <sylvain.thenault at logilab.fr>:

> On 24 février 17:43, aurélien campéas wrote:
> > Hi Sylvain,
>
> Hi Aurélien,
>
> > > That would be great if some people could take a look at it and may be
> take
> > > some
> > > time to champion one store or another, as I suppose that the current
> > > "generic"
> > > implementation could be optimized depending on each store
> particularities.
> > >
> > > You'll find setup instructions in the README file. The results.txt file
> > > states
> > > the overall goals I'm looking for, and provides some results and
> > > discussion. You
> > > should probably at least take a look at this if you're interested:
> > > http://hg.logilab.org/users/sthenault/dibench/file/tip/results.txt
> >
> > I am interested in the memory consumption issue of fastimport.
> > However I stop it at around 8mins run time (without seeing a memory
> > spike, but with Christophe fix applied).
> >
> > Upon investigation I suspect the FastExtEntitiesImporter is suboptimal,
> > though I have yet to understand what it does...
> >
> > I will dig into this.
>
> Great. As I said, it's much likely that the FastExtEntitiesImporter could
> be
> optimized for the need of one store or another. You're welcome to provide
> an
> implementation tweaked for the fastimport store. We should probably do that
> before starting discussing of a common API.
>
> > > You're much welcome to comment about any point on the list or to
> provide
> > > some
> > >
> > >
> > Small note: the patch to handle only eids in fastimport is moot provided
> a
> > small patch
> > on dibench that sends entities instead of eids.
> > I don't think it will affect performance (though I should test it) since
> at
> > relation insertion
> > time the entity is actually cached on the cnx anyway.
>
> Though imo the patch is not that intrusive and makes the fastimport store
> more
> similar to other stores (which may not use entities at all, and so without
> anything in the cnx cache). We'll probably want this at some point.
>
> --
> Sylvain Thénault, LOGILAB, Paris (01.45.32.03.12) - Toulouse
> (05.62.17.16.42)
> Formations Python, Debian, Méth. Agiles: http://www.logilab.fr/formations
> Développement logiciel sur mesure:       http://www.logilab.fr/services
> CubicWeb, the semantic web framework:    http://www.cubicweb.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.cubicweb.org/pipermail/cubicweb/attachments/20150225/c8623696/attachment-0186.html>


More information about the Cubicweb mailing list