[Cubicweb] work on dataimport stores
aurelien.campeas at gmail.com
Wed Feb 25 13:08:30 CET 2015
Right now, fastimport does not provide a store api, but I plan to do so (if
to call a "store" the thing that lives in cw/dataimport).
This will reduce the need to write complicated code to use it and provide a
drop-in replacement for other stores.
2015-02-25 8:10 GMT+01:00 Sylvain Thénault <sylvain.thenault at logilab.fr>:
> On 24 février 17:43, aurélien campéas wrote:
> > Hi Sylvain,
> Hi Aurélien,
> > > That would be great if some people could take a look at it and may be
> > > some
> > > time to champion one store or another, as I suppose that the current
> > > "generic"
> > > implementation could be optimized depending on each store
> > >
> > > You'll find setup instructions in the README file. The results.txt file
> > > states
> > > the overall goals I'm looking for, and provides some results and
> > > discussion. You
> > > should probably at least take a look at this if you're interested:
> > > http://hg.logilab.org/users/sthenault/dibench/file/tip/results.txt
> > I am interested in the memory consumption issue of fastimport.
> > However I stop it at around 8mins run time (without seeing a memory
> > spike, but with Christophe fix applied).
> > Upon investigation I suspect the FastExtEntitiesImporter is suboptimal,
> > though I have yet to understand what it does...
> > I will dig into this.
> Great. As I said, it's much likely that the FastExtEntitiesImporter could
> optimized for the need of one store or another. You're welcome to provide
> implementation tweaked for the fastimport store. We should probably do that
> before starting discussing of a common API.
> > > You're much welcome to comment about any point on the list or to
> > > some
> > >
> > >
> > Small note: the patch to handle only eids in fastimport is moot provided
> > small patch
> > on dibench that sends entities instead of eids.
> > I don't think it will affect performance (though I should test it) since
> > relation insertion
> > time the entity is actually cached on the cnx anyway.
> Though imo the patch is not that intrusive and makes the fastimport store
> similar to other stores (which may not use entities at all, and so without
> anything in the cnx cache). We'll probably want this at some point.
> Sylvain Thénault, LOGILAB, Paris (01.45.32.03.12) - Toulouse
> Formations Python, Debian, Méth. Agiles: http://www.logilab.fr/formations
> Développement logiciel sur mesure: http://www.logilab.fr/services
> CubicWeb, the semantic web framework: http://www.cubicweb.org
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Cubicweb