[Cubicweb] Using fastimport

Christophe de Vienne christophe at unlish.com
Wed Jan 7 13:55:33 CET 2015


A follow-up on the memory leak:

Aurélien and I had a closer look and found how to avoid the memory leak:

    After calling insert_entities, do a "cnx.user._cw_related_cache.clear()"


The result is great, I can import 133000 records in less than 5m instead
of more than 17m without fastimport.

Cheers,

Christophe

Le 07/01/2015 08:42, aurélien campéas a écrit :
> 
> 
> 2015-01-07 0:21 GMT+01:00 Christophe de Vienne <christophe at unlish.com
> <mailto:christophe at unlish.com>>:
> 
>     Just FYI, there is a memory leak somewhere.
> 
>     I do batch insertions, the batch size is 250, and the more it goes, to
>     slowest it becomes, and the memory usage jump to several gigabytes (more
>     than 10G) after ~36000 insertions.
> 
>     I don't have this problem with a classic create_entity.
> 
> 
> You will have to show some bits of code to help address this.
>  
> 
> 
>     Cheers,
> 
>     Christophe
> 
>     Le 07/01/2015 00:02, Christophe de Vienne a écrit :
>     > Also,
>     >
>     > 3. Why does FlushController take a session and not a cnx ?
> 
> 
> Because it is still cw 3.17 compatible.
>  
> 
>     >
>     > Le 06/01/2015 23:51, Christophe de Vienne a écrit :
>     >> Hi everyone,
>     >>
>     >> I am trying to use the fastimport module and have some questions.
>     >>
>     >> 1. Why are some deferred hooks sent to a CWWorkerTask ?
>     >>    Can I avoid that ?
>     >>
>     >> 2. Why is there a dependency on numpy ? It feels a bit heavy.
>     >>
>     >>
>     >> Cheers,
>     >>
>     >> Christophe
>     >>
>     >
> 
> 
>     _______________________________________________
>     Cubicweb mailing list
>     Cubicweb at lists.cubicweb.org <mailto:Cubicweb at lists.cubicweb.org>
>     https://lists.cubicweb.org/mailman/listinfo/cubicweb
> 
> 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: OpenPGP digital signature
URL: <http://lists.cubicweb.org/pipermail/cubicweb/attachments/20150107/35a695a4/attachment-0182.sig>


More information about the Cubicweb mailing list