[Cubicweb] MassiveObjectStore usage

Vincent Michel vincent.michel at logilab.fr
Thu May 22 13:56:59 CEST 2014


Hi Cyril,

The meta data associated with the entities table should have been flushed.
You could  try:

  * to add "autoflush_metadata=True"  in the MassiveObjectStore:

/store = MassiveObjectStore(session, replace_sep=' ', 
/autoflush_metadata=True)

     even if this is currently (0.5.0) the default value.

  * to manually flush the meta data, using flush_meta_data():

/store.flush()///
/    store.commit()
     store.flush_meta_data()
     store.cleanup()/

It should solve your problem.

Best,

Vincent

On 22/05/2014 12:31, LANQUETUIT Cyril wrote:
> Bonjour,
>
> C'est bien le README de la classe MassiveObjectStore sur lequel 
> j'étais arrivé après une recherche google...
>
> j'ai essayer de procéder comme indiqué, néanmoins après l'exécution du 
> script imagen.py dans lequel
>
> /store = SQLGenObjectStore(session)/
>
> a été remplacé par
> /
> store = MassiveObjectStore(session, replace_sep=' ')/
>
> les appels suivants sont bien effectués:
> /
> store.flush()
> store.commit()
> store.cleanup()/
>
> cependant l'interface CubicWeb me retourne toujour l'erreur:
>
>
>     /une erreur est survenue/
>
> /UnknownEid: No entity with eid 70239 in the repository/
> ------------------------------------------------------------------------
>
> /No entity with eid 70239 in the repository/
>
> /*File* 
> */usr/lib/python2.7/dist-packages/cubicweb/web/application.py*, *line* 
> *456*, *function* *core_handle*:
>   result = controller.publish(rset=rset)
> [+] *File* 
> */usr/lib/python2.7/dist-packages/cubicweb/web/views/basecontrollers.py*, 
> *line* *136*, *function* *publish*:
>   rset=rset, view=view)
> [+] *File* */usr/lib/python2.7/dist-packages/cubicweb/cwvreg.py*, 
> *line* *398*, *function* *main_template*:
>   res = obj.render(**kwargs)
> [+] *File* */usr/lib/python2.7/dist-packages/cubicweb/view.py*, *line* 
> *137*, *function* *render*:
>   view_func(**context)
> [+] *File* 
> */home/cl239099/cubicweb/cubes/squareui/views/basetemplates.py*, 
> *line* *27*, *function* *call*:
>   self.template_page_content(view)
> [+] *File* 
> */home/cl239099/cubicweb/cubes/squareui/views/basetemplates.py*, 
> *line* *75*, *function* *template_page_content*:
>   self.content_column(view, content_cols)
> [+] *File* 
> */home/cl239099/cubicweb/cubes/squareui/views/basetemplates.py*, 
> *line* *110*, *function* *content_column*:
>   view.render(w=w)
> [+] *File* */usr/lib/python2.7/dist-packages/cubicweb/view.py*, *line* 
> *137*, *function* *render*:
>   view_func(**context)
> [+] *File* 
> */home/cl239099/cubicweb/cubes/bootstrap/views/baseviews.py*, *line* 
> *57*, *function* *call*:
>   self._cw.view(self.item_vid, self.cw_rset, row=i, col=0, w=w)
> [+] *File* */usr/lib/python2.7/dist-packages/cubicweb/req.py*, *line* 
> *349*, *function* *view*:
>   return view.render(w=w, **kwargs)
> [+] *File* */usr/lib/python2.7/dist-packages/cubicweb/view.py*, *line* 
> *137*, *function* *render*:
>   view_func(**context)
> [+] *File* 
> */usr/lib/python2.7/dist-packages/cubicweb/web/views/baseviews.py*, 
> *line* *414*, *function* *cell_call*:
>   self.wview('listitem', self.cw_rset, row=row, col=col, **kwargs)
> [+] *File* */usr/lib/python2.7/dist-packages/cubicweb/view.py*, *line* 
> *243*, *function* *wview*:
>   self._cw.view(__vid, rset, __fallback_vid, w=self.w, **kwargs)
> [+] *File* */usr/lib/python2.7/dist-packages/cubicweb/req.py*, *line* 
> *349*, *function* *view*:
>   return view.render(w=w, **kwargs)
> [+] *File* */usr/lib/python2.7/dist-packages/cubicweb/view.py*, *line* 
> *137*, *function* *render*:
>   view_func(**context)
> [+] *File* 
> */usr/lib/python2.7/dist-packages/cubicweb/web/views/baseviews.py*, 
> *line* *342*, *function* *cell_call*:
>   self.wview(vid, self.cw_rset, row=row, col=col, **kwargs)
> [+] *File* */usr/lib/python2.7/dist-packages/cubicweb/view.py*, *line* 
> *243*, *function* *wview*:
>   self._cw.view(__vid, rset, __fallback_vid, w=self.w, **kwargs)
> [+] *File* */usr/lib/python2.7/dist-packages/cubicweb/req.py*, *line* 
> *349*, *function* *view*:
>   return view.render(w=w, **kwargs)
> [+] *File* */usr/lib/python2.7/dist-packages/cubicweb/view.py*, *line* 
> *137*, *function* *render*:
>   view_func(**context)
> [+] *File* 
> */home/cl239099/cubicweb/cubes/medicalexp/views/secondary.py*, *line* 
> *42*, *function* *cell_call*:
>   self.w(u'<h4><a href="%s">%s</a></h4>' % (entity.absolute_url(), 
> entity.dc_title()))
> [+] *File* */usr/lib/python2.7/dist-packages/cubicweb/entity.py*, 
> *line* *685*, *function* *absolute_url*:
>   sourcemeta = self.cw_metainformation()['source']
> [+] *File* 
> */usr/lib/pymodules/python2.7/logilab/common/decorators.py*, *line* 
> *66*, *function* *wrapped*:
>   return self.__call__(*args, **kwargs)
> [+] *File* 
> */usr/lib/pymodules/python2.7/logilab/common/decorators.py*, *line* 
> *60*, *function* *__call__*:
>   value = __me.callable(self, *args)
> [+] *File* */usr/lib/python2.7/dist-packages/cubicweb/entity.py*, 
> *line* *647*, *function* *cw_metainformation*:
>   res = self._cw.describe(self.eid, asdict=True)
> [+] *File* */usr/lib/python2.7/dist-packages/cubicweb/dbapi.py*, 
> *line* *412*, *function* *describe*:
>   return self.cnx.describe(eid, asdict)
> [+] *File* */usr/lib/python2.7/dist-packages/cubicweb/dbapi.py*, 
> *line* *548*, *function* *decorator*:
>   return func(self, *args, **kwargs)
> [+] *File* */usr/lib/python2.7/dist-packages/cubicweb/dbapi.py*, 
> *line* *754*, *function* *describe*:
>   metas = self._repo.describe(self.sessionid, eid, **self._txid())
> [+] *File* 
> */usr/lib/python2.7/dist-packages/cubicweb/server/repository.py*, 
> *line* *811*, *function* *describe*:
>   return self.type_and_source_from_eid(eid, session)
> [+] *File* 
> */usr/lib/python2.7/dist-packages/cubicweb/server/repository.py*, 
> *line* *1046*, *function* *type_and_source_from_eid*:
>   session, eid)
> [+] *File* 
> */usr/lib/python2.7/dist-packages/cubicweb/server/sources/native.py*, 
> *line* *876*, *function* *eid_type_source*:
>   res = self._eid_type_source(session, eid, sql)
> [+] *File* 
> */usr/lib/python2.7/dist-packages/cubicweb/server/sources/native.py*, 
> *line* *871*, *function* *_eid_type_source*:
>   raise UnknownEid(eid)
> [+] /
> /*CubicWeb version:* 3.17.14
> *Cube brainomics version:* 0.8.0
> *Cube genomics version:* 0.7.0
> *Cube medicalexp version:* 0.8.0
> *Cube neuroimaging version:* 0.5.0
> *Cube questionnaire version:* 0.5.0
> *Cube squareui version:* 0.2.1
> *Cube bootstrap version:* 0.4.0
> *Cube card version:* 0.5.3
> *Cube comment version:* 1.10.0
> *Cube file version:* 1.15.0
> *Cube jqplot version:* 0.4.0/
>
> En essayant de comprendre ce qui s'est passé j'ai juste réussit à 
> vérifier que l'entité 70239 (ce n'est pas la seule mais comme c'est la 
> première de la liste des subject c'est celle-ci qu'indique le message 
> d'erreur) est bien présente dans la table subject mais en revanche 
> absente de la table entities:
>
> test110=# select * from entities where eid = 70239;
>  eid | type | source | asource | mtime | extid
> -----+------+--------+---------+-------+-------
> (0 rows)
>
> Ne sachant pas comment cette table doit être remplie je peux 
> difficilement investiguer plus loin,
> Merci d'avance si vous pouvez aider à éclaircir et éventuellement 
> corriger ce disfonctionnement dans l'utilisation du MassiveObjectStore
>
> Cordialement,
> Cyril
> ------------------------------------------------------------------------
> *De :* Vincent Michel [vincent.michel at logilab.fr]
> *Envoyé :* mercredi 21 mai 2014 14:05
> *À :* LANQUETUIT Cyril
> *Objet :* Re: [Cubicweb] Misunderstanding of dataimport 
> _create_copyfrom_buffer function
>
> Salut Cyril,
>
> Regarde plutôt la docstring de la classe dans le code, ou le README, 
> ils sont probablement
> plus à jour que google.
>
> Sinon:
>
>  * ca ne marche pas avec les relations inlined. Meme avec le 
> init_rtype_table.
>     Donc, pour les relations inlined, tu peux les créer en passant 
> l'eid dans le create_entity.
>     Par exemple, si related_study est inlined dans une entitéé 
> Subject, tu peux faire,
>     si tu as l'eid de l'étude sous la main:
>
>       store.create_entity('Subject', ...., related_study=eid_of_the_study)
>
>  * les convert_relations ne doivent être utilisées que pour la 
> création de relations (non inlined)
>     qui ne se basent pas sur des eids. Je ne pense pas que ca soit 
> utile dans ton cas.
>
> Cordialement,
>
> Vincent
>
> On 21/05/2014 11:49, LANQUETUIT Cyril wrote:
>> Bonjour Vincent,
>>
>> Bon effectivement le MassiveObjectStore nécessite un peu plus 
>> d'efforts pour le mettre en œuvre que le SQLGenObjectStore...
>>
>> l'initialisation est faite:
>>
>> /  store = MassiveObjectStore(session, replace_sep=' ')/
>>
>> une recherche google m'a apporté l'information suivante:
>> * inlined-relations are not supported in the `relate` method.
>>
>> dans l'exemple associé ces relations étaient précisées de la façon suivante:
>> /    store.init_rtype_table('ExternalResource', 'related_study', 
>> 'Study')/
>>
>> c'est donc ce que j'ai fait pour toutes les relations utilisées dans 
>> imagen.py par des appels du type "store.relate"
>>
>> j'obtenais néanmoins toujours un message d'erreur:
>>
>> /File 
>> "/usr/lib/python2.7/dist-packages/cubicweb/server/sources/native.py", 
>> line 743, in doexec
>>     cursor.execute(str(query), args)
>> psycopg2.ProgrammingError: relation "related_study_relation" does not 
>> exist
>> LINE 1: INSERT INTO related_study_relation (eid_from, eid_to) SELECT.../
>>
>> la commande psql me permet de constater qu'effectivement la table 
>> related_study_relation n'existe pas, elle a bien été créée mais avec 
>> un suffixe "_tmp"
>>
>> dans l'exemple trouvé sur google, entre le flush et le cleanup se trouve:
>>   # Convert the relation
>>      store.convert_relations('Person', 'lives', 'Location')
>>
>> j'ai donc ajouté des lignes du type:
>> /
>>      store.convert_relations('ExternalResource', 'related_study', 'Study')/
>> pour toute les relations initialisées par "store.init_rtype_table" en 
>> espérant que les tables de relation suffixées de "_tmp" se retrouvent 
>> sans ce suffixe mais c'est encore insuffisant...
>>
>> Le problème semble se produire entre le flush et le cleanup, si tu as 
>> une idée pour corriger ce dysfonctionnement j'essayerai de 
>> l'appliquer, sinon on gardera le SQLGenObjectStore et les scripts 
>> seront juste lents à exécuter,
>>
>> Thank's,
>> Cyril
>> ------------------------------------------------------------------------
>> *De :* Vincent Michel [vincent.michel at logilab.fr]
>> *Envoyé :* mardi 20 mai 2014 14:34
>> *À :* LANQUETUIT Cyril
>> *Cc :* PAPADOPOULOS ORFANOS Dimitri
>> *Objet :* Re: [Cubicweb] Misunderstanding of dataimport 
>> _create_copyfrom_buffer function
>>
>> Salut Cyril,
>>
>> Je vais faire en français, ca sera plus facile :)
>>
>> Le MassiveObjectStore a besoin d'un peu plus d'appels de fonctions 
>> que le SQLGenObjectStore.
>> En particulier, il faut bien effectuer un appel à .flush() pour qu'il 
>> flush les méta-data,
>> ainsi qu'un appel (unique, à la fin de l'import) à .cleanup().
>>
>> N'hésites pas si il reste un problème.
>>
>> Cordialement,
>>
>> Vincent
>>
>> On 20/05/2014 14:13, LANQUETUIT Cyril wrote:
>>> Hello Vincent,
>>>
>>> Thank's for your answer,
>>>
>>> I've try to use MassiveObjectStore by passing a specified 
>>> replace_sep for the initialisation of the object as following:
>>>
>>> /    store = MassiveObjectStore(session, replace_sep=' ')
>>>
>>> /that is to say a space instead of '',  this line just supply the 
>>> former one:
>>>
>>> /    store = SQLGenObjectStore(session)/
>>>
>>> in imagen.py script, so the error described in the last mail does 
>>> not appear anymore and the script is executed of course quicker than 
>>> with SQLGenObjectStore but some error persist in the base, when I 
>>> try to visualise subjects in CubicWeb I get this following error 
>>> message:
>>>
>>>
>>>     /une erreur est survenue/
>>>
>>> /UnknownEid: No entity with eid 70239 in the repository/
>>> ------------------------------------------------------------------------
>>>
>>> /No entity with eid 70239 in the repository/
>>>
>>> /
>>> /
>>>
>>> /[...] <-- a stacktrace that I don't really understand and I prefere 
>>> to sumarise like this
>>> /
>>>
>>> /
>>> *CubicWeb version:* 3.17.14
>>> *Cube brainomics version:* 0.8.0
>>> *Cube genomics version:* 0.7.0
>>> *Cube medicalexp version:* 0.8.0
>>> *Cube neuroimaging version:* 0.5.0
>>> *Cube questionnaire version:* 0.5.0
>>> *Cube squareui version:* 0.2.1
>>> *Cube bootstrap version:* 0.4.0
>>> *Cube card version:* 0.5.3
>>> *Cube comment version:* 1.10.0
>>> *Cube file version:* 1.15.0
>>> *Cube jqplot version:* 0.4.0/
>>>
>>> Entity 70239 is the first subject inserted in the base, I can see it 
>>> with psql as :
>>>
>>>     cw_identifier    | cw_surname | cw_firstname | cw_gender | 
>>> cw_date_of_birth | cw_handedness | cw_creation_date   | cw_cwuri | 
>>> cw_modification_date    | cw_eid
>>> ---------------------+------------+--------------+-----------+------------------+---------------+---------------------+-------------------------------------------------------+----------------------------+--------
>>>  IMAGEN_000064851530 |            | | female    |                  | 
>>> right         | 2014-05-20 00:00:00 | 
>>> _auto_generated81d7c2bc89924f1d8ac021b3009e6b3b63036 | 2014-05-20 
>>> 00:00:00        |  70239
>>>
>>> So I don't understand the error message...
>>>
>>> This error does not appear when we use SQLGenObjectStore so if there 
>>> is a correction to bring to the script for enable MassiveObjectStore 
>>> to do the stuff as well as the last stable version of the script 
>>> I'll accept your advices, in the other case we certainly may keep 
>>> the script as it is, that is to say in a version which work slowly 
>>> but properly
>>>
>>> Thank's again,
>>> Cyril
>>> ------------------------------------------------------------------------
>>> *De :* Vincent Michel [vincent.michel at logilab.fr]
>>> *Envoyé :* mardi 20 mai 2014 12:26
>>> *À :* LANQUETUIT Cyril; cubicweb at lists.cubicweb.org
>>> *Objet :* Re: [Cubicweb] Misunderstanding of dataimport 
>>> _create_copyfrom_buffer function
>>>
>>> Hi Cyril,
>>>
>>> The _create_copyfrom_buffer of cubicweb/dataimport.py is indeed a 
>>> bit tricky...
>>>
>>> This function creates a CSV string that will be pushed in postgresql 
>>> using COPY FROM.
>>> However, if the separator used in the CSV string already exists in 
>>> some attributes, it will crash. Thus it is possible to give a 
>>> replace_sep to replace the separator by this given string.
>>>
>>> Indeed, the correct clause should be :
>>> /if (replace_sep != None):
>>>
>>> /
>>> I have created a ticket on this issue 
>>> (https://www.cubicweb.org/3845572),
>>> and I think that the whole function should be rewritten.
>>>
>>> Best,
>>>
>>> Vincent
>>>
>>>
>>>
>>>
>>> On 19/05/2014 14:24, LANQUETUIT Cyril wrote:
>>>> Hello,
>>>>
>>>> In function _create_copyfrom_buffer of file dataimport.py, in the 
>>>> following part of code:
>>>> /
>>>>                         # If a replace_sep is given, replace
>>>>                         # the separator instead of returning None
>>>>                         # (and thus avoid empty buffer)
>>>>                         if replace_sep:
>>>>                             value = value.replace(_char, replace_sep)
>>>>                         else:
>>>>                             return/
>>>>
>>>> when the separator is set to '' (the default value when MassiveObjectStore is used for store)
>>>> the replace_sep is evaluate to false so the else clause is executed, the function _create_copyfrom_buffer return an empty buffer
>>>> then this following error is raised:
>>>>
>>>> /File "/home/cl239099/cubicweb/cubes/dataio/dataimport.py", line 831, in flush_entities
>>>>      raise ValueError('Error in buffer creation for etype %s' % etype)
>>>> ValueError: Error in buffer creation for etype Question/
>>>>
>>>> I don't understand exactly what the code should do or should be but if the clause
>>>> /
>>>> if replace_sep:/
>>>>
>>>> is replaced by something like
>>>> /
>>>> if (replace_sep != None):
>>>> /
>>>> the else clause won't be executed in this case...
>>>>
>>>> please could you examinate this issue and correct it if needed or let me know what is wrong in the precedent discribed comportement/
>>>> /
>>>> Thank's,
>>>> Cyril
>>>>
>>>>
>>>> _______________________________________________
>>>> Cubicweb mailing list
>>>> Cubicweb at lists.cubicweb.org
>>>> http://lists.cubicweb.org/mailman/listinfo/cubicweb
>>>
>>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.cubicweb.org/pipermail/cubicweb/attachments/20140522/23eef6da/attachment-0102.html>


More information about the Cubicweb mailing list