<html>
  <head>
    <meta content="text/html; charset=windows-1252"
      http-equiv="Content-Type">
  </head>
  <body text="#000000" bgcolor="#FFFFFF">
    <div class="moz-cite-prefix">Hi Cyril,<br>
      <br>
      The meta data associated with the entities table should have been
      flushed.<br>
      You could  try:<br>
      <br>
       * to add "autoflush_metadata=True"  in the MassiveObjectStore:<br>
      <br>
          <i>store = MassiveObjectStore(session, replace_sep=' ', </i>autoflush_metadata=True)<br>
      <br>
          even if this is currently (0.5.0) the default value.<br>
      <br>
       * to manually flush the meta data, using flush_meta_data():<br>
      <br>
          <i>store.flush()</i><i><i><br>
        </i>    store.commit()<br>
            store.flush_meta_data()<br>
            store.cleanup()</i><br>
      <br>
      It should solve your problem.<br>
      <br>
      Best,<br>
      <br>
      Vincent<br>
      <br>
      On 22/05/2014 12:31, LANQUETUIT Cyril wrote:<br>
    </div>
    <blockquote
      cite="mid:1AB37ADB88BD0343B8A7CEB0471DF423C7E8D8@EXDAG0-B1.intra.cea.fr"
      type="cite">
      <meta http-equiv="Content-Type" content="text/html;
        charset=windows-1252">
      <div style="direction: ltr;font-family: Tahoma;color:
        #000000;font-size: 10pt;">Bonjour,<br>
        <br>
        C'est bien le README de la classe MassiveObjectStore sur lequel
        j'étais arrivé après une recherche google...<br>
        <br>
        j'ai essayer de procéder comme indiqué, néanmoins après
        l'exécution du script imagen.py dans lequel<br>
        <br>
        <i>store = SQLGenObjectStore(session)</i><br>
        <br>
        a été remplacé par<br>
        <i><br>
          store = MassiveObjectStore(session, replace_sep=' ')</i><br>
        <br>
        les appels suivants sont bien effectués:<br>
        <i><br>
          store.flush()<br>
          store.commit()<br>
          store.cleanup()</i><br>
        <br>
        cependant l'interface CubicWeb me retourne toujour l'erreur:<br>
        <h2><font size="1"><i>une erreur est survenue</i></font></h2>
        <div class="tb"><font size="1"><i>UnknownEid: No entity with eid
              70239 in the repository</i></font></div>
        <hr>
        <div class="tb">
          <p class="error"><font size="1"><i>No entity with eid 70239 in
                the repository</i></font></p>
          <div class="error_traceback"><font size="1"><i><b>File</b> <b
                  class="file">/usr/lib/python2.7/dist-packages/cubicweb/web/application.py</b>,
                <b>line</b> <b class="line">456</b>, <b>function</b> <b
                  class="function">core_handle</b>:<br>
                  result = controller.publish(rset=rset)<br>
                [<a moz-do-not-send="true">+</a>] <b>File</b> <b
                  class="file">/usr/lib/python2.7/dist-packages/cubicweb/web/views/basecontrollers.py</b>,
                <b>line</b> <b class="line">136</b>, <b>function</b> <b
                  class="function">publish</b>:<br>
                  rset=rset, view=view)<br>
                [<a moz-do-not-send="true">+</a>] <b>File</b> <b
                  class="file">/usr/lib/python2.7/dist-packages/cubicweb/cwvreg.py</b>,
                <b>line</b> <b class="line">398</b>, <b>function</b> <b
                  class="function">main_template</b>:<br>
                  res = obj.render(**kwargs)<br>
                [<a moz-do-not-send="true">+</a>] <b>File</b> <b
                  class="file">/usr/lib/python2.7/dist-packages/cubicweb/view.py</b>,
                <b>line</b> <b class="line">137</b>, <b>function</b> <b
                  class="function">render</b>:<br>
                  view_func(**context)<br>
                [<a moz-do-not-send="true">+</a>] <b>File</b> <b
                  class="file">/home/cl239099/cubicweb/cubes/squareui/views/basetemplates.py</b>,
                <b>line</b> <b class="line">27</b>, <b>function</b> <b
                  class="function">call</b>:<br>
                  self.template_page_content(view)<br>
                [<a moz-do-not-send="true">+</a>] <b>File</b> <b
                  class="file">/home/cl239099/cubicweb/cubes/squareui/views/basetemplates.py</b>,
                <b>line</b> <b class="line">75</b>, <b>function</b> <b
                  class="function">template_page_content</b>:<br>
                  self.content_column(view, content_cols)<br>
                [<a moz-do-not-send="true">+</a>] <b>File</b> <b
                  class="file">/home/cl239099/cubicweb/cubes/squareui/views/basetemplates.py</b>,
                <b>line</b> <b class="line">110</b>, <b>function</b> <b
                  class="function">content_column</b>:<br>
                  view.render(w=w)<br>
                [<a moz-do-not-send="true">+</a>] <b>File</b> <b
                  class="file">/usr/lib/python2.7/dist-packages/cubicweb/view.py</b>,
                <b>line</b> <b class="line">137</b>, <b>function</b> <b
                  class="function">render</b>:<br>
                  view_func(**context)<br>
                [<a moz-do-not-send="true">+</a>] <b>File</b> <b
                  class="file">/home/cl239099/cubicweb/cubes/bootstrap/views/baseviews.py</b>,
                <b>line</b> <b class="line">57</b>, <b>function</b> <b
                  class="function">call</b>:<br>
                  self._cw.view(self.item_vid, self.cw_rset, row=i,
                col=0, w=w)<br>
                [<a moz-do-not-send="true">+</a>] <b>File</b> <b
                  class="file">/usr/lib/python2.7/dist-packages/cubicweb/req.py</b>,
                <b>line</b> <b class="line">349</b>, <b>function</b> <b
                  class="function">view</b>:<br>
                  return view.render(w=w, **kwargs)<br>
                [<a moz-do-not-send="true">+</a>] <b>File</b> <b
                  class="file">/usr/lib/python2.7/dist-packages/cubicweb/view.py</b>,
                <b>line</b> <b class="line">137</b>, <b>function</b> <b
                  class="function">render</b>:<br>
                  view_func(**context)<br>
                [<a moz-do-not-send="true">+</a>] <b>File</b> <b
                  class="file">/usr/lib/python2.7/dist-packages/cubicweb/web/views/baseviews.py</b>,
                <b>line</b> <b class="line">414</b>, <b>function</b> <b
                  class="function">cell_call</b>:<br>
                  self.wview('listitem', self.cw_rset, row=row, col=col,
                **kwargs)<br>
                [<a moz-do-not-send="true">+</a>] <b>File</b> <b
                  class="file">/usr/lib/python2.7/dist-packages/cubicweb/view.py</b>,
                <b>line</b> <b class="line">243</b>, <b>function</b> <b
                  class="function">wview</b>:<br>
                  self._cw.view(__vid, rset, __fallback_vid, w=self.w,
                **kwargs)<br>
                [<a moz-do-not-send="true">+</a>] <b>File</b> <b
                  class="file">/usr/lib/python2.7/dist-packages/cubicweb/req.py</b>,
                <b>line</b> <b class="line">349</b>, <b>function</b> <b
                  class="function">view</b>:<br>
                  return view.render(w=w, **kwargs)<br>
                [<a moz-do-not-send="true">+</a>] <b>File</b> <b
                  class="file">/usr/lib/python2.7/dist-packages/cubicweb/view.py</b>,
                <b>line</b> <b class="line">137</b>, <b>function</b> <b
                  class="function">render</b>:<br>
                  view_func(**context)<br>
                [<a moz-do-not-send="true">+</a>] <b>File</b> <b
                  class="file">/usr/lib/python2.7/dist-packages/cubicweb/web/views/baseviews.py</b>,
                <b>line</b> <b class="line">342</b>, <b>function</b> <b
                  class="function">cell_call</b>:<br>
                  self.wview(vid, self.cw_rset, row=row, col=col,
                **kwargs)<br>
                [<a moz-do-not-send="true">+</a>] <b>File</b> <b
                  class="file">/usr/lib/python2.7/dist-packages/cubicweb/view.py</b>,
                <b>line</b> <b class="line">243</b>, <b>function</b> <b
                  class="function">wview</b>:<br>
                  self._cw.view(__vid, rset, __fallback_vid, w=self.w,
                **kwargs)<br>
                [<a moz-do-not-send="true">+</a>] <b>File</b> <b
                  class="file">/usr/lib/python2.7/dist-packages/cubicweb/req.py</b>,
                <b>line</b> <b class="line">349</b>, <b>function</b> <b
                  class="function">view</b>:<br>
                  return view.render(w=w, **kwargs)<br>
                [<a moz-do-not-send="true">+</a>] <b>File</b> <b
                  class="file">/usr/lib/python2.7/dist-packages/cubicweb/view.py</b>,
                <b>line</b> <b class="line">137</b>, <b>function</b> <b
                  class="function">render</b>:<br>
                  view_func(**context)<br>
                [<a moz-do-not-send="true">+</a>] <b>File</b> <b
                  class="file">/home/cl239099/cubicweb/cubes/medicalexp/views/secondary.py</b>,
                <b>line</b> <b class="line">42</b>, <b>function</b> <b
                  class="function">cell_call</b>:<br>
                  self.w(u'<h4><a
                href="%s">%s</a></h4>' %
                (entity.absolute_url(), entity.dc_title()))<br>
                [<a moz-do-not-send="true">+</a>] <b>File</b> <b
                  class="file">/usr/lib/python2.7/dist-packages/cubicweb/entity.py</b>,
                <b>line</b> <b class="line">685</b>, <b>function</b> <b
                  class="function">absolute_url</b>:<br>
                  sourcemeta = self.cw_metainformation()['source']<br>
                [<a moz-do-not-send="true">+</a>] <b>File</b> <b
                  class="file">/usr/lib/pymodules/python2.7/logilab/common/decorators.py</b>,
                <b>line</b> <b class="line">66</b>, <b>function</b> <b
                  class="function">wrapped</b>:<br>
                  return self.__call__(*args, **kwargs)<br>
                [<a moz-do-not-send="true">+</a>] <b>File</b> <b
                  class="file">/usr/lib/pymodules/python2.7/logilab/common/decorators.py</b>,
                <b>line</b> <b class="line">60</b>, <b>function</b> <b
                  class="function">__call__</b>:<br>
                  value = __me.callable(self, *args)<br>
                [<a moz-do-not-send="true">+</a>] <b>File</b> <b
                  class="file">/usr/lib/python2.7/dist-packages/cubicweb/entity.py</b>,
                <b>line</b> <b class="line">647</b>, <b>function</b> <b
                  class="function">cw_metainformation</b>:<br>
                  res = self._cw.describe(self.eid, asdict=True)<br>
                [<a moz-do-not-send="true">+</a>] <b>File</b> <b
                  class="file">/usr/lib/python2.7/dist-packages/cubicweb/dbapi.py</b>,
                <b>line</b> <b class="line">412</b>, <b>function</b> <b
                  class="function">describe</b>:<br>
                  return self.cnx.describe(eid, asdict)<br>
                [<a moz-do-not-send="true">+</a>] <b>File</b> <b
                  class="file">/usr/lib/python2.7/dist-packages/cubicweb/dbapi.py</b>,
                <b>line</b> <b class="line">548</b>, <b>function</b> <b
                  class="function">decorator</b>:<br>
                  return func(self, *args, **kwargs)<br>
                [<a moz-do-not-send="true">+</a>] <b>File</b> <b
                  class="file">/usr/lib/python2.7/dist-packages/cubicweb/dbapi.py</b>,
                <b>line</b> <b class="line">754</b>, <b>function</b> <b
                  class="function">describe</b>:<br>
                  metas = self._repo.describe(self.sessionid, eid,
                **self._txid())<br>
                [<a moz-do-not-send="true">+</a>] <b>File</b> <b
                  class="file">/usr/lib/python2.7/dist-packages/cubicweb/server/repository.py</b>,
                <b>line</b> <b class="line">811</b>, <b>function</b> <b
                  class="function">describe</b>:<br>
                  return self.type_and_source_from_eid(eid, session)<br>
                [<a moz-do-not-send="true">+</a>] <b>File</b> <b
                  class="file">/usr/lib/python2.7/dist-packages/cubicweb/server/repository.py</b>,
                <b>line</b> <b class="line">1046</b>, <b>function</b>
                <b class="function">type_and_source_from_eid</b>:<br>
                  session, eid)<br>
                [<a moz-do-not-send="true">+</a>] <b>File</b> <b
                  class="file">/usr/lib/python2.7/dist-packages/cubicweb/server/sources/native.py</b>,
                <b>line</b> <b class="line">876</b>, <b>function</b> <b
                  class="function">eid_type_source</b>:<br>
                  res = self._eid_type_source(session, eid, sql)<br>
                [<a moz-do-not-send="true">+</a>] <b>File</b> <b
                  class="file">/usr/lib/python2.7/dist-packages/cubicweb/server/sources/native.py</b>,
                <b>line</b> <b class="line">871</b>, <b>function</b> <b
                  class="function">_eid_type_source</b>:<br>
                  raise UnknownEid(eid)<br>
                [<a moz-do-not-send="true">+</a>] </i></font></div>
        </div>
        <font size="1"><i><b>CubicWeb version:</b> 3.17.14<br>
            <b>Cube brainomics version:</b> 0.8.0<br>
            <b>Cube genomics version:</b> 0.7.0<br>
            <b>Cube medicalexp version:</b> 0.8.0<br>
            <b>Cube neuroimaging version:</b> 0.5.0<br>
            <b>Cube questionnaire version:</b> 0.5.0<br>
            <b>Cube squareui version:</b> 0.2.1<br>
            <b>Cube bootstrap version:</b> 0.4.0<br>
            <b>Cube card version:</b> 0.5.3<br>
            <b>Cube comment version:</b> 1.10.0<br>
            <b>Cube file version:</b> 1.15.0<br>
            <b>Cube jqplot version:</b> 0.4.0</i></font><br>
        <br>
        En essayant de comprendre ce qui s'est passé j'ai juste réussit
        à vérifier que l'entité 70239 (ce n'est pas la seule mais comme
        c'est la première de la liste des subject c'est celle-ci
        qu'indique le message d'erreur) est bien présente dans la table
        subject mais en revanche absente de la table entities:<br>
        <br>
        test110=# select * from entities where eid = 70239;<br>
         eid | type | source | asource | mtime | extid <br>
        -----+------+--------+---------+-------+-------<br>
        (0 rows)<br>
        <br>
        Ne sachant pas comment cette table doit être remplie je peux
        difficilement investiguer plus loin,<br>
        Merci d'avance si vous pouvez aider à éclaircir et
        éventuellement corriger ce disfonctionnement dans l'utilisation
        du MassiveObjectStore<br>
        <br>
        Cordialement,<br>
        Cyril<br>
        <div style="font-family: Times New Roman; color: #000000;
          font-size: 16px">
          <hr tabindex="-1">
          <div style="direction: ltr;" id="divRpF177885"><font
              color="#000000" face="Tahoma" size="2"><b>De :</b> Vincent
              Michel [<a class="moz-txt-link-abbreviated" href="mailto:vincent.michel@logilab.fr">vincent.michel@logilab.fr</a>]<br>
              <b>Envoyé :</b> mercredi 21 mai 2014 14:05<br>
              <b>À :</b> LANQUETUIT Cyril<br>
              <b>Objet :</b> Re: [Cubicweb] Misunderstanding of
              dataimport _create_copyfrom_buffer function<br>
            </font><br>
          </div>
          <div>
            <div class="moz-cite-prefix">Salut Cyril,<br>
              <br>
              Regarde plutôt la docstring de la classe dans le code, ou
              le README, ils sont probablement<br>
              plus à jour que google.<br>
              <br>
              Sinon:<br>
              <br>
               * ca ne marche pas avec les relations inlined. Meme avec
              le init_rtype_table.<br>
                  Donc, pour les relations inlined, tu peux les créer en
              passant l'eid dans le create_entity.<br>
                  Par exemple, si related_study est inlined dans une
              entitéé Subject, tu peux faire,<br>
                  si tu as l'eid de l'étude sous la main:<br>
              <br>
                    store.create_entity('Subject', ....,
              related_study=eid_of_the_study)<br>
              <br>
               * les convert_relations ne doivent être utilisées que
              pour la création de relations (non inlined)<br>
                  qui ne se basent pas sur des eids. Je ne pense pas que
              ca soit utile dans ton cas.<br>
              <br>
              Cordialement,<br>
              <br>
              Vincent<br>
              <br>
              On 21/05/2014 11:49, LANQUETUIT Cyril wrote:<br>
            </div>
            <blockquote type="cite">
              <div style="direction:ltr; font-family:Tahoma;
                color:#000000; font-size:10pt">Bonjour Vincent,<br>
                <br>
                Bon effectivement le MassiveObjectStore nécessite un peu
                plus d'efforts pour le mettre en
                <span class="st">œ</span>uvre que le
                SQLGenObjectStore...<br>
                <br>
                l'initialisation est faite:<br>
                <br>
                  <i>  store = MassiveObjectStore(session, replace_sep='
                  ')</i><br>
                <br>
                une recherche google m'a apporté l'information suivante:<br>
                <pre>* inlined-relations are not supported in the `relate` method.<font face="Tahoma">

dans l'exemple associé ces relations étaient précisées de la façon suivante:</font>
</pre>
                <i>    store.init_rtype_table('ExternalResource',
                  'related_study', 'Study')</i><br>
                <br>
                c'est donc ce que j'ai fait pour toutes les relations
                utilisées dans imagen.py par des appels du type
                "store.relate"<br>
                <br>
                j'obtenais néanmoins toujours un message d'erreur:<br>
                <br>
                <i>File
                  "/usr/lib/python2.7/dist-packages/cubicweb/server/sources/native.py",
                  line 743, in doexec<br>
                      cursor.execute(str(query), args)<br>
                  psycopg2.ProgrammingError: relation
                  "related_study_relation" does not exist<br>
                  LINE 1: INSERT INTO related_study_relation (eid_from,
                  eid_to) SELECT...</i><br>
                <br>
                la commande psql me permet de constater qu'effectivement
                la table related_study_relation n'existe pas, elle a
                bien été créée mais avec un suffixe "_tmp"<br>
                <br>
                dans l'exemple trouvé sur google, entre le flush et le
                cleanup se trouve:<br>
                <pre> # Convert the relation
    store.convert_relations('Person', 'lives', 'Location')

<font face="Tahoma">j'ai donc ajouté des lignes du type:</font>
<font face="Tahoma"><i>
    store.convert_relations('ExternalResource', 'related_study', 'Study')</i></font>
</pre>
                pour toute les relations initialisées par
                "store.init_rtype_table" en espérant que les tables de
                relation suffixées de "_tmp" se retrouvent sans ce
                suffixe mais c'est encore insuffisant...<br>
                <br>
                Le problème semble se produire entre le flush et le
                cleanup, si tu as une idée pour corriger ce
                dysfonctionnement j'essayerai de l'appliquer, sinon on
                gardera le SQLGenObjectStore et les scripts seront juste
                lents à exécuter,<br>
                <br>
                Thank's,<br>
                Cyril<br>
                <div style="font-family:Times New Roman; color:#000000;
                  font-size:16px">
                  <hr tabindex="-1">
                  <div id="divRpF891938" style="direction:ltr"><font
                      color="#000000" face="Tahoma" size="2"><b>De :</b>
                      Vincent Michel [<a moz-do-not-send="true"
                        class="moz-txt-link-abbreviated"
                        href="mailto:vincent.michel@logilab.fr"
                        target="_blank">vincent.michel@logilab.fr</a>]<br>
                      <b>Envoyé :</b> mardi 20 mai 2014 14:34<br>
                      <b>À :</b> LANQUETUIT Cyril<br>
                      <b>Cc :</b> PAPADOPOULOS ORFANOS Dimitri<br>
                      <b>Objet :</b> Re: [Cubicweb] Misunderstanding of
                      dataimport _create_copyfrom_buffer function<br>
                    </font><br>
                  </div>
                  <div>
                    <div class="moz-cite-prefix">Salut Cyril,<br>
                      <br>
                      Je vais faire en français, ca sera plus facile :)<br>
                      <br>
                      Le MassiveObjectStore a besoin d'un peu plus
                      d'appels de fonctions que le SQLGenObjectStore.<br>
                      En particulier, il faut bien effectuer un appel à
                      .flush() pour qu'il flush les méta-data,<br>
                      ainsi qu'un appel (unique, à la fin de l'import) à
                      .cleanup().<br>
                      <br>
                      N'hésites pas si il reste un problème.<br>
                      <br>
                      Cordialement,<br>
                      <br>
                      Vincent<br>
                      <br>
                      On 20/05/2014 14:13, LANQUETUIT Cyril wrote:<br>
                    </div>
                    <blockquote type="cite">
                      <div style="direction:ltr; font-family:Tahoma;
                        color:#000000; font-size:10pt">Hello Vincent,<br>
                        <br>
                        Thank's for your answer,<br>
                        <br>
                        I've try to use MassiveObjectStore by passing a
                        specified replace_sep for the initialisation of
                        the object as following:<br>
                        <br>
                        <i>    store = MassiveObjectStore(session,
                          replace_sep=' ')<br>
                          <br>
                        </i>that is to say a space instead of '',  this
                        line just supply the former one:<br>
                        <br>
                        <i>    store = SQLGenObjectStore(session)</i><br>
                        <br>
                        in imagen.py script, so the error described in
                        the last mail does not appear anymore and the
                        script is executed of course quicker than with
                        SQLGenObjectStore but some error persist in the
                        base, when I try to visualise subjects in
                        CubicWeb I get this following error message:<br>
                        <br>
                        <h2><i>une erreur est survenue</i></h2>
                        <div class="tb"><i>UnknownEid: No entity with
                            eid 70239 in the repository</i></div>
                        <hr>
                        <p class="error"><i>No entity with eid 70239 in
                            the repository</i></p>
                        <p class="error"><i><br>
                          </i></p>
                        <p class="error"><i>[...] <-- a stacktrace
                            that I don't really understand and I prefere
                            to sumarise like this<br>
                          </i></p>
                        <i><br>
                          <b>CubicWeb version:</b> 3.17.14<br>
                          <b>Cube brainomics version:</b> 0.8.0<br>
                          <b>Cube genomics version:</b> 0.7.0<br>
                          <b>Cube medicalexp version:</b> 0.8.0<br>
                          <b>Cube neuroimaging version:</b> 0.5.0<br>
                          <b>Cube questionnaire version:</b> 0.5.0<br>
                          <b>Cube squareui version:</b> 0.2.1<br>
                          <b>Cube bootstrap version:</b> 0.4.0<br>
                          <b>Cube card version:</b> 0.5.3<br>
                          <b>Cube comment version:</b> 1.10.0<br>
                          <b>Cube file version:</b> 1.15.0<br>
                          <b>Cube jqplot version:</b> 0.4.0</i><br>
                        <br>
                        Entity 70239 is the first subject inserted in
                        the base, I can see it with psql as :<br>
                        <br>
                            cw_identifier    | cw_surname | cw_firstname
                        | cw_gender | cw_date_of_birth | cw_handedness |
                        cw_creation_date   | cw_cwuri |   
                        cw_modification_date    | cw_eid
                        <br>
---------------------+------------+--------------+-----------+------------------+---------------+---------------------+-------------------------------------------------------+----------------------------+--------<br>
                         IMAGEN_000064851530 |            |             
                        | female    |                  | right         |
                        2014-05-20 00:00:00 |
                        _auto_generated81d7c2bc89924f1d8ac021b3009e6b3b63036 
                        | 2014-05-20 00:00:00        |  70239<br>
                        <br>
                        So I don't understand the error message...<br>
                        <br>
                        This error does not appear when we use
                        SQLGenObjectStore so if there is a correction to
                        bring to the script for enable
                        MassiveObjectStore to do the stuff as well as
                        the last stable version of the script I'll
                        accept your advices, in the other case we
                        certainly may keep the script as it is, that is
                        to say in a version which work slowly but
                        properly<br>
                        <br>
                        Thank's again,<br>
                        Cyril<br>
                        <div style="font-family:Times New Roman;
                          color:#000000; font-size:16px">
                          <hr tabindex="-1">
                          <div id="divRpF778268" style="direction:ltr"><font
                              color="#000000" face="Tahoma" size="2"><b>De
                                :</b> Vincent Michel [<a
                                moz-do-not-send="true"
                                class="moz-txt-link-abbreviated"
                                href="mailto:vincent.michel@logilab.fr"
                                target="_blank">vincent.michel@logilab.fr</a>]<br>
                              <b>Envoyé :</b> mardi 20 mai 2014 12:26<br>
                              <b>À :</b> LANQUETUIT Cyril; <a
                                moz-do-not-send="true"
                                class="moz-txt-link-abbreviated"
                                href="mailto:cubicweb@lists.cubicweb.org"
                                target="_blank">
                                cubicweb@lists.cubicweb.org</a><br>
                              <b>Objet :</b> Re: [Cubicweb]
                              Misunderstanding of dataimport
                              _create_copyfrom_buffer function<br>
                            </font><br>
                          </div>
                          <div>
                            <div class="moz-cite-prefix">Hi Cyril,<br>
                              <br>
                              The _create_copyfrom_buffer of
                              cubicweb/dataimport.py is indeed a bit
                              tricky...<br>
                              <br>
                              This function creates a CSV string that
                              will be pushed in postgresql using COPY
                              FROM.<br>
                              However, if the separator used in the CSV
                              string already exists in some attributes,
                              it will crash. Thus it is possible to give
                              a replace_sep to replace the separator by
                              this given string.<br>
                              <br>
                              Indeed, the correct clause should be :<br>
                              <pre class="literal-block"><i>if (replace_sep != None):

</i></pre>
                              I have created a ticket on this issue (<a
                                moz-do-not-send="true"
                                class="moz-txt-link-freetext"
                                href="https://www.cubicweb.org/3845572"
                                target="_blank">https://www.cubicweb.org/3845572</a>),<br>
                              and I think that the whole function should
                              be rewritten.<br>
                              <br>
                              Best,<br>
                              <br>
                              Vincent<br>
                              <br>
                              <br>
                              <br>
                              <br>
                              On 19/05/2014 14:24, LANQUETUIT Cyril
                              wrote:<br>
                            </div>
                            <blockquote type="cite">
                              <style id="owaParaStyle" type="text/css">
<!--
p
        {margin-top:0;
        margin-bottom:0}
p
        {margin-top:0;
        margin-bottom:0}
body
        {direction:ltr;
        font-family:Tahoma;
        color:#000000;
        font-size:10pt}
p
        {margin-top:0;
        margin-bottom:0}
body
        {scrollbar-base-color:undefined;
        scrollbar-highlight-color:undefined;
        scrollbar-darkshadow-color:undefined;
        scrollbar-arrow-color:undefined}
body
        {scrollbar-base-color:undefined;
        scrollbar-highlight-color:undefined;
        scrollbar-darkshadow-color:undefined;
        scrollbar-arrow-color:undefined}
body
        {scrollbar-base-color:undefined;
        scrollbar-highlight-color:undefined;
        scrollbar-darkshadow-color:undefined;
        scrollbar-arrow-color:undefined}
body
        {scrollbar-base-color:undefined;
        scrollbar-highlight-color:undefined;
        scrollbar-darkshadow-color:undefined;
        scrollbar-arrow-color:undefined}
body
        {direction:ltr;
        font-family:Tahoma;
        color:#000000;
        font-size:10pt}
p
        {margin-top:0;
        margin-bottom:0}
body
        {scrollbar-base-color:undefined;
        scrollbar-highlight-color:undefined;
        scrollbar-darkshadow-color:undefined;
        scrollbar-arrow-color:undefined}
body
        {scrollbar-base-color:undefined;
        scrollbar-highlight-color:undefined;
        scrollbar-darkshadow-color:undefined;
        scrollbar-arrow-color:undefined}
body
        {scrollbar-base-color:undefined;
        scrollbar-highlight-color:undefined;
        scrollbar-darkshadow-color:undefined;
        scrollbar-arrow-color:undefined}
body
        {scrollbar-base-color:undefined;
        scrollbar-highlight-color:undefined;
        scrollbar-darkshadow-color:undefined;
        scrollbar-arrow-color:undefined}
body
        {scrollbar-base-color:undefined;
        scrollbar-highlight-color:undefined;
        scrollbar-darkshadow-color:undefined;
        scrollbar-arrow-color:undefined}
-->
BODY {direction: ltr;font-family: Tahoma;color: #000000;font-size: 10pt;}P {margin-top:0;margin-bottom:0;}BODY {scrollbar-base-color:undefined;scrollbar-highlight-color:undefined;scrollbar-darkshadow-color:undefined;scrollbar-track-color:undefined;scrollbar-arrow-color:undefined}BODY {scrollbar-base-color:undefined;scrollbar-highlight-color:undefined;scrollbar-darkshadow-color:undefined;scrollbar-track-color:undefined;scrollbar-arrow-color:undefined}BODY {scrollbar-base-color:undefined;scrollbar-highlight-color:undefined;scrollbar-darkshadow-color:undefined;scrollbar-track-color:undefined;scrollbar-arrow-color:undefined}</style>
                              <div style="direction:ltr;
                                font-family:Tahoma; color:#000000;
                                font-size:10pt"><tt>Hello,<br>
                                  <br>
                                  In function _create_copyfrom_buffer of
                                  file dataimport.py, in the following
                                  part of code:</tt><br>
                                <i><br>
                                                          # If a
                                  replace_sep is given, replace<br>
                                                          # the
                                  separator instead of returning None<br>
                                                          # (and thus
                                  avoid empty buffer)<br>
                                                          if
                                  replace_sep:<br>
                                                              value =
                                  value.replace(_char, replace_sep)<br>
                                                          else:<br>
                                                              return</i><br>
                                <br>
                                <div style="font-family:Times New Roman;
                                  color:#000000; font-size:16px">
                                  <div>
                                    <div style="direction:ltr;
                                      font-family:Tahoma; color:#000000;
                                      font-size:10pt">
                                      <pre class="literal-block">when the separator is set to '' (the default value when MassiveObjectStore is used for store)
the replace_sep is evaluate to false so the else clause is executed, the function _create_copyfrom_buffer return an empty buffer
then this following error is raised:

<i>File "/home/cl239099/cubicweb/cubes/dataio/dataimport.py", line 831, in flush_entities
    raise ValueError('Error in buffer creation for etype %s' % etype)
ValueError: Error in buffer creation for etype Question</i>

I don't understand exactly what the code should do or should be but if the clause
<i>
if replace_sep:</i>

is replaced by something like
<i>
if (replace_sep != None):
</i>
the else clause won't be executed in this case...

please could you examinate this issue and correct it if needed or let me know what is wrong in the precedent discribed comportement<i>
</i>
Thank's,
Cyril
</pre>
                                    </div>
                                  </div>
                                </div>
                              </div>
                              <br>
                              <fieldset class="mimeAttachmentHeader"
                                target="_blank"></fieldset>
                              <br>
                              <pre>_______________________________________________
Cubicweb mailing list
<a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:Cubicweb@lists.cubicweb.org" target="_blank">Cubicweb@lists.cubicweb.org</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://lists.cubicweb.org/mailman/listinfo/cubicweb" target="_blank">http://lists.cubicweb.org/mailman/listinfo/cubicweb</a>
</pre>
                            </blockquote>
                            <br>
                          </div>
                        </div>
                      </div>
                    </blockquote>
                    <br>
                  </div>
                </div>
              </div>
            </blockquote>
            <br>
          </div>
        </div>
      </div>
    </blockquote>
    <br>
  </body>
</html>