[Cubicweb] datafeed

[David Douard]

On 04/12/2013 09:36, Dimitri Papadopoulos Orfanos wrote:
> I've found a few words on datafeed in the Logilab blog about CubicWeb 3.11:
>     http://www.cubicweb.org/blogentry/1512775
> 
>     A new 'datafeed' source was introduced, inspired by the
>     soon to be deprecated datafeed cube. It needs polishing
>     but sets the foundation for advanced semantic web
>     applications that import content from others site using
>     simple http request.
> 
>     A 'datafeed' source is associated to a parser that
>     analyses the imported data and then creates/updates
>     entities accordingly. There is currently a single parser
>     in the core that imports CubicWeb-generated xml and needs
>     to be configured with a mapping information that defines
>     how relations are to be followed. It provides a viable
>     alternative to 'pyrorql' sources. Other parsers to import
>     RDF, RSS, etc should come soon.
> 
> From what I gather datafeed can help import data from structured Web
> sites. It doesn't help importing data from files (we have to parse at
> least CSV, XML, DICOM files), does it?

Not really, datafeed really is about importing (part of) an external
database in a CW instance. datafeed can update entities as long as it
can identify a unique identifier for that entity in the remote database
(typically an URI).

> In our application we repeatedly scan a directory for new files and data
> appended to existing files. Can datafeed help in that case? I'm thinking
> here about the "creates/updates entities accordingly" part of datafeed.


At update time, it needs to be able to ask the objects in the remote
database that have been created/modified since last synchronization. All
these stuff you won't (easily) have from files. What you need is a set
of (smart) importation scripts.

Maybe Vincent can help you here.


David



> Le 04/12/2013 09:25, Dimitri Papadopoulos Orfanos a écrit :
>> Dear all,
>>
>> I am interested in datafeed, a recent addition to the core CubicWeb
>> libraries, because we are assigned the task to import data from
>> different sources into our database.
>>
>> There is not much information on datafeed except for the source code. I
>> wonder whether any of you CubicWeb wizards could:
>> * explain the purpose and benefits of datafeed in one or two sentences,
>> * give a few hints on the usage of datafeed.
>>
> 


-- 

David DOUARD		 LOGILAB
Directeur du département Outils & Systèmes

+33 1 45 32 03 12	 david.douard at logilab.fr
+33 1 83 64 25 26	 http://www.logilab.fr/id/david.douard

Formations - http://www.logilab.fr/formations
Développements - http://www.logilab.fr/services
Gestion de connaissances - http://www.cubicweb.org/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: david_douard.vcf
Type: text/x-vcard
Size: 302 bytes
Desc: not available
URL: <http://lists.cubicweb.org/pipermail/cubicweb/attachments/20131216/ba2a75ef/attachment-0273.vcf>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 263 bytes
Desc: OpenPGP digital signature
URL: <http://lists.cubicweb.org/pipermail/cubicweb/attachments/20131216/ba2a75ef/attachment-0271.sig>