[Cubicweb] Loading slow a big table using default view

Jinpeng Li mr.li.jinpeng at gmail.com
Thu Jan 23 12:24:52 CET 2014


Dear developers,

We found a problem that it is too slow to show a big table using default
view. I wrote a description as below with Markdown description. I hope that
the description is clear enough to understand. Don't hesitate to let me
know if you don't understand any part.

Best,
Jinpeng
Brainomics in Neurospin in CEA

Create a new cube as:

```
$ cubicweb-ctl newcube test_big_table
Enter a short description for your cube: dddd
```

Modify test_big_table/schema.py as:
```
from yams.buildobjs import (EntityType, RelationDefinition,
                            SubjectRelation, String, RichString,
                            BigInt, Int, Float, Boolean)
class DataEntity(EntityType):
    """ Test big table"""
    name = String(maxsize=256)  # HotSpot
    data = Float(required=True)
```

Create an instance as i_test_big_table:
```
$ cubicweb-ctl create test_big_table i_test_big_table
$ cubicweb-ctl start i_test_big_table
```

Create a python script "import_db_big_table.py" as below:

```
from cubicweb.dataimport import SQLGenObjectStore
if __name__ == "__main__":
    store = SQLGenObjectStore(session)
    for i in xrange(1000000):
        if i % 100000 == 0:
            print i
            store.flush()
        data = {"name": "%d" % i, "data": float(i)}
        store.create_entity('DataEntity', **data)
    store.flush()
```

Run this script to import all the data:
```
$ cubicweb-ctl shell i_test_big_table import_db_big_table.py
```

Go to the web page (http://127.0.0.1:8080) to see DataEntity, it will take
more than 5 seconds to open pages with only first 40 data items among 1
million data items. If there are 10 million data, it will take more than 1
minute. In fact, if we execute rql in shell mode and only get first 40
items, it will show it very fast:

```
$ cubicweb-ctl shell i_test_big_table
>>> res = session.execute("Any E where E is DataEntity")
>>> entities = res.entities()
>>> entities.next()
<Entity DataEntity 656551 ['cwuri', 'data', 'name', 'modification_date',
'creation_date'] at 55907984>
```

We guess that the default view may try to load all the data once and show
only first 40 items. In our real database, we have 10 million data, it will
take more than 1 minute to open web page where only first 40 items have
been shown. It would be to better to improve the default view with loading
limited number data items.  If there is another possible reason, please let
us know. Thanks in advance.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.cubicweb.org/pipermail/cubicweb/attachments/20140123/7d681d4d/attachment-0164.html>


More information about the Cubicweb mailing list