Page 1 of 1

Import shared/starred items from GReader: Postgres support

Posted: 28 Mar 2013, 15:27
by nic
Just to let you know: People are helping me to make the import of starred and shared articles from Google Reader work not only for MySQL databases, but also Postgres.
See progress here: https://github.com/nhoening/gritttt-rss/issues/37

The import script is here: https://github.com/nhoening/gritttt-rss ... der-import

Re: Import shared/starred items from GReader: Postgres suppo

Posted: 28 Mar 2013, 15:33
by fox
Can someone share an example of this so I could look into making an import plugin already? I totally promise not to laugh at your silly starred items.

Re: Import shared/starred items from GReader: Postgres suppo

Posted: 28 Mar 2013, 15:35
by nic
I could give you a JSON snippet from my old withered Google account. But why would you need an example when you can see in my import script what the relevant parts are that need to be pulled out from it?

Re: Import shared/starred items from GReader: Postgres suppo

Posted: 28 Mar 2013, 15:38
by fox
I dunno, it feels easier somehow. Anyway nvm, I just starred some articles and made my own.

Re: Import shared/starred items from GReader: Postgres suppo

Posted: 28 Mar 2013, 16:27
by fox

Re: Import shared/starred items from GReader: Postgres suppo

Posted: 28 Mar 2013, 16:37
by nic
Cool. No shared items, though?

Re: Import shared/starred items from GReader: Postgres suppo

Posted: 28 Mar 2013, 16:41
by fox
Do those have different json format? I don't have any.

Re: Import shared/starred items from GReader: Postgres suppo

Posted: 28 Mar 2013, 16:42
by fox
Nah, same format. Imported fine. They are set starred, but it could be made optional.

Edit: I'll make it go by filename, if starred.json then set starred. Updated in trunk.

Edit2: made a separate thread as to not shit up this one.

Re: Import shared/starred items from GReader: Postgres suppo

Posted: 28 Mar 2013, 17:08
by Saliency
If it is easy, in addition to shared items, an import of labels would also be nice.

That said this will only be used mostly over the next three months..

Re: Import shared/starred items from GReader: Postgres suppo

Posted: 29 Mar 2013, 13:57
by mtavares
I tried the import implemented by nic gritttt-rss but it failed.
tt-rss v1.7.5, php 5.4.13, postgresql 9.2.3; dedicated hosting and multi user tt-rss
Runnig import.py with python 2.7.3 (doesn't seem to work with python 3) results in a gritttt-import.sql of 11MB (yes, long time greader here...) with 14792 inserts.
Then running "psql -U tt-rss -f gritttt-import.sql tt-rss &> log" the transaction is aborted because:
psql:gritttt-import.sql:63434: ERROR: index row size 4072 exceeds maximum 2712 for index "ttrss_entries_guid_key"
HINT: Values larger than 1/3 of a buffer page cannot be indexed.
Consider a function index of an MD5 hash of the value, or use full text indexing.

So my guess is that the index ttrss_entries_guid is using a btree and my stared/shared feeds include some very long entry.

I managed to isolate the "ofending" entry in the import generated sql and in the shared.json. Here are relevant parts (as far as I can tell):
* insert statement http://dpaste.org/ZwaPI/raw/
* original ofending entry http://bpaste.net/show/87488/ (notice that it's invalid json)
* corrected ofending entry http://bpaste.net/show/87490/

(had to use dpaste for the sql statement as bpaste decided that it contained spam.. whatever).

So the problem here might be the my takeout of google reader that is not a valid json at all. Nice. I'll try to edit and validate this huge json file and start the import again...

Re: Import shared/starred items from GReader: Postgres suppo

Posted: 31 Mar 2013, 14:47
by fox
Edit: in effort to not shit up this thread, posts related to the other plugin moved to the plugin thread.

viewtopic.php?f=1&t=1573