Purging long feeds

Development-related discussion, including bundled plugins
xtaz
Bear Rating Master
Bear Rating Master
Posts: 174
Joined: 24 Dec 2009, 16:48

Purging long feeds

Postby xtaz » 19 Nov 2010, 15:24

Am I right in thinking old posts are only purged once they are older than the purge time and also when they are no longer in the downloaded feed? I have just noticed that one of the feeds I subscribe to ( http://www.thinkbroadband.com/rss/full/ ) currently has about 524 entries in it dating back to march 2009. Obviously this is very bad practice for the person who did this but I've noticed my tt-rss has every entry in it and hasn't purged the older ones.

I'm thinking that if this continues for several more years, or several other feeds did the same thing then tt-rss would get to a state where it was slowing down because of all the entries and was in dire need of a purge. I'm wondering if it might be better to change the purge code a little so that it also purged older ones which were still part of the feed. I guess the reason it doesn't do this is because tt-rss would then just see them as new entries again and re-import them and show them as new posts. Something to think about? Maybe have a cache of imported post ID's, or a configurable limit on the number of new posts to import?

User avatar
fox
^ me reading your posts ^
Posts: 6318
Joined: 27 Aug 2005, 22:53
Location: Saint-Petersburg, Russia
Contact:

Re: Purging long feeds

Postby fox » 19 Nov 2010, 15:55

Am I right in thinking old posts are only purged once they are older than the purge time and also when they are no longer in the downloaded feed


Yes. Otherwise you'll get duplicate articles on every feed update.

Maybe have a cache of imported post ID's, or a configurable limit on the number of new posts to import?


Personally I don't see the value of adding complexity by something like that given the (vanishingly small) number of feeds that have humongous amounts of obsolete entries. You'll need a lot of feeds like that to make any difference on tt-rss performance. Also, taking into account that unread counters are cached into separate table, I'm not sure that would even affect UI responsibility so much, unless your database server is really slow.

xtaz
Bear Rating Master
Bear Rating Master
Posts: 174
Joined: 24 Dec 2009, 16:48

Re: Purging long feeds

Postby xtaz » 19 Nov 2010, 18:50

Cool. If you are happy with that then it's not a problem. I just know how much you push the point that people need to purge or else there are large performance implications so thought I would mention it in case it did become a problem.

User avatar
fox
^ me reading your posts ^
Posts: 6318
Joined: 27 Aug 2005, 22:53
Location: Saint-Petersburg, Russia
Contact:

Re: Purging long feeds

Postby fox » 19 Nov 2010, 19:05

Well, if it was a really widespread issue, it might have required some kind of tweak. Luckily content providers are quite keen on minimizing their traffic bills. :)


Return to “Development”

Who is online

Users browsing this forum: No registered users and 4 guests