Error 403 / 400 with several feeds

[aratistern]

I have a problem with several feeds since 6 February 2017. I'm on my own server, running Ubuntu 16.10, and TT-RSS updated to latest git. I use the standard theme and no third-party plugins.

Code: Select all

PHP Version => 7.0.15-0ubuntu0.16.10.2
PostgreSQL 9.4.7 on x86_64-unknown-linux-gnu, compiled by gcc (Ubuntu 5.2.1-22ubuntu2) 5.2.1 20151010, 64-bit
nginx version: nginx/1.10.1 (Ubuntu)
Tiny Tiny RSS v17.1 (e6d66fe)   © 2005-2017   Andrew Dolgov
Description: Ubuntu 16.10
Release: 16.10
Codename: yakkety

Can anyone give me a hint? Many thanks...

One example is https://kodi.tv/feed/
Output from https://fakecake.org/myfeedsucks/

Processing feed: https://kodi.tv/feed/
Fetching URL (UA: Tiny Tiny RSS/17.1 (http://tt-rss.org/))
OK. Here's what we received:

Code: Select all

<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0"
   xmlns:content="http://purl.org/rss/1.0/modules/content/"
   xmlns:wfw="http://wellformedweb.org/CommentAPI/"
   xmlns:dc="http://purl.org/dc/elements/1.1/"
   xmlns:atom="http://www.w3.org/2005/Atom"
   xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
   xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
   >
<channel>
   <title>Kodi</title>
   <atom:link href="https://kodi.tv/feed/" rel="self" type="application/rss+xml" />
   <link>https://kodi.tv</link>
   <description>Open Source Home Theater Software</description>
   <lastBuildDate>Thu, 23 Feb 2017 19:40:42 +0000</lastBuildDate>
   <language>en-US</language>
   <sy:updatePeriod>hourly</sy:updatePeriod>
   <sy:updateFrequency>1</sy:updateFrequency>
   <generator>http://wordpress.org/?v=3.5.2</generator>
      <item>
      <title>Kodi v17.1 “Krypton” Release Candidate 1</title>
      <link>https://kodi.tv/kodi-v17-1-krypton-release-candidate-1/</link>
      <comments>https://kodi.tv/kodi-v17-1-krypton-release-candidate-1/#comments</comments>
      <pubDate>Thu, 23 Feb 2017 19:40:42 +0000</pubDate>
      <dc:creator>Martijn Kaijser</dc:creator>
            <category><![CDATA[Site News]]></category>

      <guid isPermaLink="false">https://kodi.tv/?p=7241</guid>
      <description><![CDATA[This is the Release Candidate for our upcoming bugfix release v17.1 “Krypton” which contains our continuous effort to further improve the v17 release. Our team will certainly try to tackle as much of the reported problems as possible with the limited resources we have. We do want to note that since we are just a small …]]></description>
            <content:encoded>[...]

Parsing...
Parsed successfully.
Title: Kodi
Articles:
ID: https://kodi.tv/?p=7241
Date: 1487878842
Link: https://kodi.tv/kodi-v17-1-krypton-release-candidate-1/
Title: Kodi v17.1 “Krypton” Release Candidate 1
Description: This is the Release Candidate for our upcoming bugfix release v17.1 “Krypton” which contains our continuous effort to further improve the v17 release. Our team will certainly try to tackle as much of the reported problems as possible with the limited resources we have. We do want to note that since we are just a small …
Content:

The feed debugger gives the following output:

Feed Debugger: 249: Kodi
[13:42:23/21535] start
[13:42:23/21535] local cache will not be used for this feed
[13:42:23/21535] fetching [https://kodi.tv/feed/]...
[13:42:24/21535] fetch done.
[13:42:24/21535] unable to fetch: HTTP Code: 403 [403]

Another example is http://www.bildblog.de/feed/
Output from https://fakecake.org/myfeedsucks/

Processing feed: http://www.bildblog.de/feed/
Fetching URL (UA: Tiny Tiny RSS/17.1 (http://tt-rss.org/))
OK. Here's what we received:

Code: Select all

<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
   xmlns:content="http://purl.org/rss/1.0/modules/content/"
   xmlns:wfw="http://wellformedweb.org/CommentAPI/"
   xmlns:dc="http://purl.org/dc/elements/1.1/"
   xmlns:atom="http://www.w3.org/2005/Atom"
   xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
   xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
   >

<channel>
   <title>BILDblog</title>
   <atom:link href="http://www.bildblog.de/feed/" rel="self" type="application/rss+xml" />
   <link>http://www.bildblog.de</link>
   <description>Ein Watchblog für deutsche Medien</description>
   <lastBuildDate>Fri, 24 Feb 2017 11:41:46 +0000</lastBuildDate>
   <language>de-DE</language>
   <sy:updatePeriod>hourly</sy:updatePeriod>
   <sy:updateFrequency>1</sy:updateFrequency>
   <generator>https://wordpress.org/?v=4.7.2</generator>
   <item>[...]

Parsing...
Parsed successfully.
Title: BILDblog
Articles:
ID: http://www.bildblog.de/?p=87190
Date: 1487922898
Link: http://www.bildblog.de/87190/hand-vorn- ... -hoffmann/
Title: Kopp Verlag, „Bild“-Ombudsmann, Klagesportgruppe Hoffmann [...]

The feed debugger gives the following output:

Feed Debugger: 117: BILDBlog
[13:48:56/22832] start
[13:48:56/22832] local cache will not be used for this feed
[13:48:56/22832] fetching [http://www.bildblog.de/feed/]...
[13:48:56/22832] fetch done.
[13:48:56/22832] unable to fetch: HTTP Code: 400 [400]

The TT-RSS is full of errors like:
Error:

Code: Select all

E_DEPRECATED (8192)

Filename:

Code: Select all

plugins/af_lang_detect/languagedetect/Text/LanguageDetect/Parser.php:33

Message:

Code: Select all

Methods with the same name as their class will not be constructors in a future version of PHP; Text_LanguageDetect_Parser has a deprecated constructor
1. plugins/af_lang_detect/languagedetect/LanguageDetect.php(25): ttrss_error_handler(8192, Methods with the same name as their class will not be constructors in a future version of PHP; Text_LanguageDetect_Parser has a deprecated constructor, plugins/af_lang_detect/languagedetect/Text/LanguageDetect/Parser.php, 33, Array)
2. plugins/af_lang_detect/languagedetect/LanguageDetect.php(25): require_once()
3. plugins/af_lang_detect/init.php(17): require_once(/var/www/ttrss/plugins/af_lang_detect/languagedetect/LanguageDetect.php)
4. classes/pluginhost.php(193): init([PluginHost])
5. include/rssfuncs.php(370): load(af_comics,af_fsckportal,af_lang_detect,af_readability,af_redditimgur,af_tumblr_1280,af_unburn,bookmarklets,fever,googlereaderkeys,import_export,mailto,shorten_expanded, 3, 1)
6. include/rssfuncs.php(212): update_rss_feed(216, 1, )
7. update.php(195): update_daemon_common(50)

[fox]

maybe you should look up http code 403

[aratistern]

Yep, but I had both 400 and 403 errors.

Anyway, I figured it out. Since many RSS feeds move behind Cloudflare, I implemented this tip on 6 February: viewtopic.php?p=23694#p23694

But what you can do is to use curl to download problematic RSS [...] Then point tt-rss to your local domain as a source for RSS.

... or just define SELF_USER_AGENT in config.php and TT-RSS will use that.

As you guessed yourself in that thread, it broke many other feeds. Or is there a bullet-proof value for that variable?

[fox]

maybe it's time to either prevent an easy modification of user agent or, at least, state that this makes their configuration unsupported

i had my doubts from the start tbh

e: support rules updated