using af_feedmod

Support requests, bug reports, etc. go here. Dedicated servers / VDS hosting only
thermionic
Bear Rating Trainee
Bear Rating Trainee
Posts: 42
Joined: 15 May 2013, 13:50

using af_feedmod

Postby thermionic » 16 May 2013, 18:59

Hi

Please forgive my coding/json inabilities, I managed to get ttrss running with nginx on Linux, but I'm completely out of my depth with json.

I'm trying and failing to understand how to use af_feedmod.

If I understand it correctly, the af_feedmod should allow a "complete" feed to be displayed in ttrss instead of just a "synopsis" feed, the problem being I'm not adept enough to follow the instructions at https://github.com/mbirth/ttrss_plugin-af_feedmod

I attempted to use the example for The Oatmeal as a base

I created a file ioshints.info.json in af_feedmod/mods

Code: Select all

{
    "name": "ioshints",
    "feed": "http://blog.ioshints.info/feeds/posts/default",
    "match": "CiscoIosHintsAndTricks",
    "config": {
        "type": "xpath",
        "xpath": "div[@id='CiscoIosHintsAndTricks']"
    }
}

I then created the following in the feedmod tab

Code: Select all

{
    "CiscoIosHintsAndTricks": {
        "type": "xpath",
        "xpath": "div[@id='CiscoIosHintsAndTricks']"
    }
}

However, unsurprisingly, it doesn't work :-(

I now it's down to my understanding of what is required by the plugin, but the readme is way over my head. I've had a google for a more complete worked example, but come up with nothing so far.

If anyone can either point me to a "dummys guide" or explain what information I need, that would be great :-)

Cheers

feader
Bear Rating Master
Bear Rating Master
Posts: 160
Joined: 26 Dec 2012, 20:03

Re: using af_feedmod

Postby feader » 16 May 2013, 20:26

You got this all wrong :roll:

Don't bother with creating external files, work only with the FeedMod tab. There is an example configuration right on the github page. The thing you probably want is

Code: Select all

{
    "blog.ioshints.info": {
        "type": "xpath",
        "xpath": "div[@id='post-body']"
    }
}

The array name is used to find the right feed, it must be contained in the feed URL. The part where you have some work to do is to find the correct XPATH expression (did that for you :mrgreen: ). Check w3c for some introduction. If you use Firefox, you may find this Add-On very useful, Chrome stuff is listed on github.

Hope it helps. :wink:

thermionic
Bear Rating Trainee
Bear Rating Trainee
Posts: 42
Joined: 15 May 2013, 13:50

Re: using af_feedmod

Postby thermionic » 16 May 2013, 21:43

More demonstrations of my ignorance, but I am trying to learn (yes I know, very trying...)

I had created the .json file from back tracking the "golem0Bde0C" example, "golem0Bde0C" obviously not being a hostname...

Should the "mods" directory be more accurately called a "samples/examples" directory ?

I don't follow what is meant by "array name"

What determines if it is "xpath": "div[@class='aaa']" or "xpath": "bbb" or "xpath": "div[@id='ccc']" ? and what determines what the aaa, bbb or ccc should be ?

I tried looking at the source for The Oatmean xml feed and could only find "comic" in the URL, not speaking or reading German made the others a little too complex for me to follow.

I tried the code that you posted and it had no effect, then I remembered that I had read that af_feedmod does its work before the the feed goes into the database, so unsubscribed and re-subscribed, but it still only has the synopsis.

From using Chrome "Inspect element" on the feedburner page that the URL redirects to (http://feeds.feedburner.com/CiscoIosHintsAndTricks) I tried replacing "xpath": "div[@id='post-body']" with "xpath": "div[@id='bodyblock']", then I tried using "xpath": "div[@class='itemtitle']", then I tried using the redirected URL http://feeds.feedburner.com/CiscoIosHintsAndTricks for the feed and changing the hostname in the feedmod tab, but this also appeared to have no effect

Just to confirm, I unsubscribed and re-subscribed with each change.

Also just to confirm, I'm running 1.7.9 and I have the following other plugins enabled
auth_internal
updater
af_buttersafe
af_explosm
af_feedmod
af_gocomics
af_pennyarcade
af_redditimgur
af_unburn
embed_original
ff_xmllint
note
videoframes

Apologies if I'm coming across as an idiot

feader
Bear Rating Master
Bear Rating Master
Posts: 160
Joined: 26 Dec 2012, 20:03

Re: using af_feedmod

Postby feader » 16 May 2013, 22:07

Read the main page very carefully, he calls it "array key". Inform yourself what a XPath actually is, and what this plug-in does with the xpath you give to it (xpath specifies which part of the "article page" is fetched).

User avatar
fox
^ me reading your posts ^
Posts: 6318
Joined: 27 Aug 2005, 22:53
Location: Saint-Petersburg, Russia
Contact:

Re: using af_feedmod

Postby fox » 16 May 2013, 22:09

Let's be fair, xpath is not for normal people. :)

feader
Bear Rating Master
Bear Rating Master
Posts: 160
Joined: 26 Dec 2012, 20:03

Re: using af_feedmod

Postby feader » 16 May 2013, 22:42

fox wrote:Let's be fair, xpath is not for normal people. :)

Even I got some part of it, eventually :wink:
But yeah, the wikipedia entry is very technical, a newbie should better read the w3schools link in my first post, and play with the browser extensions to get a feeling for it.

lotrfan
Bear Rating Disaster
Bear Rating Disaster
Posts: 73
Joined: 18 Mar 2013, 04:42

Re: using af_feedmod

Postby lotrfan » 16 May 2013, 22:48

Try

Code: Select all

{
    "CiscoIosHintsAndTricks": {
        "type": "xpath",
        "xpath": "div[@class='post-body']"
    }
}


edit: I'm using af_unburn, so the URL my be different for you.

How I came up with the above:

If you debug the feed update by pressing "f D" when you have the feed open (that's a capital 'D', so it's actually "f shift+d"), you will see something like

Code: Select all

[18:33:02/19284] date 1368680760 [2013/05/16 05:06:00]
[18:33:02/19284] title Tail-f Network Control System – the First Impressions
[18:33:02/19284] link http://feedproxy.google.com/~r/CiscoIosHintsAndTricks/~3/FoEctOnetwork-control-system-first.html
[18:33:02/19284] author Ivan Pepelnjak
[18:33:02/19284] num_comments: 0
[18:33:02/19284] looking for tags...
[18:33:02/19284] tags found: network management,sdn,openflow
[18:33:02/19284] done collecting data.
[18:33:02/19284] applying plugin filters..
[18:33:02/19284] plugin data: feedmod,2:unburn,2:
[18:33:02/19284] base guid found, checking for user record

The third line ("link ...") is the . This is what TT-RSS thinks that the article link is. You want to pick something unique from that URL. I chose "CiscoIosHintsAndTricks" --- this is what's in the second line of the JSON above. If you look of the source of that URL, you will see that the article is wrapped in

Code: Select all

<div class='post-body'>
...
</div>

The identifier is therefore that the class is 'post-body'. Using the example XPath's on github, the expression is

Code: Select all

div[@class='post-body']


(As noted in the posts above, XPath can be tricky, especially if you're not used to working with XML. The browser extensions can definitely help a lot.)

thermionic
Bear Rating Trainee
Bear Rating Trainee
Posts: 42
Joined: 15 May 2013, 13:50

Re: using af_feedmod

Postby thermionic » 16 May 2013, 23:13

I used "array name" following your post...

I still haven't got a clue by what is meant by array name/key

As your code didn't appear to work, I tried several different alternatives that looked reasonable from my examination of the source, that also did not work, suggesting that I re-read a manual page that I've already said I am unable to

I'm beginning to get a handle on what xpath is, however, getting to something that works...

If I wanted to select the top/newest link, I'm guessing that it would be

Code: Select all

//*[@id="bodyblock"]/ul[1]/li/h4/a

and the second and third ones etc would be

Code: Select all

//*[@id="bodyblock"]/ul[2]/li/h4/a
//*[@id="bodyblock"]/ul[3]/li/h4/a

which as I want it to be able to find all of them, I think might be be

Code: Select all

//*[@id="bodyblock"]/ul/li/h4/a

which then needs to be translated to work with the plugin, the closest I can get to would be

Code: Select all

 "xpath": "div[@id='bodyblock'/ul/li/h4/a]"


However I still don't know if I should be using

the hostname
the redirected hostname
the unique part of the redirected URL
something else

thermionic
Bear Rating Trainee
Bear Rating Trainee
Posts: 42
Joined: 15 May 2013, 13:50

Re: using af_feedmod

Postby thermionic » 16 May 2013, 23:51

@lotrfan

That worked perfectly :-)

Many thanks for the explanation.

If I understand you correctly, the "unique id" is used to "discover" which links to enumerate, and the xpath is used to specify what to get on the actual destination?

feader
Bear Rating Master
Bear Rating Master
Posts: 160
Joined: 26 Dec 2012, 20:03

Re: using af_feedmod

Postby feader » 16 May 2013, 23:54

We're getting into the right direction, I think … This JSON object

Code: Select all

{
"heise.de": {
    "type": "xpath",
    "xpath": "div[@class='meldung_wrapper']",
    "force_charset": "utf-8"
}
}

contains another JSON object which is named "heise.de", and this is the object name/key that the author and I falsely called array name/key (see wikipedias JSON for that). The object name determines on which URLs the Plug-in does its magic:
The array key [sic!] is part of the URL of the article links(!). You'll notice the golem0Bde0C in the last entry: That's because all their articles link to something like
http://rss.feedsportal.com/c/33374/f/578068/p/1/s/3f6db44e/l/0L0Sgolem0Bde0Cnews0Cthis0Eis0Ean0Eexample0A10Erss0Bhtml/story01.htm
and to have the plugin match that URL and not interfere with other feeds using feedsportal.com, I used the part golem0Bde0C.

See lotrfan's post for your example, my code didn't match because I didn't bother to subscribe to the feed and look at the URLs, so I provided a wrong object name. Sorry for that :oops:

For the XPath thing: The browser plugins can be helpful (at the very least to get a starting point in the source), but as lotrfan suggested, it is probably better to take a good look at the source code of the page to construct a good node specifier. You need some working knowledge about CSS/HTML for that (what classes & ids are).

lotrfan
Bear Rating Disaster
Bear Rating Disaster
Posts: 73
Joined: 18 Mar 2013, 04:42

Re: using af_feedmod

Postby lotrfan » 17 May 2013, 00:22

thermionic wrote:If I understand you correctly, the "unique id" is used to "discover" which links to enumerate, and the xpath is used to specify what to get on the actual destination?

Yes, that is correct -- the "array key" (in this case, "CiscoIosHintsAndTricks") makes sure that only the right articles are replaced.

"array key", as mentioned previously, is a bit of a misnomer, but I guess JSON's "{ ... }" construct could be thought of defining an associative array...


On a semi-but-not-really related note, if you're using the git version (of TT-RSS), you probably want to change (around line 75)

Code: Select all

if (version_compare(VERSION, '1.7.9', '>=')) {

to

Code: Select all

if (version_compare(VERSION_STATIC, '1.7.9', '>=')) {

The test fails when comparing with VERSION when running git. It's not a huge deal, but it does let the plugin use the nicer fetching functions (i.e., it uses cURL).

thermionic
Bear Rating Trainee
Bear Rating Trainee
Posts: 42
Joined: 15 May 2013, 13:50

Re: using af_feedmod

Postby thermionic » 17 May 2013, 00:58

I've got it :-)

I'm doing a worked example for two sites which I'll post the Knowledge Base

I have however hit a small question, what if the destination site needs more than one xpath for its articles ?

As an example, some posts might be div id="comic" and others div id="blog"

feader
Bear Rating Master
Bear Rating Master
Posts: 160
Joined: 26 Dec 2012, 20:03

Re: using af_feedmod

Postby feader » 17 May 2013, 02:25

thermionic wrote:As an example, some posts might be div id="comic" and others div id="blog"

There is the or operator '|', see here. But af_feedmod has a dirty little secret: As yet, not all valid XPath expressions are supported, don't know about that one.

thermionic
Bear Rating Trainee
Bear Rating Trainee
Posts: 42
Joined: 15 May 2013, 13:50

Re: using af_feedmod

Postby thermionic » 17 May 2013, 02:49

I tried "xpath": "div[@id='comic' | @id='blog']" which validates as json, but didn't have any joy :-(

lotrfan
Bear Rating Disaster
Bear Rating Disaster
Posts: 73
Joined: 18 Mar 2013, 04:42

Re: using af_feedmod

Postby lotrfan » 17 May 2013, 03:00

Try

Code: Select all

"xpath": "div[@id='comic'] | //div[@id='blog']"


Not sure if it will work, but it's worth a shot...


Return to “Support”

Who is online

Users browsing this forum: No registered users and 11 guests