Twitter RSS Feeds

Support requests, bug reports, etc. go here. Dedicated servers / VDS hosting only
User avatar
sleeper_service
Bear Rating Overlord
Bear Rating Overlord
Posts: 884
Joined: 30 Mar 2013, 23:50
Location: Dallas, Texas

Re: Twitter RSS Feeds

Postby sleeper_service » 27 Jun 2013, 01:03

craywolf wrote:
Also, those differences are next to impossible to read :)


The first three lines starting with ">" were added to the header output under the <feed> tag (line 23). The $feedContent line (#33 in the original) starting with "<" was removed from the file, and replaced with the line below it. The "print('..." line (#37 in the original) starting with "<" was removed from the file, and replaced with the line below it.


I find context diffs easier to read ;)

craywolf
Mr. Awesome
Posts: 97
Joined: 19 Mar 2013, 18:07

Re: Twitter RSS Feeds

Postby craywolf » 27 Jun 2013, 01:12

sleeper_service wrote:I find context diffs easier to read ;)


Mmh... you're probably right. I don't do this stuff often. I'm a sysadmin, not a developer. I know enough PHP to be dangerous, and I know how to Google things and read RFCs. I just today figured out how to send a pull request via github.

I have some programming experience, and some web dev experience, they're just not something I do regularly.

User avatar
sleeper_service
Bear Rating Overlord
Bear Rating Overlord
Posts: 884
Joined: 30 Mar 2013, 23:50
Location: Dallas, Texas

Re: Twitter RSS Feeds

Postby sleeper_service » 27 Jun 2013, 02:57

craywolf wrote:
sleeper_service wrote:I find context diffs easier to read ;)


Mmh... you're probably right. I don't do this stuff often. I'm a sysadmin, not a developer. I know enough PHP to be dangerous, and I know how to Google things and read RFCs. I just today figured out how to send a pull request via github.

I have some programming experience, and some web dev experience, they're just not something I do regularly.


you just described me.

jdelamater99
Bear Rating Master
Bear Rating Master
Posts: 109
Joined: 11 Apr 2013, 17:45

Re: Twitter RSS Feeds

Postby jdelamater99 » 27 Jun 2013, 05:10

craywolf wrote:
jdelamater99 wrote:I've been testing the output with the w3 validator since v.5, and all my tests have been valid/well-formed XML from that point on.


Then there is something wildly different between the script you're running, and the script I'm running. I replaced my altered script with the one from your github repo, pasted the output of http://mydomain.tld/twitter-to-rss/?screen_name=wilw into the validator, and here's what it has to say:

Sorry

This feed does not validate.
line 3, column 0: Missing feed element: id [help]
<feed xmlns="http://www.w3.org/2005/Atom">
line 3, column 0: Missing feed element: updated [help]
<feed xmlns="http://www.w3.org/2005/Atom">
line 12, column 41: updated must be an RFC-3339 date-time: Wed Jun 26 20:29:50 +0000 2013 (20 occurrences) [help]
<updated>Wed Jun 26 20:29:50 +0000 2013</updated>


That can't just be a difference in PHP version or something, unless your PHP randomly inserts valid <id> and <updated> tags into output.



trés bizarre. It validates fine for me. :?

here's the xml output on my server. Although, I do agree that the superfolous html/body/p tags can go. the reason they are there in the first place is because they were in the twitter rss feed I had cached and was using as reference.

Code: Select all

<?xml version="1.0" encoding="utf-8"?>
<?xml-stylesheet type="text/xsl" href="atom-to-html.xsl"?>
<feed xmlns="http://www.w3.org/2005/Atom">
<title>@wilw</title>

   <entry>
      <id>https://twitter.com/wilw/statuses/350055466312802304</id>
      <link href="https://twitter.com/wilw/statuses/350055466312802304" rel="alternate" type="text/html"/>
      <title>wilw: I know it’s stupid, but I always feel like a bit of a Jedi when I swat a fly out of mid air. Or a ninja. Oh! Or a Ninjedi.</title>
      <summary type="html"><![CDATA[wilw: I know it’s stupid, but I always feel like a bit of a Jedi when I swat a fly out of mid air. Or a ninja. Oh! Or a Ninjedi.]]></summary>
      <content type="html"><![CDATA[<html><body><p></p><p>I know it’s stupid, but I always feel like a bit of a Jedi when I swat a fly out of mid air. Or a ninja. Oh! Or a Ninjedi.</p></body></html>]]></content>
      <updated>Thu Jun 27 00:58:21 +0000 2013</updated>
      <author><name></name></author>
   </entry>

   <entry>
      <id>https://twitter.com/wilw/statuses/350027566205255680</id>
      <link href="https://twitter.com/wilw/statuses/350027566205255680" rel="alternate" type="text/html"/>
      <title>wilw: SQUEEEEEE!!! http://t.co/9FgDhrptDY Thank you, @AshleyEsqueda and @Team_Barrowman</title>
      <summary type="html"><![CDATA[wilw: SQUEEEEEE!!! http://t.co/9FgDhrptDY Thank you, @AshleyEsqueda and @Team_Barrowman]]></summary>
      <content type="html"><![CDATA[<html><body><p></p><p>SQUEEEEEE!!! <a href="http://t.co/9FgDhrptDY">http://t.co/9FgDhrptDY</a> Thank you, @AshleyEsqueda and @Team_Barrowman</p></body></html>]]></content>
      <updated>Wed Jun 26 23:07:29 +0000 2013</updated>
      <author><name></name></author>
   </entry>
</feed>
<!-- vim:ft=xml -->
Attachments
@wilw.jpg
@wilw.jpg (39.69 KiB) Viewed 3643 times

craywolf
Mr. Awesome
Posts: 97
Joined: 19 Mar 2013, 18:07

Re: Twitter RSS Feeds

Postby craywolf » 27 Jun 2013, 05:19

Looks like that's just an XML validator. Being valid XML isn't the same as being valid Atom. Check it out in the validator I linked, which is specifically for Atom and RSS.

http://validator.w3.org/feed/

jdelamater99
Bear Rating Master
Bear Rating Master
Posts: 109
Joined: 11 Apr 2013, 17:45

Re: Twitter RSS Feeds

Postby jdelamater99 » 27 Jun 2013, 05:28

Ahhhhhh. I've been using the wrong validator

I've been using the w3.org one, but for xml, not Atom.

blerg. :(

craywolf
Mr. Awesome
Posts: 97
Joined: 19 Mar 2013, 18:07

Re: Twitter RSS Feeds

Postby craywolf » 27 Jun 2013, 06:57

jdelamater99 wrote:Ahhhhhh. I've been using the wrong validator


Eh, no worries. Good news is, the work to make it pass is already done. :D

Also, I don't know if this is helpful to you, but the output of the Twitter oembed API call looks nice. Gets you a nicely-formatted HTML version of the tweet including links for mentions and hashtags.

Unfortunately running into the rate-limiting would probably be an issue, if the script loads too many tweets at once or is reloaded too often. The response pretty much needs to be cached, and that would add a lot of complexity to your script. But I figured I'd throw it out there.

jdelamater99
Bear Rating Master
Bear Rating Master
Posts: 109
Joined: 11 Apr 2013, 17:45

Re: Twitter RSS Feeds

Postby jdelamater99 » 27 Jun 2013, 07:59

craywolf wrote:
jdelamater99 wrote:Ahhhhhh. I've been using the wrong validator


Eh, no worries. Good news is, the work to make it pass is already done. :D

Also, I don't know if this is helpful to you, but the output of the Twitter oembed API call looks nice. Gets you a nicely-formatted HTML version of the tweet including links for mentions and hashtags.

Unfortunately running into the rate-limiting would probably be an issue, if the script loads too many tweets at once or is reloaded too often. The response pretty much needs to be cached, and that would add a lot of complexity to your script. But I figured I'd throw it out there.


ok, now it outputs valid feed xml. I did use a bit of your code (thank you, btw), but I also ended up rewriting pretty much the entire beginning section of the feed xml.

Also incorporated a fix for the multiple searches, although the use of the hash in the URL does work, and this is a browser limitation.
?q=#monkeys

When you mouse over a hash tag on twitter, and you see twitter.com/search?q=#Thermostat, the link is actually pointing to the urlencoded version of that link, which is twitter.com/search?q=%23Thermostat. When you put the unencoded hashtag (#) in the search query, the browser interprets that as a page anchor, and doesn't pass anything after that to the server-side interpreter.

Can probably work that out with some javascript, but putting it at the top of the page breaks the validation, and putting it at the bottom of the page results in a double API call, neither of which is optimal. As it stands right now, you can't use the actual hashtag (#) in the search query.

The other major change is instead of each use case having it's own xml parser, they all now reference a common parser. This way, when I get around to adding in the embed images, I'll only have to do that once, instead of 4 times.

craywolf
Mr. Awesome
Posts: 97
Joined: 19 Mar 2013, 18:07

Re: Twitter RSS Feeds

Postby craywolf » 27 Jun 2013, 17:33

jdelamater99 wrote:ok, now it outputs valid feed xml. I did use a bit of your code (thank you, btw), but I also ended up rewriting pretty much the entire beginning section of the feed xml.


Nice changes overall, but you've gotta fix the feed ID. You're setting it to the URL of the first tweet, meaning it's going to change on every update. The feed ID needs to be constant through the life of the feed, it cannot change when the feed updates. After reading into the spec, I'd suggest this, which is a bit modified from my last post and is based on these tag URI recommendations:

Code: Select all

print('<id>tag:twitter.com,2006:/'.$twitter_data[0]['user']['screen_name'].'</id>'. PHP_EOL);

// Output: <id>tag:twitter.com,2006:/wilw</id>


The ID tags in feed.php, list.php, and home.php should be changed similarly. Looks like the one in search.php is OK.

Using the permalink as an entry ID is not recommended either, though it is technically correct and shouldn't cause any problems. But, here's a version that meets those same tag URI recommendations:

Code: Select all

print('         <id>tag:twitter.com,' . date("Y-m-d", strtotime($twitter_data[$i]['created_at'])) . ':/' . $twitter_data[$i]['user']['screen_name'] . '/statuses/' . $twitter_data[$i]['id_str'] . '</id>'. PHP_EOL);

// Output: <id>tag:twitter.com,2013-06-27:/wilw/statuses/350124047360724993</id>


That gets you an ID that is derived from the permalink URL in a predictable way, but isn't a permalink URL. The entry ID change is admittedly a bit of pedantry, but the feed ID definitely needs to be altered.

When you put the unencoded hashtag (#) in the search query, the browser interprets that as a page anchor, and doesn't pass anything after that to the server-side interpreter.


Ugh, I should have known that. I was expecting the browser to take care of urlencoding it (like it does for spaces) but forgot it's a special case.

EDIT: Also, tt-rss isn't picking up on your <link> element correctly (the one that goes to the twitter feed). Removing the rel= and type= attributes fixes that. I'm pretty sure that having rel="alternate" in there is valid, but it's also optional, so no harm in removing it.

EDIT 2: Adding this before the output might be a good idea too. With this, there's literally nothing the validator complains about on my install anymore:

Code: Select all

header('Content-type: application/atom+xml; charset=utf-8');

jdelamater99
Bear Rating Master
Bear Rating Master
Posts: 109
Joined: 11 Apr 2013, 17:45

Re: Twitter RSS Feeds

Postby jdelamater99 » 27 Jun 2013, 18:21

Excellent feedback. Thanks. What I was intending to do with the feed id was to use the user id, not the status id, I just forgot to update the variable so that it points to the correct array item. Oh well. What you've suggested works just as well, and is probably better anyways, so I'll use that.

here are the changes that I'm proposing, want to get your feedback before I actually commit them. Mostly what you suggested, but changing the id to one that is relevant for that particular page.

index.php

Code: Select all

header('Content-type: application/atom+xml; charset=utf-8');


home.php

Code: Select all

print('<id>tag:twitter.com,2006:/home_timeline/'.$twitter_data[0]['user']['screen_name'].'</id>'. PHP_EOL);
print('<link href="https://twitter.com/"/>'. PHP_EOL);


list.php

Code: Select all

print('<id>tag:twitter.com,2006:/'.$owner.'/'.$list.'</id>'. PHP_EOL);
print('<link href="https://twitter.com/'.$twitter_data[0]['user']['screen_name'].'/'. $list .'"/>'. PHP_EOL);


search.php, just so it's consistent.

Code: Select all

print('<id>tag:twitter.com,2006:/search/'.$q.'</id>'. PHP_EOL);
print('<link href="https://twitter.com/search?q='.$q.'"/>'. PHP_EOL);


user.php

Code: Select all

print('<id>tag:twitter.com,2006:/'.$twitter_data[0]['user']['screen_name'].'</id>'. PHP_EOL);
print('<link href="https://twitter.com/'.$twitter_data[0]['user']['screen_name'].'"/>'. PHP_EOL);


feed.php

Code: Select all

print('      <id>tag:twitter.com,' . date("Y-m-d", strtotime($twitter_data[$i]['created_at'])) . ':/' . $td[$i]['user']['screen_name'] . '/statuses/' . $td[$i]['id_str'] . '</id>'. PHP_EOL);
print('      <link href="https://twitter.com/'.$td[$i]['user']['screen_name'].'/statuses/'. $td[$i]['id_str'] .'"/>'. PHP_EOL);
print('      <title>'.$td[$i]['user']['screen_name'].': '.$td[$i]['text'].'</title>'. PHP_EOL); // fixes issue with html encoded characters not displaying properly

craywolf
Mr. Awesome
Posts: 97
Joined: 19 Mar 2013, 18:07

Re: Twitter RSS Feeds

Postby craywolf » 27 Jun 2013, 18:49

jdelamater99 wrote:here are the changes that I'm proposing, want to get your feedback before I actually commit them. Mostly what you suggested, but changing the id to one that is relevant for that particular page.


Most of those match perfectly with my local version. I've noticed a lot of other things to tweak as well. Right now, I noticed you didn't replace all references of $twitter_data with $td in feeds.php.

Also, your feed <id> and <link> tags in home.php and list.php pull from the first tweet in the results, not from the owner of the home page or list.

I've sent you a pull request with the changes.

EDIT: Your way of putting the header in index.php instead of the individual pages is probably better, one sec while I update the pull request...

EDIT 2: Done

jdelamater99
Bear Rating Master
Bear Rating Master
Posts: 109
Joined: 11 Apr 2013, 17:45

Re: Twitter RSS Feeds

Postby jdelamater99 » 27 Jun 2013, 19:57

craywolf wrote:
jdelamater99 wrote:here are the changes that I'm proposing, want to get your feedback before I actually commit them. Mostly what you suggested, but changing the id to one that is relevant for that particular page.


Most of those match perfectly with my local version. I've noticed a lot of other things to tweak as well. Right now, I noticed you didn't replace all references of $twitter_data with $td in feeds.php.

Also, your feed <id> and <link> tags in home.php and list.php pull from the first tweet in the results, not from the owner of the home page or list.

I've sent you a pull request with the changes.

EDIT: Your way of putting the header in index.php instead of the individual pages is probably better, one sec while I update the pull request...

EDIT 2: Done


Yeah, I noticed the $twitter_data/$td after I posted. I've updated my local copies, and eventually github. I don't have the software on my work computer, and it's a pain to update each of the files on the website, so I'll do that at home tonight.

I'm not worried about home, because home only pulls for the authenticated user, so they are equivalent. However, I updated it so it's a little cleaner.
You are correct though on the list. I had a brain fart in regards to the data that was returned.

index.php, so it doesn't force a download when the test= parameter is set

Code: Select all

if (!isset($_GET['test'])){
   header('Content-type: application/atom+xml; charset=utf-8');
}


home.php

Code: Select all

print('<id>tag:twitter.com,2006:/home_timeline/'.$sn.'</id>'. PHP_EOL);


list.php

Code: Select all

print('<id>tag:twitter.com,2006:/'.$owner.'/'.$list.'</id>'. PHP_EOL);
print('<link href="https://twitter.com/'.$owner.'/'. $list .'"/>'. PHP_EOL);

craywolf
Mr. Awesome
Posts: 97
Joined: 19 Mar 2013, 18:07

Re: Twitter RSS Feeds

Postby craywolf » 27 Jun 2013, 20:21

jdelamater99 wrote:I'm not worried about home, because home only pulls for the authenticated user, so they are equivalent. However, I updated it so it's a little cleaner.
You are correct though on the list. I had a brain fart in regards to the data that was returned.


Hm, I know there were two places I noticed it, but I don't remember which two now. Anyway it's all in the pull request.

index.php, so it doesn't force a download when the test= parameter is set

Code: Select all

if (!isset($_GET['test'])){
   header('Content-type: application/atom+xml; charset=utf-8');
}


Good point. I've added that to the pull request too.

My version of the <id> tag for home is a bit different, neither way is better or worse. My changes to list.php are basically identical.

So if you merge in my pull request, we should both be in the same place more or less.

jdelamater99
Bear Rating Master
Bear Rating Master
Posts: 109
Joined: 11 Apr 2013, 17:45

Re: Twitter RSS Feeds

Postby jdelamater99 » 27 Jun 2013, 20:31

Merged, I think. I'm still really new to Github, so still figuring things out.

craywolf
Mr. Awesome
Posts: 97
Joined: 19 Mar 2013, 18:07

Re: Twitter RSS Feeds

Postby craywolf » 27 Jun 2013, 21:36

jdelamater99 wrote:Merged, I think. I'm still really new to Github, so still figuring things out.


No worries, so am I. And honestly it's only because I've had a slow couple of days at work that I've been able to contribute anything. Anyway, looks like you merged it successfully.

Also, consider adding a .gitignore file with these contents:

Code: Select all

config.php
.ht*


That way you can run the script right out of a cloned repo, and git won't complain that you haven't added or committed added those files.


Return to “Support”

Who is online

Users browsing this forum: No registered users and 6 guests