krypted.com

Tiny Deathstars of Foulness

Safari can subscribe to RSS feeds; so can Mail. Podcast Producer is an RSS or XML feed as are the feeds created by blog and wiki services in Mac OS X Server. And then of course, RSS and ATOM come pre-installed with practically every blogging and wiki tool on the market. Those doing mass deployment and scripting work can make use of automatically connecting users to and caching information found in these RSS feeds. If you have 40,000 students, or even 250 employees, it is easier to send a script to those computers than to open the Mail or Safari client on each and subscribe to an RSS feed. Additionally, pubsub offers what I like to call Yet Another Scripting Interface to RSS (my acronym here is meant to sound a bit like yessir).  Pubsub caches the feeds both within the SQLite database and in the form of XML files. Because pubsub caches data onto the client it can be parsed more quickly than using other tools, allowing a single system to do much more than if a feed were being accessed over the Internet. Using pubsub We’ll start by looking at some simple RSS management from the command line to aid in a quest at better understanding of the underpinnings of Mac OS X’s built-in RSS functionalities. The PubSub framework stores feeds and associated content in a SQLite database. Interacting with the database directly can be a bit burdensome. The easiest way to manage RSS from Mac OS X is using a command called pubsub. First off, let’s take a look at all of the RSS feeds that the current user is subscribed to by opening terminal and simply typing pubsub followed by the list verb: pubsub list You should then see output of the title and url of each RSS feed that mail and safari are subscribed to. You’ll also see how long each article is kept in the expiry option and the interval with which the applications check for further updates in the refresh option. You can also see each application that can be managed with pubsub by running the same command with clients appended to the end of it (clients are how pubsub refers to applications whose subscriptions it can manage): pubsub list clients To then just look at only feeds in Safari: pubsub list client com.apple.safari And Mail: pubsub list client com.apple.mail Each of the above commands will provide a URL for the feed. This url can be used to show each entry, or article in the feed. Extract the URL and then you can use the list verb to see each feed entry, which Apple consistently calls episodes both within PubSub, in databases and on the Podcast Producer server side of things but yet somehow calls an entry here (consistency people). To see a list of entries for a given URL: pubsub list http://googleenterprise.blogspot.com/atom.xml Episodes will be listed in 40 character hex keys, similar to other ID space mechanisms used by Apple. To then see each episode, or entry, use the list verb, followed by entry and then that key: pubsub list entry 5fcef167d77c8c00d7ff041a869d45445cc4ae42 To subscribe to a pubsub, use the –client option to identify which application to subscribe in along with the subscribe verb, followed by the URL of the feed: pubsub --client com.apple.mail subscribe http://krypted.com/LameAssFeed.xml To unsubscribe, simply use pubsub followed by the unsubscribe verb and then the url of the feed: pubsub unsubscribe http://example.com/UninterestingFeed.xml Ofline Databases and Imaging While these can be run against a typical running system, they cannot be run against a sqlite database that is sitting in all of your users home folders nor can they be run against a database in a user template home on a client. Therefore, to facilitate imaging, you can run sqlite3 commands against  database directly. The database, stored in ~/Library/PubSub/Database/Database.sqlite3. To see the clients (the equivalent of `pubsub list clients`): sqlite3 /Volumes/Image/Username/Library/PubSub/Database/Database.sqlite3 'SELECT * FROM clients' To see each feed: sqlite3 /Volumes/Image/Username/Library/PubSub/Database/Database.sqlite3 'SELECT * FROM feeds' To see each entry: sqlite3 /Volumes/Image/Username/Library/PubSub/Database/Database.sqlite3 'SELECT * FROM entries' To see the column headers for each: sqlite3 /Volumes/Image/Username/Library/PubSub/Database/Database.sqlite3 'PRAGMA TABLE_INFO(Clients)'; sqlite3 /Volumes/Image/Username/Library/PubSub/Database/Database.sqlite3 'PRAGMA TABLE_INFO(Feeds)'; sqlite3 /Volumes/Image/Username/Library/PubSub/Database/Database.sqlite3 'PRAGMA TABLE_INFO(Subscriptions)'; sqlite3 /Volumes/Image/Username/Library/PubSub/Database/Database.sqlite3 'PRAGMA TABLE_INFO(Entries)'; sqlite3 /Volumes/Image/Username/Library/PubSub/Database/Database.sqlite3 'PRAGMA TABLE_INFO(Enclosures)'; sqlite3 /Volumes/Image/Username/Library/PubSub/Database/Database.sqlite3 'PRAGMA TABLE_INFO(Authors)'; sqlite3 /Volumes/Image/Username/Library/PubSub/Database/Database.sqlite3 'PRAGMA TABLE_INFO(Contents)';     sqlite3 /Volumes/Image/Username/Library/PubSub/Database/Database.sqlite3 'PRAGMA TABLE_INFO(SyncInfo)'; To narrow an ID down to a specific row within any of these searches add a WHERE followed by the column within the table you’d like to search. For example, if we wanted to only see the article with the identifier of 5b84e609317fb3fb77011c2d26efd26a337d5d7d sqlite3 --line /Volumes/Image/Username/Library/PubSub/Database/Database.sqlite3 'SELECT * FROM entries WHERE identifier="5b84e609317fb3fb77011c2d26efd26a337d5d7d"' Note: Sqlite3 can use the –line option to show each entry in an XML feed per line. Dumping pubsub to be Parsed By Other Tools Pubsub can also be used as a tool to supply feeds and parse them. You can extract conversations only matching specific patterns and text or email yourself that they occurred without a lot of fanfare. You can also dump the entire feed’s cached data by specifying the dump verb without the entry or identifier but instead the URL: pubsub dump http://googleenterprise.blogspot.com/atom.xml Once dumped you can parse the XML into other tools easily. Or to dump specific entries to XML for parsing by another tool using syntax similar to the list entry syntax: pubsub dump entry 5fcef167d77c8c00d7ff041a869d45445cc4ae42 Because these feeds have already been cached on the local client and because some require authentication and other expensive (in terms of script run-time) processes to aggregate or search, looking at the files is an alternative way of doing so. Instant refreshes can also be performed using pubsub’s refresh verb followed by a URL: pubsub refresh Also, feeds are cached to ~/Library/PubSub/Feeds, where they are nested within a folder with the name of the unique ID of the feed (row 2 represents the unique ID whereas row 1 represents the row). Each episode, or post can then be read by entry ID. Yhose entries are basic xml files. You can also still programatically interface with RSS using curl. For example: curl --silent "http://${server}.myschool.org/search/cpg?query=%22random+curse+word%22&catAbbreviation=cpg&addThree=&format=rss" | grep "item rdf:about=" | cut -c 18-100 | sed -e "s/"//g" | sed -e "s/>//g"

June 10th, 2011

Posted In: Mac OS X, Mac OS X Server, Mass Deployment

Tags: , , , , , , , ,

MySpace won’t die. Good. Competition breeds innovation and if social networks are to continue to become more and more useful then a somewhat healthy MySpace is simply going to keep the cog wheels turning in the otherwise spiderweb filled heads of talented engineers. MySpace introduced applications awhile back and there is now a pretty ample number, although nothing close to what Facebook has. I suppose there’s something to be said for being a day late and a dollar short, eh? Most of the applications, as with Facebook, aren’t that useful. Games to help you waste time and quizzes to help you inform your friends and coworkers that you haven’t stepped out in the sunlight in far too long. But there is a specific application that anyone with an RSS feed that wants people to see it might find interesting (after all according to Alexa.com, MySpace is still the 18th ranked website in the world). The RSS Reader application, by Ideamesh, Inc can be used to add a blog or anything else that can be aggregated through a public RSS feed, to your MySpace page. It’s pretty easy to use and provides a quick and easy application box to show the feed. This RSS reader isn’t everything you might want though. As far as I can tell, it only supports a single stream. It has options for reformatting text and links that are displayed, but not a lot. It doesn’t have any features for limiting feeds, although that is something you can usually do with the path to the feed if you want to. But it is free, it is somewhat useful and the application framework that MySpace has introduced is a great step in the right direction to perhaps help MySpace to slow their slide into oblivion. Either way it’s nice to finally have a little more integration across all the sites out there, even if MySpace continues to drop in the rankings…

February 6th, 2010

Posted In: sites, Social Networking

Tags: , , , ,

RSS feeds are pretty darn useful for a lot of things. And WordPress makes them really, really easy. If you want to insert an rss feed somewhere then according to the type of feed you need, you can just use a pretty repeatable pattern to do so. Basically, following the site you would use /wp-rss.php for rss, /wp-rss2.php for rss2 or /wp-atom.php for Atom feeds. For example, to get a feed of this site in rss you could use the following: http://krypted.com/wp-rss.php Or rss2: http://krypted.com/wp-rss2.php Or rdf: http://krypted.com/wp-rdf.php Or Atom: http://krypted.com/wp-atom.php

January 16th, 2010

Posted In: WordPress

Tags: , , , , ,

RSS is an incredibly powerful way to manage content. Using RSS you can provide a feed to users and your website simultaneously.  You can then have items your site, such as WordPress dynamically generate pages for browsers using items published in the feed and have users able to view the feed without seeing the rich media objects that you might also put on the site. A basic RSS feed might include something like the following:
<item> <title>My Article</title> <link>http://www.krypted.com/article1</link> <description>Some article on my site.</description> </item>
Each of the above items is a field that has been defined in the rss file that is used to view your feed, similar to an LDAP schema, headers on an Excel spreadsheet or table headers in a database.  MagpieRSS is a php script that parses these RSS feeds. It is fast, flexible and can be as easy to integrate into your site as:
require(‘rss_fetch.inc’); $rss = fetch_rss($url);
There are 4 scripts in Magpie (so I guess it’s more of a framework really):
  • rss_fetch.inc – gets an object or array of objects from an RSS feed.
  • rss_parse.inc – parses an RSS object, and therefore provides flexibility in dealing with the feed.
  • rss_cache.inc – caches RSS objects.
  • rss_utils.inc – provides other basic functions of dealing with RSS originated objects.
Magpie can then be inserted into a page and display the items in feeds, even multiple feeds if you have content from a few different sites you would like to display.  You can also parse this information, so that you should articles that have a specific pattern in them.  This allows for syndication of content across a number of systems.

August 9th, 2009

Posted In: sites, WordPress

Tags: , , ,

A blog is a stream of conscious.  To some degree, so is an rss feed.  If you are browsing it in rss then your reader will more than likely allow you to limit which articles you wish to see based on their tags or other information gleaned from the rss feed.  For example, you can use feed://krypted.com/feed to access this page without graphics, which looks at the rss feed rather than the richer media content, including the css and other elements that cause graphics and the such to appear on the site. You can then limit your results to any field that is specified in the index.rss file.  In this case, you may choose to say, limit to the articles which I have tagged with Xsan.  To do so, you would append a ? to indicate that you are searching for content, followed by the field that you will be searching which is in this case the word tag.  Then you need to specify what you are searching for; for example, tag=xsan would limit the results to only articles which have been tagged as being about Xsan.  Searches are literal, presenting a good argument why spaces and special characters should not be used when tagging assets. You can then perform compound searches.  To search for content that matches two different tags you would use an ampersand (&) to delimit the searches.  For example, you could do feed://krypted.com/feed?tag=xsan&tag=unix which would search for all articles tagged as Xsan and Unix. A number of solutions are based on the concept of an rss (or at least atom) feed.  For example, Podcast Producer submits data to the weblog services of Mac OS X which can then be viewed as a feed rather than using the nice graphical interface that Apple provides (which is by the way very fast).  You can also use rss with Final Cut Server (with a lot of tinkering) and with WordPress and a number of other popular sites.  While I used the example of tag= above, this is not specified as part of the RFC spec for rss and so might not work across all rss engines, although it does work with weblog and WordPress, the CMS that this site is built on.  Twitter and a number of other sites rely on and foster rss as part of their architecture and APIs. The rss feed option therefore opens up endless possibilities for flexible integration between solutions by parsing feeds and even aggregating multiple feeds.  In companies, this can allow you to have a news feed that is integrated into, for example, a SharePoint server.  This might be useful if you have an announcement that has been tagged for only specific offices.  Each staff member might have a feed in their portal splash page that only displays items tagged for their office, their position title or even their department.  I would assume that a new national medical record solution will be a feed and each doctor that a patient visits will view those records in a feed (it’s lightweight and can have links to actual files such as x-rays, scanned/signed insurance forms, etc).  In educational institutions this opens up the possibility of aggregating feeds from multiple Mac OS X Servers running Podcast Producer or the wiki+blog services into a course management solution, such as Moodle, Blackboard or D2L. Aggregation will more than likely have a resounding impact on how we access and how we ultimately find relevant content.  You can subscribe to a Twitter feed, to YouTube, Digg and limit your results to those that match patterns that are consistent with what you are interested in displaying.  But aggregation of content when it is not appropriate or not acceptable can lead to issues with digital rights.  This is one of the key aspects of what the Associated Press has a problem with right now.  One solution they have come up with is to inject information into feeds which then can help track potential aggregators (which is according to their acceptable use policy, against the law).  Other organizations want you to use their RSS feeds, but to only show the first paragraph or so of content, linking to the original site for the full article.  They are saying yes, please link to our content so that we can have more subscribers, but please do not display our entire articles, or if you do please pay us a royalty for having done so. If you own all of the content though, digital rights of the assets (aka articles) is not an issue.  But whose job is it to make sure that you own all of the assets.  How do you track which of those assets are yours?  How do you limit the content that is appropriate for each user in your organization so as not to incur massive quantities of white noise, but still give the power to inform users and readily access content?  These are all questions that creators of a lot of data will be asking (if they are not already) and there is no magic answer to any of these questions.  The key though, is to think of the questions and how they relate to you.

August 7th, 2009

Posted In: sites, Social Networking, WordPress

Tags: , , , , , , ,

Atom, an alternative to RSS, is used extensively in Final Cut Server.  Atom allows one to, for example show a list of assets and parse the information in a manner that makes sense to a browser to read.  Atom also makes internationalization easier for developers.  For more information on Atom see it’s wikipedia entry at http://en.wikipedia.org/wiki/Atom_(standard) In the px PostgreSQL database for Final Cut Server, you’ll notice the pxatommap, which is used by Final Cut Server to parse the nearly 2,000 fields of data used by Final Cut Server into information easily readable by clients using Atom, similar to how the widget that I recently posted on this site uses RSS to display articles published through the site.

November 17th, 2008

Posted In: Final Cut Server

Tags: , ,