Add automatic cross posting from my Sanity/Gatsby Blog to Dev.to
Knut (@kmelve) made the great suggestion to me recently that setting up cross posting of my blog posts to Dev.to would reach a broader audience. Doing this manually is a bit cumbersome, so I had planned to automate the process, but had not researched the details yet. Knut, was kind enough to include a great tutorial, however, and so I decided there was no time like the present.
Therefore, today, as part of my personal #100DaysOfGatsby journey, I'm going to set up cross-posting of my blog to Dev.to.
Quick history lesson - What is cross-posting?
Cross-posting is a term for posting something in multiple places at the same time (or having automation that re-posts for you to the same effect). The term goes back to days when Usenet was the main internet forum. Usenet is split into a huge amount of areas for specific topics called newsgroups. When you wrote a message for Usenet, it was called a post. You posted your post to the newsgroup that was the most relevant, and if your message had relevance to other newsgroups as well, you could post it to multiple at the same time; this was called cross-posting.
Back to today - how am I going to achieve this with my blog?
The basic idea is that I will add an RSS feed to the blog, and then subscribe to that feed from my Dev.to account. Dev.to will then monitor the RSS feed and create new articles for any new posts it sees on my site. Sounds simple right? Turns out it is a little more complicated when using Sanity due to the Portable Text objects. RSS wants HTML and not Portable Text. The extra work is a small price to pay, however, to get the awesome benefits of Portable Text.
Let's get started!
Step 1 - Add canonical URLs to blog posts
Despite its cryptic name, a canonical URL is simply a way for you to add information to a web page that indicates where the TRUE (or authoritative) source of that page is.
Why is this important? Because it tells search pages where the primary content is found and that any cross-post is a legitimate copy. Setting your site as canonical for your posts means you get credit from Google, drives traffic to your site and builds your personal brand.
Adding this to Gatsby is simple enough using a plugin aptly named gatsby-plugin-react-helmet-canonical-urls (note: I’m using this plugin since I’m also using react-helmet. If you are not using react-helmet, there is a different plugin to use instead - gatsby-plugin-canonical-urls)
I start by changing to the web directory of my mono-repo and then install the plugin.
npm install -save gatsby-plugin-react-helmet-canonical-urlsNow I will add the plugin configuration to web/gatsby.config.
plugins: [
{
resolve: `gatsby-plugin-react-helmet-canonical-urls`,
options: {
// Change `siteUrl` to your domain
siteUrl: `https://christianlobaugh.com`,
// Query string parameters are included by default.
// Set `stripQueryString: true` if you don't want `/blog`
// and `/blog?tag=foobar` to be indexed separately
stripQueryString: true
}
}
]Now I have the canonical url added to the HEAD of all of my pages.
Step 2 - Add an RSS Feed to the site.
As you would guess, Gatsby has a plugin for this as well - gatsby-plugin-feed
Install:
npm install -save gatsby-plugin-feedThen I need to add the plugin configutation to web/gatsby-config.js.
This is the most complex piece of the puzzle and requires the most heavy lifting. The basic steps are to write a query to get the data the RSS plugin needs, and then to serialize that data. Since I am using data from Sanity that is Portable Text, however, I will need to translate that portable text to HTML, which is an extra step. Fortunately Sanity provides a nice library that can do exactly that.
I will start by installing it to my web directory.
npm install -save @sanity/block-content-to-htmlNext, I identify the data I need for Dev.to and then write my query to get the data from Sanity.
It's important to note that gatsby-plugin-feed has a defined set of values you can use. If you want to use anything different, you will need to use the "custom_elements" field.
Here's a breakdown of the data I think I need with the plugin variable names assigned:
// Site level data:
title [siteMetadata.title]
description [ siteMetadata.description ]
siteUrl [ siteMetadata.siteUrl ]
site_url: siteUrl [ siteMetadata.siteUrl ]
//Post level data:
title [ allSanityPost.title ]
url [ allSanityPost.title.url ]
guid [ allSanityPost.title.url ]
date [ allSanityPost.publishedAt ]
// The following fields are Portable text
// and will need to be converted to HTML
description [ allSanityPost._rawExcerpt ]
custom_elements{content} [ allSanityPost._rawBody ]
Now that I know what data I need, I will build my graphql query. A couple of notes:
- I'm pulling the site data from the siteMetadata block in the gatsby-config, not from Sanity. You can create these fields and pull them from Sanity if you prefer.
- Use your dev GraphiQL editor found at http://localhost:8000/___graphql when you are running Gatsby in dev mode. This is invaluable for getting your queries correct before putting them in your code.
- Any time you are dealing with a portable text field, you should use the _raw version of that field that Sanity provides. If there are references in your portable text you will need to add the resolveReferences prop. (see my example below for usage)
//Site level query:
query: ` {
site {
siteMetadata {
title
description
siteUrl
site_url: siteUrl
}
}`
//Post level query
query: `{
allSanityPost(sort: {fields: publishedAt, order: DESC}) {
edges {
node {
_rawExcerpt
_rawBody(resolveReferences: {maxDepth: 10})
title
publishedAt
slug {
current
}
}
}
}
}
`,
The last, and most complicated step, is to serialize this data properly for the RSS feed. Since a few of my fields are Portable Text objects and not strings this will require a bit of extra code to be added to my gatsby-config.js to convert these to HTML.
I start by adding some needed libraries and functions to the top of the gatsby-config.js file. I am mostly needing PortableText function from the @sanity/block-content-to-html library and the imageUrlBuilder function from @sanity/image-url.
Now I can write the config for the gatsby-plugin-feed. The code is fairly straight forward:
Run the queries defined > filter the posts > iterate through the posts > pull out needed fields and assign to appropriate feed variables.
The tricky part is that I need to convert the Portable Text fields to HTML. The block-content-to-html library's PortableText funtion comes to the rescue. It knows how to handle the basic html styles, but needs some help with images, and with custom objects that I've added. I do this by adding serialization instructions to the Portable Text call.
Note: For my youtube-embed custom object I just add a link to the embedded video with a description. There may be a way to serialize this to a proper video container that will work with the RSS feed and will be read appropriately by Dev.to. It's simple enough to edit in the draft on DEV.to for now so I'm handing that code exercise off to Future Me.
Here is my entire gatsby-config.js. See the feeds section of the gatsby-plugin-feed config section to see the main logic processing and the Portable Text serialization.
// Load variables from `.env` as soon as possible
require("dotenv").config({
path: `.env.${process.env.NODE_ENV || "development"}`
});
const clientConfig = require("./client-config");
const isProd = process.env.NODE_ENV === "production";
// for Portable Text Serialization
const PortableText = require("@sanity/block-content-to-html");
const imageUrlBuilder = require("@sanity/image-url");
const h = PortableText.h;
const imageUrlFor = source => imageUrlBuilder(clientConfig.sanity).image(source);
// Helper functions for Portable Text
const { format, isFuture } = require("date-fns");
function filterOutDocsPublishedInTheFuture({ publishedAt }) {
return !isFuture(publishedAt);
}
function getBlogUrl(publishedAt, slug) {
return `/blog/${format(publishedAt, "YYYY/MM")}/${slug.current || slug}/`;
}
module.exports = {
siteMetadata: {
title: "Christian Lobaugh",
siteUrl: "https://www.christianlobaugh.com",
description: "The blog and website of Christian Lobaugh"
},
plugins: [
"gatsby-plugin-postcss",
"gatsby-plugin-netlify",
"gatsby-plugin-react-helmet",
{
resolve: "gatsby-source-sanity",
options: {
...clientConfig.sanity,
token: process.env.SANITY_READ_TOKEN,
watchMode: !isProd,
overlayDrafts: !isProd
}
},
{
resolve: `gatsby-plugin-react-helmet-canonical-urls`,
options: {
// Change `siteUrl` to your domain
siteUrl: `https://christianlobaugh.com`,
// Query string parameters are inclued by default.
// Set `stripQueryString: true` if you don't want `/blog`
// and `/blog?tag=foobar` to be indexed separately
stripQueryString: true
}
},
{
resolve: `gatsby-plugin-feed`,
options: {
query: `
{
site {
siteMetadata {
title
description
siteUrl
site_url: siteUrl
}
}
}
`,
feeds: [
{
serialize: ({ query: { site, allSanityPost = [] } }) => {
return allSanityPost.edges
.filter(({ node }) => filterOutDocsPublishedInTheFuture(node))
.filter(({ node }) => node.slug)
.map(({ node }) => {
const { title, publishedAt, slug, _rawBody, _rawExcerpt } = node;
const url = site.siteMetadata.siteUrl + getBlogUrl(publishedAt, slug.current);
return {
title: title,
date: publishedAt,
url,
guid: url,
description: PortableText({
blocks: _rawExcerpt,
serializers: {
types: {
code: ({ node }) =>
h("pre", h("code", { lang: node.language }, node.code))
}
}
}),
custom_elements: [
{
"content:encoded": PortableText({
blocks: _rawBody,
serializers: {
types: {
code: ({ node }) =>
h("pre", h("code", { lang: node.language }, node.code)),
mainImage: ({ node }) =>
h("img", {
src: imageUrlFor(node.asset).url()
}),
authorReference: ({ node }) => h("p", "Author: " + node.author.name),
youtube: ({ node }) =>
h(
"a",
{
href: node.url
},
'Watch this video on YouTube'
)
}
}
})
}
]
};
});
},
query: `{
allSanityPost(sort: {fields: publishedAt, order: DESC}) {
edges {
node {
_rawExcerpt
_rawBody(resolveReferences: {maxDepth: 10})
title
publishedAt
slug {
current
}
}
}
}
}
`,
output: "/rss.xml",
title: "Christian Lobaugh - RSS Feed",
// optional configuration to insert feed reference in pages:
// if `string` is used, it will be used to create RegExp and then test if pathname
// of current page satisfied this regular expression;
// if not provided or `undefined`, all pages will have feed reference inserted
match: "^/blog/"
}
]
}
}
]
};
Important Note: The RSS feed only works when your site is in production mode. You will not see it when you do a gatsby dev, or npm run dev, but only when you’ve done a gatsby build. To see the RSS feed in your local dev environment, run:
gatsby build && gatsby serveNow you can see your rss.xml at localhost:9000/rss.xml.
Step 3 - Setup Dev.to to consume RSS feed
- Log in to Dev.to account
- Go to: Settings > Publishing from RSS or https://dev.to/settings/publishing-from-rss
- Add “RSS Feed URL” (for me that's. https://christianlobaugh.com/rss.xml)
- Check the box "Mark the RSS source as canonical URL by default"
- Uncheck the box "Replace self-referential links with DEV-specific links"
- Click “Submit” or "Update" (whichever you are offered)
Now all of my blog posts are immediately posted to my DEV.to drafts folder for me to review before posting them live. I can give Dev.to admins permission to do this, but I'm going to hold off on that for now. I want to see how much editing that my drafts will need. Once I've confirmed that my RSS feed is creating posts where the edits are zero or minimal, then I will look at skipping the draft step.
Step 4 - Review your blog posts on DEV.to
Go to your Dev.to dashboard and you will see a list of posts. For the ones in draft mode, pull them up and review. If you are happy, edit the post and change the published: field in the front-matter from false to true and then click save.
Step 5 - Bask in the glow of a Dev.to article that was (almost completely) auto generated from your blog
References that helped me create this:
https://tueri.io/blog/2019-06-06-how-to-automatically-cross-post-from-your-gatsbyjs-blog-with-rss/
https://www.gatsbyjs.org/packages/gatsby-plugin-feed/
https://www.gatsbyjs.org/packages/gatsby-plugin-react-helmet-canonical-urls/