Skip to navigation Skip to main content
Eleventy
Eleventy Documentation
Blog
Help Eleventy: Fill out Survey Awesome!
Versions
Stable
3.0.0
Toggle Menu
Compare npm install
Eleventy 5.81s
Remix 40.14s

Fetch

Contents

This documentation is written for Eleventy Fetch v5.0+, requiring Node 18 or newer.

Fetch network resources and cache them locally to avoid bombarding your API (or other remote or rate-limited resources). Do this at configurable intervals (not with every build): once per minute, once per hour, once per day, or however often you like!

Successful API requests allow working offline too!

This plugin can save any kind of asset—JSON, HTML, images, videos, etc.

  • Fetch a remote URL and saves it to a local cache.
  • If the remote server goes down or linkrots away—we keep and continue to use the local asset. Save remote images!
  • If a network connection fails (or if you’re offline), Fetch will continue to use the cached request (even if the cache is expired) and make a new request when the network connectivity is restored.
  • Concurrency limits to avoid making too many network requests simultaneously.
  • Uses built-in fetch Added in Fetch 5.0 (previously node-fetch)
  • Supports and caches HTTP verbs separately (think POST versus GET) Added in Fetch 5.0

Installation

npm install @11ty/eleventy-fetch

This plugin was formerly known as @11ty/eleventy-cache-assets and was renamed to @11ty/eleventy-fetch with v3: Video Changelog.

WARNING:
Important Security and Privacy Notice

This plugin caches complete network responses. Unless you’re willing to perform a full review of everything this plugin caches to disk for privacy and security exposure, it is strongly recommended that you add the .cache folder to your .gitignore file so that network responses aren’t checked in to your git repository.

Are you 100% sure that private e-mail addresses aren’t being returned from a cached API? I’m guessing no—add .cache to your .gitignore file. Right now. Do it.

Usage

Cache a JSON file from an API

Consider the following example, perhaps in an Eleventy Global Data File like _data/githubRepos.js.

import Fetch from "@11ty/eleventy-fetch";

export default async function () {
	let url = "https://api.github.com/repos/11ty/eleventy";

	let json = await Fetch(url, {
		duration: "1d", // save for 1 day
		type: "json", // we’ll parse JSON for you
	});

	return json;
};
const Fetch = require("@11ty/eleventy-fetch");

module.exports = async function () {
	let url = "https://api.github.com/repos/11ty/eleventy";

	let json = await Fetch(url, {
		duration: "1d", // save for 1 day
		type: "json", // we’ll parse JSON for you
	});

	return json;
};

The first argument to Eleventy Fetch can be:

  • a string (pointing to a URL)
  • a URL instance Added in Fetch 5.0
  • a custom function (async-friendly) Added in Fetch 5.0

Options

Verbose Output

Option to log requested remote URLs to the console.

  • verbose: true (default: false) Added in Fetch 3.0

Change the Cache Duration

After this amount of time has passed, we’ll make a new network request to the URL to fetch fresh data.

The duration option supports the following shorthand values:

  • s is seconds (e.g. duration: "43s")
  • m is minutes (e.g. duration: "2m")
  • h is hours (e.g. duration: "99h")
  • d is days (The default is duration: "1d")
  • w is weeks, or shorthand for 7 days (e.g. duration: 2w is 14 days)
  • y is years, or shorthand for 365 days (not exactly one year) (e.g. duration: 2y is 730 days)

Special values:

  • duration: "*" will never fetch new data (after the first success).
  • duration: "0s" will always fetch new data (works with any unit, e.g. "0m", "0h").

Type

  • type: "buffer" (default)
  • type: "json"
  • type: "text"
  • type: "xml" (alias for text) Added in Fetch 5.0
  • type: "parsed-xml" (uses parse-xml to return a JavaScript representation of the XML) Added in Fetch 5.0

Return Type Added in Fetch 5.0

  • returnType: undefined (default) returns the processed body of the request specific to the type
  • returnType: "response" returns a cached object with url, status, headers, and body properties.

What happens when a request fails?

  1. If this is the first ever request to this URL (no entry exists in your cache folder), it will fail. Use a try/catch if you’d like to handle this gracefully.
  2. If a failure happens and a cache entry already exists (even if it’s expired), it will use the cached entry.
  3. If you prefer the build to fail when your API requests fail, leave out the try catch and let the error throw without handling it!

Consider the following global data file in Eleventy (e.g. _data/github.js):

import Fetch from "@11ty/eleventy-fetch";

export default async function () {
	try {
		let url = "https://api.github.com/repos/11ty/eleventy";

		/* This returns a promise */
		return Fetch(url, {
			duration: "1d",
			type: "json",
		});
	} catch (e) {
		return {
			// my failure fallback data
		};
	}
};
const Fetch = require("@11ty/eleventy-fetch");

module.exports = async function () {
	try {
		let url = "https://api.github.com/repos/11ty/eleventy";

		/* This returns a promise */
		return Fetch(url, {
			duration: "1d",
			type: "json",
		});
	} catch (e) {
		return {
			// my failure fallback data
		};
	}
};

Running this on your Build Server

This documentation has moved to the Deployment page.

More Examples

Cache a Remote Image

This is what eleventy-img uses internally.

import Fetch from "@11ty/eleventy-fetch";

let url = "https://www.zachleat.com/img/avatar-2017-big.png";
let imageBuffer = await Fetch(url, {
	duration: "1d",
	type: "buffer",
});
// Use imageBuffer as an input to the `sharp` plugin, for example

// (Example truncated)
const Fetch = require("@11ty/eleventy-fetch");

let url = "https://www.zachleat.com/img/avatar-2017-big.png";
let imageBuffer = await Fetch(url, {
	duration: "1d",
	type: "buffer",
});
// Use imageBuffer as an input to the `sharp` plugin, for example

// (Example truncated)

Fetch Google Fonts CSS

Also a good example of using fetchOptions to pass in a custom user agent. Full option list is available on the fetch documentation on MDN.

import Fetch from "@11ty/eleventy-fetch";

let url = "https://fonts.googleapis.com/css?family=Roboto+Mono:400&display=swap";
let fontCss = await Fetch(url, {
	duration: "1d",
	type: "text",
	fetchOptions: {
		headers: {
			// lol
			"user-agent":
				"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36",
		},
	},
});
const Fetch = require("@11ty/eleventy-fetch");

let url = "https://fonts.googleapis.com/css?family=Roboto+Mono:400&display=swap";
let fontCss = await Fetch(url, {
	duration: "1d",
	type: "text",
	fetchOptions: {
		headers: {
			// lol
			"user-agent":
				"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36",
		},
	},
});

Fetching GitHub Stars for a repo

Advanced Usage

Add a custom timeout

Use AbortSignal to supply a timeout for the request. Read more about AbortSignal on MDN.

import Fetch from "@11ty/eleventy-fetch";

await Fetch("https://…", {
	fetchOptions: {
		signal: AbortSignal.timeout(5000)
	},
});
const Fetch = require("@11ty/eleventy-fetch");

await Fetch("https://…", {
	fetchOptions: {
		signal: AbortSignal.timeout(5000)
	},
});

Cache Directory

The directory option let’s you change where the cache is stored. It is strongly recommended that you add this folder to your .gitignore file.

WARNING:
Read the Important Security and Privacy Notice.
import Fetch from "@11ty/eleventy-fetch";

await Fetch("https://…", {
	directory: ".cache",
});
const Fetch = require("@11ty/eleventy-fetch");

await Fetch("https://…", {
	directory: ".cache",
});

If you want to use this utility inside of a Netlify Function (or AWS Lambda), use a writeable location (/tmp/) like directory: "/tmp/.cache/". You can also use dryRun: true to skip writing to the file system.

Remove URL query params from Cache Identifier

Added in Fetch 2.0.3 If your fetched URL contains some query parameters that aren’t relevant to the identifier used in the cache, remove them using the removeUrlQueryParams option. This is useful if an API adds extra junk to your request URLs.

  • removeUrlQueryParams: true (false is default)
import Fetch from "@11ty/eleventy-fetch";

await Fetch(
	"https://www.zachleat.com/img/avatar-2017-big.png?Get=rid&of=these",
	{
		removeUrlQueryParams: true,
	}
);
const Fetch = require("@11ty/eleventy-fetch");

await Fetch(
	"https://www.zachleat.com/img/avatar-2017-big.png?Get=rid&of=these",
	{
		removeUrlQueryParams: true,
	}
);

Note that query params are removed before—and are relevant to how—the cache key is calculated.

Change the cache filename

This controls the name of the files inside your cache directory.

import Fetch from "@11ty/eleventy-fetch";

await Fetch("https://…", {
	filenameFormat: function(cacheKey, hash) {
		// do not include the file extension
		return `custom-name-${key}-${hash}`
	}
});
const Fetch = require("@11ty/eleventy-fetch");

await Fetch("https://…", {
	filenameFormat: function(cacheKey, hash) {
		// do not include the file extension
		return `custom-name-${key}-${hash}`
	}
});

Manually store your own data in the cache

Added in Fetch 5.0 You can pass a function (async-friendly) in as your source to run your own logic and return any arbitrary data. You must supply a unique key for the request in the requestId property. Consider the following Global Data File:

import Fetch from "@11ty/eleventy-fetch";

export default function() {
	return Fetch(async function() {
		// do some expensive operation here, this is simplified for brevity
		let fakeTwitterApiContents = { followerCount: 1000 };

		return fakeTwitterApiContents;
	}, {
		// must supply a unique id for the callback
		requestId: "zachleat_twitter_followers"
	});
}
const Fetch = require("@11ty/eleventy-fetch");

module.exports = function() {
	return Fetch(async function() {
		// do some expensive operation here, this is simplified for brevity
		let fakeTwitterApiContents = { followerCount: 1000 };

		return fakeTwitterApiContents;
	}, {
		// must supply a unique id for the callback
		requestId: "zachleat_twitter_followers"
	});
}

Even lower-level access to the cache

You probably won’t need to do this. It’s more straightforward to pass in a function source. Consider the following Global Data File:

import { AssetCache } from "@11ty/eleventy-fetch";

export default async function () {
	// Pass in your unique custom cache key
	// (normally this would be tied to your API URL)
	let asset = new AssetCache("zachleat_twitter_followers");

	// check if the cache is fresh within the last day
	if (asset.isCacheValid("1d")) {
		// return cached data.
		return asset.getCachedValue(); // a promise
	}

	// do some expensive operation here, this is simplified for brevity
	let fakeTwitterApiContents = { followerCount: 1000 };

	await asset.save(fakeTwitterApiContents, "json");

	return fakeTwitterApiContents;
};
const { AssetCache } = require("@11ty/eleventy-fetch");

module.exports = async function () {
	// Pass in your unique custom cache key
	// (normally this would be tied to your API URL)
	let asset = new AssetCache("zachleat_twitter_followers");

	// check if the cache is fresh within the last day
	if (asset.isCacheValid("1d")) {
		// return cached data.
		return asset.getCachedValue(); // a promise
	}

	// do some expensive operation here, this is simplified for brevity
	let fakeTwitterApiContents = { followerCount: 1000 };

	await asset.save(fakeTwitterApiContents, "json");

	return fakeTwitterApiContents;
};

Change Global Concurrency

import Fetch from "@11ty/eleventy-fetch";
Fetch.concurrency = 4; // default is 10

DEBUG mode

DEBUG=Eleventy:Fetch npx @11ty/eleventy

Other pages in Official Plugins: