Week 10 - Fetch, HTTP, & REST

This week we will start to write and run code that reaches beyond the browser. We will introduce a long list of acronyms and give you lots of topics to discuss with your family and friends that will leave them with quizzical expressions on their faces.

If you haven’t felt like a real developer yet, then this week will change that.

Client and Server

The terms client and server will be something that we use frequently from this point onwards.

The client is the browser. It sends requests for resources to web servers.

The server is the computer that holds web pages and resources for your web apps.

The web server is the program listening for requests that come from clients.

AJAX

AJAX stands for Asynchronous Javascript And XML.

In the late nineties, it was not possible to download new data and apply that to the current webpage. If you wanted updated results you had to refresh the whole page.

What is AJAX

What we will be discussing over the new few weeks is the process of how to make requests from webpages for new data or new files from a remote webserver.

It is with JavaScript that we will be requesting the new content and then injecting it into your pages.

XMLHttpRequest

The original way to make requests to a remote server to have files sent was an object called XMLHttpRequest.

This is NOT how we will be talking to remote servers.

let xhr = new XMLHttpRequest();
xhr.open('POST', url);
xhr.addEventListener('readystatechange', function (ev) {
  //this function runs each time readystate changes
});
xhr.addEventListener('error', function (ev) {
  //this function runs if an error happens when called xhr.send()
  //xhr.readyState - 2: sent, 3: partial response, or 4: complete
  //xhr.status - the http status code like 200, 404, 401, 500, etc
});
xhr.send(data);
//data is the data we are sending to the server - FormData or JSON string, etc

The limitations of the XMLHttpRequest object were mostly to do with security.

Along with ES6, and the addition of Promises allowed for a reworking of the whole process with a new method called fetch.

xmlhttprequest

Fetch

The NEW method that replaced XMLHttpRequest. The good news here is that if you are already familiar with the syntax for Promises then you already know most of the syntax for fetch.

let url = 'https://www.example.com/api/';
fetch(url)
  .then((response) => {
    //we have a response from the server
    if (!response.ok) throw new Error(response.statusText);
    //if we got a bad response then we can trigger the catch
    //if the response was a JSON file then we can extract the contents with:
    return response.json();
    //extract the JSON String from the body of the response
    //convert the JSON String into an object
  })
  .then((data) => {
    //data will be the Object that was extracted from the JSON String
  })
  .catch((err) => {
    //this code runs if there was a network error preventing sending the request
    //OR if the response gave a bad status code
  });

When we send a request to a webserver it can be for ANY type of file - html, xml, json, css, js, png, jpg, avi, mpg, mp3, or anything else. When you make the request you should be aware of what kind of file will be send back.

In the example above we used request.json() to extract the JSON data from the response. If we were getting some other kind of text file, like CSS, HTML, or XML then we should use response.text(). If we were getting a binary file like an image or audio file then we would use response.blob() to get the binary data. Blob stands for Binary Large OBject.

Regardless of whether you are using fetch or XMLHttpRequest to get the file from the server we are working with the same technology. We are making an HTTP Request, which contains headers and, potentially, a body. The request is sent to a web server over the internet and then we get back an HTTP Response, which contains headers and has a body.

Think of making Request like mailing a letter to someone.

• The headers are the things that get written on the outside of the envelope.
• The body is the contents inside the envelope.
• Once the letter is sealed and addressed you stick it in a mail box. The postal service will take your letter, interpret what you wrote on the envelope and figure out how to get your letter to the correct address.
• The street address and city and province help you understand where you are sending your letter.
• The postal code is what the postal service actually uses to figure our how to route your letter.
• You don’t need to understand how the letter gets to the other address.
• The envelope should have a return address on the outside of the envelope so that a response can be sent back to you.

Fetch in 5 under minutes

JSON

JSON - JavaScript Object Notation is the most popular format for sending data between clients and servers. It is called JSON because it uses a JavaScript-compatible syntax for encoding the information inside the file.

Official JSON website

However, it is NOT JavaScript. It is just a text file with a single long string. For this exact reason, we cannot save things like functions or DOM elements inside of JSON files. We can only save String, Number, Boolean, and null (Primitive values) plus Array literals and Object literals.

The JSON file format is used by localStorage and sessionStorage to hold data in the browser. More on this next week.

The primary differences between a JS object and a JSON object are:

1. All object keys must be wrapped in double quotes.
2. All string values must be wrapped in double quotes.
3. No trailing commas are allowed after array or object values.
4. No comments are allowed in the file.
5. No variable declarations.

Here is a JavaScript Object:

let obj = {
  name: 'Joanne',
  id: 123,
  active: true,
  courses: ['HRT100', 'HRT200', 'HRT300'],
};

and here is the same information as JSON:

{
  "name": "Joanne",
  "id": 123,
  "active": true,
  "courses": ["HRT100", "HRT200", "HRT300"]
}

Notice all the double quotes around all the string values. No quotes around the number or boolean values.

JSON vs JavaScript Object Literals

XML

XML - eXtensible Markup Language, created in 1998, was the first file format that was used for client-side web development for the transfer of data between clients and servers. As the name suggests, it is a MarkUp language. Angle brackets < > are used to wrap the tag names which are used to label and describe the information in the file.

The most important rule for writing XML files is Human Readable.

This one rule meant that XML rapidly became a very popular format with the thousands of new developers who started working in web development in the late 90s and early 2000s. The format was adopted by nearly all major software providers and is still widely used today.

An example of the widespread support for XML was the Microsoft adoption of it as a wrapper for all their MS Office files in Office 2007. With this release file formats changed from .doc to .docx and .xls to .xslx and so on. The name change reflected that XML had become a core part of the file format. A .docx file is really just a .doc file, wrapped inside of an XML file and then zipped. All the new features for MS Word have been added via the XML portion of the file.

The basic rules for writing well-formed (valid) XML are:

1. Only one root element. Think <html> tag.
2. Tags must be properly nested.
3. All tags must be self-closing or have a closing tag.
4. All tags are case-sensitive.
5. Attribute values must be wrapped in double-quotes.
6. No overlapping tags. If tag <b> opens inside of tag <a>, then </b> must close before </a>.
7. Special characters must be written with character entities like &.
8. XML preserves whitespace.
9. Attribute names must all be unique within a single element.
10. If you have a lot of special characters, like code sample, then you should wrap the special text in a CDATA block. Here is an example of a CDATA block.

<script><![CDATA[
  if (x < 10) {
    alert('Hello');
  }
]]></script>

JSON overtook XML as the most popular web development format during the last decade because it was Developer Readable and because the file size was noticeably smaller than XML.

Here is the same data as above, as an XML file.

<?xml version="1.0" encoding="utf-8" xmlns="https://com.algonquincollege/student">
<student>
  <name>Joanne</name>
  <id>123</id>
  <active>true</active>
  <courses>
    <course>HRT100</course>
    <course>HRT200</course>
    <course>HRT300</course>
  </courses>
</student>

You can see how much more typing is required to output that small amount of information.

Fetching and Reading XML

Request Objects

When you make a fetch call, very often you are only providing a URL to the method. However, the fetch method will actually create a new Request() object on your behalf, using all the default values plus your URL.

If you need to you can create your own Request object.

let request = new Request();
//fetch also accepts a Request object instead of a URL object or URL string.
fetch(request)
  .then((response) => {})
  .then((body) => {})
  .catch(console.warn);

Inside the Request object you can define what the headers are and what the body is.

MDN reference for Request Object. You can use this reference to find all the properties and methods of a Request object.

Fetch with Request and Header Objects

Response Objects

Typically you will be working with a response object that gets returned to the first then method after a fetch.

If you need to, like in a Service Worker when you want to send something to the browser that you are creating in response to a request, you can create your own Response Object.

let response = new Response();

MDN reference for a Response Object. You can use this reference page to find all the properties and methods for a Response Object.

Body Objects

The Body object is the container for any file or large block of data being transferred between a client and a server. Both the Response and the Request objects have a body property that is used to access the Body object.

MDN reference for the body property of the Response object

The body property can contain one of the following data types:

• Blob - for binary files like images
• BufferSource - like an array of data but binary
• FormData - generally used for the information from a form
• ReadableStream - large chunk of text or numerical data
• URLSearchParams - Query String formatted string
• USVString - an encoded string

json(), text(), and blob() Methods

When a Response comes back to the browser, to either your main script or a service worker, the most common datatypes that we receive are:

• a json file
• a text file (CSS, XML, or HTML)
• an image (Blob)

Because of that, there are three specific methods that we can use to extract the contents of those files, from the body of the Response. We use the Response.json() method to convert the contents of the JSON file into a JavaScript Object. We use the Response.text() method to read the contents of the CSS, XML, or HTML file into a string. We use the Response.blob() method to extract the binary data from a binary file (like an image) into a Blob object.

All three of the methods are asynchronous and return a Promise. So, we use them inside a then() method and return the Promise that they create. That way it gets passed to the next then() in the chain. The next then() will receive the Object, String, or Binary content from the method.

fetch(request)
  .then((response) => {
    //only the first return actually runs
    return response.text();
    return response.blob();
    return response.json();
  })
  .then((body) => {
    //body is the formatted contents returned from one of those methods
  });

If we want to use the Blob as the source for an image element on our page then we need to use the URL.createObjectURL() method.

document.getElementById('dynamicImage').src = URL.createObjectURL(blob);

It is worth noting that there is also a formData() method that will extract the text from the body as if it were a FormData object. Also, you have an arrayBuffer() method available to use if you want the file contents as an ArrayBuffer instead of a Blob.

A Response object can only be used for one purpose - providing content to the webpage or saving it to the Cache API. If you need multiple copies of a response object you can use the clone() method to create that copy.

Header Objects

Inside your HTTP Request and HTTP Response, the Head holds the meta information about the request or response. What address is it being sent from, what address it is being sent to, whether it is encrypted, what type of file is contained in the body, the file size of the body, the encoding type, if it is compressed, cookies, what is the expiry date of the response, and much more. All the values in the Head are called Headers.

Parts of the URL

The QueryString is one of the Headers. It is a string that contains a series of name value pairs. There is an = sign between each name and value. And there is an & ampersand between each of the pairs. The values in the QueryString need to be URL encoded to avoid causing issues with other programs, through special characters, who might read the string.

The official object for a QueryString is a URLSearchParams object.

encoding and decoding URIs

The Method verb (GET, POST, PUT, PATCH, DELETE, HEAD, OPTIONS) is one of the Headers. The Method is important on the Server because it gives the server an idea of how to handle the request. You will learn more about this in MAD9124.

The Body is the container for the file that is being uploaded or downloaded. The Body can be empty if there is no file or data being uploaded or downloaded. With a GET Request the Body is always empty.

The two places that you can put information to send with a Request or Response are in the QueryString or in the Body. The QueryString is limited to about 2000 characters. The Body is limited in total size on most servers to files that are roughly 20MB. This limit can be changed through server-side settings.

The Header object is like an array of name value pairs. Many of the headers will be created by the browser and sent automatically. These are values that you are not allowed to edit. Things like the IP address cannot be altered in your script.

List of restricted header names

In addition, you are not allowed to programmatically edit the set-cookie or set-cookie2 headers in a Response object.

The Header object has an append method that we can use to add or edit header values.

let h = new Headers();
h.append('content-type', 'application/json');
h.append('content-length', 13234); //file size in bytes

When we are uploading a file, it means that we are adding a file to the Body and we should include headers that indicate the type and size of the attached file.

Here is a list of possible headers

Many of the existing headers can also be accessed through the .headers property on the Request and Response objects and its .get() method.

fetch(url).then((response) => {
  response.headers.get('Content-Type');
  response.headers.get('Content-Encoding');
  response.headers.get('Age');
  response.headers.get('Date');
  response.headers.get('X-my-custom-header');
});

The other methods on the Header object include has(), entries(), and delete(). Here is the link to the get method.

It is worth noting that not all headers are accessible through JavaScript. This is for security reasons. Some are not able to be changed or deleted through script and some are not allow to be read.

Headers and Why They Matter

URL and Location Objects

URLs can be thought of as just strings but they do have clearly defined parts: the domain, the path, the hash, the querystring(search), the protocol, and the port.

JavaScript has a top level object window.location that contains all these parts. If you console.log the location object you will see all the different components that make it up. One way that you could built a url is to take or create a Location object and set the values for its different components and then use the full value of the href property string.

MDN reference for Location Object

Another approach is to use the URL object to create one. It takes two parameters that make it work a lot more like a typical development situation, where you have a base url and then an endpoint with or without a QueryString.

const BASEURL = 'http://somedomain.com';
//a global variable that holds the base url for your API

let endpoint = '/api/people/23?api-key=8768374823';
let url = new URL(endpoint, BASEURL);
//first we have a url object to pass to a Request or fetch call.

let str = url.toString();
//here we have a String built from the url object

let req = new Request(url);
// OR
req = new Request(str);
//here we have a Request object with the url inside it

fetch(url).then((resp) => {
  //we can pass a url directly to fetch
  console.log('fetch with url', resp.status);
});

fetch(req).then((resp) => {
  //we can pass the request object that was given a url or string
  console.log('fetch with url in request', resp.status);
});

fetch(str).then((resp) => {
  //we can pass the string to the fetch
  console.log('fetch with url as string', resp.status);
});

The URL property of a Request or Location object, just like the URL object itself has a bunch of properties that you can access to directly read the different parts of a URL. (Otherwise you would have to parse the string yourself.)

• hash is the part that begins with #.
• host is a hostname with a port (if specified).
• hostname is the host without the port.
• href is the entire url string with all the parts.
• origin includes the scheme, domain, and port.
• pathname is the path and filename starting with /.
• port is the port number part of the url. Eg: 80 or 5500.
• protocol is the scheme, like http:, ftp:, https:, blob:, or file:.
• search is the querystring for the url and is either an empty string or includes the ? at the start.
• searchParams is a read-only SearchParams object with the elements in the querystring.

MDN reference for URL

FormData Objects

The FormData object is a great way to bundle an entire HTML form, including any hidden input elements, into a format that can be set as the value for a Request body.

let myform = document.getElementById('myform'); //a form element on the page
let fd = new FormData(myform);
//this will append all the name value pairs for the elements in the form.

2 lines of code and your entire form is bundled and ready to be uploaded.

Important

You must give each of your form input, select, and textarea elements a name attribute. Without the name attribute it will not get added to the FormData object.

You should always do validation on the data and not blindly upload it.

Now, you can also use the FormData object to bundle data that is not part of a form, or a mix of form data and other variables.

let fd = new FormData();
fd.append('name', 'value');
fd.append('email', document.getElementById('email').value);

This FormData object with the two name-value pairs can now be passed to the body property of a Request or options object for a fetch call.

What is a FormData Object

Effective use of FormData

Uploading data with FormData

URLSearchParams

The URLSearchParams object is the object that can be used to hold or build a QueryString. It can also be used as an alternative to FormData when bundling data to upload to the server. It can be used as part of the URL, in the headers, or in the body.

MDN reference for URLSearchParams

It is named as URLSearchParams because it represents the value held in the search property of the Location object, which is the top-level object that holds all the information about the webpage’s url, hash, protocol, port, domain, path, etc.

MDN Location reference

It works much the same way as a Headers or FormData object do.

let search = new URLSearchParams();
search.set('key', 'value');
search.set('name', 'Karim');
search.has('key'); //true
search.get('name'); //Karim
search.sort(); //sorts the existing values by their key
search.forEach((val, key) => console.log(key, val));
search.delete('key');
search.toString(); //gives you the full string (without a `?`)

There is also an append method that works like set but will allow for duplicate entries with the same key.

It is an iterable object, which means it can be used in a for...of loop.

When you want to add a URLSearchParams string to the URL or to the body, use the toString method. Remember to add the ? in front of it if you are using it as part of the URL.

URL and URLSearchParams

USVString

A USVString is a Unicode Scalar Value String. Basically it is a string designed for efficient text processing. It uses UTF-16 instead of UTF-8 for the holding of string values. This means that code-points that need 16-bit values to be represented can be saved as a single character. Chinese characters and Emojis both fall into this category. JavaScript will internally handle the conversion between UTF-8 strings and UTF-16 strings when you use the USVString.

When you see that a USVString is being used for an API, you can just think of it as a string that is encoded to work well in fetch calls and URLs.

Fetch All Together

Now that you know all the parts of an HTTP Request, you can build your own Request object and pass it to a fetch call.

The following example shows how to combine all the different parts into a Request object that gets passed to the fetch method. Only one value is being appended or added to the Headers, FormData, or URLSearchParms object for brevity’s sake.

let head = new Headers(); //`append` what you need
head.append('x-custom-header', 'Memento');

let fd = new FormData(); //`append` what you need
fd.append('name', 'value');

let search = new URLSearchParams(); //`set` what you need
search.set('api-key', 'some-value');

let baseURL = `https://www.example.com`;
let relativePath = `./api/endpoint`;
let url = new URL(`${relativePath}?${search}`, baseURL);
// or new URL(path+querystring, baseurl).toString();
//call the toString method to create the DOMString to pass to the Request object

let req = new Request(url, {
  headers: head,
  method: 'POST',
  mode: 'cors',
  body: fd,
});

fetch(req)
  .then((response) => {
    //response to the fetch
  })
  .then()
  .catch();

Everything you need to know to master the Fetch API

Fetch

So, now that you are aware of all the parts of the world of fetch, let’s start to use the fetch() method and talk to some real APIs.

There are a few ways that the initial fetch method call can be made depending on how much information you need to pass and what needs to be customized.

//1. send a string to the fetch method
let urlString = 'http://www.example.com/api';
fetch(urlString);
//uses GET as the default method. No extra headers set. No data being sent to the server

//2. send a URL object to the fetch method
let url = new URL();
fetch(url);
//same as version 1

//3. send a Request Object that contains the url
let req = new Request(url);
//Request object can also have an options param with data and headers and non-GET method
fetch(req);

If you are only requesting to receive data from a web server (a GET request), and the server does not need an API key or any authorization headers, and no data is being uploaded then any of these can be used.

It is only if you start to upload data or customize headers that you need to add a Headers object or define the body contents.

IDE, API, Library, Framework, and SDK

Ever wonder what the difference is between these five things - IDE, API, Library, Framework, and SDK?

While there are different interpretations with different types of programming and disagreements between developers about the finer points, here is a general reference to what each is:

IDE is an Integrated Development Environment. This is a text editor on steroids. It has features that help developers write and compile their code, as well as, manage all their projects. Usually, they have integrations with Source Control (Git) too.

API is an Application Programming Interface. This is a group of functions or server-side endpoints that let another program access a service or data-source.

Library is one or more files that use the same programming language as your project. They can be included in your project to provide pre-tested code that will speed up your development work.

Framework is similar to a library but will typically also include UI components, design assets, best practice guidance, and conventions to follow.

SDK is a software development kit. It tends to be the largest of all of these. It will often include a library or framework. Most importantly, it will include tools that you need in order to develop for your target platforms, such as a compiler or testing tools.

JSONPlaceHolder

This is a free website that developers often use to test that their code is working.

http://jsonplaceholder.typicode.com/

They have a static series of datasets that will be returned when you make fetch calls to their endpoints.

• Users http://jsonplaceholder.typicode.com/users: 10 users
• Albums http://jsonplaceholder.typicode.com/albums: 100 albums
• Photos http://jsonplaceholder.typicode.com/photos: 5000 photos
• Todo http://jsonplaceholder.typicode.com/todos: 200 todos
• Posts http://jsonplaceholder.typicode.com/posts: 100 posts
• Comments http://jsonplaceholder.typicode.com/comments: 500 comments

You can make a GET request to any of those endpoints and you will get the same response each time.

If you want to pretend to do an upload a new object of one of those types then make a POST request and include the data you want to upload in the body of your request. You will get a 201 success message from the server. It doesn’t actually add anything to the dataset. It just pretends.

If you want to get the details of a single item from one of those sets then just add the id of the item you want at the end of the endpoint and make a GET request.

You can also make DELETE requests with the id to pretend to delete one or PUT or PATCH requests with the id to pretend to do an update.

RandomFox

https://randomfox.ca/

Need a random image of a fox? We got you fam!

Make a fetch GET request to this endpoint https://randomfox.ca/floof/ and you will be sent a JSON response that looks like this:

{
  "image": "https://randomfox.ca/images/8.jpg",
  "link": "https://randomfox.ca/?i=8"
}

RandomDog

https://random.dog/

Need an image of a random dog? Search no more!

Send a GET request with fetch to this endpoint https://random.dog/woof.json.

You will get a JSON response that looks like this:

{
  "fileSizeBytes": 95056,
  "url": "https://random.dog/1f3fcc44-3b7c-4268-92a9-a6faa6f75547.jpg"
}

Creating Dynamic Content

Once you have a grasp on how to match a fetch request to a web server, then it is time to start looking at how to update your webpage content with the data that came back from the webserver.

//make a request to an API that will return some JSON data
fetch(url)
  .then((response) => {
    if (!response.ok) throw new Error('Data request failed');
    return response.json();
  })
  .then((body) => {
    //body will be the object with the data from the server
    //Where do you want to add the new data?
    //Are you replacing old HTML or adding to existing HTML?
    //Is there a template to use in building the new HTML?
    //Which approach do you want to use when building the HTML?
    //Does `body` contain an array to loop through?
  })
  .catch((err) => {
    //handle the error somehow
    //tell the user
    //write a message about the failure...
  });

We discussed best practices for dynamically adding new HTML from an array /modules/week5/#dynamic-html-best-practices in week 5.

Build HTML with fetch

Real World Fetch

Promise all and multiple files

Reddit REST API

One of the cool features built into the Reddit website, is that you can take nearly every URL from any subreddit and just add .json to the end of the URL to load a JSON file version of all the information on the page.

This means that you can make fetch() calls to the home page of a subreddit and load a JSON file with a current list of all the posts to that subreddit.

Say, for example you took the LearnJavaScript subreddit page - https://www.reddit.com/r/learnjavascript/ and then replace the final / with .json at the end of that string. Also switch the www to api. Then you would have the url to use in the fetch call.

const url = `https://api.reddit.com/r/learnjavascript.json`;
fetch(url)
  .then((response) => {
    if (!response.ok) throw new Error('Unable to fetch the URL');
    return response.json();
  })
  .then((data) => {
    //data will be the JavaScript object created from the JSON string returned from reddit.com
  })
  .catch((err) => {
    console.error(err.message);
  });

Try just pasting this URL https://api.reddit.com/r/learnjavascript/.json into your browser and see the results that are loaded.

Github REST API

Github Public Repo REST APIs

Github has a number of public APIs that we can use to fetch information about Repositories or Users or more. You can make a fetch call to a url like https://api.github.com/users/prof3ssorSt3v3/repos to get a JSON file with a list of the all the repos for Steve Griffith’s Github account. The URL https://api.github.com/users/maddprof1/repos returns a JSON file with a list of all the repos for Tony Davidson.

So, as long as you know the username, you can get a list of all that person’s public repos.

Try loading either of those URLs into the browser and look at the JSON results that are displayed.

Github also provides their own JavaScript library called Octokit, which can be used to make calls to the Github APIs.

With the Octokit library, instead of calling fetch, you would import the library, create an Octokit object and then call the request() method, with the desired endpoint URL.

import { Octokit } from 'https://cdn.skypack.dev/octokit';
//import the Octokit function
//then create an instance of the Octokit object
const octokit = new Octokit({}); //the {} object allows for passing in of options

octokit
  .request('GET /repos/{owner}/{repo}', {
    owner: 'octocat', //will become the {owner} part of the url
    repo: 'Spoon-Knife', //will become the {repo} part of the url
    sort: 'updated',
  })
  .then((response) => {
    console.log(response.status); //status code
    console.log(response.data); //same as the data object you get in fetch from response.json()
  });

Http Cat

Looking for a fun way to learn and remember the Http Status Codes? https://http.cat/ will give you the full list.

If you want a single status code with it’s corresponding image, then just add the status code to the end of the url. Eg:

https://http.cat/404

The Cat API

The Cat API is an API that you can use to retrieve lots of cat images. You can search for cats based on breed or category of picture. You can retrieve random images or download them in sets called pages.

You need to request an API key from the company. Go here to request a free API.

Here is the official documentation for using the API.

Authentication

Every request that you make for cat images needs to include the API key in either your Request Headers OR in the query string.

//the header version
let headers = new Headers();
header.append('x-api-key', 'your api key goes here');
let url = new URL(`https://api.thecatapi.com/v1/images/search?`);
let req = new Request(url, {
  headers: header,
});

//the querystring version
let url = new URL(`https://api.thecatapi.com/v1/images/search?api_key=YOUR_API_KEY`);
let req = new Request(url);

Get the Category List

When doing a search for images, if you want to filter by category then you first need to get the list of possible category ids.

//url to get the list of categories
let url = `https://api.thecatapi.com/v1/categories`;

The resulting JSON returned from the server will be an Array of objects. Each object will contain a name and the category id.

Search Params

When your make a request for images you need to provide parameters for the search in your querystring, as part of the request url. These are your possible querystring parameters.

• limit Eg: ?limit=20
• page Eg: ?page=0
• order Eg: ?order=ASC or DESC or RAND
• has_breeds Eg: has_breeds=0
• breed_ids Eg: ?breed_ids=beng,abys
• category_ids Eg: ?category_ids=3,5,15

You can find more information about these on the Basics: Getting Images.

Example Results

After you make your HTTP Request for the search you will get a JSON response from the server.

[
  {
    "id": "ebv",
    "url": "https://cdn2.thecatapi.com/images/ebv.jpg",
    "width": 176,
    "height": 540,
    "breeds": [],
    "favourite": {}
  }
]

Your JSON data will be an array of objects. Each object will have an id which will also be the name of the image. Unfortunately, there are no names associated with each image. So, if you want names then you need to generate those yourself. However, you can use the id value as the key for an image in an object.

Download Progress

The old XMLHttpRequest object had a progress event that let you measure the progress of your upload of data to the server. This cannot be done with fetch. However, we are able to measure the progress of a download of any file from a server.

Note: This will not work if the server is not sending the content-length header with it’s response. If you are building a server-side API, it is a good idea to include content-type and content-length headers with all your responses.

The response.body object that we get as a response to calling the fetch() method is a ReadableStream object. This means that you are actually getting a stream of data coming from the web server.

When the first then() method is triggered by fetch() it means that we have a Response object. The Response object will contain all the headers from the server. However, we don’t necessarily have all the contents of the downloaded file in the body property.

This is why we have to call response.json() or response.text() or response.blob(), which are all asynchronous methods that return a Promise. They are ALL waiting for the rest of the body to be downloaded before they can extract the content and trigger the next then() method.

fetch(url)
  .then((response) => {
    //we now have a Response Object
    //it has all the Headers
    //If cached, we could have the whole file
    //If not cached we are still waiting for the contents of the response.body
    //call an async method to extract the contents from response.body
    return response.json();
    //this method does not resolve until all the contents have been downloaded and extracted.
  })
  .then((content) => {
    //NOW we have all the contents extracted from the body.
  })
  .catch((err) => {
    //handle errors
  });

So, to determine the progress of your download from the server we need to know both the total filesize to download PLUS the amount that you have downloaded so far.

This example will be specifically for a text file, like JSON, XML, or HTML. A Binary file like an image would have a few differences with the TypedArray.

First we will access the Reader object to get each chunk of data from the stream as it is downloaded. Then we go to the Content-Length header to find the size being downloaded.

//inside the first then()
//get the reader
const reader = response.body.getReader();
//get the total size of the file from the `Content-Length` header
const totalBytes = +response.headers.get('Content-Length');
// the unary plus operator in front of response.headers.get will convert the value to a Number

MDN Reference for Unary Plus operator

Next we want to start reading the chunks and add each of their byte sizes to a current size variable. The current size is the number of bytes downloaded so far in the stream. We also will need an array to hold all the chunks downloaded so far. After the loop we will combine all the chunks into a single block which will be our file body.

We will use the read() method that returns an object with two properties - done and value. The done value is a Boolean indicating if you have reached the end of the Stream yet. The value is the latest chunk of data from the server.

Add the value.length to our total bytes so far and add the latest chunk to our array of chunks.

let currentBytes = 0;
let chunks = []; // our Array of data that makes up the whole body
//create a loop that will keep looping until you tell it to stop
while (true) {
  const { done, value } = await reader.read();
  if (done) {
    break; //exit the loop if the reader's `done` property is true
  }
  chunks.push(value); //add the newest chunk to our Array
  currentBytes += value.length; //figure out the new amount downloaded so far.

  console.log(`Received ${currentBytes} of ${totalBytes}`);
  //output the percentage or values somehow on your page
}

Once the loop has been exited it means that we have the whole file. We need to take all our chunks from our Array and put them into a single 8-bit TypedArray. This will be the actual file that we can then call methods on like json() or text().

//after the loop
let wholeFile = new Uint8Array(totalBytes); // TypedArray
let position = 0; //position in the TypedArray
for (let chunk of chunks) {
  wholeFile.set(chunk, position); // place a chunk into the wholeFile TypedArray
  position += chunk.length; //move to the next position in the TypedArray
}

When the for of loop is completed, we will have a wholeFile TypedArray which will contain all the binary data for our Response.body.

The last step is to tell the browser how to read the file. In other words, is it a UTF-8 text file or a binary file, etc. Say that we are dealing with a JSON file. We need to turn the contents of the TypedArray binary data into a UTF-8 string. Then we can parse the String from JSON to a JS Object.

//turn the TypedArray into a utf-8 String
let str = new TextDecoder('utf-8').decode(wholeFile);
//parse the string as a JSON string and return to the next then()
return JSON.parse(str);

The returned object created by calling JSON.parse() on our utf-8 string will be passed to the next then() method and become that content variable.

Using for await of

There is a new variation of for of called for await of. The new version still loops through iterable objects returning all the values, but the iterables can be asychronous. This means you could be using Promises or fetch calls to get the values. The for await of loop will wait for each value to resolve before it increments to the next one.

Normal loops like for, for in, while and for of all aim to finish as quickly as possible. If the values that they are looping over are asynchronous Promises then the values you see in your loop will all be unresolved Promise. Which isn’t very useful.

So, in the progress example above with the ReadableStream Response.body, you could use a for await of loop instead of the while loop.

Here is the new version of all the code from above using a for await of loop.

//with for await (of)
fetch(url)
  .then(response=>{
    if(!response.ok) throw new Error(response.statusText);
    const reader = response.body.getReader();
    let currentBytes = 0;
    const totalBytes = +response.headers.get('Content-Length');
    let wholeFile = new Uint8Array(totalBytes); // TypedArray
    let position = 0;
    //loop while getting chunks
    for await ({done, value} of reader.read()) {
      //output the percentage or values somehow on your page
      currentBytes += value.length; //figure out the new amount downloaded so far.
      console.log(`Received ${currentBytes} of ${totalBytes}`);
      // add the chunk into the wholeFile TypedArray
      wholeFile.set(value, position);
      position += value.length; //move to the next position in the TypedArray

      if (done) {
        break; //exit the loop if the reader's `done` property is true
      }
    }
    //parse the string as a JSON string and return to the next then()
    let str = new TextDecoder('utf-8').decode(wholeFile);
    return JSON.parse(str);
  })
  .then((contents) => {
    //we have the contents of the file as a JS Object
  })
  .catch((err) => console.warn);

The last two lines inside the first then() are assuming that we are dealing with a text file, specifically a JSON file.

If you were dealing with a binary file like an image then we would want to create a Blob object instead of a string.

fetch(url)
  .then((response) => {
    // ...all the other code remains the same
    let blob = new Blob(wholeFile);
    return blob; //pass the binary large object to the second then()
    // If we weren't monitoring progress then we could just use
    // return response.blob();
  })
  .then((blob) => {
    //blob is the BLOB object with the image data
    let img = document.querySelector('#targetImage');
    //use URL.createObjectURL to turn the binary data into something that can be loaded by an <img>
    img.src = URL.createObjectURL(blob);
  })
  .catch((err) => console.warn);

If your fetch is going to be retrieving both text or binary items then you can add a test to see what kind of file is being downloaded.

fetch(url).then((response) => {
  if (!response.ok) throw new Error(response.statusText);
  //get the content-type header
  const fileType = response.headers.get('Content-type');
  //do different things depending...
  if (!fileType) {
    //missing a content-type header...
  } else if (fileType.includes(`image/`) || fileType.includes(`video/`) || fileType.includes(`audio/`)) {
    //we are dealing with an image, audio or video file
  } else if (fileType.includes('application/json') || fileType.includes('text/')) {
    //json, html, txt, xml
  } else {
    //not one of the desired types
  }
});

Tasks

Todo This Week

• Attend class
• Read all the content from Weeks 10 and 11.
• Prepare questions to ask in class.