[Shopify] API (REST) Pagination

Today I’ll summarize about Shopify pagination. The explanation of the code is ʻASP.Net Core 3.1 C #`.

Pagination?

Pagination is, as the translation, pagination. For example, when you request something via API, even if 1 million rows of data are returned, it can not be processed, and even from the Shopify server that returns the response, the load is too heavy, so it is distributed (divided into pages) Make a response (reply to the request).

For example, you might have information about 100,000 customers when you get a list of customers in customersavedsearch in Shopify.

Click here for details on customers aved search
https://shopify.dev/docs/admin-api/rest/reference/customers/customersavedsearch#other-2020-07

And although there are 100,000 people, it returns only 50 people by default and 250 people at the maximum. So you need to page through.

Page feed mechanism (roughly)

When you send a request to Shopify, the response (reply) header has a value that indicates the next page. The mechanism is to read the value on the next page and send the next request.

Then you might think that you should loop, but be aware that processing this request in a loop may exceed Shopify’s API limit. In the case of REST API, the upper limit is 2 requests per second, so it is recommended to control so that requests are sent every few seconds.

See below for the upper limit.
https://shopify.dev/concepts/about-apis/rate-limits

Request Properties (Request to Shopify API Service)

The parameters you can specify when sending a request to the Shopify API are:

Example: When you want 3 records to be returned on one page

https://{yourshop}.myshopify.com/admin/api/2020-07/customer_saved_searches/{customer_saved_search_id}/customers.json?limit=3

The above is used to specify how many records should be returned on one page at the time of initial request.

If you want to see the next page, you also need to specify page_info. The value of page_info should read the header information contained in the API response. (It will be later)

The properties related to pagination that can be specified at the time of request are as follows.

Property	Description
page_info	The ID that specifies the page. Specify the page ID included in the API response here.
limit	Specify the number of records you want to return
fields	Specify the property you want to return (supported API is limited)

Response property (response from Shopify API Service)

The postman request and response are below

Decrypt the header properties below. The property in the header is Link. When there is no next page, there is no Link property itself.

<https://{yourshop}.myshopify.com/admin/api/2020-07/customer_saved_searches/{customer_saved_search_id}/customers.json?limit=3&page_info=eyJxdWVyeSI6ImNvdW50cnk6XCJKYXBhblwiIiwibGFzdF9pZCI6MzcwODgxOTg5ODUyNywibGFzdF92YWx1ZSI6MTU5NjUwNDY1NzAwMCwiZGlyZWN0aW9uIjoibmV4dCJ9>; rel="next"

Read the page_info = part from the above value.

Extracting page_info

It is extracted with the code below.

  public string ExtractPageInfo(string url)
        {
            // return string that is page_info=
            var pageInfo = "";

            // strip down <>
            url = url.Replace("<", "");
            url = url.Replace(">", "");

            var key = "page_info=";

            Match match = Regex.Match(url, key, RegexOptions.IgnoreCase);

            if (match.Success)
            {
              
                var length = key.Length;

                pageInfo = url.Substring(match.Index + length, url.Length - (match.Index + length));
            }

            return pageInfo;
        }

Commentary

I would like to explain with the actual code. The environment below is ASP.NET Core 3.1 C #.

Whole (for the time being)

First of all, the whole code is below.

 public async Task<ShopifyCustomerSavedSearcheCustomers> GetListOfUsersEmailInShopifyCustomerSavedSearcheAsync(string storeUrl, 
            string nameOfCustomerSavedSearches, string pageInfo)
        {
            // read 250 at time

            var endPoint = apiVersion + "/customer_saved_searches/" + nameOfCustomerSavedSearches + "/customers.json";

            var store = _organizationHandlers.GetOrganization(storeUrl);

            var rawStoreLog = new Dictionary<string, string>();
            rawStoreLog.Add("Store", JsonConvert.SerializeObject(store));

            _telemetry.TrackEvent("customer_saved_searches.json - store", rawStoreLog);

            var ShopifyCustomerSavedSearcheCustomers = new ShopifyCustomerSavedSearcheCustomers();
          
            try
            {
                using (var httpClient = new HttpClient())
                {
                    httpClient.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
                    httpClient.DefaultRequestHeaders.Add("X-Shopify-Access-Token", store.AccessToken);

                    var requestUrl = "https://" + storeUrl + endPoint + "?limit=250&fields=email,first_name,last_name";

                    if(!string.IsNullOrEmpty(pageInfo))
                    {
                        requestUrl += "&page_info=" + pageInfo;
                    }

                    using (var response = await httpClient.GetAsync(requestUrl))
                    {
                        string apiResponse = await response.Content.ReadAsStringAsync();
                        
                        // Try and get link property of the header
                        var apiHeaderCollectionLinkKeyValuePairs = from l in response.Headers where l.Key == "Link" select l;
                        var apiHeaderCollectionLinkKeyValuePairString = "";

                        if (apiHeaderCollectionLinkKeyValuePairs.Any())
                        {
                            // Insert page_info (if any)
                            apiHeaderCollectionLinkKeyValuePairString = apiHeaderCollectionLinkKeyValuePairs.FirstOrDefault().Value.First();
                        }

                        var rawLog = new Dictionary<string, string>();
                        rawLog.Add("ApiResponse", apiResponse);

                        _telemetry.TrackEvent("customer_saved_searches.json", rawLog);

                        ShopifyCustomerSavedSearcheCustomers = JsonConvert.DeserializeObject<ShopifyCustomerSavedSearcheCustomers>(apiResponse);

                        // Append page_info
                        ShopifyCustomerSavedSearcheCustomers.link = apiHeaderCollectionLinkKeyValuePairString;
                    }
                }

                return ShopifyCustomerSavedSearcheCustomers;
            }
            catch (Exception e)
            {
                var log = new Dictionary<string, string>();
                log.Add("Message", e.Message);

                _telemetry.TrackEvent("GetShopifyCustomerSavedSearchesAsync", log);
            }

            return ShopifyCustomerSavedSearcheCustomers;

        }

      
    }

Method definition

First from here

public async Task<ShopifyCustomerSavedSearcheCustomers> GetListOfUsersEmailInShopifyCustomerSavedSearcheAsync(string storeUrl, 
            string nameOfCustomerSavedSearches, string pageInfo)

PageInfo is provided in the input value of this message. The assumption here is that pageInfo itself is passed to this method. Note that this method itself does not process pagination. The method of actually pagination to ensure independence is in the next higher layer.

Object preparation

In the next part, create an object to be used for the processing to be performed.

var endPoint = apiVersion + "/customer_saved_searches/" + nameOfCustomerSavedSearches + "/customers.json";

            var store = _organizationHandlers.GetOrganization(storeUrl);

            var rawStoreLog = new Dictionary<string, string>();
            rawStoreLog.Add("Store", JsonConvert.SerializeObject(store));

            _telemetry.TrackEvent("customer_saved_searches.json - store", rawStoreLog);

            var ShopifyCustomerSavedSearcheCustomers = new ShopifyCustomerSavedSearcheCustomers();

Here, the acquisition of the merchant information saved in the DB and the log to Application Insight for error detection and debug are described.

Prepare a class to receive a response

ShopifyCustomerSavedSearcheCustomers is a received object and is defined as follows.

  public class ShopifyCustomerSavedSearcheCustomers
    {
        public List<ShopifyCustomerSavedSearcheCustomer> customers { get; set; }
        public string link { get; set; }
    }

    public class ShopifyCustomerSavedSearcheCustomer
    {
        public string email { get; set; }
        public string first_name { get; set; }
        public string last_name { get; set; }
    }

Shoify API Request

This is the part that is actually requesting Shopify from the next.

  try
            {
                using (var httpClient = new HttpClient())
                {
                    httpClient.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
                    httpClient.DefaultRequestHeaders.Add("X-Shopify-Access-Token", store.AccessToken);

                    var requestUrl = "https://" + storeUrl + endPoint + "?limit=250&fields=email,first_name,last_name";

                    if(!string.IsNullOrEmpty(pageInfo))
                    {
                        requestUrl += "&page_info=" + pageInfo;
                    }

                    using (var response = await httpClient.GetAsync(requestUrl))
                    {
                        string apiResponse = await response.Content.ReadAsStringAsync();
                        
                        // Try and get link property of the header
                        var apiHeaderCollectionLinkKeyValuePairs = from l in response.Headers where l.Key == "Link" select l;
                        var apiHeaderCollectionLinkKeyValuePairString = "";

                        if (apiHeaderCollectionLinkKeyValuePairs.Any())
                        {
                            // Insert page_info (if any)
                            apiHeaderCollectionLinkKeyValuePairString = apiHeaderCollectionLinkKeyValuePairs.FirstOrDefault().Value.First();
                        }

                        var rawLog = new Dictionary<string, string>();
                        rawLog.Add("ApiResponse", apiResponse);

                        _telemetry.TrackEvent("customer_saved_searches.json", rawLog);

                        ShopifyCustomerSavedSearcheCustomers = JsonConvert.DeserializeObject<ShopifyCustomerSavedSearcheCustomers>(apiResponse);

                        // Append page_info
                        ShopifyCustomerSavedSearcheCustomers.link = apiHeaderCollectionLinkKeyValuePairString;
                    }
                }

                return ShopifyCustomerSavedSearcheCustomers;
            }
            catch (Exception e)
            {
                var log = new Dictionary<string, string>();
                log.Add("Message", e.Message);

                _telemetry.TrackEvent("GetShopifyCustomerSavedSearchesAsync", log);
            }

Shopify API Request (header)

Let’s take a closer look at the above code.

 using (var httpClient = new HttpClient())
                {
                    httpClient.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
                    httpClient.DefaultRequestHeaders.Add("X-Shopify-Access-Token", store.AccessToken);

                    var requestUrl = "https://" + storeUrl + endPoint + "?limit=250&fields=email,first_name,last_name";

                    if(!string.IsNullOrEmpty(pageInfo))
                    {
                        requestUrl += "&page_info=" + pageInfo;
                    }

                    using (var response = await httpClient.GetAsync(requestUrl))
                    {
                        string apiResponse = await response.Content.ReadAsStringAsync();
                        
                        // Try and get link property of the header
                        var apiHeaderCollectionLinkKeyValuePairs = from l in response.Headers where l.Key == "Link" select l;
                        var apiHeaderCollectionLinkKeyValuePairString = "";

                        if (apiHeaderCollectionLinkKeyValuePairs.Any())
                        {
                            // Insert page_info (if any)
                            apiHeaderCollectionLinkKeyValuePairString = apiHeaderCollectionLinkKeyValuePairs.FirstOrDefault().Value.First();
                        }

                        var rawLog = new Dictionary<string, string>();
                        rawLog.Add("ApiResponse", apiResponse);

                        _telemetry.TrackEvent("customer_saved_searches.json", rawLog);

                        ShopifyCustomerSavedSearcheCustomers = JsonConvert.DeserializeObject<ShopifyCustomerSavedSearcheCustomers>(apiResponse);

                        // Append page_info
                        ShopifyCustomerSavedSearcheCustomers.link = apiHeaderCollectionLinkKeyValuePairString;
                    }
                }

                return ShopifyCustomerSavedSearcheCustomers;
            }

Here, AccessToken is specified in the header part, and the Url character string for the request is generated. If pageInfo is not blank or null, add it to the end of the request URL string.

httpClient.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
                    httpClient.DefaultRequestHeaders.Add("X-Shopify-Access-Token", store.AccessToken);

                    var requestUrl = "https://" + storeUrl + endPoint + "?limit=250&fields=email,first_name,last_name";

 if(!string.IsNullOrEmpty(pageInfo))
                    {
                        requestUrl += "&page_info=" + pageInfo;
                    }

Response processing

Next, make the actual request and process the returned value. This time, the property called Link in the header part is also read. This Link will not be included in the response value without the next page, so search with the linq syntax to see if it is in the results .Any () part.

After that, the response string itself is recorded in application insight for debug, and json is mapped to the predefined project. And it is accompanied by the value of the last link.

 using (var response = await httpClient.GetAsync(requestUrl))
                    {
                        string apiResponse = await response.Content.ReadAsStringAsync();
                        
                        // Try and get link property of the header
                        var apiHeaderCollectionLinkKeyValuePairs = from l in response.Headers where l.Key == "Link" select l;
                        var apiHeaderCollectionLinkKeyValuePairString = "";

                        if (apiHeaderCollectionLinkKeyValuePairs.Any())
                        {
                            // Insert page_info (if any)
                            apiHeaderCollectionLinkKeyValuePairString = apiHeaderCollectionLinkKeyValuePairs.FirstOrDefault().Value.First();
                        }

                        var rawLog = new Dictionary<string, string>();
                        rawLog.Add("ApiResponse", apiResponse);

                        _telemetry.TrackEvent("customer_saved_searches.json", rawLog);

                        ShopifyCustomerSavedSearcheCustomers = JsonConvert.DeserializeObject<ShopifyCustomerSavedSearcheCustomers>(apiResponse);

                        // Append page_info
                        ShopifyCustomerSavedSearcheCustomers.link = apiHeaderCollectionLinkKeyValuePairString;

In this way, check the header value of the response and perform pagination.

That’s all for today.

Recruitment

The Transcosmos Technology Institute is still looking for developers and project managers to work with Shopify. Please refer to here.
http://t-rnd.com/Jobs/