Jianghu Digital Platform
Login

HttpClient

12003

In the era of the Internet, countless services communicate based on the HTTP protocol, and it is very common for web applications to call backend HTTP services.

To facilitate this, the framework has implemented an HttpClient based on the built-in urllib, allowing applications to easily perform any HTTP requests.

Using HttpClient through app

When the application initializes, the framework automatically initializes the HttpClient to app.httpclient. Additionally, a method app.curl(url, options) is added, which is equivalent to app.httpclient.request(url, options).

This allows for convenient use of the app.curl method to complete an HTTP request.

// app.js
module.exports = app => {
  app.beforeStart(async () => {
    // Example: Read the version information from https://registry.npmmirror.com/egg/latest at startup
    const result = await app.curl('https://registry.npmmirror.com/egg/latest', {
      dataType: 'json',
    });
    app.logger.info('Latest Egg version: %s', result.data.version);
  });
};

Using HttpClient through ctx

The framework also provides ctx.curl(url, options) and ctx.httpclient in the Context to maintain a consistent usage experience with that under app. This makes it very convenient to use the ctx.curl() method to complete an HTTP request wherever there is a Context (such as in a controller).

// app/controller/npm.js
class NpmController extends Controller {
  async index() {
    const ctx = this.ctx;

    // Example: Request information about an npm module
    const result = await ctx.curl('https://registry.npmmirror.com/egg/latest', {
      // Automatically parse JSON response
      dataType: 'json',
      // 3 seconds timeout
      timeout: 3000,
    });

    ctx.body = {
      status: result.status,
      headers: result.headers,
      package: result.data,
    };
  }
}

Basic HTTP Requests

HTTP has been widely used. Although there are various request methods for HTTP, the fundamental principles remain the same. We will first explain the basic four request methods as examples and gradually discuss more complex application scenarios.

The following examples will initiate requests to https://httpbin.org within the controller code.

GET

Data retrieval is almost always done using GET requests. It is the most common and widely used method in the HTTP world. Its request parameters are also the easiest to construct.

// app/controller/npm.js
class NpmController extends Controller {
  async get() {
    const ctx = this.ctx;
    const result = await ctx.curl('https://httpbin.org/get?foo=bar');
    ctx.status = result.status;
    ctx.set(result.headers);
    ctx.body = result.data;
  }
}
  • A GET request does not require setting the options.method parameter; the default method of HttpClient will be set to GET.
  • The return value result will contain three properties: status, headers, and data.
    • status: Response status code, such as 200, 302, 404, 500, etc.
    • headers: Response headers, similar to { 'content-type': 'text/html', ... }.
    • data: Response body; by default, HttpClient does not perform any processing and will directly return data in Buffer type.
      Once options.dataType is set, HttpClient will process data accordingly based on this parameter.

For a complete explanation of the request parameters options and return value result, please refer to the section Detailed Explanation of options Parameters below.

POST

Creating data scenarios generally use POST requests, which have an additional request body parameter compared to GET.

For example, sending a JSON body:

// app/controller/npm.js
class NpmController extends Controller {
  async post() {
    const ctx = this.ctx;
    const result = await ctx.curl('https://httpbin.org/post', {
      // Must specify method
      method: 'POST',
      // Inform HttpClient to send in JSON format via contentType
      contentType: 'json',
      data: {
        hello: 'world',
        now: Date.now(),
      },
      // Clearly inform HttpClient to process the returned response body in JSON format
      dataType: 'json',
    });
    ctx.body = result.data;
  }
}

The following sections will also detail how to implement form submissions and file uploads using POST.

PUT

PUT is similar to POST but is more suitable for updating and replacing data semantics. Apart from needing to set the method parameter to PUT, other parameters are almost identical to POST.

// app/controller/npm.js
class NpmController extends Controller {
  async put() {
    const ctx = this.ctx;
    const result = await ctx.curl('https://httpbin.org/put', {
      // Must specify method
      method: 'PUT',
      // Inform HttpClient to send in JSON format via contentType
      contentType: 'json',
      data: {
        update: 'foo bar',
      },
      // Clearly inform HttpClient to process the response body in JSON format
      dataType: 'json',
    });
    ctx.body = result.data;
  }
}

DELETE

To delete data, a DELETE request is chosen. It typically does not require adding a request body, but HttpClient does not impose any restrictions on this.

// app/controller/npm.js
class NpmController extends Controller {
  async del() {
    const ctx = this.ctx;
    const result = await ctx.curl('https://httpbin.org/delete', {
      // Must specify method
      method: 'DELETE',
      // Clearly inform HttpClient to process the response body in JSON format
      dataType: 'json',
    });
    ctx.body = result.data;
  }
}

Advanced HTTP Requests

In real application scenarios, there are also some more complex HTTP requests.

Form Submission

Form submissions designed for browsers (excluding files) typically require submitting request data in the format of content-type: application/x-www-form-urlencoded.

// app/controller/npm.js
class NpmController extends Controller {
  async submit() {
    const ctx = this.ctx;
    const result = await ctx.curl('https://httpbin.org/post', {
      // Must specify method, supports POST, PUT, and DELETE
      method: 'POST',
      // No need to set contentType; HttpClient will default to sending requests in application/x-www-form-urlencoded format
      data: {
        now: Date.now(),
        foo: 'bar',
      },
      // Clearly inform HttpClient to process the response body in JSON format
      dataType: 'json',
    });
    ctx.body = result.data.form;
    // The response will ultimately be similar to the following result:
    // {
    //   "foo": "bar",
    //   "now": "1483864184348"
    // }
  }
}

Uploading Files via Multipart

When a Form submission includes files, the request data format must be submitted as multipart/form-data.

urllib has a built-in formstream module to help us generate a consumable form object.

// app/controller/http.js
class HttpController extends Controller {
  async upload() {
    const { ctx } = this;

    const result = await ctx.curl('https://httpbin.org/post', {
      method: 'POST',
      dataType: 'json',
      data: {
        foo: 'bar',
      },
      
      // Single file upload
      files: __filename,
      
      // Multiple file uploads
      // files: {
      //   file1: __filename,
      //   file2: fs.createReadStream(__filename),
      //   file3: Buffer.from('mock file content'),
      // },
    });

    ctx.body = result.data.files;
    // The response will ultimately be similar to the following result:
    // {
    //   "file": "use strict; const For...."
    // }
  }
}

Uploading Files via Stream

In the Node.js world, streams are actually the mainstream. If the server supports streaming uploads, the most friendly way is to send streams directly. Streams will be sent using the Transfer-Encoding: chunked transfer encoding format, and this conversion is automatically implemented by the HTTP module.

// app/controller/npm.js
const fs = require('fs');
const FormStream = require('formstream');
class NpmController extends Controller {
  async uploadByStream() {
    const ctx = this.ctx;
    // Upload the current file itself for testing
    const fileStream = fs.createReadStream(__filename);
    // httpbin.org does not support stream mode, using local stream interface instead
    const url = `${ctx.protocol}://${ctx.host}/stream`;
    const result = await ctx.curl(url, {
      // Must specify method, supports POST, PUT
      method: 'POST',
      // Submit in stream mode
      stream: fileStream,
    });
    ctx.status = result.status;
    ctx.set(result.headers);
    ctx.body = result.data;
    // The response will ultimately be similar to the following result:
    // {"streamSize":574}
  }
}

Detailed Explanation of options Parameters

Due to the complexity of HTTP requests, the options parameter of httpclient.request(url, options) can be very extensive. Next, we will explain each optional parameter's actual use with parameter descriptions and code examples.

Default Global Configuration for HttpClient

// config/config.default.js
exports.httpclient = {
  // Whether to enable local DNS caching, default is off. When enabled, it has two features:
  // 1. All DNS queries will prioritize using the cache by default, even if the DNS query fails, it will not affect the application.
  // 2. For the same domain name, it will only query once within the dnsCacheLookupInterval (default 10s).
  enableDNSCache: false,
  // Minimum interval time for DNS queries for the same domain name
  dnsCacheLookupInterval: 10000,
  // Maximum number of domain names cached in DNS, default is 1000
  dnsCacheMaxLength: 1000,

  request: {
    // Default request timeout
    timeout: 3000
  },

  httpAgent: {
    // Default to enable http KeepAlive feature
    keepAlive: true,
    // Idle KeepAlive socket can survive for a maximum of 4 seconds
    freeSocketTimeout: 4000,
    // When a socket has no activity for more than 30 seconds, it will be treated as a timeout
    timeout: 30000,
    // Maximum number of sockets allowed to create
    maxSockets: Number.MAX_SAFE_INTEGER,
    // Maximum number of idle sockets
    maxFreeSockets: 256
  },

  httpsAgent: {
    // Default to enable https KeepAlive feature
    keepAlive: true,
    // Idle KeepAlive socket can survive for a maximum of 4 seconds
    freeSocketTimeout: 4000,
    // When a socket has no activity for more than 30 seconds, it will be treated as a timeout
    timeout: 30000,
    // Maximum number of sockets allowed to create
    maxSockets: Number.MAX_SAFE_INTEGER,
    // Maximum number of idle sockets
    maxFreeSockets: 256
  }
};

Applications can override this configuration through config/config.default.js.

data: Object

The data to be sent in the request, automatically selecting the correct data processing method based on method.

  • GET, HEAD: Processed by querystring.stringify(data) and appended to the URL's query parameters.
  • POST, PUT, DELETE, etc.: Need to be further judged and processed based on contentType.
    • contentType = json: Processed by JSON.stringify(data) and sent as body.
    • Others: Processed by querystring.stringify(data) and sent as body.
// GET + data
ctx.curl(url, {
  data: { foo: 'bar' }
});

// POST + data
ctx.curl(url, {
  method: 'POST',
  data: { foo: 'bar' }
});

// POST + JSON + data
ctx.curl(url, {
  method: 'POST',
  contentType: 'json',
  data: { foo: 'bar' }
});

dataAsQueryString: Boolean

If dataAsQueryString=true is set, even in a POST request, options.data will be processed by querystring.stringify and appended to the URL's query parameters.

This setting is suitable for scenarios where data needs to be sent as stream and additional request parameters are passed in the form of URL query:

ctx.curl(url, {
  method: 'POST',
  dataAsQueryString: true,
  data: {
    // Usually authorization parameters, such as access token
    accessToken: 'some access token value'
  },
  stream: myFileStream
});

content: String|Buffer

The request body to be sent. If this parameter is set, the data parameter will be ignored.

ctx.curl(url, {
  method: 'POST',
  // Directly send raw XML data without special processing by HttpClient
  content: '<xml><hello>world</hello></xml>',
  headers: {
    'content-type': 'text/html'
  }
});

files: Mixed

File uploads, supporting the following formats: String | ReadStream | Buffer | Array | Object.

ctx.curl(url, {
  method: 'POST',
  files: '/path/to/read',
  data: {
    foo: 'other fields',
  },
});

Multiple file uploads:

ctx.curl(url, {
  method: 'POST',
  files: {
    file1: '/path/to/read',
    file2: fs.createReadStream(__filename),
    file3: Buffer.from('mock file content'),
  },
  data: {
    foo: 'other fields',
  },
});

stream: ReadStream

Sets a readable data stream for sending the request body, with a default value of null. Once this parameter is set, HttpClient will ignore data and content.

ctx.curl(url, {
  method: 'POST',
  stream: fs.createReadStream('/path/to/read'),
});

writeStream: WriteStream

Sets a writable data stream for receiving response data, with a default value of null. Once this parameter is set, the return value result.data will be set to null because the data has been written to writeStream.

ctx.curl(url, {
  writeStream: fs.createWriteStream('/path/to/store'),
});

consumeWriteStream: Boolean

Whether to wait for writeStream to finish writing completely before considering the response received, default is true. It is recommended to keep the default value unless you are aware of its potential side effects.

method: String

Sets the request method, default is GET. Supports GET, POST, PUT, DELETE, PATCH, and all HTTP methods.

contentType: String

Sets the request data format, default is undefined. HttpClient will automatically set based on data and content. If data is an object, it defaults to form. Supports json format.

For example, sending data in JSON format:

ctx.curl(url, {
  method: 'POST',
  data: {
    foo: 'bar',
    now: Date.now(),
  },
  contentType: 'json',
});

dataType: String

Sets the response data format, default is to not process and directly return buffer. Supports text and json.

Note: If set to json, a parsing failure will throw a JSONResponseFormatError exception.

const jsonResult = await ctx.curl(url, {
  dataType: 'json',
});
console.log(jsonResult.data);

const htmlResult = await ctx.curl(url, {
  dataType: 'text',
});
console.log(htmlResult.data);

fixJSONCtlChars: Boolean

Whether to automatically filter special control characters (U+0000～U+001F), default is false. Some CGI systems may return JSON containing these characters.

ctx.curl(url, {
  fixJSONCtlChars: true,
  dataType: 'json',
});

headers: Object

Custom request headers.

ctx.curl(url, {
  headers: {
    'x-foo': 'bar',
  },
});

timeout: Number|Array

Request timeout, default is [5000, 5000], which means the connection timeout is 5 seconds, and the response timeout is 5 seconds.

ctx.curl(url, {
  // Connection timeout 3 seconds, response timeout 3 seconds
  timeout: 3000
});

ctx.curl(url, {
  // Connection timeout 1 second, response timeout 30 seconds, for larger response scenarios
  timeout: [1000, 30000]
});

agent: HttpAgent

Allows overriding the default HttpAgent through this parameter. If you do not want to enable KeepAlive, you can set this parameter to false.

ctx.curl(url, {
  agent: false
});

httpsAgent: HttpsAgent

Allows overriding the default HttpsAgent through this parameter. If you do not want to enable KeepAlive, you can set this parameter to false.

ctx.curl(url, {
  httpsAgent: false
});

auth: String

Basic Authentication parameters, which will send the login information in plain text via the Authorization request header.

ctx.curl(url, {
  // The parameter must be set in the format `user:password`
  auth: 'foo:bar'
});

digestAuth: String

Digest Authentication parameters. Setting this parameter will automatically attempt to generate the Authorization request header for a 401 response and request again with authorization.

ctx.curl(url, {
  // The parameter must be set in the format `user:password`
  digestAuth: 'foo:bar'
});

followRedirect: Boolean

Whether to automatically follow 3xx redirect responses, default is false.

ctx.curl(url, {
  followRedirect: true
});

maxRedirects: Number

Sets the maximum number of automatic redirects to avoid infinite loops, default is 10 times. This parameter should not be set too high; it only takes effect when followRedirect=True.

ctx.curl(url, {
  followRedirect: true,
  // Automatically redirect a maximum of 5 times
  maxRedirects: 5
});

formatRedirectUrl: Function(from, to)

Allows customizing the implementation of URL concatenation for 302, 301, etc. redirects through formatRedirectUrl, default is url.resolve(from, to).

ctx.curl(url, {
  formatRedirectUrl: (from, to) => {
    // For example, you can correct incorrectly redirected URLs here
    if (to === '//foo/') {
      to = '/foo';
    }
    return url.resolve(from, to);
  }
});

beforeRequest: Function(options)

Before the request is officially sent, HttpClient will attempt to call the beforeRequest hook, allowing us to make final modifications to the request parameters here.

ctx.curl(url, {
  beforeRequest: (options) => {
    // For example, you can set a global request ID here for log tracking
    options.headers['x-request-id'] = uuid.v1();
  }
});

streaming: Boolean

Whether to directly return the response stream, default is false. Once streaming is enabled, HttpClient will return immediately after obtaining the response object res, at which point result.headers and result.status can be read, but the data data has not been read yet.

const result = await ctx.curl(url, {
  streaming: true
});

console.log(result.status, result.data);
// result.res is a ReadStream object
ctx.body = result.res;

Note: If res is not directly passed to body, we must consume this stream and handle the error event properly.

gzip: Boolean

Whether to support gzip response format, default is false. When gzip is enabled, HttpClient will automatically set the Accept-Encoding: gzip request header and will automatically decompress data with the Content-Encoding: gzip response header.

ctx.curl(url, {
  gzip: true,
});

timing: Boolean

Whether to enable timing measurements for each stage of the request, default is false. When timing is enabled, you can obtain the timing measurement values for each stage of this HTTP request (in milliseconds) through result.res.timing. With these measurement values, we can easily locate which stage of the request is the slowest. The effect is similar to Chrome network timing.

Timing measurement values breakdown:

  • queuing: Time taken to allocate the socket
  • dnslookup: Time taken for DNS queries
  • connected: Time taken for the socket's three-way handshake connection to succeed
  • requestSent: Time taken to complete sending the request data
  • waiting: Time taken to receive the first byte of response data
  • contentDownload: Time taken to receive all response data completely
const result = await ctx.curl(url, {
  timing: true,
});
console.log(result.res.timing);
// {
//   "queuing": 29,
//   "dnslookup": 37,
//   "connected": 370,
//   "requestSent": 1001,
//   "waiting": 1833,
//   "contentDownload": 3416
// }

ca, rejectUnauthorized, pfx, key, cert, passphrase, ciphers, and secureProtocol

These parameters are all passed through to the HTTPS module. For specifics, refer to https.request(options, callback).

Debugging Assistance

If you need to debug requests made by HttpClient, you can add the following configuration to config.local.js:

// config.local.js
module.exports = () => {
  const config = {};

  // add http_proxy to httpclient
  if (process.env.http_proxy) {
    config.httpclient = {
      request: {
        enableProxy: true,
        rejectUnauthorized: false,
        proxy: process.env.http_proxy,
      },
    };
  }

  return config;
};

Then start a packet capture tool, such as Charles or Fiddler. Start the application with the following command:

$ http_proxy=http://127.0.0.1:8888 npm run dev

After the operation is complete, all requests sent through HttpClient can be viewed in the packet capture tool.

Common Errors

Connection Timeout

  • Exception Name: ConnectionTimeoutError
  • Scenario: Usually occurs due to slow DNS queries or slow network between the client and server.
  • Troubleshooting Suggestion: Increase the timeout parameter appropriately.

Service Response Timeout

  • Exception Name: ResponseTimeoutError
  • Scenario: Occurs when the network between the client and server is slow or when the response data is large.
  • Troubleshooting Suggestion: Increase the timeout parameter appropriately.

Service Actively Closes Connection

  • Exception Name: ResponseError, code: ECONNRESET
  • Scenario: The server actively closes the socket connection, causing HTTP request chain abnormalities.
  • Troubleshooting Suggestion: Check if there are network anomalies on the server side.

Service Unreachable

  • Exception Name: RequestError, code: ECONNREFUSED, status: -1
  • Scenario: The IP or port of the requested URL cannot be connected.
  • Troubleshooting Suggestion: Ensure the IP or port settings are correct.

Domain Name Does Not Exist

  • Exception Name: RequestError, code: ENOTFOUND, status: -1
  • Scenario: The domain name of the requested URL cannot be resolved through DNS.
  • Troubleshooting Suggestion: Ensure the domain name exists and check the DNS service configuration.

JSON Response Data Format Error

  • Exception Name: JSONResponseFormatError
  • Scenario: Thrown when dataType=json is set but the response data is not in JSON format.
  • Troubleshooting Suggestion: Ensure the server returns correctly formatted JSON data.

Global request and response Events

In enterprise application scenarios, there is often a need for unified tracer logging. To facilitate unified listening for HttpClient's requests and responses at the app level, we have defined global request and response events to expose these two events.

    init options
        |
        V
    emit `request` event
        |
        V
    send request and receive response
        |
        V
    emit `response` event
        |
        V
       end

request Event: Occurs Before Network Operations

Before the request is sent, a request event will be triggered, allowing interception of the request.

app.httpclient.on('request', (req) => {
  req.url; // Request URL
  req.ctx; // Current context that initiated this request

  // You can set some trace headers here for full-link tracking
});

response Event: Occurs After Network Operations End

After the request ends, a response event will be triggered, allowing external subscriptions to this event to log information.

app.httpclient.on('response', (result) => {
  result.res.status; // Response status code
  result.ctx; // Current context that initiated this request
  result.req; // Corresponding req object, which is the req from the request event
});

Example Code

Complete example code can be found at eggjs/examples/httpclient.

Other reference links: