Guide: Node Script to Export All Channels

To allow applications to have quick access to their data, Stream offers an ExportChannels endpoint.

It's a simple endpoint, however it is limited to 25 channels. In the case that a larger export is required (ex. export of all channels of a particular team), it can be cumbersome to manually deal with all of this channel data. 

To make this easier, here is a script that automates most of the process. 

The script parses through an array of Channel CIDs and formats each CID into an object that the ExportChannels endpoint can use, and each object is put into array. The array is then split into sub-arrays with a length of 25, and given to exportChannels. From there, it generates a list of task IDs and ultimately returns a list of resolved async requests that include the task result, including the URL with exported channel data. 

Note: To generate the initial list of channels, you can use this guide to generate a list of a large selection of channels: Node Script to Paginate All Channels

// Step 1: Prepare channels array:

const channels = [
  "channelType:channelId",
  "channelType2:channelId2",
  ...
];

//Step 2: Format Channels:

const parseChannels = () => {
  let mutatedChannels = channels;
  const channelArray = [];

  while (mutatedChannels.length) {
    let counter = 0;
    let exportFormat = { type: null, id: null };
    let splitArray = channels[counter].split(":");
    exportFormat.type = splitArray[0];
    exportFormat.id = splitArray[1];
    channelArray.push(exportFormat);
    mutatedChannels.shift();
    counter++;
  }

let data;

const chunkChannelArrayIntoSetsOf25 = (arr, size) =>
  (data = Array.from({ length: Math.ceil(arr.length / size) }, (v, i) =>
  arr.slice(i * size, i * size + size)
));

return chunkChannelArrayIntoSetsOf25(channelArray, 25);
};

parseChannels()

const runExports = async () => {
  const channelsInSetsOf25 = [...returned data from previous function...]

  let taskIds = []

  for (let i = 0; i < channelsInSetsOf25.length; i++) {
    const response = await serverClient.exportChannels(channelsInSetsOf25[i], {
      version: "v2",
      include_truncated_messages: true
    });
    taskIds.push(response.task_id)
  }


  let tasks = [];
  
  const getTaskStatus = async (taskId) => {
    return await serverClient.getExportChannelStatus(taskId);
  };

  taskIds.forEach((taskId) => {
    tasks.push(getTaskStatus(taskId));
  });

  tasks = Promise.all(tasks);
  return tasks;
};