├── .nvmrc ├── test ├── resources │ ├── cycle1.m3u8 │ ├── cycle2.m3u8 │ ├── empty.m3u8 │ ├── windows.m3u8 │ ├── path-testing.m3u8 │ ├── with-sub-manifest │ │ ├── var256000 │ │ │ └── playlist.m3u8 │ │ ├── var386000 │ │ │ └── playlist.m3u8 │ │ ├── var500000 │ │ │ └── playlist.m3u8 │ │ └── playlist.m3u8 │ ├── duplicate-manifests │ │ ├── manifest0 │ │ │ └── rendition.m3u8 │ │ ├── manifest1 │ │ │ └── rendition.m3u8 │ │ ├── manifest2 │ │ │ └── rendition.m3u8 │ │ └── master.m3u8 │ ├── simple.m3u8 │ ├── fmp4.m3u8 │ ├── path-query.m3u8 │ ├── tilde-query-param.m3u8 │ ├── dash.mpd │ ├── sidx.mpd │ └── long-path.m3u8 └── unit │ └── walk-manifest.spec.js ├── .npmignore ├── .travis.yml ├── .editorconfig ├── .gitignore ├── src ├── index.js ├── utils.js ├── cli.js ├── write-data.js └── walk-manifest.js ├── LICENSE ├── CONTRIBUTING.md ├── README.md ├── package.json └── CHANGELOG.md /.nvmrc: -------------------------------------------------------------------------------- 1 | lts/* 2 | -------------------------------------------------------------------------------- /test/resources/cycle1.m3u8: -------------------------------------------------------------------------------- 1 | #EXTM3U 2 | #EXT-X-VERSION:3 3 | #EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=300000 4 | cycle2.m3u8 -------------------------------------------------------------------------------- /test/resources/cycle2.m3u8: -------------------------------------------------------------------------------- 1 | #EXTM3U 2 | #EXT-X-VERSION:3 3 | #EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=300000 4 | cycle1.m3u8 -------------------------------------------------------------------------------- /test/resources/empty.m3u8: -------------------------------------------------------------------------------- 1 | #EXTM3U 2 | #EXT-X-PLAYLIST-TYPE:VOD 3 | #EXT-X-TARGETDURATION:8 4 | #EXT-X-VERSION:6 5 | #EXT-X-INDEPENDENT-SEGMENTS 6 | #EXT-X-MEDIA-SEQUENCE:0 -------------------------------------------------------------------------------- /.npmignore: -------------------------------------------------------------------------------- 1 | # Intentionally left blank, so that npm does not ignore anything by default, 2 | # but relies on the package.json "files" array to explicitly define what ends 3 | # up in the package. 4 | -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | sudo: false 2 | dist: trusty 3 | language: node_js 4 | node_js: 5 | - 'lts/*' 6 | before_install: 7 | - npm install -g greenkeeper-lockfile@1 8 | after_script: greenkeeper-lockfile-upload 9 | 10 | -------------------------------------------------------------------------------- /test/resources/windows.m3u8: -------------------------------------------------------------------------------- 1 | #EXTM3U 2 | #EXT-X-PLAYLIST-TYPE:VOD 3 | #EXT-X-TARGETDURATION:8 4 | #EXT-X-VERSION:6 5 | #EXT-X-INDEPENDENT-SEGMENTS 6 | #EXT-X-MEDIA-SEQUENCE:0 7 | #EXTINF:2.000, 8 | foo\\bar\\chunk_0.ts 9 | #EXTINF:1.999, 10 | foo/bar/chunk_1.ts 11 | -------------------------------------------------------------------------------- /.editorconfig: -------------------------------------------------------------------------------- 1 | # http://editorconfig.org 2 | root = true 3 | 4 | [*] 5 | charset = utf-8 6 | end_of_line = lf 7 | indent_style = space 8 | indent_size = 2 9 | insert_final_newline = true 10 | trim_trailing_whitespace = true 11 | 12 | [*.md] 13 | trim_trailing_whitespace = false 14 | -------------------------------------------------------------------------------- /test/resources/path-testing.m3u8: -------------------------------------------------------------------------------- 1 | #EXTM3U 2 | #EXT-X-PLAYLIST-TYPE:VOD 3 | #EXT-X-TARGETDURATION:8 4 | #EXT-X-VERSION:6 5 | #EXT-X-INDEPENDENT-SEGMENTS 6 | #EXT-X-MEDIA-SEQUENCE:0 7 | chunk_0.ts 8 | #EXTINF:2.000, 9 | /test/chunk_1.ts 10 | #EXTINF:1.999, 11 | test/chunk_2.ts 12 | #EXTINF:2.000, 13 | http://manifest-list-test.com/test/chunk_3.ts 14 | #EXTINF:2.000, 15 | 16 | -------------------------------------------------------------------------------- /test/resources/with-sub-manifest/var256000/playlist.m3u8: -------------------------------------------------------------------------------- 1 | #EXTM3U 2 | #EXT-X-TARGETDURATION:4 3 | #EXT-X-MEDIA-SEQUENCE:0 4 | #EXT-X-VERSION:3 5 | #EXTINF:4.166, 6 | seg0.ts 7 | #EXTINF:4.166, 8 | seg1.ts 9 | #EXTINF:4.166, 10 | seg2.ts 11 | #EXTINF:4.166, 12 | seg3.ts 13 | #EXTINF:4.166, 14 | seg4.ts 15 | #EXTINF:4.166, 16 | seg5.ts 17 | #EXTINF:4.166, 18 | seg6.ts 19 | #EXTINF:1.404, 20 | seg7.ts 21 | #EXT-X-ENDLIST 22 | -------------------------------------------------------------------------------- /test/resources/with-sub-manifest/var386000/playlist.m3u8: -------------------------------------------------------------------------------- 1 | #EXTM3U 2 | #EXT-X-TARGETDURATION:4 3 | #EXT-X-MEDIA-SEQUENCE:0 4 | #EXT-X-VERSION:3 5 | #EXTINF:4.166, 6 | seg0.ts 7 | #EXTINF:4.166, 8 | seg1.ts 9 | #EXTINF:4.166, 10 | seg2.ts 11 | #EXTINF:4.166, 12 | seg3.ts 13 | #EXTINF:4.166, 14 | seg4.ts 15 | #EXTINF:4.166, 16 | seg5.ts 17 | #EXTINF:4.166, 18 | seg6.ts 19 | #EXTINF:1.094, 20 | seg7.ts 21 | #EXT-X-ENDLIST 22 | -------------------------------------------------------------------------------- /test/resources/with-sub-manifest/var500000/playlist.m3u8: -------------------------------------------------------------------------------- 1 | #EXTM3U 2 | #EXT-X-TARGETDURATION:4 3 | #EXT-X-MEDIA-SEQUENCE:0 4 | #EXT-X-VERSION:3 5 | #EXTINF:4.166, 6 | seg0.ts 7 | #EXTINF:4.166, 8 | seg1.ts 9 | #EXTINF:4.166, 10 | seg2.ts 11 | #EXTINF:4.166, 12 | seg3.ts 13 | #EXTINF:4.166, 14 | seg4.ts 15 | #EXTINF:4.166, 16 | seg5.ts 17 | #EXTINF:4.166, 18 | seg6.ts 19 | #EXTINF:1.013, 20 | seg7.ts 21 | #EXT-X-ENDLIST 22 | -------------------------------------------------------------------------------- /test/resources/with-sub-manifest/playlist.m3u8: -------------------------------------------------------------------------------- 1 | #EXTM3U 2 | #EXT-X-STREAM-INF:PROGRAM-ID=2000,BANDWIDTH=256000,CODECS="avc1.64001e,mp4a.40.2",RESOLUTION=720x576 3 | var256000/playlist.m3u8 4 | #EXT-X-STREAM-INF:PROGRAM-ID=2000,BANDWIDTH=386000,CODECS="avc1.64001e,mp4a.40.2",RESOLUTION=720x576 5 | var386000/playlist.m3u8 6 | #EXT-X-STREAM-INF:PROGRAM-ID=2000,BANDWIDTH=500000,CODECS="avc1.64001e,mp4a.40.2",RESOLUTION=720x576 7 | var500000/playlist.m3u8 8 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # OS 2 | Thumbs.db 3 | ehthumbs.db 4 | Desktop.ini 5 | .DS_Store 6 | ._* 7 | 8 | # Editors 9 | *~ 10 | *.swp 11 | *.tmproj 12 | *.tmproject 13 | *.sublime-* 14 | .idea/ 15 | .project/ 16 | .settings/ 17 | .vscode/ 18 | 19 | # Logs 20 | logs 21 | *.log 22 | npm-debug.log* 23 | 24 | # Dependency directories 25 | bower_components/ 26 | node_modules/ 27 | hls-fetcher/ 28 | 29 | # Build-related directories 30 | dist/ 31 | docs/api/ 32 | test/dist/ 33 | .eslintcache 34 | .yo-rc.json 35 | -------------------------------------------------------------------------------- /test/resources/duplicate-manifests/manifest0/rendition.m3u8: -------------------------------------------------------------------------------- 1 | #EXTM3U 2 | #EXT-X-VERSION:3 3 | #EXT-X-PLAYLIST-TYPE:VOD 4 | #EXT-X-MEDIA-SEQUENCE:0 5 | #EXT-X-TARGETDURATION:10 6 | #EXTINF:9.985, 7 | segment0.ts 8 | #EXTINF:9.985, 9 | segment1.ts 10 | #EXTINF:9.985, 11 | segment2.ts 12 | #EXTINF:9.985, 13 | segment3.ts 14 | #EXTINF:9.985, 15 | segment4.ts 16 | #EXTINF:9.985, 17 | segment5.ts 18 | #EXTINF:9.985, 19 | segment6.ts 20 | #EXTINF:9.985, 21 | segment7.ts 22 | #EXTINF:6.177, 23 | segment8.ts 24 | #EXT-X-ENDLIST 25 | -------------------------------------------------------------------------------- /test/resources/duplicate-manifests/manifest1/rendition.m3u8: -------------------------------------------------------------------------------- 1 | #EXTM3U 2 | #EXT-X-VERSION:3 3 | #EXT-X-PLAYLIST-TYPE:VOD 4 | #EXT-X-MEDIA-SEQUENCE:0 5 | #EXT-X-TARGETDURATION:10 6 | #EXTINF:9.985, 7 | segment0.ts 8 | #EXTINF:9.985, 9 | segment1.ts 10 | #EXTINF:9.985, 11 | segment2.ts 12 | #EXTINF:9.985, 13 | segment3.ts 14 | #EXTINF:9.985, 15 | segment4.ts 16 | #EXTINF:9.985, 17 | segment5.ts 18 | #EXTINF:9.985, 19 | segment6.ts 20 | #EXTINF:9.985, 21 | segment7.ts 22 | #EXTINF:6.177, 23 | segment8.ts 24 | #EXT-X-ENDLIST 25 | -------------------------------------------------------------------------------- /test/resources/duplicate-manifests/manifest2/rendition.m3u8: -------------------------------------------------------------------------------- 1 | #EXTM3U 2 | #EXT-X-VERSION:3 3 | #EXT-X-PLAYLIST-TYPE:VOD 4 | #EXT-X-MEDIA-SEQUENCE:0 5 | #EXT-X-TARGETDURATION:10 6 | #EXTINF:9.985, 7 | segment0.ts 8 | #EXTINF:9.985, 9 | segment1.ts 10 | #EXTINF:9.985, 11 | segment2.ts 12 | #EXTINF:9.985, 13 | segment3.ts 14 | #EXTINF:9.985, 15 | segment4.ts 16 | #EXTINF:9.985, 17 | segment5.ts 18 | #EXTINF:9.985, 19 | segment6.ts 20 | #EXTINF:9.985, 21 | segment7.ts 22 | #EXTINF:6.177, 23 | segment8.ts 24 | #EXT-X-ENDLIST 25 | -------------------------------------------------------------------------------- /test/resources/simple.m3u8: -------------------------------------------------------------------------------- 1 | #EXTM3U 2 | #EXT-X-PLAYLIST-TYPE:VOD 3 | #EXT-X-TARGETDURATION:8 4 | #EXT-X-VERSION:6 5 | #EXT-X-INDEPENDENT-SEGMENTS 6 | #EXT-X-MEDIA-SEQUENCE:0 7 | #EXTINF:2.000, 8 | chunk_0.ts 9 | #EXTINF:2.000, 10 | chunk_1.ts 11 | #EXTINF:1.999, 12 | chunk_2.ts 13 | #EXTINF:2.000, 14 | chunk_3.ts 15 | #EXTINF:1.999, 16 | chunk_4.ts 17 | #EXTINF:2.000, 18 | chunk_5.ts 19 | #EXTINF:2.000, 20 | chunk_6.ts 21 | #EXTINF:1.999, 22 | chunk_7.ts 23 | #EXTINF:2.000, 24 | chunk_8.ts 25 | #EXTINF:1.999, 26 | chunk_9.ts 27 | #EXTINF:2.000, 28 | chunk_10.ts 29 | -------------------------------------------------------------------------------- /test/resources/fmp4.m3u8: -------------------------------------------------------------------------------- 1 | #EXTM3U 2 | #EXT-X-PLAYLIST-TYPE:VOD 3 | #EXT-X-TARGETDURATION:8 4 | #EXT-X-VERSION:6 5 | #EXT-X-INDEPENDENT-SEGMENTS 6 | #EXT-X-MEDIA-SEQUENCE:0 7 | #EXT-X-MAP:URI="init.mp4" 8 | #EXTINF:2.000, 9 | chunk_0.m4s 10 | #EXTINF:2.000, 11 | chunk_1.m4s 12 | #EXTINF:1.999, 13 | chunk_2.ts 14 | #EXTINF:2.000, 15 | chunk_3.ts 16 | #EXTINF:1.999, 17 | chunk_4.m4s 18 | #EXTINF:2.000, 19 | chunk_5.m4s 20 | #EXTINF:2.000, 21 | chunk_6.ts 22 | #EXTINF:1.999, 23 | chunk_7.ts 24 | #EXTINF:2.000, 25 | chunk_8.m4s 26 | #EXTINF:1.999, 27 | chunk_9.ts 28 | #EXTINF:2.000, 29 | chunk_10.ts 30 | -------------------------------------------------------------------------------- /src/index.js: -------------------------------------------------------------------------------- 1 | /* eslint-disable no-console */ 2 | const WalkManifest = require('./walk-manifest'); 3 | const WriteData = require('./write-data'); 4 | 5 | const main = function(options) { 6 | console.log('Gathering Manifest data...'); 7 | const settings = {decrypt: options.decrypt, basedir: options.output, uri: options.input}; 8 | 9 | return WalkManifest(settings) 10 | .then(function(resources) { 11 | console.log('Downloading additional data...'); 12 | return WriteData(options.decrypt, options.concurrency, resources); 13 | }); 14 | }; 15 | 16 | module.exports = main; 17 | module.exports.WalkManifest = WalkManifest; 18 | -------------------------------------------------------------------------------- /test/resources/duplicate-manifests/master.m3u8: -------------------------------------------------------------------------------- 1 | #EXTM3U 2 | #EXT-X-VERSION:4 3 | #EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="audio-0",NAME="en",DEFAULT=YES,AUTOSELECT=YES,LANGUAGE="en",URI="manifest0/rendition.m3u8" 4 | #EXT-X-STREAM-INF:PROGRAM-ID=0,BANDWIDTH=70400,CODECS="mp4a.40.2",AUDIO="audio-0",CLOSED-CAPTIONS=NONE 5 | manifest0/rendition.m3u8 6 | #EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="audio-1",NAME="en",DEFAULT=YES,AUTOSELECT=YES,LANGUAGE="en",URI="manifest1/rendition.m3u8" 7 | #EXT-X-STREAM-INF:PROGRAM-ID=0,BANDWIDTH=105600,CODECS="mp4a.40.2",AUDIO="audio-1",CLOSED-CAPTIONS=NONE 8 | manifest1/rendition.m3u8 9 | #EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="audio-2",NAME="en",DEFAULT=YES,AUTOSELECT=YES,LANGUAGE="en",URI="manifest2/rendition.m3u8" 10 | #EXT-X-STREAM-INF:PROGRAM-ID=0,BANDWIDTH=139700,CODECS="mp4a.40.2",AUDIO="audio-2",CLOSED-CAPTIONS=NONE 11 | manifest2/rendition.m3u8 12 | -------------------------------------------------------------------------------- /test/resources/path-query.m3u8: -------------------------------------------------------------------------------- 1 | #EXTM3U 2 | #EXT-X-VERSION:3 3 | #EXT-X-PLAYLIST-TYPE:VOD 4 | #EXT-X-MEDIA-SEQUENCE:0 5 | #EXT-X-TARGETDURATION:10 6 | #EXTINF:9.985, 7 | https://example.com/segment0.ts?foo=/do-not/include-me/in-output 8 | #EXTINF:9.985, 9 | https://example.com/segment1.ts?foo=/do-not/include-me/in-output 10 | #EXTINF:9.985, 11 | https://example.com/segment2.ts?foo=/do-not/include-me/in-output 12 | #EXTINF:9.985, 13 | https://example.com/segment3.ts?foo=/do-not/include-me/in-output 14 | #EXTINF:9.985, 15 | https://example.com/segment4.ts?foo=/do-not/include-me/in-output 16 | #EXTINF:9.985, 17 | https://example.com/segment5.ts?foo=/do-not/include-me/in-output 18 | #EXTINF:9.985, 19 | https://example.com/segment6.ts?foo=/do-not/include-me/in-output 20 | #EXTINF:9.985, 21 | https://example.com/segment7.ts?foo=/do-not/include-me/in-output 22 | #EXTINF:6.177, 23 | https://example.com/segment8.ts?foo=/do-not/include-me/in-output 24 | #EXT-X-ENDLIST 25 | -------------------------------------------------------------------------------- /src/utils.js: -------------------------------------------------------------------------------- 1 | const url = require('url'); 2 | const path = require('path'); 3 | const fs = require('fs'); 4 | 5 | const Utils = { 6 | joinUri(absolute, relative) { 7 | const parse = url.parse(absolute); 8 | 9 | parse.pathname = path.join(parse.pathname, relative); 10 | return url.format(parse); 11 | }, 12 | getUriPath(uri, base) { 13 | base = base || ''; 14 | const parse = url.parse(uri); 15 | 16 | return path.relative('.', path.join(base, parse.pathname)); 17 | }, 18 | isAbsolute(uri) { 19 | const parsed = url.parse(uri); 20 | 21 | if (parsed.protocol) { 22 | return true; 23 | } 24 | return false; 25 | }, 26 | fileExists(file) { 27 | try { 28 | return fs.statSync(file).isFile(); 29 | } catch (e) { 30 | return false; 31 | } 32 | }, 33 | localize(parentUri, childUri) { 34 | return path.join(path.basename(parentUri, path.extname(parentUri)), path.basename(childUri)); 35 | } 36 | 37 | }; 38 | 39 | module.exports = Utils; 40 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright (c) brandonocasey 2 | 3 | Permission is hereby granted, free of charge, to any person obtaining a copy 4 | of this software and associated documentation files (the "Software"), to deal 5 | in the Software without restriction, including without limitation the rights 6 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 7 | copies of the Software, and to permit persons to whom the Software is 8 | furnished to do so, subject to the following conditions: 9 | 10 | The above copyright notice and this permission notice shall be included in all 11 | copies or substantial portions of the Software. 12 | 13 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 14 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 15 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 16 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 17 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 18 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 19 | SOFTWARE. 20 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # CONTRIBUTING 2 | 3 | We welcome contributions from everyone! 4 | 5 | ## Getting Started 6 | 7 | Make sure you have Node.js 4.8 or higher and npm installed. 8 | 9 | 1. Fork this repository and clone your fork 10 | 1. Install dependencies: `npm install` 11 | 1. Run a development server: `npm start` 12 | 13 | ### Making Changes 14 | 15 | Refer to the [video.js plugin conventions][conventions] for more detail on best practices and tooling for video.js plugin authorship. 16 | 17 | When you've made your changes, push your commit(s) to your fork and issue a pull request against the original repository. 18 | 19 | ### Running Tests 20 | 21 | Testing is a crucial part of any software project. For all but the most trivial changes (typos, etc) test cases are expected. Tests are run in actual browsers using [Karma][karma]. 22 | 23 | - In all available and supported browsers: `npm test` 24 | - In a specific browser: `npm run test:chrome`, `npm run test:firefox`, etc. 25 | - While development server is running (`npm start`), navigate to [`http://localhost:9999/test/`][local] 26 | 27 | 28 | [karma]: http://karma-runner.github.io/ 29 | [local]: http://localhost:9999/test/ 30 | [conventions]: https://github.com/videojs/generator-videojs-plugin/blob/master/docs/conventions.md 31 | -------------------------------------------------------------------------------- /src/cli.js: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env node 2 | /* eslint-disable no-console */ 3 | 4 | const path = require('path'); 5 | const start = require('./index'); 6 | const pessimist = require('pessimist') 7 | .usage('Fetch and save the contents of an HLS playlist locally.\nUsage: $0 ') 8 | .alias('i', 'input') 9 | .demand('i') 10 | .describe('i', 'uri to m3u8 (required)') 11 | .alias('o', 'output') 12 | .default('o', './hls-fetcher') 13 | .describe('o', "output path (default:'./hls-fetcher')") 14 | .alias('c', 'concurrency') 15 | .default('c', Infinity) 16 | .describe('c', 'number of simultaneous fetches (default: Infinity)') 17 | .alias('d', 'decrypt') 18 | .default('d', false) 19 | .describe('d', 'decrypt and remove enryption from manifest (default: false)') 20 | .argv; 21 | 22 | // Make output path 23 | const output = path.resolve(pessimist.o); 24 | const startTime = Date.now(); 25 | const options = { 26 | input: pessimist.i, 27 | output, 28 | concurrency: pessimist.c, 29 | decrypt: pessimist.d 30 | }; 31 | 32 | start(options).then(function() { 33 | const timeTaken = ((Date.now() - startTime) / 1000).toFixed(2); 34 | 35 | console.log('Operation completed successfully in', timeTaken, 'seconds.'); 36 | process.exit(0); 37 | }).catch(function(error) { 38 | console.error('ERROR', error); 39 | process.exit(1); 40 | }); 41 | -------------------------------------------------------------------------------- /test/resources/tilde-query-param.m3u8: -------------------------------------------------------------------------------- 1 | #EXTM3U 2 | #EXT-X-VERSION:3 3 | #EXT-X-PLAYLIST-TYPE:VOD 4 | #EXT-X-MEDIA-SEQUENCE:0 5 | #EXT-X-TARGETDURATION:10 6 | #EXTINF:9.985, 7 | https://example.com/segment0.ts?foo_token=exp=4444444444~pool=/123456788787387878728/*~nice=some-uuid-my-dude 8 | #EXTINF:9.985, 9 | https://example.com/segment1.ts?foo_token=exp=4444444444~pool=/123456788787387878728/*~nice=some-uuid-my-dude 10 | #EXTINF:9.985, 11 | https://example.com/segment2.ts?foo_token=exp=4444444444~pool=/123456788787387878728/*~nice=some-uuid-my-dude 12 | #EXTINF:9.985, 13 | https://example.com/segment3.ts?foo_token=exp=4444444444~pool=/123456788787387878728/*~nice=some-uuid-my-dude 14 | #EXTINF:9.985, 15 | https://example.com/segment4.ts?foo_token=exp=4444444444~pool=/123456788787387878728/*~nice=some-uuid-my-dude 16 | #EXTINF:9.985, 17 | https://example.com/segment5.ts?foo_token=exp=4444444444~pool=/123456788787387878728/*~nice=some-uuid-my-dude 18 | #EXTINF:9.985, 19 | https://example.com/segment6.ts?foo_token=exp=4444444444~pool=/123456788787387878728/*~nice=some-uuid-my-dude 20 | #EXTINF:9.985, 21 | https://example.com/segment7.ts?foo_token=exp=4444444444~pool=/123456788787387878728/*~nice=some-uuid-my-dude 22 | #EXTINF:6.177, 23 | https://example.com/segment8.ts?foo_token=exp=4444444444~pool=/123456788787387878728/*~nice=some-uuid-my-dude 24 | #EXT-X-ENDLIST 25 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | 2 | # HLS-FETCHER 3 | 4 | [![Build Status](https://travis-ci.org/videojs/hls-fetcher.svg?branch=master)](https://travis-ci.org/videojs/hls-fetcher) 5 | [![Greenkeeper badge](https://badges.greenkeeper.io/videojs/hls-fetcher.svg)](https://greenkeeper.io/) 6 | [![Slack Status](http://slack.videojs.com/badge.svg)](http://slack.videojs.com) 7 | 8 | [![NPM](https://nodei.co/npm/hls-fetcher.png?downloads=true&downloadRank=true)](https://nodei.co/npm/hls-fetcher/) 9 | 10 | Maintenance Status: Stable 11 | 12 | A simple CLI tool to fetch an entire hls manifest and it's segments and save it all locally. 13 | 14 | 15 | 16 | **Table of Contents** *generated with [DocToc](https://github.com/thlorenz/doctoc)* 17 | 18 | - [Installation](#installation) 19 | - [Command Line Usage](#command-line-usage) 20 | 21 | 22 | 23 | ## Installation 24 | 25 | ``` bash 26 | $ [sudo] npm install hls-fetcher -g 27 | ``` 28 | 29 | ### Command Line Usage 30 | 31 | **Example** 32 | ``` 33 | hls-fetcher -i http://example.com/hls_manifest.m3u8 34 | ``` 35 | 36 | **Options** 37 | ``` 38 | $ hls-fetcher 39 | Usage: hls-fetcher 40 | 41 | Options: 42 | -i, --input uri to m3u8 (required) 43 | -o, --output output path (default:'./') 44 | -c, --concurrency number of simultaneous fetches (default: 5) 45 | ``` 46 | -------------------------------------------------------------------------------- /test/resources/dash.mpd: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | -------------------------------------------------------------------------------- /test/resources/sidx.mpd: -------------------------------------------------------------------------------- 1 | 2 | 3 | ./ 4 | 5 | 6 | 7 | German_Forest_SHORT_1v0-avc1.42c01e-68s-848x476-h264-1500000bps_seg.mp4 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | German_Forest_Short_Poem_english-en-68s-2-lc-128000bps_seg.mp4 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | German_Forest_Short_Poem_german-de-68s-2-lc-128000bps_seg.mp4 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | -------------------------------------------------------------------------------- /package.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "hls-fetcher", 3 | "version": "2.3.0", 4 | "description": "Fetch HLS segments from an m3u8 playlist", 5 | "main": "src/index.js", 6 | "scripts": { 7 | "docs:toc": "doctoc README.md", 8 | "lint": "vjsstandard", 9 | "pretest": "npm run lint", 10 | "start": "npm run test -- --watch", 11 | "test": "NODE_ENV=test mocha test/unit", 12 | "update-changelog": "conventional-changelog -p videojs -i CHANGELOG.md -s", 13 | "preversion": "npm test", 14 | "version": "is-prerelease || npm run update-changelog && git add CHANGELOG.md", 15 | "prepublishOnly": "vjsverify --skip-es-check" 16 | }, 17 | "bin": { 18 | "hls-fetcher": "./src/cli.js" 19 | }, 20 | "repository": { 21 | "type": "git", 22 | "url": "git+https://github.com/videojs/hls-fetcher.git" 23 | }, 24 | "contributors": [ 25 | "Jon-Carlos Rivera (http://jon-carlos.com/)", 26 | "Brandon Casey (https://github.com/brandonocasey)" 27 | ], 28 | "license": "MIT", 29 | "bugs": { 30 | "url": "https://github.com/videojs/hls-fetcher/issues" 31 | }, 32 | "homepage": "https://github.com/videojs/hls-fetcher#readme", 33 | "dependencies": { 34 | "aes-decrypter": "^3.1.2", 35 | "bluebird": "^3.7.2", 36 | "filenamify": "^4.3.0", 37 | "m3u8-parser": "^4.7.0", 38 | "mkdirp": "^1.0.4", 39 | "mpd-parser": "^0.16.0", 40 | "pessimist": "^0.3.5", 41 | "request": "^2.87.0", 42 | "requestretry": "^5.0.0" 43 | }, 44 | "devDependencies": { 45 | "conventional-changelog-cli": "^2.1.1", 46 | "conventional-changelog-videojs": "^3.0.0", 47 | "doctoc": "^2.0.0", 48 | "husky": "^6.0.0", 49 | "lint-staged": "^11.0.0", 50 | "mocha": "^8.4.0", 51 | "nock": "^13.0.11", 52 | "not-prerelease": "^1.0.1", 53 | "npm-merge-driver-install": "^2.0.1", 54 | "videojs-generator-verify": "~3.0.3", 55 | "videojs-standard": "^8.0.4" 56 | }, 57 | "generator-videojs-plugin": { 58 | "version": "7.3.2" 59 | }, 60 | "browserslist": [ 61 | "defaults", 62 | "ie 11" 63 | ], 64 | "vjsstandard": { 65 | "ignore": [ 66 | "dist", 67 | "docs", 68 | "test/dist" 69 | ] 70 | }, 71 | "files": [ 72 | "CONTRIBUTING.md", 73 | "docs/", 74 | "src/", 75 | "test/" 76 | ], 77 | "husky": { 78 | "hooks": { 79 | "pre-commit": "lint-staged" 80 | } 81 | }, 82 | "lint-staged": { 83 | "*.js": [ 84 | "vjsstandard --fix", 85 | "git add" 86 | ], 87 | "README.md": [ 88 | "npm run docs:toc", 89 | "git add" 90 | ] 91 | } 92 | } 93 | -------------------------------------------------------------------------------- /test/resources/long-path.m3u8: -------------------------------------------------------------------------------- 1 | #EXTM3U 2 | #EXT-X-PLAYLIST-TYPE:VOD 3 | #EXT-X-TARGETDURATION:8 4 | #EXT-X-VERSION:6 5 | #EXT-X-INDEPENDENT-SEGMENTS 6 | #EXT-X-MEDIA-SEQUENCE:0 7 | chunk_0.ts?token=OYK%40%3F%2BrjKeGaskhhsmf8E7aoUftbIfXZo0ucm9qebBFsXG5yepliwyfwKIf4zTGqMocHsTJePF91V17ZJ4h8A7mS3ysNSOcjKQT2oAVJmfD3vJIwdDfD0mZqlA9jQOOWwXnLy0UQtn9V2eYXlNAdIc9w8yDFPxDp509vJC9lurHWewql6eg22drnACC2rEDOXYit0I3CqOaVRvLSIqG0quUda5CoDn7vmaCBlvsBA0MEoWQSG0TEmDtdTT6DP8vUCC7BTtr9Zaxo5l9QYnWyNMzZNszjijCoKq8LsAi95WIo2n9 8 | 9 | #EXTINF:2.000, 10 | /testOYK%40%3F%2BrjKeGaskhhsmf8E7aoUftbIfXZo0ucm9qebBFsXG5yepliwyfwKIf4zTGqMocHsTJePF91V17ZJ4h8A7mS3ysNSOcjKQT2oAVJmfD3vJIwdDfD0mZqlA9jQOOWwXnLy0UQtn9V2eYXlNAdIc9w8yDFPxDp509vJC9lurHWewql6eg22drnACC2rEDOXYit0I3CqOaVRvLSIqG0quUda5CoDn7vmaCBlvsBA0MEoWQSG0TEmDtdTT6DP8vUCC7BTtr9Zaxo5l9QYnWyNMzZNszjijCoKq8LsAi95WIo2n9/chunk_1.ts?token=OYK%40%3F%2BrjKeGaskhhsmf8E7aoUftbIfXZo0ucm9qebBFsXG5yepliwyfwKIf4zTGqMocHsTJePF91V17ZJ4h8A7mS3ysNSOcjKQT2oAVJmfD3vJIwdDfD0mZqlA9jQOOWwXnLy0UQtn9V2eYXlNAdIc9w8yDFPxDp509vJC9lurHWewql6eg22drnACC2rEDOXYit0I3CqOaVRvLSIqG0quUda5CoDn7vmaCBlvsBA0MEoWQSG0TEmDtdTT6DP8vUCC7BTtr9Zaxo5l9QYnWyNMzZNszjijCoKq8LsAi95WIo2n9 11 | 12 | #EXTINF:1.999, 13 | testOYK%40%3F%2BrjKeGaskhhsmf8E7aoUftbIfXZo0ucm9qebBFsXG5yepliwyfwKIf4zTGqMocHsTJePF91V17ZJ4h8A7mS3ysNSOcjKQT2oAVJmfD3vJIwdDfD0mZqlA9jQOOWwXnLy0UQtn9V2eYXlNAdIc9w8yDFPxDp509vJC9lurHWewql6eg22drnACC2rEDOXYit0I3CqOaVRvLSIqG0quUda5CoDn7vmaCBlvsBA0MEoWQSG0TEmDtdTT6DP8vUCC7BTtr9Zaxo5l9QYnWyNMzZNszjijCoKq8LsAi95WIo2n9/testOYK%40%3F%2BrjKeGaskhhsmf8E7aoUftbIfXZo0ucm9qebBFsXG5yepliwyfwKIf4zTGqMocHsTJePF91V17ZJ4h8A7mS3ysNSOcjKQT2oAVJmfD3vJIwdDfD0mZqlA9jQOOWwXnLy0UQtn9V2eYXlNAdIc9w8yDFPxDp509vJC9lurHWewql6eg22drnACC2rEDOXYit0I3CqOaVRvLSIqG0quUda5CoDn7vmaCBlvsBA0MEoWQSG0TEmDtdTT6DP8vUCC7BTtr9Zaxo5l9QYnWyNMzZNszjijCoKq8LsAi95WIo2n9/chunk_2.ts?token=OYK%40%3F%2BrjKeGaskhhsmf8E7aoUftbIfXZo0ucm9qebBFsXG5yepliwyfwKIf4zTGqMocHsTJePF91V17ZJ4h8A7mS3ysNSOcjKQT2oAVJmfD3vJIwdDfD0mZqlA9jQOOWwXnLy0UQtn9V2eYXlNAdIc9w8yDFPxDp509vJC9lurHWewql6eg22drnACC2rEDOXYit0I3CqOaVRvLSIqG0quUda5CoDn7vmaCBlvsBA0MEoWQSG0TEmDtdTT6DP8vUCC7BTtr9Zaxo5l9QYnWyNMzZNszjijCoKq8LsAi95WIo2n9 14 | #EXTINF:2.000, 15 | http://manifest-list-test.com/testOYK%40%3F%2BrjKeGaskhhsmf8E7aoUftbIfXZo0ucm9qebBFsXG5yepliwyfwKIf4zTGqMocHsTJePF91V17ZJ4h8A7mS3ysNSOcjKQT2oAVJmfD3vJIwdDfD0mZqlA9jQOOWwXnLy0UQtn9V2eYXlNAdIc9w8yDFPxDp509vJC9lurHWewql6eg22drnACC2rEDOXYit0I3CqOaVRvLSIqG0quUda5CoDn7vmaCBlvsBA0MEoWQSG0TEmDtdTT6DP8vUCC7BTtr9Zaxo5l9QYnWyNMzZNszjijCoKq8LsAi95WIo2n9/chunk_3.ts?token=OYK%40%3F%2BrjKeGaskhhsmf8E7aoUftbIfXZo0ucm9qebBFsXG5yepliwyfwKIf4zTGqMocHsTJePF91V17ZJ4h8A7mS3ysNSOcjKQT2oAVJmfD3vJIwdDfD0mZqlA9jQOOWwXnLy0UQtn9V2eYXlNAdIc9w8yDFPxDp509vJC9lurHWewql6eg22drnACC2rEDOXYit0I3CqOaVRvLSIqG0quUda5CoDn7vmaCBlvsBA0MEoWQSG0TEmDtdTT6DP8vUCC7BTtr9Zaxo5l9QYnWyNMzZNszjijCoKq8LsAi95WIo2n9 16 | #EXTINF:2.000, 17 | 18 | -------------------------------------------------------------------------------- /src/write-data.js: -------------------------------------------------------------------------------- 1 | /* eslint-disable no-console */ 2 | const Promise = require('bluebird'); 3 | const mkdirp = require('mkdirp'); 4 | const request = require('requestretry'); 5 | const fs = Promise.promisifyAll(require('fs')); 6 | const AesDecrypter = require('aes-decrypter').Decrypter; 7 | const path = require('path'); 8 | 9 | const writeFile = function(file, content) { 10 | return mkdirp(path.dirname(file)).then(function() { 11 | return fs.writeFileAsync(file, content); 12 | }).then(function() { 13 | console.log('Finished: ' + path.relative('.', file)); 14 | }); 15 | }; 16 | 17 | const requestFile = function(uri) { 18 | const options = { 19 | uri, 20 | // 60 seconds timeout 21 | timeout: 60000, 22 | // treat all responses as a buffer 23 | encoding: null, 24 | // retry 1s after on failure 25 | retryDelay: 1000 26 | }; 27 | 28 | return new Promise(function(resolve, reject) { 29 | request(options, function(err, response, body) { 30 | if (err) { 31 | return reject(err); 32 | } 33 | return resolve(body); 34 | }); 35 | }); 36 | }; 37 | 38 | const toUint8Array = function(nodeBuffer) { 39 | return new Uint8Array(nodeBuffer.buffer, nodeBuffer.byteOffset, nodeBuffer.byteLength / Uint8Array.BYTES_PER_ELEMENT); 40 | }; 41 | 42 | const decryptFile = function(content, encryption) { 43 | return new Promise(function(resolve, reject) { 44 | /* eslint-disable no-new */ 45 | // this is how you use it, its kind of bad but :shrug: 46 | new AesDecrypter(toUint8Array(content), encryption.bytes, encryption.iv, function(err, bytes) { 47 | if (err) { 48 | return reject(err); 49 | } 50 | return resolve(Buffer.from(bytes)); 51 | }); 52 | /* eslint-enable no-new */ 53 | }); 54 | }; 55 | 56 | const WriteData = function(decrypt, concurrency, resources) { 57 | const inProgress = []; 58 | const operations = []; 59 | 60 | resources.forEach(function(r) { 61 | if (r.content) { 62 | operations.push(function() { 63 | return writeFile(r.file, r.content); 64 | }); 65 | } else if (r.uri && r.key && decrypt) { 66 | operations.push(function() { 67 | return requestFile(r.uri).then(function(content) { 68 | return decryptFile(content, r.key); 69 | }).then(function(content) { 70 | return writeFile(r.file, content); 71 | }); 72 | }); 73 | } else if (r.uri && inProgress.indexOf(r.uri) === -1) { 74 | operations.push(function() { 75 | return requestFile(r.uri).then(function(content) { 76 | return writeFile(r.file, content); 77 | }); 78 | }); 79 | inProgress.push(r.uri); 80 | } 81 | }); 82 | 83 | return Promise.map(operations, function(o) { 84 | return Promise.join(o()); 85 | }, {concurrency}).all(function(o) { 86 | console.log('DONE!'); 87 | return Promise.resolve(); 88 | }); 89 | }; 90 | 91 | module.exports = WriteData; 92 | -------------------------------------------------------------------------------- /CHANGELOG.md: -------------------------------------------------------------------------------- 1 | 2 | # [2.3.0](https://github.com/videojs/hls-fetcher/compare/v2.2.2...v2.3.0) (2021-05-19) 3 | 4 | ### Bug Fixes 5 | 6 | * replace relative uri before converting to absolute ([#44](https://github.com/videojs/hls-fetcher/issues/44)) ([1b08ca6](https://github.com/videojs/hls-fetcher/commit/1b08ca6)) 7 | 8 | ### Chores 9 | 10 | * update packages ([3010f4c](https://github.com/videojs/hls-fetcher/commit/3010f4c)) 11 | 12 | 13 | ## [2.2.2](https://github.com/videojs/hls-fetcher/compare/v2.2.1...v2.2.2) (2020-12-02) 14 | 15 | ### Bug Fixes 16 | 17 | * better errors, correctly join uri with relative hash/query ([4024a13](https://github.com/videojs/hls-fetcher/commit/4024a13)) 18 | * handle tilde/paths in the query string, cleanup tests ([#43](https://github.com/videojs/hls-fetcher/issues/43)) ([4a528ce](https://github.com/videojs/hls-fetcher/commit/4a528ce)) 19 | 20 | 21 | ## [2.2.1](https://github.com/videojs/hls-fetcher/compare/v2.2.0...v2.2.1) (2019-09-06) 22 | 23 | ### Bug Fixes 24 | 25 | * windows paths and test console error ([#39](https://github.com/videojs/hls-fetcher/issues/39)) ([ae14253](https://github.com/videojs/hls-fetcher/commit/ae14253)) 26 | 27 | 28 | # [2.2.0](https://github.com/videojs/hls-fetcher/compare/v2.1.0...v2.2.0) (2019-09-05) 29 | 30 | ### Features 31 | 32 | * fetch mpd/dash playlists ([#35](https://github.com/videojs/hls-fetcher/issues/35)) ([f6b4101](https://github.com/videojs/hls-fetcher/commit/f6b4101)) 33 | 34 | ### Bug Fixes 35 | 36 | * Use filenamify to sanitize filenames, and cut to 255 chars ([#36](https://github.com/videojs/hls-fetcher/issues/36)) ([e8114da](https://github.com/videojs/hls-fetcher/commit/e8114da)) 37 | * Fix hls decryption 38 | 39 | 40 | # [2.1.0](https://github.com/videojs/hls-fetcher/compare/v2.0.2...v2.1.0) (2019-06-11) 41 | 42 | ### Features 43 | 44 | * support init segment downloading ([#33](https://github.com/videojs/hls-fetcher/issues/33)) ([36ccd60](https://github.com/videojs/hls-fetcher/commit/36ccd60)) 45 | 46 | ### Bug Fixes 47 | 48 | * path resolution for relative urls using leading slash ([#30](https://github.com/videojs/hls-fetcher/issues/30)) ([f9b1da7](https://github.com/videojs/hls-fetcher/commit/f9b1da7)) 49 | * Remove the postinstall script to prevent install issues ([#28](https://github.com/videojs/hls-fetcher/issues/28)) ([2de6167](https://github.com/videojs/hls-fetcher/commit/2de6167)) 50 | 51 | ### Chores 52 | 53 | * Update tooling to fit new plugin generator v7 standards ([1b94e31](https://github.com/videojs/hls-fetcher/commit/1b94e31)) 54 | 55 | ### Documentation 56 | 57 | * removing maintainers section ([#29](https://github.com/videojs/hls-fetcher/issues/29)) ([1ff905d](https://github.com/videojs/hls-fetcher/commit/1ff905d)) 58 | * **README:** Remove videojs-errors description ([#31](https://github.com/videojs/hls-fetcher/issues/31)) ([5b9596e](https://github.com/videojs/hls-fetcher/commit/5b9596e)) 59 | 60 | -------------------------------------------------------------------------------- /src/walk-manifest.js: -------------------------------------------------------------------------------- 1 | /* eslint-disable no-console */ 2 | const m3u8 = require('m3u8-parser'); 3 | const mpd = require('mpd-parser'); 4 | const request = require('requestretry'); 5 | const url = require('url'); 6 | const path = require('path'); 7 | const querystring = require('querystring'); 8 | const filenamify = require('filenamify'); 9 | 10 | // replace invalid http/fs characters with valid representations 11 | const fsSanitize = function(filepath) { 12 | return path.normalize(filepath) 13 | // split on \, \\, or / 14 | .split(/\\\\|\\|\//) 15 | // max filepath is 255 on OSX/linux, and 260 on windows, 255 is fine for both 16 | // replace invalid characters with nothing 17 | .map((p) => filenamify(querystring.unescape(p), {replacement: '', maxLength: 255})) 18 | // join on OS specific path seperator 19 | .join(path.sep); 20 | }; 21 | 22 | const urlBasename = function(uri) { 23 | const parsed = url.parse(uri); 24 | const pathname = parsed.pathname || parsed.path.replace(parsed.query || '', ''); 25 | const query = (parsed.query || '').split(/\\\\|\\|\//).join(''); 26 | const basename = path.basename(pathname) + query; 27 | 28 | return fsSanitize(basename); 29 | }; 30 | 31 | const joinURI = function(absolute, relative) { 32 | const abs = url.parse(absolute); 33 | const rel = url.parse(relative); 34 | 35 | abs.pathname = path.resolve(abs.pathname, rel.pathname); 36 | 37 | abs.query = rel.query; 38 | abs.hash = rel.hash; 39 | 40 | return url.format(abs); 41 | }; 42 | 43 | const isAbsolute = function(uri) { 44 | const parsed = url.parse(uri); 45 | 46 | if (parsed.protocol) { 47 | return true; 48 | } 49 | return false; 50 | }; 51 | 52 | const mediaGroupPlaylists = function(mediaGroups) { 53 | const playlists = []; 54 | 55 | ['AUDIO', 'VIDEO', 'CLOSED-CAPTIONS', 'SUBTITLES'].forEach(function(type) { 56 | const mediaGroupType = mediaGroups[type]; 57 | 58 | if (mediaGroupType && !Object.keys(mediaGroupType).length) { 59 | return; 60 | } 61 | 62 | for (const group in mediaGroupType) { 63 | for (const item in mediaGroupType[group]) { 64 | const props = mediaGroupType[group][item]; 65 | 66 | playlists.push(props); 67 | } 68 | } 69 | }); 70 | return playlists; 71 | }; 72 | 73 | const parseM3u8Manifest = function(content) { 74 | const parser = new m3u8.Parser(); 75 | 76 | parser.push(content); 77 | parser.end(); 78 | return parser.manifest; 79 | }; 80 | 81 | const collectPlaylists = function(parsed) { 82 | return [] 83 | .concat(parsed.playlists || []) 84 | .concat(mediaGroupPlaylists(parsed.mediaGroups || {}) || []) 85 | .reduce(function(acc, p) { 86 | acc.push(p); 87 | 88 | if (p.playlists) { 89 | acc = acc.concat(collectPlaylists(p)); 90 | } 91 | return acc; 92 | }, []); 93 | }; 94 | 95 | const parseMpdManifest = function(content, srcUrl) { 96 | const parsedManifestInfo = mpd.inheritAttributes(mpd.stringToMpdXml(content), { 97 | manifestUri: srcUrl 98 | }); 99 | const mpdPlaylists = mpd.toPlaylists(parsedManifestInfo.representationInfo); 100 | 101 | const m3u8Result = mpd.toM3u8(mpdPlaylists); 102 | const m3u8Playlists = collectPlaylists(m3u8Result); 103 | 104 | m3u8Playlists.forEach(function(m) { 105 | const mpdPlaylist = m.attributes && mpdPlaylists.find(function(p) { 106 | return p.attributes.id === m.attributes.NAME; 107 | }); 108 | 109 | if (mpdPlaylist) { 110 | m.dashattributes = mpdPlaylist.attributes; 111 | } 112 | // add sidx to segments 113 | if (m.sidx) { 114 | // fix init segment map if it has one 115 | if (m.sidx.map && !m.sidx.map.uri) { 116 | m.sidx.map.uri = m.sidx.map.resolvedUri; 117 | } 118 | 119 | m.segments.push(m.sidx); 120 | } 121 | }); 122 | 123 | return m3u8Result; 124 | }; 125 | 126 | const parseKey = function(requestOptions, basedir, decrypt, resources, manifest, parent) { 127 | return new Promise(function(resolve, reject) { 128 | 129 | if (!manifest.parsed.segments[0] || !manifest.parsed.segments[0].key) { 130 | return resolve({}); 131 | } 132 | const key = manifest.parsed.segments[0].key; 133 | 134 | let keyUri = key.uri; 135 | 136 | // if we are not decrypting then we just download the key 137 | if (!decrypt) { 138 | // put keys in parent-dir/key-name.key 139 | key.file = basedir; 140 | if (parent) { 141 | key.file = path.dirname(parent.file); 142 | } 143 | key.file = path.join(key.file, urlBasename(key.uri)); 144 | 145 | manifest.content = Buffer.from(manifest.content.toString().replace( 146 | key.uri, 147 | path.relative(path.dirname(manifest.file), key.file) 148 | )); 149 | 150 | if (!isAbsolute(keyUri)) { 151 | keyUri = joinURI(path.dirname(manifest.uri), keyUri); 152 | } 153 | key.uri = keyUri; 154 | resources.push(key); 155 | return resolve(key); 156 | } 157 | 158 | requestOptions.url = keyUri; 159 | requestOptions.encoding = null; 160 | 161 | // get the aes key 162 | request(requestOptions) 163 | .then(function(response) { 164 | if (response.statusCode !== 200) { 165 | const keyError = new Error(response.statusCode + '|' + keyUri); 166 | 167 | console.error(keyError); 168 | return reject(keyError); 169 | } 170 | 171 | const keyContent = response.body; 172 | 173 | key.bytes = new Uint32Array([ 174 | keyContent.readUInt32BE(0), 175 | keyContent.readUInt32BE(4), 176 | keyContent.readUInt32BE(8), 177 | keyContent.readUInt32BE(12) 178 | ]); 179 | 180 | // remove the key from the manifest 181 | manifest.content = Buffer.from(manifest.content.toString().replace( 182 | new RegExp('.*' + key.uri + '.*'), 183 | '' 184 | )); 185 | 186 | resolve(key); 187 | }) 188 | .catch(function(err) { 189 | // TODO: do we even care about key errors; currently we just keep going and ignore them. 190 | const keyError = new Error(err.message + '|' + keyUri); 191 | 192 | console.error(keyError, err); 193 | reject(keyError); 194 | }); 195 | }); 196 | }; 197 | 198 | const walkPlaylist = function(options) { 199 | return new Promise(function(resolve, reject) { 200 | 201 | const { 202 | decrypt, 203 | basedir, 204 | uri, 205 | parent = false, 206 | manifestIndex = 0, 207 | onError = function(err, errUri, resources, res, rej) { 208 | // Avoid adding the top level uri to nested errors 209 | if (err.message.includes('|')) { 210 | rej(err); 211 | } else { 212 | rej(new Error(err.message + '|' + errUri)); 213 | } 214 | }, 215 | visitedUrls = [], 216 | requestTimeout = 1500, 217 | requestRetryMaxAttempts = 5, 218 | dashPlaylist = null, 219 | requestRetryDelay = 5000 220 | } = options; 221 | 222 | let resources = []; 223 | const manifest = {parent}; 224 | 225 | if (uri) { 226 | manifest.uri = uri; 227 | manifest.file = path.join(basedir, urlBasename(uri)); 228 | } 229 | 230 | let existingManifest; 231 | 232 | // if we are not the master playlist 233 | if (dashPlaylist && parent) { 234 | manifest.file = parent.file; 235 | manifest.uri = parent.uri; 236 | existingManifest = visitedUrls[manifest.uri]; 237 | } else if (parent) { 238 | manifest.file = path.join( 239 | basedir, 240 | path.dirname(path.relative(basedir, parent.file)), 241 | 'manifest' + manifestIndex, 242 | path.basename(manifest.file) 243 | ); 244 | 245 | const file = existingManifest && existingManifest.file || manifest.file; 246 | const relativePath = path.relative(path.dirname(parent.file), file); 247 | 248 | // replace original uri in file with new file path 249 | parent.content = Buffer.from(parent.content.toString().replace(manifest.uri, relativePath)); 250 | 251 | // get the real uri of this playlist 252 | if (!isAbsolute(manifest.uri)) { 253 | manifest.uri = joinURI(path.dirname(parent.uri), manifest.uri); 254 | } 255 | 256 | existingManifest = visitedUrls[manifest.uri]; 257 | } 258 | 259 | if (!dashPlaylist && existingManifest) { 260 | console.error(`[WARN] Trying to visit the same uri again; skipping to avoid getting stuck in a cycle: ${manifest.uri}`); 261 | return resolve(resources); 262 | } 263 | visitedUrls[manifest.uri] = manifest; 264 | 265 | let requestPromise; 266 | 267 | if (dashPlaylist) { 268 | requestPromise = Promise.resolve({statusCode: 200}); 269 | } else { 270 | requestPromise = request({ 271 | url: manifest.uri, 272 | timeout: requestTimeout, 273 | maxAttempts: requestRetryMaxAttempts, 274 | retryDelay: requestRetryDelay 275 | }); 276 | } 277 | 278 | requestPromise.then(function(response) { 279 | if (response.statusCode !== 200) { 280 | const manifestError = new Error(response.statusCode + '|' + manifest.uri); 281 | 282 | manifestError.reponse = {body: response.body, headers: response.headers}; 283 | return onError(manifestError, manifest.uri, resources, resolve, reject); 284 | } 285 | // Only push manifest uris that get a non 200 and don't timeout 286 | let dash; 287 | 288 | if (!dashPlaylist) { 289 | resources.push(manifest); 290 | 291 | manifest.content = response.body; 292 | if ((/^application\/dash\+xml/i).test(response.headers['content-type']) || (/^\<\?xml/i).test(response.body)) { 293 | dash = true; 294 | manifest.parsed = parseMpdManifest(manifest.content, manifest.uri); 295 | } else { 296 | manifest.parsed = parseM3u8Manifest(manifest.content); 297 | } 298 | } else { 299 | dash = true; 300 | manifest.parsed = dashPlaylist; 301 | } 302 | 303 | manifest.parsed.segments = manifest.parsed.segments || []; 304 | manifest.parsed.playlists = manifest.parsed.playlists || []; 305 | manifest.parsed.mediaGroups = manifest.parsed.mediaGroups || {}; 306 | 307 | const initSegments = []; 308 | 309 | manifest.parsed.segments.forEach(function(s) { 310 | if (s.map && s.map.uri && !initSegments.some((m) => s.map.uri === m.uri)) { 311 | manifest.parsed.segments.push(s.map); 312 | initSegments.push(s.map); 313 | } 314 | }); 315 | 316 | const playlists = manifest.parsed.playlists.concat(mediaGroupPlaylists(manifest.parsed.mediaGroups)); 317 | 318 | parseKey({ 319 | time: requestTimeout, 320 | maxAttempts: requestRetryMaxAttempts, 321 | retryDelay: requestRetryDelay 322 | }, basedir, decrypt, resources, manifest, parent).then(function(key) { 323 | // SEGMENTS 324 | manifest.parsed.segments.forEach(function(s, i) { 325 | if (!s.uri) { 326 | return; 327 | } 328 | // put segments in manifest-name/segment-name.ts 329 | s.file = path.join(path.dirname(manifest.file), urlBasename(s.uri)); 330 | 331 | if (manifest.content) { 332 | manifest.content = Buffer.from(manifest.content.toString().replace( 333 | s.uri, 334 | path.relative(path.dirname(manifest.file), s.file) 335 | )); 336 | } 337 | 338 | if (!isAbsolute(s.uri)) { 339 | s.uri = joinURI(path.dirname(manifest.uri), s.uri); 340 | } 341 | if (key) { 342 | s.key = key; 343 | s.key.iv = s.key.iv || new Uint32Array([0, 0, 0, manifest.parsed.mediaSequence, i]); 344 | } 345 | resources.push(s); 346 | }); 347 | 348 | // SUB Playlists 349 | const subs = playlists.map(function(p, z) { 350 | if (!p.uri && !dash) { 351 | return Promise.resolve(resources); 352 | } 353 | return walkPlaylist({ 354 | dashPlaylist: dash ? p : null, 355 | decrypt, 356 | basedir, 357 | uri: p.uri, 358 | parent: manifest, 359 | manifestIndex: z, 360 | onError, 361 | visitedUrls, 362 | requestTimeout, 363 | requestRetryMaxAttempts, 364 | requestRetryDelay 365 | }); 366 | }); 367 | 368 | Promise.all(subs).then(function(r) { 369 | const flatten = [].concat.apply([], r); 370 | 371 | resources = resources.concat(flatten); 372 | resolve(resources); 373 | }).catch(function(err) { 374 | onError(err, manifest.uri, resources, resolve, reject); 375 | }); 376 | }); 377 | }) 378 | .catch(function(err) { 379 | onError(err, manifest.uri, resources, resolve, reject); 380 | }); 381 | }); 382 | }; 383 | 384 | module.exports = walkPlaylist; 385 | -------------------------------------------------------------------------------- /test/unit/walk-manifest.spec.js: -------------------------------------------------------------------------------- 1 | /* eslint-env mocha */ 2 | /* eslint-disable max-nested-callbacks */ 3 | const assert = require('assert'); 4 | const nock = require('nock'); 5 | 6 | nock.disableNetConnect(); 7 | nock.enableNetConnect(/localhost/); 8 | const walker = require('../../src/walk-manifest'); 9 | 10 | const TEST_URL = 'http://manifest-list-test.com'; 11 | 12 | const countResources = function({ 13 | extensions, 14 | resources, 15 | file = true, 16 | uri = true, 17 | total = true, 18 | uniqueFile = true 19 | }) { 20 | extensions = extensions || ['m4s', 'm3u8', 'ts', 'mp4', 'mpd', 'm4a']; 21 | const uniqueFiles = []; 22 | const counts = {}; 23 | 24 | if (total) { 25 | counts.total = resources.length; 26 | } 27 | 28 | resources.forEach(function(item) { 29 | if (uniqueFile && uniqueFiles.indexOf(item.file) !== -1) { 30 | return; 31 | } 32 | uniqueFiles.push(item.file); 33 | extensions.forEach(function(ext) { 34 | if (file && !item.file.includes(`.${ext}`)) { 35 | return; 36 | } 37 | 38 | if (uri && !item.uri.includes(`.${ext}`)) { 39 | return; 40 | } 41 | 42 | counts[ext] = counts[ext] || 0; 43 | counts[ext]++; 44 | }); 45 | }); 46 | 47 | return counts; 48 | }; 49 | 50 | const customError = function(errors) { 51 | return function(err, uri, resources, resolve) { 52 | // Avoid adding the top level uri to nested errors 53 | if (err.message.includes('|')) { 54 | errors.push(err); 55 | } else { 56 | errors.push(new Error(err.message + '|' + uri)); 57 | } 58 | 59 | resolve(resources); 60 | }; 61 | }; 62 | 63 | describe('walk-manifest', function() { 64 | describe('walkPlaylist', function() { 65 | /* eslint-disable no-console */ 66 | beforeEach(function() { 67 | this.oldError = console.error; 68 | 69 | console.error = () => {}; 70 | }); 71 | afterEach(function() { 72 | console.error = this.oldError; 73 | if (!nock.isDone()) { 74 | this.test.error(new Error('Not all nock interceptors were used!')); 75 | nock.cleanAll(); 76 | } 77 | }); 78 | /* eslint-enable no-console */ 79 | 80 | it('should return just top level error for bad m3u8 uri', function() { 81 | nock(TEST_URL) 82 | .get('/test.m3u8') 83 | .reply(500); 84 | 85 | const options = {decrypt: false, basedir: '.', uri: TEST_URL + '/test.m3u8', requestRetryMaxAttempts: 0}; 86 | 87 | return walker(options) 88 | .catch(function(err) { 89 | assert.equal(err.message, '500|' + TEST_URL + '/test.m3u8'); 90 | assert(err.reponse); 91 | }); 92 | }); 93 | 94 | it('should return just m3u8 for empty m3u8', function() { 95 | nock(TEST_URL) 96 | .get('/test.m3u8') 97 | .replyWithFile(200, `${process.cwd()}/test/resources/empty.m3u8`); 98 | 99 | const options = {decrypt: false, basedir: '.', uri: TEST_URL + '/test.m3u8', requestRetryMaxAttempts: 0}; 100 | 101 | return walker(options) 102 | .then(function(resources) { 103 | const counts = countResources({resources}); 104 | 105 | assert.deepEqual(counts, { 106 | total: 1, 107 | m3u8: 1 108 | }); 109 | }); 110 | }); 111 | 112 | it('should return just segments for simple m3u8', function() { 113 | nock(TEST_URL) 114 | .get('/test.m3u8') 115 | .replyWithFile(200, `${process.cwd()}/test/resources/simple.m3u8`); 116 | 117 | const options = {decrypt: false, basedir: '.', uri: TEST_URL + '/test.m3u8', requestRetryMaxAttempts: 0}; 118 | 119 | return walker(options) 120 | .then(function(resources) { 121 | const counts = countResources({resources}); 122 | 123 | assert.deepEqual(counts, { 124 | m3u8: 1, 125 | ts: 11, 126 | total: 12 127 | }); 128 | }); 129 | }); 130 | 131 | it('should return just segments for m3u8 with windows paths', function() { 132 | nock(TEST_URL) 133 | .get('/test.m3u8') 134 | .replyWithFile(200, `${process.cwd()}/test/resources/windows.m3u8`); 135 | 136 | const options = {decrypt: false, basedir: '.', uri: TEST_URL + '/test.m3u8', requestRetryMaxAttempts: 0}; 137 | 138 | return walker(options) 139 | .then(function(resources) { 140 | const counts = countResources({resources}); 141 | 142 | assert.deepEqual(counts, { 143 | m3u8: 1, 144 | ts: 2, 145 | total: 3 146 | }); 147 | assert.equal(resources[1].file, 'chunk_0.ts'); 148 | assert.equal(resources[2].file, 'chunk_1.ts'); 149 | }); 150 | }); 151 | 152 | it('should return correct paths for m3u8', function() { 153 | nock(TEST_URL) 154 | .get('/test/test.m3u8') 155 | .replyWithFile(200, `${process.cwd()}/test/resources/path-testing.m3u8`); 156 | 157 | const options = {decrypt: false, basedir: '.', uri: TEST_URL + '/test/test.m3u8', requestRetryMaxAttempts: 0}; 158 | 159 | return walker(options) 160 | .then(function(resources) { 161 | const counts = countResources({resources}); 162 | 163 | assert.deepEqual(counts, { 164 | m3u8: 1, 165 | ts: 4, 166 | total: 5 167 | }); 168 | assert.equal(resources[0].uri, `${TEST_URL}/test/test.m3u8`); 169 | assert.equal(resources[1].uri, `${TEST_URL}/test/chunk_0.ts`); 170 | assert.equal(resources[2].uri, `${TEST_URL}/test/chunk_1.ts`); 171 | assert.equal(resources[3].uri, `${TEST_URL}/test/test/chunk_2.ts`); 172 | assert.equal(resources[4].uri, `${TEST_URL}/test/chunk_3.ts`); 173 | }); 174 | }); 175 | 176 | it('should shorten paths that will be too long', function() { 177 | // string used in long-path.m3u8 178 | const longPathRandom = 'OYK%40%3F%2BrjKeGaskhhsmf8E7aoUftbIfXZo0ucm9qebBFsXG5yepliwyfwKIf4zTGqMocHsTJePF91V17ZJ4h8A7mS3ysNSOcjKQT2oAVJmfD3vJIwdDfD0mZqlA9jQOOWwXnLy0UQtn9V2eYXlNAdIc9w8yDFPxDp509vJC9lurHWewql6eg22drnACC2rEDOXYit0I3CqOaVRvLSIqG0quUda5CoDn7vmaCBlvsBA0MEoWQSG0TEmDtdTT6DP8vUCC7BTtr9Zaxo5l9QYnWyNMzZNszjijCoKq8LsAi95WIo2n9'; 179 | const chunkPath = `chunk_@.ts?token=${longPathRandom}` 180 | .replace('?token=OYK%40%3F%2B', 'token=OYK@+') 181 | .substring(0, 255); 182 | const manifestUri = `test.m3u8?token=${longPathRandom}`; 183 | const manifestPath = manifestUri 184 | .replace('?token=OYK%40%3F%2B', 'token=OYK@+') 185 | .substring(0, 255); 186 | 187 | nock(TEST_URL) 188 | .get(`/test/${manifestUri}`) 189 | .replyWithFile(200, `${process.cwd()}/test/resources/long-path.m3u8`); 190 | 191 | const options = { 192 | decrypt: false, 193 | basedir: '.', 194 | uri: `${TEST_URL}/test/${manifestUri}`, 195 | requestRetryMaxAttempts: 0 196 | }; 197 | 198 | return walker(options) 199 | .then(function(resources) { 200 | const counts = countResources({resources}); 201 | 202 | assert.deepEqual(counts, { 203 | m3u8: 1, 204 | ts: 4, 205 | total: 5 206 | }); 207 | assert.equal(resources[0].file, manifestPath, 'manifest'); 208 | assert.equal(resources[1].file, chunkPath.replace('@', '0'), 'chunk 0'); 209 | assert.equal(resources[2].file, chunkPath.replace('@', '1'), 'chunk 1'); 210 | assert.equal(resources[3].file, chunkPath.replace('@', '2'), 'chunk 2'); 211 | assert.equal(resources[4].file, chunkPath.replace('@', '3'), 'chunk 3'); 212 | }); 213 | }); 214 | 215 | it('should return fmp4/ts segments and init segment for fmp4 m3u8', function() { 216 | nock(TEST_URL) 217 | .get('/test.m3u8') 218 | .replyWithFile(200, `${process.cwd()}/test/resources/fmp4.m3u8`); 219 | 220 | const options = {decrypt: false, basedir: '.', uri: TEST_URL + '/test.m3u8', requestRetryMaxAttempts: 0}; 221 | 222 | return walker(options) 223 | .then(function(resources) { 224 | const counts = countResources({resources}); 225 | 226 | assert.deepEqual(counts, { 227 | m3u8: 1, 228 | m4s: 5, 229 | ts: 6, 230 | mp4: 1, 231 | total: 13 232 | }); 233 | }); 234 | }); 235 | 236 | it('should follow http redirects for simple m3u8', function() { 237 | nock(TEST_URL) 238 | .get('/test.m3u8') 239 | .reply(302, undefined, {location: TEST_URL + '/redirect.m3u8'}) 240 | .get('/redirect.m3u8') 241 | .replyWithFile(200, `${process.cwd()}/test/resources/simple.m3u8`); 242 | 243 | const options = {decrypt: false, basedir: '.', uri: TEST_URL + '/test.m3u8', requestRetryMaxAttempts: 0}; 244 | 245 | return walker(options) 246 | .then(function(resources) { 247 | const counts = countResources({resources}); 248 | 249 | assert.deepEqual(counts, { 250 | m3u8: 1, 251 | ts: 11, 252 | total: 12 253 | }); 254 | }); 255 | }); 256 | 257 | it('should not get stuck and short circuit for a cycle and throw no errors', function() { 258 | nock(TEST_URL) 259 | .get('/cycle1.m3u8') 260 | .replyWithFile(200, `${process.cwd()}/test/resources/cycle1.m3u8`) 261 | .get('/cycle2.m3u8') 262 | .replyWithFile(200, `${process.cwd()}/test/resources/cycle2.m3u8`); 263 | 264 | const errors = []; 265 | const options = { 266 | decrypt: false, 267 | basedir: '.', 268 | uri: TEST_URL + '/cycle1.m3u8', 269 | requestRetryMaxAttempts: 0 270 | }; 271 | 272 | return walker(options) 273 | .then(function(resources) { 274 | const counts = countResources({resources}); 275 | 276 | assert.deepEqual(counts, { 277 | m3u8: 2, 278 | total: 2 279 | }); 280 | 281 | // no errors on cycle 282 | assert.equal(errors.length, 0); 283 | }); 284 | }); 285 | 286 | it('should return top level error if server takes too long to respond with top level m3u8 on default onError', function() { 287 | nock(TEST_URL) 288 | .get('/test.m3u8') 289 | .delayConnection(100) 290 | .replyWithFile(200, `${process.cwd()}/test/resources/simple.m3u8`); 291 | 292 | const options = { 293 | decrypt: false, 294 | basedir: '.', 295 | uri: TEST_URL + '/test.m3u8', 296 | requestRetryMaxAttempts: 0, 297 | requestTimeout: 10, 298 | requestRetryDelay: 10 299 | }; 300 | 301 | return walker(options) 302 | .catch(function(err) { 303 | assert.equal(err.message, 'ESOCKETTIMEDOUT|' + TEST_URL + '/test.m3u8'); 304 | }); 305 | }); 306 | 307 | it('should return error in resources not top error if server takes too long to respond m3u8 on custom onError', function() { 308 | nock(TEST_URL) 309 | .get('/test.m3u8') 310 | .delayConnection(100) 311 | .replyWithFile(200, `${process.cwd()}/test/resources/simple.m3u8`); 312 | 313 | const errors = []; 314 | const options = { 315 | decrypt: false, 316 | basedir: '.', 317 | uri: TEST_URL + '/test.m3u8', 318 | onError: customError(errors), 319 | requestRetryMaxAttempts: 0, 320 | requestTimeout: 10, 321 | requestRetryDelay: 10 322 | }; 323 | 324 | return walker(options) 325 | .then(function(resources) { 326 | assert.equal(resources.length, 0); 327 | assert(errors.find(o => o.message === 'ESOCKETTIMEDOUT|' + TEST_URL + '/test.m3u8')); 328 | }); 329 | }); 330 | 331 | it('should return just original m3u8 for invalid m3u8 and not break', function() { 332 | nock(TEST_URL) 333 | .get('/test.m3u8') 334 | .reply(200, 'not a valid m3u8'); 335 | 336 | const options = {decrypt: false, basedir: '.', uri: TEST_URL + '/test.m3u8', requestRetryMaxAttempts: 0}; 337 | 338 | return walker(options) 339 | .then(function(resources) { 340 | const counts = countResources({resources}); 341 | 342 | assert.deepEqual(counts, { 343 | m3u8: 1, 344 | total: 1 345 | }); 346 | }); 347 | }); 348 | 349 | it('should return just segments for m3u8 with sub playlists', function() { 350 | nock(TEST_URL) 351 | .get('/test.m3u8') 352 | .replyWithFile(200, `${process.cwd()}/test/resources/with-sub-manifest/playlist.m3u8`) 353 | .get('/var256000/playlist.m3u8') 354 | .replyWithFile(200, `${process.cwd()}/test/resources/with-sub-manifest/var256000/playlist.m3u8`) 355 | .get('/var386000/playlist.m3u8') 356 | .replyWithFile(200, `${process.cwd()}/test/resources/with-sub-manifest/var386000/playlist.m3u8`) 357 | .get('/var500000/playlist.m3u8') 358 | .replyWithFile(200, `${process.cwd()}/test/resources/with-sub-manifest/var500000/playlist.m3u8`); 359 | 360 | const options = {decrypt: false, basedir: '.', uri: TEST_URL + '/test.m3u8', requestRetryMaxAttempts: 0}; 361 | 362 | return walker(options) 363 | .then(function(resources) { 364 | const counts = countResources({resources}); 365 | 366 | assert.deepEqual(counts, { 367 | m3u8: 4, 368 | ts: 24, 369 | total: 28 370 | }); 371 | }); 372 | }); 373 | 374 | it('should return just segments for m3u8 with sub playlists with a redirect', function() { 375 | nock(TEST_URL) 376 | .get('/test.m3u8') 377 | .replyWithFile(200, `${process.cwd()}/test/resources/with-sub-manifest/playlist.m3u8`) 378 | .get('/var256000/playlist.m3u8') 379 | .replyWithFile(200, `${process.cwd()}/test/resources/with-sub-manifest/var256000/playlist.m3u8`) 380 | .get('/var386000/playlist.m3u8') 381 | .replyWithFile(200, `${process.cwd()}/test/resources/with-sub-manifest/var386000/playlist.m3u8`) 382 | .get('/var500000/playlist.m3u8') 383 | .reply(302, undefined, {location: TEST_URL + '/redirect.m3u8'}) 384 | .get('/redirect.m3u8') 385 | .replyWithFile(200, `${process.cwd()}/test/resources/with-sub-manifest/var500000/playlist.m3u8`); 386 | 387 | const options = {decrypt: false, basedir: '.', uri: TEST_URL + '/test.m3u8', requestRetryMaxAttempts: 0}; 388 | 389 | return walker(options) 390 | .then(function(resources) { 391 | const counts = countResources({resources}); 392 | 393 | assert.deepEqual(counts, { 394 | m3u8: 4, 395 | ts: 24, 396 | total: 28 397 | }); 398 | }); 399 | }); 400 | 401 | it('should for one sub playlist getting 404 should get top level 404 error on default onError', function() { 402 | nock(TEST_URL) 403 | .get('/test.m3u8') 404 | .replyWithFile(200, `${process.cwd()}/test/resources/with-sub-manifest/playlist.m3u8`) 405 | .get('/var256000/playlist.m3u8') 406 | .reply(404) 407 | .get('/var386000/playlist.m3u8') 408 | .replyWithFile(200, `${process.cwd()}/test/resources/with-sub-manifest/var386000/playlist.m3u8`) 409 | .get('/var500000/playlist.m3u8') 410 | .replyWithFile(200, `${process.cwd()}/test/resources/with-sub-manifest/var500000/playlist.m3u8`); 411 | 412 | const options = {decrypt: false, basedir: '.', uri: TEST_URL + '/test.m3u8', requestRetryMaxAttempts: 0}; 413 | 414 | return walker(options) 415 | .catch(function(err) { 416 | assert.equal(err.message, '404|' + TEST_URL + '/var256000/playlist.m3u8'); 417 | }); 418 | }); 419 | 420 | it('should for one sub playlist getting 404 get 404 error but the rest of valid resources on custom onError', function() { 421 | nock(TEST_URL) 422 | .get('/test.m3u8') 423 | .replyWithFile(200, `${process.cwd()}/test/resources/with-sub-manifest/playlist.m3u8`) 424 | .get('/var256000/playlist.m3u8') 425 | .reply(404) 426 | .get('/var386000/playlist.m3u8') 427 | .replyWithFile(200, `${process.cwd()}/test/resources/with-sub-manifest/var386000/playlist.m3u8`) 428 | .get('/var500000/playlist.m3u8') 429 | .replyWithFile(200, `${process.cwd()}/test/resources/with-sub-manifest/var500000/playlist.m3u8`); 430 | 431 | const errors = []; 432 | const options = { 433 | decrypt: false, 434 | basedir: '.', 435 | uri: TEST_URL + '/test.m3u8', 436 | onError: customError(errors), 437 | requestRetryMaxAttempts: 0 438 | }; 439 | 440 | return walker(options) 441 | .then(function(resources) { 442 | const counts = countResources({resources}); 443 | 444 | assert.deepEqual(counts, { 445 | m3u8: 3, 446 | ts: 16, 447 | total: 19 448 | }); 449 | 450 | resources.forEach(function(item) { 451 | // We shouldn't get the bad manifest 452 | assert(item.uri !== TEST_URL + '/var256000/playlist.m3u8'); 453 | }); 454 | }); 455 | }); 456 | 457 | it('should for one sub playlist getting 500 should get top level 500 error on default onError', function() { 458 | nock(TEST_URL) 459 | .get('/test.m3u8') 460 | .replyWithFile(200, `${process.cwd()}/test/resources/with-sub-manifest/playlist.m3u8`) 461 | .get('/var256000/playlist.m3u8') 462 | .reply(500) 463 | .get('/var386000/playlist.m3u8') 464 | .replyWithFile(200, `${process.cwd()}/test/resources/with-sub-manifest/var386000/playlist.m3u8`) 465 | .get('/var500000/playlist.m3u8') 466 | .replyWithFile(200, `${process.cwd()}/test/resources/with-sub-manifest/var500000/playlist.m3u8`); 467 | 468 | const options = { 469 | decrypt: false, 470 | basedir: '.', 471 | uri: TEST_URL + '/test.m3u8', 472 | requestRetryMaxAttempts: 0 473 | }; 474 | 475 | return walker(options) 476 | .catch(function(err) { 477 | assert.equal(err.message, '500|' + TEST_URL + '/var256000/playlist.m3u8'); 478 | }); 479 | }); 480 | 481 | it('should for one sub playlist getting 500 should get top level 500 error on default onError and with even on 2 retry', function() { 482 | nock(TEST_URL) 483 | .get('/test.m3u8') 484 | .replyWithFile(200, `${process.cwd()}/test/resources/with-sub-manifest/playlist.m3u8`) 485 | .get('/var256000/playlist.m3u8') 486 | .times(2) 487 | .reply(500) 488 | .get('/var386000/playlist.m3u8') 489 | .replyWithFile(200, `${process.cwd()}/test/resources/with-sub-manifest/var386000/playlist.m3u8`) 490 | .get('/var500000/playlist.m3u8') 491 | .replyWithFile(200, `${process.cwd()}/test/resources/with-sub-manifest/var500000/playlist.m3u8`); 492 | 493 | const options = { 494 | decrypt: false, 495 | basedir: '.', 496 | uri: TEST_URL + '/test.m3u8', 497 | requestRetryMaxAttempts: 2, 498 | requestRetryDelay: 10 499 | }; 500 | 501 | return walker(options) 502 | .catch(function(err) { 503 | assert.equal(err.message, '500|' + TEST_URL + '/var256000/playlist.m3u8'); 504 | }); 505 | }); 506 | 507 | it('should for one sub playlist getting 500 get 500 error but the rest of valid resources on custom onError', function() { 508 | nock(TEST_URL) 509 | .get('/test.m3u8') 510 | .replyWithFile(200, `${process.cwd()}/test/resources/with-sub-manifest/playlist.m3u8`) 511 | .get('/var256000/playlist.m3u8') 512 | .reply(500) 513 | .get('/var386000/playlist.m3u8') 514 | .replyWithFile(200, `${process.cwd()}/test/resources/with-sub-manifest/var386000/playlist.m3u8`) 515 | .get('/var500000/playlist.m3u8') 516 | .replyWithFile(200, `${process.cwd()}/test/resources/with-sub-manifest/var500000/playlist.m3u8`); 517 | 518 | const errors = []; 519 | const options = { 520 | decrypt: false, 521 | basedir: '.', 522 | uri: TEST_URL + '/test.m3u8', 523 | onError: customError(errors), 524 | requestRetryMaxAttempts: 0 525 | }; 526 | 527 | return walker(options) 528 | .then(function(resources) { 529 | // 3 m3u8 and 8 * 2 segments 530 | const setResources = new Set(resources); 531 | 532 | assert.equal(setResources.size, 19); 533 | 534 | assert(errors.find(o => o.message === '500|' + TEST_URL + '/var256000/playlist.m3u8')); 535 | resources.forEach(function(item) { 536 | assert(item.uri.includes('.ts') || item.uri.includes('.m3u8')); 537 | // We shouldn't get the bad manifest 538 | assert(item.uri !== TEST_URL + '/var256000/playlist.m3u8'); 539 | }); 540 | }); 541 | }); 542 | 543 | it('should for one sub playlist throwing error should get top level error default onError', function() { 544 | nock(TEST_URL) 545 | .get('/test.m3u8') 546 | .replyWithFile(200, `${process.cwd()}/test/resources/with-sub-manifest/playlist.m3u8`) 547 | .get('/var256000/playlist.m3u8') 548 | .replyWithError('something awful happened') 549 | .get('/var386000/playlist.m3u8') 550 | .replyWithFile(200, `${process.cwd()}/test/resources/with-sub-manifest/var386000/playlist.m3u8`) 551 | .get('/var500000/playlist.m3u8') 552 | .replyWithFile(200, `${process.cwd()}/test/resources/with-sub-manifest/var500000/playlist.m3u8`); 553 | 554 | const options = {decrypt: false, basedir: '.', uri: TEST_URL + '/test.m3u8', requestRetryMaxAttempts: 0}; 555 | 556 | return walker(options) 557 | .catch(function(err) { 558 | assert.equal(err.message, 'something awful happened|' + TEST_URL + '/var256000/playlist.m3u8'); 559 | }); 560 | }); 561 | 562 | it('should for one sub playlist throwing error should get error but have rest of valid segments/manifests default onError', function() { 563 | nock(TEST_URL) 564 | .get('/test.m3u8') 565 | .replyWithFile(200, `${process.cwd()}/test/resources/with-sub-manifest/playlist.m3u8`) 566 | .get('/var256000/playlist.m3u8') 567 | .replyWithError('something awful happened') 568 | .get('/var386000/playlist.m3u8') 569 | .replyWithFile(200, `${process.cwd()}/test/resources/with-sub-manifest/var386000/playlist.m3u8`) 570 | .get('/var500000/playlist.m3u8') 571 | .replyWithFile(200, `${process.cwd()}/test/resources/with-sub-manifest/var500000/playlist.m3u8`); 572 | 573 | const errors = []; 574 | const options = { 575 | decrypt: false, 576 | basedir: '.', 577 | uri: TEST_URL + '/test.m3u8', 578 | onError: customError(errors), 579 | requestRetryMaxAttempts: 0 580 | }; 581 | 582 | return walker(options) 583 | .then(function(resources) { 584 | // 3 m3u8 and 8 * 2 segments 585 | const setResources = new Set(resources); 586 | 587 | assert.equal(setResources.size, 19); 588 | 589 | assert(errors.find(o => o.message === 'something awful happened|' + TEST_URL + '/var256000/playlist.m3u8')); 590 | resources.forEach(function(item) { 591 | assert(item.uri.includes('.ts') || item.uri.includes('.m3u8')); 592 | assert(item.uri !== TEST_URL + '/var256000/playlist.m3u8'); 593 | }); 594 | }); 595 | }); 596 | 597 | it('should not break if sub playlist is not a valid m3u8', function() { 598 | nock(TEST_URL) 599 | .get('/test.m3u8') 600 | .replyWithFile(200, `${process.cwd()}/test/resources/with-sub-manifest/playlist.m3u8`) 601 | .get('/var256000/playlist.m3u8') 602 | .reply(200, 'not valid m3u8') 603 | .get('/var386000/playlist.m3u8') 604 | .replyWithFile(200, `${process.cwd()}/test/resources/with-sub-manifest/var386000/playlist.m3u8`) 605 | .get('/var500000/playlist.m3u8') 606 | .replyWithFile(200, `${process.cwd()}/test/resources/with-sub-manifest/var500000/playlist.m3u8`); 607 | 608 | const options = {decrypt: false, basedir: '.', uri: TEST_URL + '/test.m3u8', requestRetryMaxAttempts: 0}; 609 | 610 | return walker(options) 611 | .then(function(resources) { 612 | // 4 m3u8 and 8 * 2 segments 613 | const setResources = new Set(resources); 614 | 615 | assert.equal(setResources.size, 20); 616 | setResources.forEach(function(item) { 617 | assert(item.uri.includes('.ts') || item.uri.includes('.m3u8')); 618 | }); 619 | // We should still get the invalid m3u8 620 | assert(resources.filter(e => e.uri === TEST_URL + '/var256000/playlist.m3u8').length > 0); 621 | }); 622 | }); 623 | 624 | it('should throw top level error if sub playlist takes too long to respond with m3u8 default onError', function() { 625 | nock(TEST_URL) 626 | .get('/test.m3u8') 627 | .replyWithFile(200, `${process.cwd()}/test/resources/with-sub-manifest/playlist.m3u8`) 628 | .get('/var256000/playlist.m3u8') 629 | .replyWithFile(200, `${process.cwd()}/test/resources/with-sub-manifest/var256000/playlist.m3u8`) 630 | .get('/var386000/playlist.m3u8') 631 | .replyWithFile(200, `${process.cwd()}/test/resources/with-sub-manifest/var386000/playlist.m3u8`) 632 | .get('/var500000/playlist.m3u8') 633 | .delayConnection(100) 634 | .replyWithFile(200, `${process.cwd()}/test/resources/with-sub-manifest/var500000/playlist.m3u8`); 635 | 636 | const options = { 637 | decrypt: false, 638 | basedir: '.', 639 | uri: TEST_URL + '/test.m3u8', 640 | requestRetryMaxAttempts: 0, 641 | requestTimeout: 10, 642 | requestRetryDelay: 10 643 | }; 644 | 645 | return walker(options) 646 | .catch(function(err) { 647 | assert.equal(err.message, 'ESOCKETTIMEDOUT|' + TEST_URL + '/var500000/playlist.m3u8'); 648 | }); 649 | }); 650 | 651 | it('should have error for sub playlist takes too long to respond with m3u8 but have rest of resources custom onError', function() { 652 | nock(TEST_URL) 653 | .get('/test.m3u8') 654 | .replyWithFile(200, `${process.cwd()}/test/resources/with-sub-manifest/playlist.m3u8`) 655 | .get('/var256000/playlist.m3u8') 656 | .replyWithFile(200, `${process.cwd()}/test/resources/with-sub-manifest/var256000/playlist.m3u8`) 657 | .get('/var386000/playlist.m3u8') 658 | .replyWithFile(200, `${process.cwd()}/test/resources/with-sub-manifest/var386000/playlist.m3u8`) 659 | .get('/var500000/playlist.m3u8') 660 | .delayConnection(100) 661 | .replyWithFile(200, `${process.cwd()}/test/resources/with-sub-manifest/var500000/playlist.m3u8`); 662 | 663 | const errors = []; 664 | const options = { 665 | decrypt: false, 666 | basedir: '.', 667 | uri: TEST_URL + '/test.m3u8', 668 | onError: customError(errors), 669 | requestRetryMaxAttempts: 0, 670 | requestTimeout: 10, 671 | requestRetryDelay: 10 672 | }; 673 | 674 | return walker(options) 675 | .then(function(resources) { 676 | const count = countResources({resources, extensions: ['ts', 'm3u8']}); 677 | 678 | assert.deepEqual(count, { 679 | ts: 16, 680 | m3u8: 3, 681 | total: 19 682 | }); 683 | assert(errors.find(o => o.message === 'ESOCKETTIMEDOUT|' + TEST_URL + '/var500000/playlist.m3u8')); 684 | resources.forEach(function(item) { 685 | // We shouldn't get the bad manifest 686 | assert(item.uri !== TEST_URL + '/var500000/playlist.m3u8'); 687 | }); 688 | }); 689 | }); 690 | 691 | it('should return segments and playlists for mpd', function() { 692 | nock(TEST_URL) 693 | .get('/dash.mpd') 694 | .replyWithFile(200, `${process.cwd()}/test/resources/dash.mpd`); 695 | 696 | const options = {decrypt: false, basedir: '.', uri: TEST_URL + '/dash.mpd', requestRetryMaxAttempts: 0}; 697 | 698 | return walker(options) 699 | .then(function(resources) { 700 | // m3u8 and 13 segments 701 | const setResources = new Set(resources); 702 | const count = {mp4: 0, m4v: 0, m4a: 0, mpd: 0}; 703 | 704 | assert.equal(setResources.size, 37); 705 | setResources.forEach(function(item) { 706 | if (item.uri.includes('.mp4')) { 707 | count.mp4 += 1; 708 | } else if (item.uri.includes('.m4v')) { 709 | count.m4v += 1; 710 | } else if (item.uri.includes('.m4a')) { 711 | count.m4a += 1; 712 | } else if (item.uri.includes('.mpd')) { 713 | count.mpd += 1; 714 | } else { 715 | assert(false, `items uri ${item.uri} was unexpected`); 716 | return; 717 | } 718 | 719 | assert(true, 'items uri was expected'); 720 | }); 721 | 722 | assert.equal(count.mp4, 6, 'mp4 count as expected'); 723 | assert.equal(count.mpd, 1, 'mpd count as expected'); 724 | assert.equal(count.m4v, 25, 'm4v count as expected'); 725 | assert.equal(count.m4a, 5, 'm4a count as expected'); 726 | }); 727 | }); 728 | 729 | it('should return segments and playlists for mpd with sidx', function() { 730 | nock(TEST_URL) 731 | .get('/sidx.mpd') 732 | .replyWithFile(200, `${process.cwd()}/test/resources/sidx.mpd`); 733 | 734 | const options = {decrypt: false, basedir: '.', uri: TEST_URL + '/sidx.mpd', requestRetryMaxAttempts: 0}; 735 | 736 | return walker(options) 737 | .then(function(resources) { 738 | // m3u8 and 13 segments 739 | const setResources = new Set(resources); 740 | const count = {mp4: 0, m4v: 0, m4a: 0, mpd: 0}; 741 | 742 | assert.equal(setResources.size, 7); 743 | setResources.forEach(function(item) { 744 | if (item.uri.includes('.mp4')) { 745 | count.mp4 += 1; 746 | } else if (item.uri.includes('.mpd')) { 747 | count.mpd += 1; 748 | } else { 749 | assert(false, `items uri ${item.uri} was unexpected`); 750 | return; 751 | } 752 | 753 | assert(true, 'items uri was expected'); 754 | }); 755 | 756 | assert.equal(count.mp4, 6, 'mp4 count as expected'); 757 | assert.equal(count.mpd, 1, 'mpd count as expected'); 758 | }); 759 | }); 760 | 761 | it('should handle tilde in the query', function() { 762 | nock(TEST_URL) 763 | .get('/tilde.m3u8?foo=~nope') 764 | .replyWithFile(200, `${process.cwd()}/test/resources/tilde-query-param.m3u8`); 765 | 766 | const options = {decrypt: false, basedir: '.', uri: TEST_URL + '/tilde.m3u8?foo=~nope', requestRetryMaxAttempts: 0}; 767 | 768 | return walker(options) 769 | .then(function(resources) { 770 | const count = countResources({resources, extensions: ['ts', 'm3u8']}); 771 | 772 | assert.deepEqual(count, { 773 | ts: 9, 774 | m3u8: 1, 775 | total: 10 776 | }); 777 | 778 | }); 779 | }); 780 | 781 | it('should handle paths in the query', function() { 782 | nock(TEST_URL) 783 | .get('/path-query.m3u8?foo=/no/dont/include') 784 | .replyWithFile(200, `${process.cwd()}/test/resources/path-query.m3u8`); 785 | 786 | const options = {decrypt: false, basedir: '.', uri: TEST_URL + '/path-query.m3u8?foo=/no/dont/include', requestRetryMaxAttempts: 0}; 787 | 788 | return walker(options) 789 | .then(function(resources) { 790 | const count = countResources({resources, extensions: ['ts', 'm3u8']}); 791 | 792 | assert.deepEqual(count, { 793 | ts: 9, 794 | m3u8: 1, 795 | total: 10 796 | }); 797 | 798 | resources.forEach(function({file}) { 799 | assert(!(/\/no\/dont\/include/).test(file), 'does not have query in file path'); 800 | }); 801 | }); 802 | }); 803 | 804 | it('handles duplicate manifests', function() { 805 | nock(TEST_URL) 806 | .get('/master.m3u8') 807 | .replyWithFile(200, `${process.cwd()}/test/resources/duplicate-manifests/master.m3u8`) 808 | .get('/manifest0/rendition.m3u8') 809 | .replyWithFile(200, `${process.cwd()}/test/resources/duplicate-manifests/manifest0/rendition.m3u8`) 810 | .get('/manifest1/rendition.m3u8') 811 | .replyWithFile(200, `${process.cwd()}/test/resources/duplicate-manifests/manifest1/rendition.m3u8`) 812 | .get('/manifest2/rendition.m3u8') 813 | .replyWithFile(200, `${process.cwd()}/test/resources/duplicate-manifests/manifest2/rendition.m3u8`); 814 | 815 | const options = {decrypt: false, basedir: '.', uri: TEST_URL + '/master.m3u8', requestRetryMaxAttempts: 0}; 816 | 817 | return walker(options) 818 | .then(function(resources) { 819 | const count = countResources({resources, extensions: ['ts', 'm3u8']}); 820 | 821 | assert.deepEqual(count, { 822 | ts: 27, 823 | m3u8: 4, 824 | total: 31 825 | }); 826 | 827 | resources.forEach(function({file}) { 828 | assert(!(/\/no\/dont\/include/).test(file), 'does not have query in file path'); 829 | }); 830 | }); 831 | }); 832 | // end walkPlaylist 833 | }); 834 | }); 835 | --------------------------------------------------------------------------------