├── .gitignore ├── test.csv ├── .github └── ISSUE_TEMPLATE │ ├── feature_request.md │ └── bug_report.md ├── create-large-csv.sh ├── plugin.php └── readme.md /.gitignore: -------------------------------------------------------------------------------- 1 | test-large.csv 2 | -------------------------------------------------------------------------------- /test.csv: -------------------------------------------------------------------------------- 1 | http://bbc.co.uk/,bbc 2 | http://bbc.co.uk/,BBC 3 | http://news.bbc.co.uk/,news 4 | http://news.bbc.co.uk/,bbcnews 5 | http://news.bbc.co.uk/,bbc-news 6 | http://itv.com/, 7 | http://www.channel4.com/,c4 8 | http://www.channel4.com/,4 9 | http://isitfridayyet.net/,friday,Friday!! 10 | -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/feature_request.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Feature request 3 | about: Suggest an idea for this project 4 | title: '' 5 | labels: '' 6 | assignees: '' 7 | 8 | --- 9 | 10 | **Is your feature request related to a problem? Please describe.** 11 | A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] 12 | 13 | **Describe the solution you'd like** 14 | A clear and concise description of what you want to happen. 15 | 16 | **Describe alternatives you've considered** 17 | A clear and concise description of any alternative solutions or features you've considered. 18 | 19 | **Additional context** 20 | Add any other context or screenshots about the feature request here. 21 | -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/bug_report.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Bug report 3 | about: Create a report to help us improve 4 | title: '' 5 | labels: '' 6 | assignees: '' 7 | 8 | --- 9 | 10 | **Describe the bug** 11 | A clear and concise description of what the bug is. 12 | 13 | **To Reproduce** 14 | Steps to reproduce the behavior: 15 | 1. Go to '...' 16 | 2. Click on '....' 17 | 3. Scroll down to '....' 18 | 4. See error 19 | 20 | **Expected behavior** 21 | A clear and concise description of what you expected to happen. 22 | 23 | **Screenshots** 24 | If applicable, add screenshots to help explain your problem. 25 | 26 | **Desktop (please complete the following information):** 27 | - OS: [e.g. iOS] 28 | - Browser [e.g. chrome, safari] 29 | - Version [e.g. 22] 30 | 31 | **Smartphone (please complete the following information):** 32 | - Device: [e.g. iPhone6] 33 | - OS: [e.g. iOS8.1] 34 | - Browser [e.g. stock browser, safari] 35 | - Version [e.g. 22] 36 | 37 | **Additional context** 38 | Add any other context about the problem here. 39 | -------------------------------------------------------------------------------- /create-large-csv.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Each 'run' generates 10 unique URLs, so if RUNS=10 you'll get 100 lines out. 4 | # Uses the command `gpw` to create random-ish data. This is not installed by default: `sudo apt-get install gpw` 5 | 6 | TIME_START=$(date +%s) 7 | FILE=test-large.csv 8 | RUNS=1000 9 | 10 | if test -f "$FILE"; then 11 | truncate -s 0 $FILE 12 | else 13 | touch $FILE 14 | fi 15 | 16 | for (( i=1; i <= $RUNS; i++)); do 17 | string=$(gpw 1 25) 18 | echo "http://$string.co.uk" >> $FILE 19 | echo "http://$string.net" >> $FILE 20 | echo "http://$string.org" >> $FILE 21 | echo "http://$string.org.uk" >> $FILE 22 | echo "http://$string.eu" >> $FILE 23 | echo "http://$string.ie" >> $FILE 24 | echo "http://$string.irish" >> $FILE 25 | echo "http://$string.cymru" >> $FILE 26 | echo "http://$string.wales" >> $FILE 27 | echo "http://$string.scot" >> $FILE 28 | 29 | if [[ $(( $i % 100 )) -eq "0" ]]; then 30 | echo "Done $(wc -l $FILE | cut -d " " -f 1) lines." 31 | fi 32 | done 33 | 34 | TIME_TAKEN=$(( ($(date +%s) - $TIME_START) )) 35 | 36 | echo 37 | echo "$(wc -l $FILE | cut -d " " -f 1) lines." 38 | 39 | echo 40 | ls -gh $FILE 41 | 42 | echo 43 | echo "Took ${TIME_TAKEN}s." 44 | -------------------------------------------------------------------------------- /plugin.php: -------------------------------------------------------------------------------- 1 | Bulk Import and Shorten 36 |

Import links as long URLs and let YOURLS shorten them for you according to your settings.

37 |

Upload a .csv file in the following format:

38 | 43 |

I don't know what will happen if two short links point to the same long link - this might or might not be allowed, according to your settings.

44 | 45 |

Import

46 | EOT; 47 | 48 | echo '
' . PHP_EOL; 49 | echo ' ' . yourls_nonce_field( 'vaughany_bias_import', 'nonce', false, false ); 50 | echo ' ' . PHP_EOL; 51 | echo ' ' . PHP_EOL; 52 | echo '
' . PHP_EOL; 53 | } 54 | 55 | function vaughany_bias_handle_post() { 56 | 57 | if ( !empty( $_FILES['import'] ) ) { 58 | if ( !empty( $_POST['nonce'] ) && yourls_verify_nonce( 'vaughany_bias_import', $_POST['nonce'] ) ) { 59 | $count = vaughany_bias_import_urls( $_FILES['import'] ); 60 | if ( $count > 0 ) { 61 | $message = $count . ' URLs imported.'; 62 | } else { 63 | $message = 'No URLs imported.'; 64 | } 65 | } 66 | } 67 | 68 | // Message 69 | if ( !empty( $message ) ) { 70 | yourls_add_notice($message); 71 | } 72 | 73 | } 74 | 75 | /** 76 | * Import the urls 77 | * @param type $import_file Uploaded file to be imported 78 | * @return int|bool Count of imported redirections or false on failure 79 | */ 80 | function vaughany_bias_import_urls( $file ) { 81 | 82 | if ( !is_uploaded_file( $file['tmp_name'] ) ) { 83 | yourls_add_notice('Not an uploaded file.'); 84 | } 85 | 86 | // Only handle .csv files. 87 | if ($file['type'] !== 'text/csv') { 88 | yourls_add_notice('Not a .csv file.'); 89 | return 0; 90 | } 91 | 92 | global $ydb; 93 | 94 | ini_set( 'auto_detect_line_endings', true ); 95 | $count = 0; 96 | $fh = fopen( $file['tmp_name'], 'r' ); 97 | $table = YOURLS_DB_TABLE_URL; 98 | 99 | // If the file handle is okay. 100 | if ( $fh ) { 101 | 102 | // Get each line in turn as an array, comma-separated. 103 | while ( $csv = fgetcsv( $fh, 1000, ',' ) ) { 104 | 105 | $url = $keyword = $title = ''; 106 | 107 | $url = trim( $csv[0] ); 108 | 109 | if ( isset( $csv[1] ) && !empty( $csv[1] ) ) { 110 | // Trim out cruft and slashes. 111 | $new_keyword = trim( str_replace( '/', '', $csv[1] ) ); 112 | 113 | // If the requested keyword is not free, use nothing. 114 | if ( yourls_keyword_is_free( $new_keyword ) ) { 115 | $keyword = $new_keyword; 116 | } 117 | } 118 | 119 | if ( isset( $csv[2] ) ) { 120 | if ( !empty ( $csv[2] ) ) { 121 | $title = trim( $csv[2] ); 122 | } else { 123 | $title = vaughany_bias_create_title_from_url( $url ); 124 | } 125 | } else { 126 | $title = vaughany_bias_create_title_from_url( $url ); 127 | } 128 | 129 | // Add a new link (passing the keyword) and get the result. 130 | $result = yourls_add_new_link( $url, $keyword, $title ); 131 | 132 | if ( $result['status'] == 'success' ) { 133 | $count++; 134 | } 135 | } 136 | 137 | } else { 138 | yourls_add_notice('File handle is bad.'); 139 | } 140 | 141 | return $count; 142 | } 143 | 144 | /** 145 | * Create a title from the URL 146 | * 147 | * @param string $url New in v0.2: Creating a title from the URL, so one is not fetched from the URL, slowing down the import. 148 | * @return string 149 | */ 150 | function vaughany_bias_create_title_from_url( string $url ) : string { 151 | return yourls_sanitize_title( str_replace(['http://', 'https://', '/'], '', $url) ); 152 | } 153 | -------------------------------------------------------------------------------- /readme.md: -------------------------------------------------------------------------------- 1 | # Bulk Import and Shorten - a YOURLS plugin 2 | 3 | Plugin for [YOURLS](http://yourls.org) 1.7.x. 4 | 5 | * Plugin URI: [github.com/vaughany/yourls-bulk-import-and-shorten](https://github.com/vaughany/yourls-bulk-import-and-shorten) 6 | * Description: A YOURLS plugin allowing importing of URLs in bulk to be shortened or (optionally) with a custom short URL. 7 | * Version: 0.4 8 | * Release date: 2020-07-31 9 | * Author: Paul Vaughan 10 | * Author URI: [github.com/vaughany](http://github.com/vaughany/) 11 | 12 | 13 | ## Description 14 | 15 | I looked at an import/export plugin for YOURLS but it only imported what it exported (for backup and restore purposes, I assume). I needed a way of getting long URLs into YOURLS in bulk, and not one at a time or via the Bookmarklets (as the links don't exist just yet). This plugin solves that problem. If you can create a CSV file, you can use this plugin. 16 | 17 | Note: CSV (comma-separated value) files are plain text files with columns separated by commas and fields deliniated by double quotes, and rows deliniated by carriage returns. MS Excel, OpenOffice/LibreOffice and Google Spreadsheets can all export data in CSV format. 18 | 19 | 20 | ## Installation 21 | 22 | 1. In `/user/plugins`, create a new folder named `bulk-import-and-shorten`. 23 | 2. Drop these files in that directory. 24 | 3. Go to the Manage Plugins page (e.g. `http://sho.rt/admin/plugins.php`) and activate the plugin. 25 | 4. Under the Manage Plugins link should be a new link called `Bulk Import and Shorten`. 26 | 5. Have fun! 27 | 28 | 29 | ## Configuration 30 | 31 | This plugin has no user-configurable options. If you know what you're doing you can add and amend this plugin, but some changes will be easier than others. If you make substantial improvements, [let me know](https://github.com/vaughany/yourls-bulk-import-and-shorten/issues). 32 | 33 | 34 | ## Use 35 | 36 | This plugin expects you to upload a CSV file with at least one column and optional second and third columns. The first column should contain the long URL of the 'target', e.g. http://bbc.co.uk. The optional second column can contain a keyword you would like to associate with this URL, if you don't want YOURLS to generate one for you. The third column contains an optional title. If one is not supplied, YOURLS generates one for you. 37 | 38 | Note: In this repository is a file called `test.csv`, which you can use as an example. 39 | 40 | 41 | ### No keywords 42 | 43 | If you don't specify a keyword in the CSV file, YOURLS will make a short URL for you. As default, this will be an integer (starting at '1' when YOURLS was first installed) and will increase by 1 with each additional URL, but you have more options: 44 | 45 | * If you have the [Random Keywords plugin](https://github.com/yourls/random-keywords) installed and activated, that will be used to generate a random alpha-numeric short URL. 46 | * If you have changed the `YOURLS_URL_CONVERT` setting (in `user/config.php`) from the default of '36' to '62' you will get an alpha-numeric short URL using both lowercase and UPPERCASE letters. 47 | * If you have the [Force Lowercase plugin](https://github.com/yourls/force-lowercase) installed and activated, the short URL will be forced into lowercase (producing the same result as if `YOURLS_URL_CONVERT` was set to the default of '36' still). 48 | 49 | 50 | ### With keywords 51 | 52 | If you supply keywords in the second column of the CSV file, then when imported, YOURLS will handle them according to the configuration settings and activated plugins. YOURLS always strips out (and never uses) any non-alpha-numeric characters (with the exception being hyphens, but read on): 53 | 54 | * If you are using the '36' setting for `YOURLS_URL_CONVERT` (in `user/config.php`) then any UPPERCASE LETTERS in your supplied keyword will be *removed* from your keyword. Not lowercased: removed. `ABCdef123` will become `def123`. `BBC` will become an empty string and will be treated as if you hadn't supplied a keyword. If you use the '62' setting (which includes both cases), this won't happen, but remember that you shouldn't change this setting after installation. Example CSV file: 55 | 56 | > `http://bbc.co.uk/,bbcnews` --> Will import as-is. 57 | 58 | > `http://bbc.co.uk/,BBC` --> UPPERCASE letters are stripped out (with '36' setting) and will import as if it had no keyword supplied. With the '62' setting, will import as-is. 59 | 60 | > `http://bbc.co.uk/,BBCnews` --> Will become `news` (with '36' setting) or will import as-is (with '62' setting). 61 | 62 | * You can get around the above by installing and activating the [Force Lowercase plugin](https://github.com/yourls/force-lowercase): 63 | 64 | > `http://bbc.co.uk/,bbcnews` --> Will import as-is. 65 | 66 | > `http://bbc.co.uk/,BBC` --> Will become 'bbc'. (If not using the plugin, will import as-is.) 67 | 68 | > `http://bbc.co.uk/,BBCnews` --> Will become 'bbcnews' and therefore a duplicate, so will not be imported. (If not using the plugin, will import as-is.) 69 | 70 | * If you have `YOURLS_UNIQUE_URLS` set to 'true' (in `user/config.php`) which is the default, then multiple short URLs pointing to the same long URL (as in the previous examples, all using the same long URL) will not be allowed, and during import will be rejected. Change the setting to 'false' to allow this: 71 | 72 | > `http://bbc.co.uk/,bbc` --> Will import fine. 73 | 74 | > `http://bbc.co.uk/,news` --> Will be rejected if `YOURLS_UNIQUE_URLS` is 'true', or allowed if 'false'. 75 | 76 | * If you want to use hyphens / dashes in your short URLs, you need to activate the 'Allow hyphens in short URLs' plugin which comes with YOURLS. 77 | 78 | > `http://bbc.co.uk/,bbc-news` --> Without the plugin activated, will become 'bbcnews'. With the plugin activated, will import as-is. 79 | 80 | I have talked through some common plugins and the outcomes of various situations, but please check carefully the short URLs of bulk-imported long URLs to ensure everything is as you expect. Due to the way plugins can hook into YOURLS I cannot know what plugins are installed and how they may affect how a short URL is created. 81 | 82 | 83 | ### Titles 84 | 85 | Normally, YOURLS will try to get a title for you by downloading the web page from the supplied URL and checking the raw HTML for a title. This is possibly the reason for slow bulk importing, when extremely large CSV files are uploaded. This plugin prevents YOURLS from doing it's job and instead creates a title from the supplied URL, or uses the one in the third column of the CSV, if present and non-empty. 86 | 87 | 88 | ## Troubleshooting 89 | 90 | One user experienced timeout issues processing a CSV file containing ~60,000 rows. I've not investigated this issue thoroughly, but the 0.2 version contains a line that creates a title from the supplied URL instead of letting YOURLS CURLing the URL and extracting a title from the returned HTML. I believe this to be faster, but I have only done a little testing. 91 | 92 | Some ideas, if processing a large CSV file: 93 | 94 | * Increase the `max_execution_time` setting in your `php.ini` file from it's default of 30 seconds. 95 | * Increase the `max_input_time` setting in your `php.ini` file from it's default of 60 seconds. 96 | 97 | **Note:** Setting either of these to 300 / 600 / 900 seconds (5 / 10 / 15 minutes) is not unreasonable. 98 | 99 | I've not investigated memory issues at this point, however I'd like to hear back if you have any experience of these (or any issues, for that matter). 100 | 101 | 102 | ## License 103 | 104 | Uses YOURLS' license, aka *"Do whatever the hell you want with it"*. I borrowed some code from others (including [GautamGupta](https://github.com/gautamgupta/yourls-Import-Export) who in turn borrowed from [John Godley](http://urbangiraffe.com/plugins/redirection/)) whose code had no licence so I can't claim this whole plugin as my own work, but the lion's share of it is. 105 | 106 | 107 | ## Bugs and Features 108 | 109 | I'm always keen to add new features, improve performance and squash bugs, so if you do any of these things, let me know. [Fork the project](https://github.com/vaughany/yourls-bulk-import-and-shorten/), make your changes and send me a pull request. If you find a bug but aren't a coder yourself, [open an issue](https://github.com/vaughany/yourls-bulk-import-and-shorten/issues) and give me as much detail as possible about the problem: 110 | 111 | * What you did 112 | * What you expected to happen 113 | * What actually happened 114 | * Steps to reproduce 115 | 116 | 117 | ## To-Do 118 | 119 | * Make theforcing of titles to either generated from the URL or specified in the CSV file optional, so that YOURLS can try to pull one from the URL's website's HTML (which is probably the preferred option, but slow when doing huge imports). 120 | 121 | 122 | ## History 123 | 124 | * **2020-07-31, v0.4:** No meaningful code changes, but added a small Bash (Linux) script to make a large single-column CSV file for testing. You should be able to run it within Bash with `./create-large-csv.sh`. You might have to `chmod +x create-large-csv.sh` first. 125 | * **2020-07-25, v0.3:** Going through the issues on GitHub and saw #3 which looked like an easy addition, so now if a third, optional field is specified in the URL, that is used as a title. 126 | * **2020-07-25, v0.2:** From a bug report via email about it running slowly processing thousands of rows, I've attempted a 'fix' by creating a title from the URL and passing that to the YOURLS function that would otherwise attempt to fetch one from the URL's HTML. 127 | * **2014-07-17, v0.1:** Still a work in progress. 128 | 129 | ## Finally... 130 | 131 | I hope you find this plugin useful. 132 | --------------------------------------------------------------------------------