├── _config.yml ├── AsyncWebAPI.jpg └── README.md /_config.yml: -------------------------------------------------------------------------------- 1 | theme: jekyll-theme-cayman -------------------------------------------------------------------------------- /AsyncWebAPI.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/magento-engcom/bulk-api/HEAD/AsyncWebAPI.jpg -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | Archived in favor of https://github.com/magento/async-import 2 | 3 | # Bulk API Project 4 | 5 | Magento uses Web API to cover the persistence operations of the business entities from the external clients. Some integrations require massive invocation of the Web API while persisting entities from the external systems. Usually it causes performance and scalability issues both on the Magento system and on the external clients. This project is intended to provide a way for persisting big amounts of data in the Magento in efficient scalable way for the external system. 6 | 7 | ## Repositories 8 | 9 | Bulk API track consists of 3 repositories: 10 | - current, https://github.com/magento-engcom/bulk-api is for documentation and tasks tracking 11 | - https://github.com/magento/bulk-api-ce is for contributions to the CE part of the Bulk API scope 12 | - https://github.com/magento/bulk-api-ee is for contribution to the EE part of the scope 13 | 14 | ## Goals 15 | 16 | 1. Implement support of asynchronous invocation of the magento Web API endpoints: requested operation should not be performed immediately, but rather saved to the queue for the later processing. 17 | 2. Support invocation of the CRUD APIs with the multiple entities in one request 18 | 3. Support of the status tracking API for asynchronous operations 19 | 4. Improve performance of persistence operations for the key entities like catalog in the Magento business logic. Resolve deadlocks. 20 | 21 | ## Design 22 | Technical vision for the Bulk API: https://github.com/magento-engcom/bulk-api/wiki 23 | High Level design for asynchronous operations: https://github.com/magento-engcom/bulk-api/wiki/Asynchronous-Web-API 24 | 25 | ## Tasks Board 26 | 27 | We are using [ZenHub](https://www.zenhub.com/) board to manage stories and tasks and build burndown chart for them. Please install browser plugin to get all the features of this application. 28 | 29 | The Kanban board: https://app.zenhub.com/workspace/o/magento-engcom/bulk-api 30 | 31 | ## Contributing 32 | 33 | Currently, all the functionality of the Message Queue and RabbitMQ integration is part of the Magento EE edition. Most stories as a part of the Bulk API track relies on the Message Queue interfaces. There is already a product decision made of moving the interfaces from the EE to CE edition of Magento. 34 | 35 | All the [Solution Partners](https://magento.com/find-a-partner) of Magento and developers affiliated with them should have access to the EE repositories for this project. If you don't have it yet, please write us at engcom@magento.com 36 | 37 | If you are not the Solution Partner but still want to contribute, completion of moving of the Message Queue interfaces to CE should be a prerequisite for this task. 38 | 39 | ## Installation 40 | 41 | ### Building the code of Magento from repositories 42 | 43 | ``` 44 | git clone https://github.com/magento/bulk-api-ce.git 45 | cd bulk-api-ce 46 | git clone https://github.com/magento/bulk-api-ee.git 47 | php -f bulk-api-ee/dev/tools/build-ee.php -- --ce-source=. --ee-source=bulk-api-ee --command=link 48 | ``` 49 | This will clone the repositories and create symlinks from the EE to CE. PLease note that EE is cloned to the subdirectory of CE. This is needed to resolve the templates files references. 50 | 51 | ### Running RabbitMQ 52 | 53 | 1. Start Docker container for RabbitMQ 3 with the management plugin and expose AMQP port and Admin UI port: 54 | ``` 55 | docker run --hostname localhost -p 5672:5672 -p 15672:15672 rabbitmq:3-management 56 | ``` 57 | 2. Go to http://localhost:15672/ login:guest, password:guest to check the statuses of the exchanges and queues 58 | 59 | ### Installing Magento connected to RabbitMQ 60 | 61 | ``` 62 | bin/magento setup:install 63 | --backend-frontname="admin" 64 | --amqp-host="localhost" 65 | --amqp-port="5672" 66 | --amqp-user="guest" 67 | --amqp-password="guest" 68 | --db-host="localhost" 69 | --db-name="bulk_api_ce" 70 | --db-user="root" 71 | --db-password="root" 72 | --admin-user="admin" 73 | --admin-password="admin123" 74 | --admin-email="vranen@example.com" 75 | --admin-firstname="Eugene" 76 | --admin-lastname="Tulika" 77 | --base-url="magento.url" 78 | --cleanup-database 79 | ``` 80 | ### Trying out PoC code 81 | 82 | 1. Setup Postman 83 | 2. Create Integration in the Magento backend 84 | 3. Send Web API request to Magento and ensure that it works 85 | ``` 86 | http://magento.loc/rest/V1/products POST 87 | 88 | { 89 | "product": { 90 | "sku": "simple-attempt-1-request-1", 91 | "type_id": "simple", 92 | "attribute_set_id": "4", 93 | "price": "12.22", 94 | "name": "new_name_simple-attempt-1-request-1", 95 | "extension_attributes": { 96 | "stock_item": { 97 | "qty": 0 98 | } 99 | }, 100 | "custom_attributes": [ 101 | { 102 | "attribute_code": "ap21_size_code", 103 | "value": "123" 104 | }, 105 | { 106 | "attribute_code": "ap21_style_code", 107 | "value": "abc" 108 | }, 109 | { 110 | "attribute_code": "special_price", 111 | "value": "12.11" 112 | }, 113 | { 114 | "attribute_code": "color", 115 | "value": "4121" 116 | }, 117 | { 118 | "attribute_code": "size", 119 | "value": "3242" 120 | } 121 | ] 122 | } 123 | } 124 | ``` 125 | 4. add `/async` prefix to the Web API endpoint `rest/V1/products` => `rest/async/V1/products` 126 | 5. Send request. Navigate to RabbitMQ and see message in the Queue 127 | 6. Navigate to the System => Bulk Actions to see the UI of bulk operations and new operation there 128 | 129 | --------------------------------------------------------------------------------