└── README.md /README.md: -------------------------------------------------------------------------------- 1 | # Bulk API Project 2 | 3 | Magento uses Web API to cover the persistence operations of the business entities from the external clients. Some integrations require massive invocation of the Web API while persisting entities from the external systems. Usually it causes perfoemance and scalability issues both on the Magento system and on the external clients. This project is intended to provide a way for persisting big amounts of data in the Magento in efficient scalable way for the external system. 4 | 5 | Documentation https://github.com/magento/bulk-api-ce/wiki 6 | 7 | ## Goals 8 | 9 | 1. Implement support of asynchronous invocation of the magento Web API endpoints: requested operation should not be performed imediatedly, but rather saved to the queue for the later processing. 10 | 2. Support invocation of the CRUD APIs with the multiple entities in one request 11 | 3. Support of the status tracking API for asyncronous operations 12 | 4. Improve performance of persistence operations for the key entities like catalog in the Magento business logic. Resolve deadlocks. 13 | 14 | ## Design 15 | Technical vision for the Bulk API: https://github.com/magento-engcom/bulk-api/wiki 16 | High Level design for asynchronous operations: https://github.com/magento-engcom/bulk-api/wiki/Asynchronous-Web-API 17 | 18 | ## Tasks Board 19 | 20 | We are using [ZenHub](https://www.zenhub.com/) board to manage stories and tasks and build burndown chart for them. Please install browser pluin to get all teh features of this application. 21 | 22 | The Kanban board: https://app.zenhub.com/workspace/o/magento-engcom/bulk-api 23 | 24 | ## Contributing 25 | 26 | Currently, all the functionality of the Message Queue and RabbitMQ integration is part of the Magento EE edition. Most stories as a part of the Bulk API track relies on the Message Queue interfaces. There is already a product decision made of moving the interfaces from the EE to CE edition of Magento. 27 | 28 | All the [Solution Partners](https://magento.com/find-a-partner) of Magento and developers affiliated with them should have access to the EE repositories for this project. If you don't have it yet, please write us at engcom@magento.com 29 | 30 | If you are not the Solution Partner but still want to contribute, completion of moving of the Message Queue interfaces to CE should be a prerequisite for this task. 31 | 32 | ## Installation 33 | 34 | ### Building the code of Magento from repositories 35 | 36 | ``` 37 | git clone https://github.com/magento/bulk-api-ce.git 38 | cd bulk-api-ce 39 | git clone https://github.com/magento/bulk-api-ee.git 40 | php -f bulk-api-ee/dev/tools/build-ee.php -- --ce-source=. --ee-source=bulk-api-ee --command=link 41 | ``` 42 | This will clone the repositories and create symlinks from the EE to CE. PLease note that EE is cloned to the subdirectory of CE. This is needed to resolve the templates files references. 43 | 44 | ### Running RabbitMQ 45 | 46 | 1. Start Docker container for RabbitMQ 3 with the management plugin and expose AMQP port and Admin UI port: 47 | ``` 48 | docker run --hostname localhost -p 5672:5672 -p 15672:15672 rabbitmq:3-management 49 | ``` 50 | 2. Go to http://localhost:15672/ login:guest, password:gust to check the statuses of the exchanges and queues 51 | 52 | ### Installing Magento connected to RabbitMQ 53 | 54 | ``` 55 | bin/magento setup:install 56 | --backend-frontname="admin" 57 | --amqp-host="localhost" 58 | --amqp-port="5672" 59 | --amqp-user="guest" 60 | --amqp-password="guest" 61 | --db-host="localhost" 62 | --db-name="bulk_api_ce" 63 | --db-user="root" 64 | --db-password="root" 65 | --admin-user="admin" 66 | --admin-password="admin123" 67 | --admin-email="vranen@gmail.com" 68 | --admin-firstname="Eugene" 69 | --admin-lastname="Tulika" 70 | --base-url="magento.url" 71 | --cleanup-database 72 | ``` 73 | ### Trying out PoC code 74 | 75 | 1. Setup Postman 76 | 2. Create Integration in the Magento backend 77 | 3. Send Web API request to Magento and ensure that it works 78 | ``` 79 | http://magento.loc/rest/V1/products POST 80 | 81 | { 82 | "product": { 83 | "sku": "simple-attempt-1-request-1", 84 | "type_id": "simple", 85 | "attribute_set_id": "4", 86 | "price": "12.22", 87 | "name": "new_name_simple-attempt-1-request-1", 88 | "extension_attributes": { 89 | "stock_item": { 90 | "qty": 0 91 | } 92 | }, 93 | "custom_attributes": [ 94 | { 95 | "attribute_code": "ap21_size_code", 96 | "value": "123" 97 | }, 98 | { 99 | "attribute_code": "ap21_style_code", 100 | "value": "abc" 101 | }, 102 | { 103 | "attribute_code": "special_price", 104 | "value": "12.11" 105 | }, 106 | { 107 | "attribute_code": "color", 108 | "value": "4121" 109 | }, 110 | { 111 | "attribute_code": "size", 112 | "value": "3242" 113 | } 114 | ] 115 | } 116 | } 117 | ``` 118 | 4. add `/async` prefix to the Web API endpoint `rest/V1/products` => `rest/async/V1/products` 119 | 5. Send request. Navigate to RabbitMQ and see message in the Queue 120 | 6. Navigate to the System => Bulk Actions to see the UI of bulk operations and new operation there 121 | 122 | --------------------------------------------------------------------------------