├── infra ├── docs │ └── SparseDesign-V2.docx ├── meetings │ ├── 006-20250408.md │ ├── 011-20250909.md │ ├── 004-20210309.md │ ├── 002-20200806.md │ ├── 001-20190701.md │ ├── 003-20201120.md │ └── 007-20250513.md └── README.md ├── models-tutorials ├── docs │ ├── ci_proposal │ │ ├── workflow.png │ │ └── components.png │ ├── onnx-workshop-modelzoo-SIG-update.pdf │ ├── ProposedModels.md │ ├── CommunityEvents.md │ └── MissionStatement.md ├── meetings │ ├── 007-20240313.md │ ├── 001-20200130.md │ ├── 003-20200413.md │ ├── 006-20200925.md │ ├── 002-20200309.md │ ├── 005-20200814.md │ └── 004-20200615.md └── README.md ├── operators ├── docs │ ├── Operator SIG ONNX Workshop April09-2020.pdf │ ├── Operator SIG ONNX Workshop August23-2019.pdf │ ├── Operator SIG ONNX Workshop October14-2020.pdf │ ├── RemovingOperatorOrFunction.md │ └── SubmittingNewOperatorOrFunction.md ├── meetings │ ├── 056-20250327.md │ ├── 011-20200129.md │ ├── 004-20190807.md │ ├── 005-20190822.md │ ├── 013-20200416.md │ ├── 014-20200429.md │ ├── 006-20190911.md │ ├── 018-20210120.md │ ├── 057-20250424.md │ ├── 017-20200922.md │ ├── 012-20200212.md │ ├── 003-20190711.md │ ├── 008-20191023.md │ ├── 007-20191002.md │ ├── 015-20200623.md │ ├── 010-20191204.md │ ├── 009-20191106.md │ ├── 002-20190626.md │ ├── 016-20200724.md │ ├── 001-20190611.md │ ├── 020-20210415.md │ ├── 035-20221027.md │ ├── 051-20240822.md │ ├── 042-20230831.md │ ├── 027-20220127.md │ ├── 030-20220526.md │ ├── 055-20250123.md │ ├── 052-20240926.md │ ├── 047-20240222.md │ ├── 022-20210610.md │ ├── 019-20210318.md │ ├── 040-20230608.md │ ├── 041-20230713.md │ ├── 053-20241024.md │ ├── 044-20231023.md │ ├── 050-20240725.md │ ├── 026-20210930.md │ ├── 032-20220728.md │ └── 039-20230504.md └── README.md ├── converters ├── docs │ ├── ConvertersDashboard.md │ ├── IssuesAndSolutions.md │ └── ConvertersList.md ├── meetings │ ├── 018-20210114.md │ ├── 019-20210212.md │ ├── 022-20210820.md │ ├── 014-20200724.md │ ├── 005-20190919.md │ ├── 014-20200612.md │ ├── 021-20210625.md │ ├── 009-20200116.md │ ├── 016-20201120.md │ ├── 007-20191024.md │ ├── 020-20210507.md │ ├── 008-20191206.md │ ├── 015-20200911.md │ ├── 013-20200522.md │ ├── 002-20190724.md │ ├── 004-20190905.md │ ├── 017-20201211.md │ ├── 003-20190814.md │ ├── 010-20200214.md │ ├── 012-20200424.md │ ├── 011-20200313.md │ └── 001-20190710.md └── README.md ├── optimizations ├── meetings │ ├── 20231005.md │ └── 20231218.md └── README.md ├── compilers ├── meetings │ └── sig-compiler-2023-04-04.md └── README.md ├── best-practices.md └── README.md /infra/docs/SparseDesign-V2.docx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/onnx/sigs/HEAD/infra/docs/SparseDesign-V2.docx -------------------------------------------------------------------------------- /models-tutorials/docs/ci_proposal/workflow.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/onnx/sigs/HEAD/models-tutorials/docs/ci_proposal/workflow.png -------------------------------------------------------------------------------- /models-tutorials/docs/ci_proposal/components.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/onnx/sigs/HEAD/models-tutorials/docs/ci_proposal/components.png -------------------------------------------------------------------------------- /models-tutorials/docs/onnx-workshop-modelzoo-SIG-update.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/onnx/sigs/HEAD/models-tutorials/docs/onnx-workshop-modelzoo-SIG-update.pdf -------------------------------------------------------------------------------- /operators/docs/Operator SIG ONNX Workshop April09-2020.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/onnx/sigs/HEAD/operators/docs/Operator SIG ONNX Workshop April09-2020.pdf -------------------------------------------------------------------------------- /operators/docs/Operator SIG ONNX Workshop August23-2019.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/onnx/sigs/HEAD/operators/docs/Operator SIG ONNX Workshop August23-2019.pdf -------------------------------------------------------------------------------- /operators/docs/Operator SIG ONNX Workshop October14-2020.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/onnx/sigs/HEAD/operators/docs/Operator SIG ONNX Workshop October14-2020.pdf -------------------------------------------------------------------------------- /infra/meetings/006-20250408.md: -------------------------------------------------------------------------------- 1 | # Tuesday April 08, 2025 at 22 CEST 2 | 3 | # Agenda 4 | * Release Preparation for 1.18 5 | * Preparation for 1.18.0rc1 6 | * Update Versions in various files in rel-1.18.0 7 | * TODO: verson updates have to be done in main 8 | 9 | # Attendees 10 | * Justin Chu 11 | * Andreas Fehlner 12 | * Ramakrishnan Sivakumar 13 | * Raghavan Naresh Sarangapani 14 | -------------------------------------------------------------------------------- /infra/meetings/011-20250909.md: -------------------------------------------------------------------------------- 1 | # Tuesday Septembert 9, 2025 at 22 CEST 2 | 3 | ## Access 4 | * use zoom-link at https://zoom-lfx.platform.linuxfoundation.org/meetings/lfai-onnx?view=month 5 | 6 | ## Participants 7 | * 8 | 9 | Next Steps 10 | * 11 | 12 | ## Agenda 13 | * 1.) Release 1.19 - Status 14 | * 2.) Release Process Review, Lessons Learned 15 | * 3.) pypi onnx organization 16 | * 4.) abi3 17 | 18 | 19 | ## pypi onnx organization 20 | 21 | ## abi3 22 | 23 | 24 | -------------------------------------------------------------------------------- /infra/meetings/004-20210309.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | # Tuesday March 09, 2021 at 9:30am PST 4 | 5 | ## Agenda 6 | * Recurrent Arch Infra Sig meetings update 7 | * Onnx upcoming release updates 8 | * Recent pipeline updates 9 | * Upcoming enhancements/updates for ONNX 10 | * Gather feedback from attendees on meeting structure for future meetings 11 | 12 | ## Attendees 13 | * Ashwini Khade (Microsoft) 14 | * Jacky Chen (Microsoft) 15 | 16 | This meeting was cancelled for the lack of quorum. We will reschedule it for next week. 17 | -------------------------------------------------------------------------------- /infra/meetings/002-20200806.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | # Tuesday August 06, 2020 at 7:00am CST 4 | 5 | The reason 6 | 1. many depencencies projects are using c++ optimizer now 7 | 2. Buggy and broken when IR version >=4 8 | 9 | Plan 10 | 1. Move before 1.9 release 11 | 2. Bug fix (support IR version >= 4 and the bug of fuse bn into conv) 12 | 3. Improve 13 | 1. New API 14 | 2. Unfold function ops 15 | 3. Pattern matching 16 | 17 | Action 18 | 1. More details plan (ETA) 19 | 2. The redesigned IR show be thought more 20 | 21 | Another thing 22 | Remove circle ci 23 | -------------------------------------------------------------------------------- /converters/docs/ConvertersDashboard.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | # Coverage reports for importers (convert ONNX to framework) 4 | | Converter | ONNX Version | Framework Version | Detailed Report | 5 | | -------------- |:------------------:|:------------------:|:------------:| 6 | |onnx-tf 1.5.0|1.5.0|Tensorflow 1.14|https://github.com/onnx/onnx-tensorflow/blob/master/doc/support_status.md| 7 | 8 | # Coverage reports for exporters (convert framework to ONNX) 9 | | Converter | Framework version | 10 | | -------------- |:------------------:| 11 | 12 | # Coverage reports for rutimes (convert ONNX to run on backend) 13 | | Runtime | ONNX version | 14 | | -------------- |:------------------:| 15 | -------------------------------------------------------------------------------- /operators/meetings/056-20250327.md: -------------------------------------------------------------------------------- 1 | # ONNX Operators SIG Meeting 2 | # Thursday, March 27th, 2025 at 9:00am PST 3 | 4 | ## Agenda 5 | 6 | * Open Discussion 7 | 8 | ## Attendees 9 | 10 | * Andreas Fehlner (TRUMPF) 11 | * Ganesan Ramalingam (Microsoft) 12 | * Michał Karzyński (Intel) 13 | * Sunny Anand (IBM) 14 | * Yuan Yao (NVIDIA) 15 | 16 | ## Meeting Minutes 17 | 18 | ### Open Discussion 19 | 20 | An open discussion was held on a number of topics: 21 | 22 | * status of the ONNX release 1.18 23 | * the idea to document a release schedule for ONNX with a regular cadence 24 | * release manager role should continue until a new manager is appointed, including the responsibility for point releases 25 | * onboarding new contributors and learning resources for ONNX 26 | -------------------------------------------------------------------------------- /operators/meetings/011-20200129.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | # Wednesday January 29, 2020 at 9:00am PST 4 | 5 | ## Agenda 6 | * ONNX Training PR: https://github.com/onnx/onnx/pull/2314 7 | * Alibaba auto generated test framework 8 | 9 | ## Attendees 10 | * Emad Barsoum (Microsoft) 11 | * Michal Karzynski (Intel) 12 | * Ganesan Ramalingam (Microsoft) 13 | * Spandan Tiwari (Microsoft) 14 | * Leonid Goldgeisser (Habana) 15 | * Weiming Zhao (Alibaba) 16 | * Wei-Sheng Chin (Microsoft) 17 | * Scott Cyphers (Intel) 18 | * Liqun Fu (Microsoft) 19 | 20 | ## Notes 21 | * Wei-Sheng gave a walkthrough the ONNX training PR. 22 | 23 | ## Action items 24 | * Continue reviewing ONNX training. 25 | * Alibaba - sharing the code, any documentation and prepare a presentation for next meeting. 26 | 27 | -------------------------------------------------------------------------------- /converters/meetings/018-20210114.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | # Thursday Jan 14 2021 5-6pm PST 4 | * https://zoom.us/j/7376656864?pwd=SjdiaGdvUFFyVXM4OUxTWndYZ2Z1Zz09 5 | 6 | ## Agenda 7 | * Greetings! 8 | * Discussion topics 9 | * Paddle to ONNX presentation 10 | 11 | ## Attendees 12 | * Chin Huang (onnx-tf, IBM) 13 | * Ti Zhou (paddle-onnx, Baidu) 14 | * Wranky (paddle-onnx, Baidu) 15 | * Lingchi Chen (paddle-onnx, Baidu) 16 | * Feifei Zhang (paddle-onnx, Baidu) 17 | * Jason (paddle-onnx, Baidu)) 18 | * Tiezhu Gao (paddle-onnx, Baidu) 19 | * Zeyu Chen (paddle-onnx, Baidu) 20 | 21 | ## Notes 22 | * Special presentation from PaddlePaddle team, recording available for replay at https://lists.lfaidata.foundation/g/onnx-sig-converters/files/ONNX%20Converters%20SIG%20Meeting%20Recordings/converters_sig_0114.mp4 23 | -------------------------------------------------------------------------------- /operators/meetings/004-20190807.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | # Wednesday August 7, 2019 at 9:00am PST 4 | 5 | ## Agenda 6 | * Discuss catching up with current operator PRs. 7 | 8 | ## Attendees 9 | * Emad Barsoum (Microsoft) 10 | * Michal Karzynski (Intel) 11 | * Wei-Sheng Chin (Microsoft) 12 | * Dilip Seqeira (NVidia) 13 | * Darren S Crews (Intel) 14 | 15 | ## Notes 16 | * Each member in the SIG, will allocate time in their calendar to review ONNX Operator issues in Github. 17 | * All operator PRs and issues need to adhere to the updated `add new operator or function document` in: 18 | * https://github.com/onnx/onnx/blob/master/docs/AddNewOp.md 19 | 20 | ## Action items 21 | * ebarsoum and Michal - will triage the operator list in Github and start assigning owners. 22 | * Next meeting: 23 | * ONNX Workshop presentation. 24 | * Check operator status. 25 | 26 | -------------------------------------------------------------------------------- /operators/meetings/005-20190822.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | # Thursday August 22, 2019 at 9:00am PST 4 | 5 | ## Agenda 6 | * Discuss current operator PRs and issues status. 7 | * ONNX Workshop SIG operator presentation and track. 8 | 9 | ## Attendees 10 | * Emad Barsoum (Microsoft) 11 | * Michal Karzynski (Intel) 12 | * Wei-Sheng Chin (Microsoft) 13 | * Dilip Sequeira (NVidia) 14 | * Spandan Tiwari (Microsoft) 15 | * Rajeev K Nalawadi (Intel) 16 | * Ganesan Ramalingam (Microsoft) 17 | * Simon Long (GraphCore) 18 | 19 | ## Notes 20 | * ONNX Workshop: 21 | * Review a list of operators in the Workshop 22 | * Support -1 officially in all OPs 23 | * Update description mean bump version number 24 | * Start assigning PR to people 25 | * Python script that general doc, need to run mannually to avoid surprises 26 | 27 | ## Action items 28 | * Start review complex PR and issues in our regular meeting 29 | 30 | -------------------------------------------------------------------------------- /operators/meetings/013-20200416.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | # Thursday April 16, 2020 at 9:00am PST 4 | 5 | ## Agenda 6 | * Code style. 7 | * Revise adding new operator requirements. 8 | * Discuss PR and Issues. 9 | 10 | ## Attendees 11 | * Emad Barsoum (Microsoft) 12 | * Spandan Tiwari (Microsoft) 13 | * Jonny Shipton (Myrtle.Ai) 14 | * Zeeshan Siddiqui (Microsoft) 15 | * Wei-Sheng Chin (Microsoft) 16 | * Ganesan Ramalingam (Microsoft) 17 | 18 | ## Notes 19 | * Most of the meetings spend on how to improve the quality of adding new operator in ONNX standard. 20 | * Generate test data from the corresponding framework that has the proposed operator. 21 | * Corner cases coverage, can we automate that? 22 | * Shape inference. 23 | * Same coverage as corresponding framework. 24 | * An appropriate place for reference implementation. 25 | 26 | ## Action items 27 | * Update the AddNewOP document with the proposed changes. 28 | -------------------------------------------------------------------------------- /operators/meetings/014-20200429.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | # Wednesady April 29, 2020 at 9:30am PST 4 | 5 | ## Agenda 6 | * Discuss preview/experimental domain. 7 | * Revise adding new operator requirements. 8 | 9 | ## Attendees 10 | * Emad Barsoum (Microsoft) 11 | * Michal Karzynski (Intel) 12 | * Spandan Tiwari (Microsoft) 13 | * Jonny Shipton (Myrtle.Ai) 14 | * Zeeshan Siddiqui (Microsoft) 15 | * Wei-Sheng Chin (Microsoft) 16 | * Ganesan Ramalingam (Microsoft) 17 | * Ke Zhang (Alibaba) 18 | * Svetlana Levitan (IBM) 19 | * Chin Huang (IBM) 20 | * Weiming Zhao (Alibaba) 21 | 22 | 23 | ## Notes 24 | * Most of the meeting spend in discussing preview/experimental domain. 25 | * Add new operator update, we almost agree on the high level. 26 | 27 | ## Action items 28 | * (Emad Barsoum) Update the AddNewOP document with the proposed changes. 29 | * (Emad Barsoum) Write a document that explain the semantic of preview domain. 30 | -------------------------------------------------------------------------------- /operators/meetings/006-20190911.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | # Wednesday September 11, 2019 at 9:00am PST 4 | 5 | ## Agenda 6 | * Reflect on last workshop 7 | * ONNX 1.6 8 | * Discuss deprecated Ops 9 | * Review a couple of PRs 10 | 11 | ## Attendees 12 | * Emad Barsoum (Microsoft) 13 | * Michal Karzynski (Intel) 14 | * Wei-Sheng Chin (Microsoft) 15 | * Dilip Sequeira (NVidia) 16 | * Spandan Tiwari (Microsoft) 17 | * Rajeev K Nalawadi (Intel) 18 | 19 | ## Notes 20 | * Discussed the feedback that we got from ONNX Workshop. 21 | * Discussed Deprecated OPs: 22 | * Any deprecated OP must have equivalent or can be compsoed by existing OPs. 23 | * Convert deprecated OPs to a function if it can be composed. 24 | * No need for a grace period. 25 | * We need to review all operators PR that are tagged for 1.6. 26 | 27 | ## Action items 28 | * ebarsoum - to write a draft document for the recommended behavior of deprecated OPs. 29 | 30 | -------------------------------------------------------------------------------- /models-tutorials/meetings/007-20240313.md: -------------------------------------------------------------------------------- 1 | # March 26, 2024 2 | 3 | ### Sig Lead 4 | 5 | * New Sig Lead: 6 | 7 | ### Model Signing (Andreas) 8 | 9 | * The Working Group for AI/ML from OpenSSF (https://github.com/ossf/ai-ml-security ) had a presentation about modeling signing on 2024-03-04 10 | * Meeting Notes of that Group: https://docs.google.com/document/d/1YNP-XJ9jpTjM6ekKOBgHH8-avAws2DVKeCpn858siiQ/edit#heading=h.etrsjlz02gla 11 | * Slides: https://docs.google.com/presentation/d/1ncyxiNFOkAauR2GFEmo09qcOs9dFN3UD861rFAs6uMk/edit?usp=sharing 12 | * Recording: https://zoom.us/rec/share/BfKHrOhxWVdjXryZuh9dzWSbXsi-Zwl8wGRwgdtH_00m2I5XHc2NI1wn-H8CJ7b0.W2strYgdt5i_H61S 13 | * Repository: https://github.com/google/model-transparency 14 | * Proposal for a SIG for model signing and transparency: https://github.com/ossf/ai-ml-security/issues/10 15 | 16 | Issues in our project 17 | 18 | * https://github.com/onnx/onnx/issues/5262 19 | * https://github.com/onnx/onnx/issues/4046 20 | * https://github.com/onnx/turnkeyml/issues/52 21 | -------------------------------------------------------------------------------- /operators/meetings/018-20210120.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | # Wednesday January 20, 2020 at 9:00am PST 4 | 5 | ## Agenda 6 | * Change of Operators SIG co-lead Emad Barsoum to Ganesan "Rama" Ramalingam. Thank you for your hard work Emad. 7 | * Set up recurrent Operators SIG meeting every 4 weeks on Thursday 8 | * Discussion of tests for older operator versions. 9 | * https://github.com/onnx/onnx/issues/2912 10 | * https://github.com/onnx/onnx/issues/2917 11 | * https://github.com/onnx/onnx/pull/3020 12 | * Discussed introducing Fourier Transform operators into ONNX 13 | * https://github.com/onnx/onnx/pull/2625 14 | 15 | ## Attendees 16 | * Ashwini Khade (Microsoft) 17 | * Emad Barsoum (Cerebras) 18 | * Evgeny Lazarev (Intel) 19 | * Evgenya Stepyreva (Intel) 20 | * Ganesan "Rama" Ramalingam (Microsoft) 21 | * Itay Hubara (Intel) 22 | * Jacky Chen (Microsoft) 23 | * Michal Karzynski (Intel) 24 | * Rajeev Nalawadi (Intel) 25 | * Scott Cyphers (Intel) 26 | * Sergey Lyalin (Intel) 27 | * Spandan Tiwari (Microsoft) 28 | -------------------------------------------------------------------------------- /optimizations/meetings/20231005.md: -------------------------------------------------------------------------------- 1 | ## SIG Optimization 10/05/2023 2 | 3 | The kick-off of the new SIG Optimizations was October 5th, at 9AM PST. https://lists.lfaidata.foundation/g/onnx-sig-optimizations/calendar. 4 | 5 | Meeting Notes from the first optimization SIG meeting 6 | 7 | Attendees : Neta Zmora (NVidia), Ramakrishna Tipireddy (General Motors) and Freddy Chiu (Intel) 8 | 9 | The optimization workgroup met for the first time to introduce the Optimizations SIG to the community. Group discussed the charter of the SIG and elicited topics from the community. We introduced the SIG’s co-chair, Neta Zmora from NVIDIA. As a SIG, we plan on meeting regularly at a monthly cadence. As next steps, we plan to do a second introductory session to get wider community engagement on topics and agenda for future discussion under this SIG. we look forward to this in November. 10 | Freddy had some challenges with setting up a zoom link which should be addressed by the next sync. Please track this channel for announcement of the next meeting in addition to the mailing list. 11 | -------------------------------------------------------------------------------- /converters/README.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | # ONNX Converters Special Interest Group (SIG) 4 | 5 | This is the working repo for the converters special interest group (SIG). 6 | This repo contains all the artifacts, materials, meeting notes and proposals regarding evolution of converters in ONNX. Feedbacks and contributions are welcome. 7 | 8 | # Slack channel 9 | Please sign up at https://slack.lfai.foundation/ and join [onnx-converters](https://lfaifoundation.slack.com/archives/C0171FSKZBN) channel. 10 | 11 | # SIG Leads 12 | 13 | * Xavier Dupre (Microsoft) 14 | * Kevin Chen (NVIDIA) 15 | 16 | # Logistics 17 | 18 | * SIG leads will drive the meeting. 19 | * Meeting annoucement will be posted in our Slack channel 20 | * Feedbacks and topic request are welcome by all. 21 | 22 | # Discussion 23 | 24 | * Slack channel: https://lfaifoundation.slack.com/archives/C0171FSKZBN 25 | * Documents and artifacts: https://github.com/onnx/sigs/tree/main/converters 26 | 27 | # Meeting notes 28 | 29 | The meeting notes can be found [here](https://github.com/onnx/sigs/tree/main/converters/meetings) 30 | 31 | -------------------------------------------------------------------------------- /operators/meetings/057-20250424.md: -------------------------------------------------------------------------------- 1 | # ONNX Operators SIG Meeting 2 | # Thursday, April 24th, 2025 at 9:00am PST 3 | 4 | ## Agenda 5 | 6 | * Update on the new ONNX GenAI working group 7 | * Infrastructure SIG updates 8 | 9 | ## Attendees 10 | 11 | * Andreas Fehlner (TRUMPF) 12 | * Ganesan Ramalingam (Microsoft) 13 | * Michał Karzyński (Intel) 14 | * Sunny Anand (IBM) 15 | * Yuan Yao (NVIDIA) 16 | 17 | ## Meeting Minutes 18 | 19 | ### ONNX GenAI working group 20 | 21 | A new working group was formed to make progress on changes to ONNX specification related to generative AI models. 22 | A kickoff meeting was held, Rama shared [slides](https://docs.google.com/presentation/d/1PYAHauEVhhdTuKMYiOsOjOVz6u8OzWka) 23 | from the meeting. 24 | 25 | 26 | ### Infrastructure SIG updates 27 | 28 | The ONNX 1.18 release was discussed. The Release Candidate RC1 was already published, RC2 is under preparation. 29 | New features will include static code analysis improvements, support for Python 3.13 support, including free-threading 30 | variant and support for Windows ARM64. Concern was raised about limited communication of ONNX releases during the 31 | RC phase and limited community testing. 32 | -------------------------------------------------------------------------------- /operators/meetings/017-20200922.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | # Tuesday September 22, 2020 at 9:00am PST 4 | 5 | ## Agenda 6 | * Discuss the following PRs: 7 | * https://github.com/onnx/onnx/pull/2922 8 | * https://github.com/onnx/onnx/pull/2284 9 | * https://github.com/onnx/onnx/issues/2159 10 | * Discussed version OPSet converter. 11 | 12 | ## Attendees 13 | * Emad Barsoum (Microsoft) 14 | * Michal Karzynski (Intel) 15 | * Spandan Tiwari (Microsoft) 16 | * Scott Cyphers (Intel) 17 | * Matteo Salvarezza 18 | * Wei-Sheng Chin (Microsoft) 19 | 20 | ## Notes 21 | * We discussed `time_major` flag for PR2922, and decided to add the flag for existing recurrent operators in order to be closer to existing framework and simplify the conversion from runtime that don't expose batch axis. 22 | * We didn't close on a decision regareding enforcing updating OPSet converter for each PR. The concern is the extra overhead added for OP PRs. 23 | * Reshape PRs, was brought up at the end of the meeting, so we didn't have time to discuss it. 24 | * https://github.com/onnx/onnx/pull/2552 25 | * https://github.com/onnx/onnx/pull/2553 26 | 27 | ## Action items 28 | * (Emad & Michal) Prepare the operator SIG presentation for the ONNX workshop. 29 | -------------------------------------------------------------------------------- /operators/meetings/012-20200212.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | # Wednesday February 12, 2020 at 9:00am PST 4 | 5 | ## Agenda 6 | * ONNX Training PR: https://github.com/onnx/onnx/pull/2314 7 | * Alibaba auto generated test framework 8 | * CircleCI stability 9 | * Operator PRs backlog. 10 | 11 | ## Attendees 12 | * Emad Barsoum (Microsoft) 13 | * Michal Karzynski (Intel) 14 | * Ganesan Ramalingam (Microsoft) 15 | * Spandan Tiwari (Microsoft) 16 | * Wei-Sheng Chin (Microsoft) 17 | * Scott Cyphers (Intel) 18 | * Liqun Fu (Microsoft) 19 | * Prasanth Pulavarthi (Microsoft) 20 | 21 | ## Notes 22 | * Discussed optimizer and loss function for the training PR. 23 | * Spend most of the time debating initializer versus mutable_initializer. 24 | * mutable_initializer need to be string corresponding to tensors in the initializer list that are mutable. 25 | * Concern about backward compatibility was raised. 26 | * Remove PyTorch tests from ONNX repo, we should block ONNX PR because of a framework specific test. 27 | * We got Alibaba presentation, but no one from Alibaba attended. 28 | 29 | ## Action items 30 | * Finish reviewing the training PR this week. 31 | * Alibaba - sharing the code, any documentation and prepare a presentation for next meeting. 32 | 33 | -------------------------------------------------------------------------------- /operators/meetings/003-20190711.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | # Thursday July 11, 2019 at 9:00am PST 4 | 5 | ## Agenda 6 | * Finalize the operator document 7 | * Discuss contributors 8 | * Discuss organize operators by domain 9 | 10 | ## Attendees 11 | * Emad Barsoum (Microsoft) 12 | * Michal Karzynski (Intel) 13 | * Ke Zhang (Microsoft) 14 | * Darren S Crews (Intel) 15 | * Ganesan Ramalingam (Microsoft) 16 | * Spandan Tiwari (Microsoft) 17 | * Wei-Sheng Chin (Microsoft) 18 | * Prasanth Pulavarthi (Microsoft) 19 | 20 | ## Notes 21 | * We finalized the new guidance for proposing and submitting a new operator, and everyone signed off on the new proposal. 22 | * We reviewed 3 PRs togethers with the lens of the new guidline. 23 | * Discuss the list of operator contributors to review PRs. 24 | * Agreed not to move existing operator to a new domain, however, training and data preprocessing operators will be added to a new domain. 25 | 26 | ## Action items 27 | * ebarsoum - turn the operator proposal to markup file and submit it to ONNX. 28 | * ebarsoum - aggregate the list of proposed contributors and share it with the community. 29 | * ebarsoum - check Alibaba auto generated ONNX operator tests. 30 | * Next meeting: 31 | * Discuss catching up on current operator PRs. 32 | 33 | -------------------------------------------------------------------------------- /models-tutorials/docs/ProposedModels.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | # Proposed Models for the Model Zoo 4 | ### Slated Models 5 | 6 | 1) EfficientNet-lite 7 | https://github.com/onnx/tensorflow-onnx/tree/master/tutorials 8 | 9 | 2) EfficientNet-det + EfficientNet-edge 10 | (Waiting for perf to improve) 11 | https://github.com/onnx/tensorflow-onnx/tree/master/tutorials 12 | 13 | 3) SSD-MobileNet v2 14 | https://github.com/onnx/tensorflow-onnx/tree/master/tutorials 15 | 16 | 4) YOLO v4 17 | 18 | 5) BERT variations 19 | - PyTorch BERT export 20 | - RoBERTa 21 | - mobile-BERT 22 | - mobile-albert 23 | 24 | 6) Other TF models 25 | - MobileNetEdgeTPU 26 | - DeepLab 27 | 28 | 7) Deep Voice 3 29 | 30 | 8) Visual Q & A - Bilinear Attention Networks 31 | https://github.com/jnhwkim/ban-vqa 32 | 33 | ### Suggestions from Svetlana Levitan, IBM 34 | 35 | 1) Audio Classifier 36 | https://github.com/IBM/MAX-Audio-Classifier 37 | 38 | 2) Toxic Comment Classifier 39 | Detect 6 types of toxicity in user comments 40 | https://developer.ibm.com/exchanges/models/all/max-toxic-comment-classifier/ 41 | 42 | 3) Text Sentiment Classifier 43 | Detect the sentiment captured in short pieces of text 44 | https://developer.ibm.com/exchanges/models/all/max-text-sentiment-classifier/ 45 | 46 | 4) Speech to Text Converter 47 | Converts spoken words into text form. 48 | https://developer.ibm.com/technologies/artificial-intelligence/models/max-speech-to-text-converter/ 49 | -------------------------------------------------------------------------------- /operators/meetings/008-20191023.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | # Wednesday October 23, 2019 at 9:00am PST 4 | 5 | ## Agenda 6 | * Review removing operator document proposal 7 | * Link: https://1drv.ms/w/s!Aq5Cu4d76dsCja9NGwFCla3uGw-9Dw?e=hZZDz4 8 | * Einsum operator 9 | * Discussion Function 10 | 11 | ## Attendees 12 | * Emad Barsoum (Microsoft) 13 | * Michal Karzynski (Intel) 14 | * Wei-Sheng Chin (Microsoft) 15 | * Spandan Tiwari (Microsoft) 16 | * Dilip Sequeira (NVidia) 17 | * Itay Hubara (Habana) 18 | * Leonid Goldgeisser (Habana) 19 | 20 | ## Notes 21 | * We discussed and agreed on the operator removal proposal. 22 | * Most of the discussion boiled down, one how to make it easy for the user to know what to do. 23 | * Archive older operator.md for each release. 24 | * Add link to changelog in operator.md. 25 | * Investigate adding the list of removed operator to the main operator.md. 26 | * Einsum 27 | * Add it as operator to ONNX. 28 | * Check if other operators can be replaced by a function. 29 | * Open question: should this be in core ONNX or different domain? 30 | * Function: didn't have time to discuss it. 31 | 32 | ## Action items 33 | * Emad Barsoum - submit the operator removal requirements to ONNX master. 34 | * Michal Karzynski - investigate adding the list of removed operator to operator.md. 35 | * Leonid Goldgeisser - to write a draft document for the dynamic shape and share it with the rest of the group. 36 | 37 | -------------------------------------------------------------------------------- /operators/meetings/007-20191002.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | # Wednesday October 2, 2019 at 9:00am PST 4 | 5 | ## Agenda 6 | * Review dynamic shape 7 | * Link: https://github.com/goldgeisser/onnx/blob/master/docs/proposals/StaticShape4DynShapeOpsProposal.md 8 | * Review deprecated document proposal 9 | * Link: https://1drv.ms/w/s!Aq5Cu4d76dsCja9NGwFCla3uGw-9Dw?e=hZZDz4 10 | 11 | ## Attendees 12 | * Emad Barsoum (Microsoft) 13 | * Michal Karzynski (Intel) 14 | * Wei-Sheng Chin (Microsoft) 15 | * Ganesan Ramalingam (Microsoft) 16 | * Spandan Tiwari (Microsoft) 17 | * Dilip Sequeira (NVidia) 18 | * Harry Kim (Intel) 19 | * Itay Hubara (Habana) 20 | * Leonid Goldgeisser (Habana) 21 | * Ofer Rosenberg (Qualcomm) 22 | * Scott Cyphers (Intel) 23 | * Shlomo Raikin (Habana) 24 | * Darren S Crews (Intel) 25 | * Akinlawon Solomon (Qualcomm) 26 | 27 | 28 | ## Notes 29 | * We spend the entire meeting discussing the dynamic shape. 30 | * Some of the discussion, where around: 31 | * Tensor properties for dynamic or static. 32 | * Define maximum size based on dimension. 33 | * Should this part of the standard? 34 | * Implementation details vs standard. 35 | * Pad or not padding 36 | * Who specify the value? And what is the end-to-end experience? 37 | * A mechanism to specify dynamic versus static. 38 | 39 | ## Action items 40 | * Leonid Goldgeisser - to write a draft document for the dynamic shape and share it with the rest of the group. 41 | 42 | -------------------------------------------------------------------------------- /infra/meetings/001-20190701.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | # Tuesday June 06, 2019 at 8:00am PST 4 | 5 | ## Agenda 6 | * SparseTensor support in ONNX. https://github.com/onnx/onnx/pull/2019. 7 | * Remove build dependencies on external frameworks. 8 | * How to pick up first group of contributors. 9 | * Merge ONNX-ML IR. 10 | 11 | ## Attendees 12 | * Lu Fang (Facebok) 13 | * Ke Zhang (Microsoft) 14 | * Emad Barsoum (Microsoft) 15 | * Darren S Crews (Intel) 16 | * Ganesan Ramalingam (Microsoft) 17 | * Wei-Sheng Chin (Microsoft) 18 | 19 | ## Notes 20 | * SparseTensor design overall looks good. SparseTensor to support weights needs to be added. A sample model using SparseTensor should be added to confirm the design. 21 | * It's agreed that the pytorch CI will be kept and it's not a "required" for merging an ONNX PR. 22 | * It's agreed to merge ONNX-ML IR. 23 | * Attendees will send a list of contributor candidates, which will be discussed in next meeting to have initial group of contributors. 24 | * ONNX compilance may be checked in several parts. 25 | * How many operators should a runtime/framework implement so that it can claim itself ONNX compliant? 26 | * Model & OP level accuracy test cases should be added to ensure a runtime/framework is ONNX compliant. ONNX model zoo models may be picked up as the initial set of models for accuracy test. 27 | 28 | ## Action items 29 | * Lu & Darren - review the SparseTensor PR. 30 | * Rama - create sample model of using SparseTensor. 31 | * Ke - Merge ONNX-ML IR. 32 | -------------------------------------------------------------------------------- /operators/meetings/015-20200623.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | # Tuesday June 23, 2020 at 9:00am PST 4 | 5 | ## Agenda 6 | * Discuss AddNewOpV3 doc (https://github.com/onnx/sigs/blob/master/operators/docs/AddNewOpV3.md). 7 | * Softmax issue (https://github.com/onnx/onnx/issues/2289#issuecomment-646967559). 8 | 9 | ## Attendees 10 | * Emad Barsoum (Microsoft) 11 | * *Michal Karzynski (Intel) 12 | * Spandan Tiwari (Microsoft) 13 | * Wei-Sheng Chin (Microsoft) 14 | * Ashwini Khade (Microsoft) 15 | * Zeeshan Siddiqui (Microsoft) 16 | 17 | ## Notes 18 | * Mostly discussed the updated AddNewOpV3 doc. At high level, we all agree, next step is to gather feedbacks before submitting an official PR. 19 | * We agree on the Softmax issue discussed in https://github.com/onnx/onnx/issues/2289#issuecomment-646967559, we should update Softmax to match other frameworks. 20 | * We discussed about enforced version conversion update with new OP PR, we need to get more information about the complexity and the need of the community. 21 | * We also discussed marking differential input and output described in: 22 | * https://github.com/onnx/onnx/pull/2794 23 | * https://github.com/onnx/onnx/pull/2723 24 | * There is an issue with the current test data being part of package. We agreed to shorten the name and in the future we need to move the test data into a separate package. 25 | 26 | ## Action items 27 | * (Emad Barsoum) Finalize AddNewOpV3. 28 | * (SIG) Review differential input/output. 29 | * (SIG) Review converter proposal. 30 | -------------------------------------------------------------------------------- /operators/meetings/010-20191204.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | # Wednesday December 4, 2019 at 9:00am PST 4 | 5 | ## Agenda 6 | * Dynamic shape proposal update: 7 | * Link: https://1drv.ms/w/s!Aq5Cu4d76dsCjbFWbJajaHnH25CkXQ?e=Dabhup 8 | * Link: https://github.com/goldgeisser/onnx/blob/master/docs/proposals/StaticShape4DynShapeOpsProposal.md 9 | * Workshop update 10 | * Alibaba auto generated test framework 11 | * Removing operator update 12 | 13 | ## Attendees 14 | * Emad Barsoum (Microsoft) 15 | * Michal Karzynski (Intel) 16 | * Ganesan Ramalingam (Microsoft) 17 | * Spandan Tiwari (Microsoft) 18 | * Leonid Goldgeisser (Habana) 19 | * Weiming Zhao (Alibaba) 20 | * Wei-Sheng Chin (Microsoft) 21 | * Scott Cyphers (Intel) 22 | * Itay Hubara (Habana) 23 | 24 | ## Notes 25 | * Remove or deprecate OPs is checked in. 26 | * Alibaba auto generated unit test framework. 27 | * Reference implementations of all Ops are needed ( 28 | most of them already imeplemented) 29 | * Discuss the technical details next meeting. 30 | * Alibaba has new version with auto-gen documentation. 31 | * Workshop in China 32 | * Well attended meeting. 33 | * Discussion new ONNX version 34 | * 1.7 plan, introduce training operators. 35 | * Procedure for deprecating operator. 36 | * Dynamic shape proposal. 37 | * Dynamic shape proposal, nothing need to change in ONNX standard. 38 | 39 | ## Action items 40 | * Alibaba - sharing the code, any documentation and prepare a presentation for next meeting. 41 | 42 | -------------------------------------------------------------------------------- /converters/meetings/019-20210212.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | # Thursday Feb 12 2021 5-6pm PST 4 | * https://zoom.us/j/7376656864?pwd=SjdiaGdvUFFyVXM4OUxTWndYZ2Z1Zz09 5 | 6 | ## Agenda 7 | * Greetings! 8 | * Discussion topics 9 | * Converters project updates 10 | * Roadmap discussion 11 | 12 | ## Attendees 13 | * Chin Huang (onnx-tf, IBM) 14 | * Guenther Schmuelling (tf2onnx, Microsoft) 15 | * Kevin Chen (Tensor-RT) 16 | * Ting Su (Matlab) 17 | 18 | ## Notes 19 | * ONNX-TF fully supports opset 13, except squeeze and unsqueeze where axes becoming an input, meaning a tensor instead list of ints. The issue is the corresponding TF APIs support only primitive types for axis. 20 | * TF2ONNX released TFLite support, being able to convert TFLite directly to ONNX, making a pass at Hugging face models, added SentencePiece as custom op 21 | * Tensor-RT is officially on opset 11, planning on support opset 13 within next release, working on QDQ tool 22 | * Matlab keeps working on operators, import in two ways, 1. functional form, up to opset 13 2. layer type of model, still working on some constraints 23 | * Would like to have Pytorch to ONNX converter updates in our SIG meetings 24 | * Frontend roadmap items: 25 | * Convert from Tensorflow-js vs onnx-js? Convert from google/jax, like jax2onnx vs jax to tf + tf2onnx? 26 | * Backend roadmap items: 27 | * Add NHWC option in ONNX, at the model level? input data format? and/or certain ops? Would be nice if the user of the model can easily know the data format. 28 | * Leverage Onnx model zoo models as standard test and verification for all backends 29 | -------------------------------------------------------------------------------- /compilers/meetings/sig-compiler-2023-04-04.md: -------------------------------------------------------------------------------- 1 | ## SIG Compiler Meeting 04/04/2023 2 | 3 | ### Attendance: 4 | Alex, Gong, Haruki, Kiyo, Nathaniel, Philip, Soren, Tung, 5 | 6 | ## Agenda: 7 | 8 | ### Goals of SIG Compiler 9 | - Philip: Provide a compiler focus to the ONNX Specs, with a focus on how to make them lean and expressive. 10 | - Soren: has practical suggestions (ranked in order of importance). 11 | - Clarify ambiguities. For example, rules about type conversions are not alwasy clear (rounding modes). To find answer, we often have to look at what ORT is doing de facto. 12 | - Provide a way to verify an op, shape inference, type. Spoke about the possiblilty of a formal description that can be used to verify an implementation. 13 | - Fix specs (scalar, inconsitencies). Issue with this is that since we aim to maintain backward compatibility, it may not simplify the task of the compiler writers. 14 | - Alex: should be more proactive with the operator committee to "proof" new ops before they come in the standard. Discussion ensued on how to do this, how relatively easy it is to generate a PR against the ONNX specs. 15 | 16 | #### Practical steps: 17 | - review new ops, look for ambiguities 18 | - may organize followup meeting among folks interested int this. 19 | 20 | ### Discussion on OpenXLA 21 | - Alexandre: goal of discussion is to see if OpenXLA is responding to a need that should also be addressed in ONNX. 22 | - Philip: ONNX customers like the fact that ONNX models are stable, and that the PyTorch to ONNX is well supported within PyTorch. Currently the path from PyTorch to StableHLO is less direct. StableHLO aim to provide stability for the OpenXLA project. 23 | -------------------------------------------------------------------------------- /operators/meetings/009-20191106.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | # Wednesday November 6, 2019 at 9:00am PST 4 | 5 | ## Agenda 6 | * Dynamic shape proposal: 7 | * Link: https://1drv.ms/w/s!Aq5Cu4d76dsCjbFWbJajaHnH25CkXQ?e=Dabhup 8 | * Link: https://github.com/goldgeisser/onnx/blob/master/docs/proposals/StaticShape4DynShapeOpsProposal.md 9 | * Removing operator update 10 | 11 | ## Attendees 12 | * Emad Barsoum (Microsoft) 13 | * Michal Karzynski (Intel) 14 | * Ganesan Ramalingam (Microsoft) 15 | * Spandan Tiwari (Microsoft) 16 | * Dilip Sequeira (NVidia) 17 | * Itay Hubara (Habana) 18 | * Scott Cyphers (Intel) 19 | * Leonid Goldgeisser (Habana) 20 | 21 | ## Notes 22 | * We discussed the dynamic versus static shape for accelerator. 23 | * We focused on what need to be in the spec and what should be an implementation detail. 24 | * Here the summary. 25 | * We will add a hint attributed to specify the max size. 26 | * This attribute is optional. 27 | * It can be set by the model creator or afterward. 28 | * The behavior of exceeding the max value is NOT defined by the standard. 29 | * There is no concept of dynamic versus static in ONNX, ONNX is a file format, it is up to the runtime to respect the hint and create a fixed size graph in-memory. 30 | * This attribute is optional and can be ignored by the converter. 31 | * The detailed design TBD. 32 | * For removed OP. 33 | * Archive older operator.md for each release. 34 | * Add remvoed/deprecated OPs at the bottem of operator.md. 35 | 36 | ## Action items 37 | * Emad Barsoum - submit the operator removal requirements to ONNX master. 38 | * Leonid Goldgeisser - to write a final draft document for the dynamic shape and share it with the rest of the group. 39 | 40 | -------------------------------------------------------------------------------- /operators/meetings/002-20190626.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | # Wednesday June 26, 2019 at 9:00am PST 4 | 5 | ## Agenda 6 | * Review the operator document 7 | * Discuss contributors 8 | * Organize operators by domain 9 | 10 | ## Attendees 11 | * Emad Barsoum (Microsoft) 12 | * Michal Karzynski (Intel) 13 | * Ke Zhang (Microsoft) 14 | * Darren S Crews (Intel) 15 | * Dilip Sequeira (NVidia) 16 | * Milan Oljaca (Qualcomm) 17 | * Wei-Sheng Chin (Microsoft) 18 | 19 | ## Notes 20 | * We spend most of the meeting going through the operator proposal document. 21 | * We need to update the text of the proposal document to clarify more about function versus operator. 22 | * Some of the points can be merged into one. 23 | * Goal: end of next week to have a sign off on the operator document. 24 | * When submitting new operator add some examples in the description text. 25 | * Domain work, sync with Edge workgroup to make sure no overlap. 26 | * Contention point, each PR need two sign off from the contributor list. 27 | * Should we require the two sign off from two different company? 28 | * Or from any two in the contributor group? 29 | * We can change the above, we can remove or add people to the contributor group. 30 | * Which approach should we start with? 31 | 32 | ## Action items 33 | * ebarsoum - update the operator proposal based on the feedbacks. 34 | * ebarsoum - check Alibaba auto generated ONNX operator tests. 35 | * Everyone - review the proposal and sign off by end of next week (Friday July 5). 36 | * Everyone - send ebarsoum the list of name to be in the initial contributor list. 37 | * ebarsoum and Michal - select 5 operators PR to discuss. 38 | * Next meeting: 39 | * Organize operators by domain. 40 | * Discuss 5 operator PRs. 41 | * Discuss contributor list and logistic. 42 | 43 | -------------------------------------------------------------------------------- /operators/meetings/016-20200724.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | # Friday July 24, 2020 at 9:00am PST 4 | 5 | ## Agenda 6 | * Adding new type to existing operator. 7 | * Discuss AddNewOpV3 doc (https://github.com/onnx/sigs/blob/master/operators/docs/AddNewOpV3.md). 8 | * Softmax: https://github.com/onnx/onnx/pull/2879 9 | * Discussed: https://github.com/onnx/onnx/issues/2912 10 | * Beam Search: extend loop. 11 | 12 | ## Attendees 13 | * Emad Barsoum (Microsoft) 14 | * Michal Karzynski (Intel) 15 | * Tom Rieck 16 | * Ashwini Khade (Microsoft) 17 | * Alvaro Lopez 18 | * Ganesan Ramalingam (Microsoft) 19 | * Spandan Tiwari (Microsoft) 20 | * Scott Cyphers (Intel) 21 | * Matteo Salvarezza 22 | * Svetlana Levitan (IBM) 23 | * Wei-Sheng Chin (Microsoft) 24 | 25 | ## Notes 26 | * We agreed to remove the restriction for bumping OPSet version when we add new type. 27 | * Assuming no logic change and we just added a new type to the operator input or output, then there is no need to increase the OPSet version number for such operator. 28 | * Discussed AddNewOpV3 doc again, will get feedbacks through PR. 29 | * Based on last meeting the updated softmax is converted to a function (https://github.com/onnx/onnx/pull/2879), we discussed how can we test that. Conclusion, evaluate the test ONNX model in ONNX Runtime offline. 30 | * We also discussed the following proposal https://github.com/onnx/onnx/issues/2912, which is about should we update test generator to be version aware. This require more discussion. 31 | * We discussed also extending Loop construct for beam search by make it accept any loop carry type. We didn't come to any conclusion, we will continue the discussion in github or gitter. 32 | 33 | ## Action items 34 | * (Emad Barsoum) Submit PR to update version.md 35 | * (Emad Barsoum) Submit PR for AddNewOpV3. 36 | -------------------------------------------------------------------------------- /converters/meetings/022-20210820.md: -------------------------------------------------------------------------------- 1 | # Friday Aug 20 2021 10:30-11:30 pm PST 2 | * https://zoom.us/j/7376656864?pwd=SjdiaGdvUFFyVXM4OUxTWndYZ2Z1Zz09 3 | 4 | ## Agenda 5 | * Greetings! 6 | * Discussion topics 7 | * Converters project updates 8 | * onnx-tf: released 1.9.0, supporting opset 14, Tensorflow 2.6 9 | * tensor-rt: focus on operator coverage, will look into participating the ONNX backend scoreboard 10 | * tf2onnx: keras to ONNX stopped, will be done using tf2onnx, still fixing GRU, developed a layout transformer to convert channel first to channel last in ONNX format, can convert JAX to ONNX 11 | * How keras models are supported? Keras operators like lstm are lowered to Tensorflow graph. So tf2onnx will match up the sub graphs/patterns and convert them to ONNX models. 12 | * How to support older versions of IR and opset? Most converters have natural support for older versions. The ONNX version converter has not been robust or up-to-date in the past. ONNX-MLIR would prefer to work with the latest ONNX version, therefore heavily depend on high quality version converter to ensure old versions can all convert to the latest. 13 | * SparseTensor became a first-class data type in ONNX 1.10, opset 15; it is however applied in only one operator Constant to turn a sparse tensor into a dense one. It can be used to minimize memory as a constent. We are interested to know if more operators would include sparse tensors to make it more useful. 14 | * Optional is a new data type in ONNX 1.10, opset 15. We are still looking into how to convert the type and operators. Would be nice to see real use cases and models. 15 | 16 | ## Attendees 17 | * Chin Huang (onnx-tf) 18 | * Guenther Schmuelling (tf2onnx) 19 | * Kevin Chen (Tensor-RT) 20 | * Alexandre Eichenberger (onnx-mlir) 21 | * Charles Volzka 22 | 23 | ## Notes 24 | 25 | -------------------------------------------------------------------------------- /converters/docs/IssuesAndSolutions.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | # What problems we are trying to solve 4 | This document is intended to capture common ONNX converter issues and provide general recommendations for all converters to address them in a similar fashion. 5 | 6 | # Issues and recommendations 7 | | Issue ID | Issue description | Recommendation | 8 | | -------------- |:------------------:|:------------------:| 9 | |14|How to handle unsupported data types between ONNX and frameworks?|Provide an auto-cast level parameter as an optional user input. The default is "no cast", meaning unsupported data types will cause an exception and fail the conversion. The other levels are "upcast", for ex. float32 to float64, and "upcast and downcast". We also recommend produce logging at the debug level whenever auto-cast occurs so that users could see clearly where the auto-cast is applied.| 10 | |18|How to handle deprecated operators?|Frontend converters: Do not add deprecated ops, with respect to the target opset version, into onnx graph. Use the new/replacement op instead. Backend converters: Run ONNX checker before conversion, the default behavior in Backend.prepare(...), which will throw an error message indicating the deprecated op. If the converter doesn't use checker, it should handle the op with the previous definition and print out a warning message state that this op is deprecated at this opset version and advise user to use the "new" op to replace this in the future.| 11 | |15|Where/how to see current state (coverage, versioning, etc) for all converters|The common landing page to view status for all converters is https://github.com/onnx/sigs/blob/master/converters/docs/ConvertersDashboard.md. Initially there is only an entry with a link to the auto-genarated report for ONNX-TF converter. It is up to each converter to provide updates to make it useful for onnx users.| 12 | -------------------------------------------------------------------------------- /infra/README.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | # ONNX Architecture & Infra Special Interest Group (SIG) 4 | 5 | This is the working repo for the architecture & infra special interest group (SIG), which is responsible for defining and maintaining the core ONNX format, the build and CI/CD systems for ONNX repositories, publishing release packages for ONNX, and creating tools to help integrate with and test against the ONNX standard. This SIG is also the defacto owner of files in the main ONNX repository unless explicitly owned by another SIG. 6 | This repo contains all the artifacts, materials, meeting notes and proposals regarding ONNX infra evolution in ONNX standard. Feedbacks and contributions are welcome. 7 | 8 | # Slack channel 9 | Please sign up at https://slack.lfai.foundation/ and join [onnx-archinfra](https://lfaifoundation.slack.com/archives/C018Y2QG7V2) channel. 10 | 11 | # SIG Leads 12 | 13 | * Andreas Fehlner (TRUMPF Laser) 01.2025 - Current 14 | * Christian Bourjau (QuantCo) 01.2026 - Current 15 | 16 | # Former SIG Leads 17 | * Xavier Dupre (Microsoft) 01.2025 - 12.2025 18 | * Liqun Fu (Microsoft) XX.XXXX - 12.2024 19 | * Ke Zhang (Ant) XX.XXXX - 12.2024 20 | 21 | # Logistics 22 | 23 | * SIG leads will drive the meeting. 24 | * Meeting annoucement will be posted in our Slack channel. 25 | * Feedbacks and topic request are welcome by all. 26 | 27 | # SIG Meeting Info 28 | 29 | * Meeting is on the 2nd Tuesday of the month at 10 pm CEST 30 | * Access link could be found at: https://zoom-lfx.platform.linuxfoundation.org/meetings/lfai-onnx?view=month 31 | 32 | # Discussion 33 | 34 | * Slack channel https://lfaifoundation.slack.com/archives/C018Y2QG7V2 35 | * Documents and artifacts: https://github.com/onnx/sigs/tree/main/infra 36 | 37 | # Meeting notes 38 | 39 | The meeting notes can be found [here](https://github.com/onnx/sigs/tree/main/infra/meetings) 40 | -------------------------------------------------------------------------------- /best-practices.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | # Best practices for SIGs 4 | 5 | ## Scheduling meetings 6 | ONNX has a global community so SIG chairs should schedule meetings to accommodate different geographies. Unfortunately, there isn't a single time that accommodates everyone in the world. Depending on the participants in the group, a specific time may be better for most people or meeting times may need to be alternated to give different geographies a chance to participate at reasonable times. 7 | 8 | ## Documents and artifacts 9 | Files that are specific to the operation of the SIG (such as meeting notes, rules, etc) should be kept in the "sigs" repository in the directory for the SIG. Workings drafts can also be maintained here until they are ready to be submitted to the main repos for broader review. 10 | 11 | ## Meeting notes 12 | Meeting notes must be kept for every meeting. Attendence should also be recorded as part of the notes. The file name should include the date to make it easy to find. 13 | 14 | ## Meeting recording 15 | At the discretion of the SIG chairs, meetings can optionally be recorded and published to the ONNX YouTube channel. We are not requiring this at this time but may do so in the future. 16 | 17 | ## Publish agenda ahead of meeting 18 | SIG chairs must publish the agenda at least 2 days ahead of a meeting. The agenda should be published as the file for the meeting notes for easy discoverability. 19 | 20 | ## Publish meeting notes soon after meeting 21 | Notes should be published within a day of the meeting. Usually notes can be published the same day because the SIG chairs would be taking notes directly in the file that has the agenda. 22 | 23 | ## Slack Channel 24 | SIG have slack channels as part of LF AI & Data. SIG leaders and participants can use the slack channels to prepare for meetings, send reminders, help with above items, and more. 25 | -------------------------------------------------------------------------------- /operators/meetings/001-20190611.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | # Tuesday June 06, 2019 at 8:00am PST 4 | 5 | ## Agenda 6 | * Introduction 7 | * Logistics 8 | * Criteria to decide which operator to add 9 | * Adding new operator 10 | * Organize operators by domain 11 | 12 | ## Attendees 13 | * Emad Barsoum (Microsoft) 14 | * Darren S Crews (Intel) 15 | * Dilip Sequeira (NVidia) 16 | * Ganesan Ramalingam (Microsoft) 17 | * Jianhao Zhang (JD) 18 | * Michal Karzynski (Intel) 19 | * Prasanth Pulavarthi (Microsoft) 20 | * Wei-Sheng Chin (Microsoft) 21 | * Ke Zhang (Microsoft) 22 | 23 | ## Notes 24 | * Move the meeting to 9am pacific time. 25 | * Start with a biweekly meeting. 26 | * Use gitter operator channel for all our online discussion (https://gitter.im/onnx/operators). 27 | * Use online world for discussion and proposal draft. 28 | * Use markup for the final proposal. 29 | * To decide which operator to add: 30 | * Need to have a model 31 | * Implemented by at least by one framework (implemented by two or more get higher priority). 32 | * Use function for composable operators. 33 | * Adding new operator 34 | * Write a detailed description about its behavior. 35 | * Write the mathematic formula or pseudo-code. 36 | * Write reference implement in Python. 37 | * Use numpy semantic if it is available in numpy. 38 | * Write a test case. 39 | * Update the documentation. 40 | * At least two sign-off from two different company. 41 | * If the operator available in more than one framework, make sure that your design is general and cover those frameworks. 42 | 43 | ## Action items 44 | * ebarsoum - write initial proposal for adding new operator. 45 | * ebarsoum - check Alibaba auto generated ONNX operator tests. 46 | * Next meeting: 47 | * Discuss adding new contributors. 48 | * Who can vote? 49 | * Organize operators by domain. 50 | -------------------------------------------------------------------------------- /operators/README.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | # ONNX Operators Special Interest Group (SIG) 4 | 5 | This is the working repo for the operators special interest group (SIG). This repo contains all the artifacts, materials, meeting notes and proposals regarding operators evolution in ONNX standard. Feedbacks and contributions are welcome. 6 | 7 | # Slack channel 8 | Please sign up at https://slack.lfai.foundation/ and join [onnx-operators](https://lfaifoundation.slack.com/archives/C018Y2RAY4C) channel. 9 | 10 | # SIG Leads 11 | 12 | * Ganesan "Rama" Ramalingam (Microsoft) 13 | * Michal Karzynski (Intel) 14 | 15 | # Logistics 16 | 17 | * SIG leads will drive the meeting. 18 | * Meeting annoucement will be posted in our Slack channel: https://lfaifoundation.slack.com/archives/C018Y2RAY4C 19 | * Feedbacks and topic request are welcome by all. 20 | * Documents and artifacts: https://github.com/onnx/sigs/tree/main/operators 21 | 22 | # SIG Meeting Info 23 | 24 | * Meeting occurs every 4 weeks on Thursday at 9 AM PST. 25 | 26 | Ways to join meeting: 27 | 28 | 1. Join from PC, Mac, iPad, or Android 29 | 30 | https://zoom-lfx.platform.linuxfoundation.org/meeting/96786144384?password=b8ffc217-d477-4a27-849f-61334ac72cb8 31 | 32 | 2. Join via audio 33 | 34 | One tap mobile: 35 | US: +12532158782,,96786144384# or +13462487799,,96786144384 36 | 37 | Or dial: 38 | US: +1 253 215 8782 or +1 346 248 7799 or +1 669 900 6833 or +1 301 715 8592 or +1 312 626 6799 or +1 646 374 8656 or 877 369 0926 (Toll Free) or 855 880 1246 (Toll Free) 39 | Canada: +1 647 374 4685 or +1 647 558 0588 or +1 778 907 2071 or +1 204 272 7920 or +1 438 809 7799 or +1 587 328 1099 or 855 703 8985 (Toll Free) 40 | 41 | Meeting ID: 96786144384 42 | 43 | Meeting Passcode: 884450 44 | 45 | 46 | International numbers: https://zoom.us/u/alwnPIaVT 47 | 48 | # Meeting notes 49 | 50 | The meeting notes can be found [here](https://github.com/onnx/sigs/tree/main/operators/meetings) 51 | 52 | -------------------------------------------------------------------------------- /converters/meetings/014-20200724.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | # Friday July 24, 2020 10:30-11:30am PST 4 | * https://zoom.us/j/7376656864 5 | 6 | ## Agenda 7 | * Greetings! 8 | * Discussion topics 9 | * Converters project updates 10 | * Optional inputs and outputs (Clip) 11 | * Scalar vs 1D tensor of size 1 12 | * Bfloat16 test and support 13 | * Inference accuracy requirements 14 | * ONNX-ML unit tests 15 | * ONNX release dependency on converters 16 | * Others 17 | 18 | ## Attendees 19 | * Chin Huang (onnx-tf, IBM) 20 | * Guenther Schmuelling (tf-onnx, Microsoft) 21 | * Winnie Tsang (onnx-tf, IBM) 22 | 23 | ## Notes 24 | * ONNX-TF converter: release 1.6.0 is completed, supporting ONNX opset 11 and release 1.6 25 | * Optional inputs can be specified with a blank input name, as seen in the Clip backend test. The backend runtimes and handlers should be aware of this treatment, despite uncommon in real models. 26 | * Scalar could be a scalar or 1D tensor of size 1 in ONNX OneHot and NonMaxSuppression implementation. But the doc states scalar only. The backend runtimes and handlers need to double check with schema before doc is totally in-sync. 27 | * bfloat16 is added to a lot of ops in opset 13. Would like to discuss how a general test could be written for all backends. 28 | * Inference accuracy requirements could be relaxed due to backend implementations of same ops might produce slightly different results. TF-ONNX considers matching results up to decimal 5. 29 | * ONNX_ML unit tests are not directly in ONNX. We will look into using ONNX runtime and SKLearn as references. 30 | * ONNX release should not have dependency on converters. For 1.7, TF-ONNX needs to make sure the converter ONNX files have the matching IR number and opset number. Would be nice to have an ONNX lookup API to provide this mapping. 31 | * Some work to support easier consumption of custom ops is underway. Would like to see it as an official ONNX project so broader community can benefit. 32 | -------------------------------------------------------------------------------- /operators/docs/RemovingOperatorOrFunction.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | # Removing operator and function 4 | There are a lot of reasons for removing ONNX operator or function, such us being replaced with different operator or can be decomposed by a set of operators. This document describes the criteria for removing an existing ONNX operator from the standard. 5 | 6 | ## Removing operator 7 | Any operator in ONNX was added because it was required by a model and/or framework. In order to deprecate such an operator we need to specify its replacement. 8 | * Operator can’t be deprecated unless there is a replacement. 9 | * Replacement can be a more general operator that supersedes the old one. 10 | * Or a set of primitive operators that together can implement the same functionality and behavior of the deprecated operator (Function). 11 | * If the deprecated operator can be decomposed by existing operators then it must be converted to a function. 12 | * If replacement isn’t in ONNX standard yet, then add the replacement operator or set of operators first. 13 | * No grace period is needed for deprecated operators. 14 | 15 | ## Removing function 16 | Function, by definition, is composed of ONNX primitives; however, function could have been accelerated by framework or runtime that support ONNX. So, removing function is not recommended, with the exception of adding another single function which supersede its functionality. 17 | 18 | ## Document removing operator or function 19 | To make sure everyone is aware of the deprecation, the following need to happen: 20 | * Any removed operator or function from ONNX need to be mentioned in the release note. 21 | * Their old documentation needs to be updated to show the new replacement and the mapping between the old to the new. 22 | * Only `def.cc` need to be remove, `old.cc` will remain. 23 | * `old.cc` need to be updated with the mapping to the replacement. 24 | * ONNX checker need to be updated to error with a proper message. 25 | * All removed operators need to be appended at the end of the `operator.md` file. 26 | -------------------------------------------------------------------------------- /infra/meetings/003-20201120.md: -------------------------------------------------------------------------------- 1 | # Friday November 20, 2020 at 10:00am PST 2 | 3 | ## Agenda 4 | * Deprecation of Circle CI 5 | * Discuss making test data optional when building from source 6 | * Discuss alternate ways of adding\packaging test data for operators 7 | 8 | 9 | ## Attendees 10 | * Ashwini Khade (Microsoft) 11 | * Jacky Chen (Microsoft) 12 | * Anna Jung (VMware) 13 | * 14 | 15 | 16 | ## Notes 17 | * Deprecate CirCle CI: 18 | Ashwini: 19 | Jacky: Current Circle CI tests pytorch\caffee2 in ONNX. This is not maintained well and we see a lot of errors because of usage of deprecated operators etc so it blocks improvements in Shape Inference. 20 | Conclusion: Circle CI is not adding to any ONNX test coverage plus this is not being maintained so we can deprecate this. 21 | 22 | * Discuss making test data optional when building from source 23 | Ashwini: Previously Michał from OP Sig had opposed this. Need to get his feedback before we conclude 24 | Anna: It makes sense 25 | Jacky: There are issues reported related to problems in installing test data. Having this option can enable users to build and install from source easily. 26 | 27 | * Discuss alternate ways of adding\packaging test data for operators 28 | Anna: Need sometime to look into this 29 | Jacky: Utilize version converter to solve this issue? (will need version converter to be always uptodate) 30 | Ashwini: May be difficult to use version converter for cases where inputs and outputs are hard coded. Planning to submit a proposal for restructing test data and options for packaging test data 31 | How to package\ship test data so that the runtimes dependant on the test data can easily pick it up. 32 | 33 | ## Action items 34 | * Ashwini: Work with Michał to understand his concerns for making test data optional when building from source. Submit a proposal for restructing test data and options for packaging test data 35 | * Anna: Understand the context behind the test data restructure and come up with ideas. 36 | * Jacky: Formalize on version converter solution 37 | * Next meeting: 38 | * Conclude on making test data optional 39 | * Discuss test data restructure and packaging 40 | * Open for more topics -------------------------------------------------------------------------------- /converters/meetings/005-20190919.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | # Thursday Sept 19, 2019 at 9:00-10:00am PST 4 | * online: https://ibm.webex.com/join/chhuang 5 | * call-in: 1-844-531-0958, 927 169 571 # 6 | 7 | ## Agenda 8 | * Greetings! 9 | * Discussion topics 10 | * Sequence ops 11 | * Training proposal update and review 12 | * ML ops 13 | * Inputs with unknown ranks 14 | * Support different system configurations 15 | * TFLite converter? 16 | * Use of custom attributes in converters 17 | 18 | ## Attendees 19 | * Chin Huang (onnx-tf, IBM) 20 | * Guenther Schmuelling (tf-onnx, Microsoft) 21 | * Winnie Tsang (onnx-tf, IBM) 22 | * Svetlana Levitan (IBM) 23 | 24 | ## Notes 25 | * We found no unit tests for the sequence ops at the operator level and will follow up to ensure standardized tests available for converters to support it. 26 | * The ONNX training proposal is consolidated into one huge PR, https://github.com/onnx/sigs/issues/12. Many SIGs, including converters, are invited to review the PR in next WG meeting. We need some frontend converters such as TF and Pytorch to evaluate the feasibility of producing the training info described in the proposal. 27 | * The ML ops are currently not processed in TF converters. We found no unit tests for them. We would like to learn which converters have ML ops support and suggest unit test contribution. 28 | * We have not seen any practical use of unknown input ranks. We would suggest the broader ONNX community to identify and share some use cases either in tests or zoo for clarity. 29 | * The backend converters might have to support different system configurations for optimization, for ex, cpu vs gpu, between the conversion and execution servers. Will continue the discussion. 30 | * Do we want to start a new TFLite converter? Currently don't see the need since all TFLite models are essentially TF models. 31 | 32 | ## Action items 33 | * 1: Investigate the feasibility of supporting ONNX training from tf and pytorch to ONNX 34 | * 2: Continue to investigate the converter reporting/dashboard 35 | * 3. Look for possible use cases for unknown rank inputs 36 | * 4. Look into unit tests for unknown shape inputs 37 | -------------------------------------------------------------------------------- /infra/meetings/007-20250513.md: -------------------------------------------------------------------------------- 1 | # Tuesday May 13, 2025 at 22 CEST 2 | 3 | ## Access 4 | * https://zoom-lfx.platform.linuxfoundation.org/meeting/97533707071?password=90fbcf4b-7cf9-4f09-992c-dc435e7794fb 5 | 6 | ## Participants 7 | * Ganesan Rama, Justin Chu, Andreas Fehlner, Ramakrishnan Sivakumar 8 | 9 | ## Agenda 10 | * 1.) Release 1.18 review 11 | * 2.) Release process optimization 12 | * 3.) Python 3.14 13 | * 4.) ONNX 1.19 ? 14 | 15 | ### 1.) Release 1.18 review ### 16 | Feedback of Krishna: 17 | * a) dedicated release team will potentials for errors, ... [Proposal by Andreas: Release team consists of a) Release manager, release before (1.18), current release (1.19), next release (1.20), b) Andreas, Justin => about 5-6 people] 18 | * b) Partner repos testing 19 | * pytorch/onnxruntime tested 20 | * 1.18.0 breaks at onnxmltools? (tools under github.com/onnx should not break...) 21 | * Better handling breaking changes...? 22 | 23 | ### 2. Release process optimization ### 24 | * publish rc-candidates to https://pypi.org/project/onnx/ instead of https://test.pypi.org/project/onnx/ ? 25 | * How do we want to ensure that no incorrect versions are sent to pypi? 26 | 27 | #### Timeline? 28 | ```mermaid 29 | graph TD 30 | A[Release Cut-Out] --> B[Stabilization
Week 1] 31 | B --> C[RC1
Week 2-3] 32 | C --> D[RC2
Week 4-5] 33 | D --> E[Final Release
Week 6] 34 | ``` 35 | 36 | ### 3. Python 3.14 support ### 37 | * Python 3.14 release beginning of October (https://peps.python.org/pep-0745/) 38 | * Andreas: I would prefer to release a version with wheels immediately afterwards 39 | * Target version: 1.19.0? or do we want to make a version before that? 40 | 41 | ### 4. ONNX 1.19 ## 42 | * Appointment of a release manager Yuan Yao 43 | 44 | ### Release Cadence Suggestion (Andreas) ### 45 | 46 | Release cut every 4 month 47 | 48 | | Minor Version | Release branch cut | Release date | First patch release date | Second patch release date| 49 | | --- | --- | --- | --- | --- | 50 | | 1.17.0 | XYZ | XYZ | Not planned | Not planned | 51 | | 1.18.0 | Mar 2025 | Mai 2025 | Not planned | Not planned | 52 | | 1.19.0 | July 2025 | August 2024 | Not planned | Not planned | 53 | | 1.20.0 | November 2025 | Dez 2025 | Not planned | Not planned | 54 | -------------------------------------------------------------------------------- /converters/meetings/014-20200612.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | # Friday June 12, 2020 10:30-11:30am PST 4 | * https://zoom.us/j/7376656864 5 | 6 | ## Agenda 7 | * Greetings! 8 | * Discussion topics 9 | * Converters project updates 10 | * Subgraph with no output data types (Range) 11 | * Inputs with unknown ranks update 12 | * Optional inputs and outputs (Clip, Compress, Slice, NonMaxSuppression, ConvInteger…) 13 | * Others 14 | 15 | ## Attendees 16 | * Chin Huang (onnx-tf, IBM) 17 | * Guenther Schmuelling (tf-onnx, Microsoft) 18 | * Kevin Chen (Nvidia) 19 | * Winnie Tsang (onnx-tf, IBM) 20 | 21 | ## Notes 22 | * ONNX-TF converter: support of ONNX opset 11 and release 1.6 will be out in a couple of weeks 23 | * TF-ONNX converter: recently completed support for ONNX 1.7 and opset 12 24 | * Tensor-RT converter: working on opset 12 ops, no immediate plan to support training ops 25 | * Custom ops are created from tf-onnx and can be handled by ONNX runtime. How other runtimes and frameworks would work? Currently onnx-tf and tensor-rt focus on standard onnx ops and do not handle custom ops. 26 | * Subgraph with no output data types doesn't work in onnx-tf. Would like to understand how tensor-rt manages the standard backend test. 27 | * Inputs with unknown ranks can be created from API but will cause an exception from the onnx checker. So we believe unknown ranks are not supported in onnx. 28 | * Optional inputs can be provided in two ways according to the IR.md spec. The use of an empty string in place of an input and output name was discussed. The tf-onnx has not applied this method. The tensor-rt and onnx-tf have not supported this case, nor seen any models with such input or output names. The conclusion and recommendation from today's discussion is to use the number of inputs to decide whether optional ipnuts are actually in use. The models need to have default values for the optional inputs in the middle positions when there is a trailing optional input with a real runtime value. That means the backends can expect all inputs specified for a node will be filled with data at the runtime. 29 | * We recommend to revisit the method of using an empty string as input/output name at ops SIG meeting. 30 | * We will continue optional outputs discussion in next meeting. 31 | -------------------------------------------------------------------------------- /models-tutorials/README.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | # ONNX Model Zoo & Tutorials Special Interest Group (SIG) 4 | 5 | This is the working repo for the ONNX model zoo and tutorials special interest group (SIG). This repo contains all the artifacts, materials, meeting notes and proposals regarding pretrained models, tutorials, and outreach materials for ONNX. 6 | 7 | The Model Zoo and Tutorials SIG is responsible for the comprehensive collection of state of the art ONNX models from a variety of sources and making it easy for users to get started with ONNX and the ecosystem around it. Concretely, this SIG has ownership over the [onnx/tutorials](https://github.com/onnx/tutorials) and [onnx/models](https://github.com/onnx/models) repositories, the official webpage [onnx/onnx.github.io](https://onnx.ai), as well as tutorials located in the deprecated [onnx/onnx-docker](https://github.com/onnx/onnx-docker) repository, instructive blog posts and conference participation. 8 | 9 | Feedback and contributions are welcome. 10 | 11 | # Slack channel 12 | Please sign up at https://slack.lfai.foundation/ and join [onnx-modelzoo](https://lfaifoundation.slack.com/archives/C018RE2BRBL) channel. 13 | 14 | # SIG Leads 15 | * Daniel Holanda Noronha (AMD): July 2024 - Current 16 | * Javier Martinez (Intel): March 2024 - Current 17 | * Ramakrishnan Sivakumar (AMD): April 2023 - July 2024 18 | * Jacky Chen (Microsoft): February 2022 - November 2023 19 | * Wenbing Li (Microsoft): September 2020 - February 2022 20 | * Vinitra Swamy (Microsoft): January 2020 - September 2020 21 | 22 | # Logistics 23 | SIG lead will drive the meeting. Meeting annoucements will be posted in our slack channel 24 | Feedback and topic requests are welcome by all. 25 | 26 | Documents and artifacts: https://github.com/onnx/sigs/tree/main/models-tutorials 27 | 28 | Meeting will take place at 4th Thursday of each month at 1pm PST (4pm EST). 29 | 30 | 31 | # Meeting notes 32 | [Mission Statement](docs/MissionStatement.md) 33 | [Community Events](docs/CommunityEvents.md) 34 | [Proposed Models](docs/ProposedModels.md) 35 | [ONNX Workshop Update](docs/onnx-workshop-modelzoo-SIG-update.pdf) 36 | 37 | [January 30, 2020](meetings/001-20200130.md) 38 | [March 09, 2020](meetings/002-20200309.md) 39 | [April 13, 2020](meetings/003-20200413.md) 40 | [June 15, 2020](meetings/004-20200615.md) 41 | [August 14, 2020](meetings/005-20200814.md) 42 | -------------------------------------------------------------------------------- /converters/meetings/021-20210625.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | # Friday June 25 2021 10:30-11:30 pm PST 4 | * https://zoom.us/j/7376656864?pwd=SjdiaGdvUFFyVXM4OUxTWndYZ2Z1Zz09 5 | 6 | ## Agenda 7 | * Greetings! 8 | * Discussion topics 9 | * Converters project updates 10 | * tf2onnx is deprecating keras2onnx, supporting Tensorflow 2.5, one issue with a single tensor > 2GB for recommender 11 | * pytorch to onnx has a new lead, a bit more difficult to track and solve converter issues since it is part of pytorch 12 | * tensor-rt 8 is projected to be out in two weeks, onnx to tensor-rt will support tensor-rt 8 right after release. tensor-rt could be a backend unit test for tf2onnx. Immediate onnx to tensor-rf supports up to opset 13. 13 | * onnx-tf is working on opset 14 new and upated ops, and looking into optimization and better NHWC suport. Would like to know whether Tensorflow addons and experimental APIs are supported in Tensorflow 2.5 and beyond. 14 | * Greenwaves states GRU doesn't seem converted properly from Tensorflow. Guenther to follow up. 15 | * Opset 14, 15 ops: Bernoulli comes with standard tests that would cause failed case for various backends due to non-deterministic nature of the operator. Backend converters need to filter out such tests or won't pass. BatchNormalization has a new attribute 'training_mode', which seemingly dictating the model is for training or inference. The converters currently have no plans to support it. 16 | * ONNX roadmap at https://onnx.ai/roadmap, or “onnx-roadmap” slack channel 17 | * NHWC support: Important for backends that prefer channel last data format. Could be implemented as ONNX utility function. Would like to see the feedback and suggestions from broader ONNX community. 18 | * Preprocessing: Images are doable, except jpeg. Audio data processing is more difficult. We could start from image processing. In fact, tf2onnx already converts Tensorflow image processing to primitive ONNX ops. 19 | * Shape inference: We see models with in-compatible input and attribute shapes. This can be detected in ONNX and ONNXRuntime shape inference utility, independent of ONNX checker. The process is in-place for tensor-rt converter and should be considered for others. 20 | 21 | ## Attendees 22 | * Chin Huang (onnx-tf) 23 | * Guenther Schmuelling (tf2onnx) 24 | * Kevin Chen (Tensor-RT) 25 | * Martin Croome (Greenwaves) 26 | 27 | ## Notes 28 | -------------------------------------------------------------------------------- /models-tutorials/docs/CommunityEvents.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | # ONNX Community Events 4 | ## Compiling a list of known upcoming ONNX events to add to the ONNX Website 5 | 6 | ### [Indy Cloud Conf - virtual](https://2020.indycloudconf.com/schedule/) 7 | **March 27, 2020 - online** 8 | 9 | 2:15 PM (EDT): Svetlana Levitan presented "Open standards for machine learning model deployment" 10 | 11 | 12 | ### [San Francisco Python Meetup Group: Learn more about continuous deployment](https://www.meetup.com/sfpython/events/xkwxvqybchbrb/) 13 | **May 13, 2020 - online** 14 | 7 pm PDT: Svetlana Levitan presented on open standards for machine learning model deployment. 15 | 16 | 17 | ### [Confer conf: Norwegian conference with focus on Data Science, AI, ML & Cloud](https://confer.no/) 18 | **June 2, 2020 - online** 19 | Svetlana Levitan presents on open standards for machine learning model deployment. 20 | 21 | 22 | ### [Women in Data Science (WIDS 2020), Silicon Valley, USA](https://events.sap.com/us/wids-2020-sv/en/home) 23 | **June 4, 2020 - Online, SAP Labs** 24 | 25 | ONNX Workshop - Vinitra Swamy, Cecilia Liu 26 | 12:30 - 1:30 PM: Open Neural Network eXchange (ONNX): Accelerate and operationalize models for deployment 27 | 28 | ONNX Talk - Faith Xu, Emma Ning 29 | 1:00 – 1:30 PM: Research to Production with Open Source Software: High performance model inferencing using ONNX 30 | 31 | 32 | ### [LF Open Source Summit NA, Virtual, Austin, TX, USA](https://events.linuxfoundation.org/open-source-summit-north-america/) 33 | **June 29 - July 2, 2020 - online** 34 | 35 | Nick Pentreath (IBM) and Svetlana Levitan (IBM) will present "Compare, Contrast, Contribute: open 36 | standards for model deployment and how you can help". 37 | 38 | 39 | ### [JSM: Joint Statistical Meeting. Virtual, US Eastern time](https://ww2.amstat.org/meetings/jsm/2020/) 40 | **August 5, 2020 - Virtual** 41 | 42 | Svetlana Levitan (IBM) and David Nichols (IBM) will present a Computer-Technology workshop 43 | "Introdution to deep learning with Watson Studio" that will include ONNX intro. 44 | 45 | 46 | ### [Codemotion Rome 2020, Rome, Italy](https://events.codemotion.com/conferences/rome/2020/agenda/12-June/) 47 | **November 28, 2020 - Rescheduled from March 28, 2020 - Engineering Department | Università degli Studi Roma Tre** 48 | Via Vito Volterra, 60 Rome, Italy 49 | 50 | 2:10 - 2:50 PM: Svetlana Levitan will present "Open standards for machine learning model deployment" 51 | -------------------------------------------------------------------------------- /converters/meetings/009-20200116.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | # Thursday Jan 16, 2020 10:45-11:30am PST 4 | * https://zoom.us/j/7376656864 5 | 6 | ## Agenda 7 | * Greetings! 8 | * Discussion topics 9 | * Training proposal update and planning 10 | * Backend scoreboard 11 | * How to support mulitple versions of a framework 12 | * Sequence ops handling 13 | * Channel first vs channel last 14 | 15 | ## Attendees 16 | * Chin Huang (onnx-tf, IBM) 17 | * Guenther Schmuelling (tf-onnx, Microsoft) 18 | * Winnie Tsang (onnx-tf, IBM) 19 | * Svetlana Lavitan (IBM) 20 | * Thomas Truong (IBM) 21 | 22 | ## Notes 23 | * We would like to clearly sych up on what converters need to provide to support ONNX Training end-to-end scenarios. Guenther commented no changes seem to the frontend converters, just producing the inference graph as is, and the runtimes would add the necessary training data to make the model trainable. Chin thought the frontend converters would generate some training data, such as optimizers, loss functions, and gradient operators, into the onnx file. We will raise the question and get clarification on the expected behaviors in the next training WG meeting. 24 | * The onnx-tf converter team developed unit tests for backend to verify the coverage for the unknown tensor shapes as inputs. The side effect in the case for Tensorflow is that the converted graph becomes much larger. 25 | * We continue to monitor the backend scoreboard, http://onnx.ai/backend-scoreboard/index.html, and hope more converters to join the scoreboard. 26 | * Both frontend and backend Tensorflow converters must support multiple versions, tf 1.x and tf 2.x, in the foreseeable future. Different approaches are taken for development, CI, and maintenance based on specific requirements and feasibility. 27 | * The ONNX sequence allows different tensor shapes in a sequence. We would like to learn the use cases and how some backend converters convert a sequence into framework operators. 28 | * Channel first (NCHW) vs channel last (NHWC) is something a lot of converters might need to deals with. When a framework such as tensorflow has the different default (NHWC) from ONNX (NCHW), data conversion needs to be properly included to ensure accuracy as well as minimal performance impact. 29 | 30 | ## Action items 31 | * 1: Identify the user experience and technical requirements of supporting ONNX training for frontend and backend converters. 32 | * 2: Investigate and share NCHW vs NHWC handling 33 | -------------------------------------------------------------------------------- /models-tutorials/meetings/001-20200130.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | # Thursday January 30, 2020 at 2:00pm PST 4 | 5 | ## Agenda 6 | * Introductions 7 | * Formalize scope of this SIG 8 | * Discuss Mission Statement and Core Priorities. Refine based on feedback. 9 | * Proposed Action Items: 10 | * Move all models in model zoo to Git LFS, standardize READMEs 11 | * Include guidelines on git commands that are relevant to LFS. 12 | * Develop a CI for Model Zoo 13 | * Base requirement: run ONNX checker on each model. 14 | * Establish guidelines for new models and tutorials to be added to the ONNX ecosystem 15 | 16 | ## Attendees 17 | * Vinitra Swamy (Microsoft) 18 | * Svetlana Levitan (IBM) 19 | * Lara Haidar (Microsoft) 20 | * Negin Raoof (Microsoft) 21 | * Ksenija Stanojevic (Microsoft) 22 | * Rajeev Rao (nVidia) 23 | * Faith Xu (Microsoft) 24 | 25 | ## Notes 26 | * Discussion on action items / mission statement 27 | * We need a clear policy for adding and deprecating models 28 | * This SIG should take responsibility for maintaining model zoo versions 29 | * Perhaps we can automate this with the version converters 30 | * This removes the burden from the contributor to deal with the maintanence costs 31 | * Can we measure Model Zoo analytics? Which models are downloaded often? Is the model zoo useful? 32 | * Outreach 33 | * Listing all the products that support ONNX / maintaining that list on the ONNX Website 34 | * Listing ONNX related events and community workshops on the ONNX website (to be added and maintained by community members) 35 | 36 | ## Recorded Meeting 37 | Topic: Model Zoo + Tutorials SIG, Meeting #1 38 | 39 | Date: Jan 30, 2020 02:00 PM Pacific Time (US and Canada) 40 | 41 | Meeting Recording: 42 | https://zoom.us/rec/share/uuBsPavr-2ZIb7fS2H7NV4MfFa-iX6a80CJLr_dcyUcwQ1tZv7FqGpEe3egaUNM 43 | 44 | ## Action Items 45 | - Vinitra - Create draft of guidelines for new model additions to the Model Zoo 46 | - XXX - feedback and revisions 47 | - Svetlana, Everyone - Add to list of [known upcoming ONNX community events](../docs/CommunityEvents.md) 48 | - XXX - Moving models to Git LFS -- download model from link, submit a PR to move it over (can be done on an individual model basis). 49 | - XXX - Investigate Model Zoo analytics on Github 50 | - Svetlana - Investigate new models to be added to the model zoo (NLP) 51 | - XXX - Revise CI proposal for Model Zoo, make initial PR 52 | 53 | ## Next Meeting 54 | End of February, early March 55 | -------------------------------------------------------------------------------- /compilers/README.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | # ONNX Compiler Special Interest Group (SIG) 4 | 5 | This is the working repo for the Compiler Special Interest Group (SIG), which is responsible for compiler solutions that optimizes and lower ONNX core format file to representations that are fed to further compilers, runtimes, or directly executed. 6 | The compiler solutions investigated by this SIG predominantly consumes ONNX core files as defined by the Architecture & Infrastrucutre SIG and uses ONNX operations as defined by the Operator SIG. 7 | This SIG is open to solutions developped in any relevant, open-sourced compiler-frameworks and may sponsor projects that work in specific compiler frameworks. 8 | This SIG also invite discussions on making the ONNX standard more amendable to compiler technology. 9 | 10 | This repo contains all the artifacts, materials, meeting notes, and proposals regarding the Compiler SIG. 11 | 12 | Feedbacks and contributions are welcome. 13 | 14 | ## Slack channel 15 | Please sign up at https://slack.lfai.foundation/ and join [onnx-compilers](https://lfaifoundation.slack.com/archives/C04Q6GVLCHX) channel. 16 | 17 | ## SIG Lead(s) 18 | 19 | * Alexandre Eichenberger (IBM) 20 | * Philip Lassen (Groq) 21 | 22 | ## Logistics 23 | 24 | * SIG lead(s) will drive the monthly Compiler SIG meeting. 25 | * Meeting annoucement will be posted in our Slack channel. 26 | * Specific Compiler-SIG projects may meet at a higher frequency. 27 | * SIG lead(s) may delegate leadership of project meetings to their respective project technical lead(s). 28 | * Monthly Compiler SIG meetings may take place at the begining of a specific project meeting. 29 | * Feedbacks and topic request are welcome by all. 30 | 31 | ## Discussion 32 | 33 | * Slack channel https://lfaifoundation.slack.com/archives/C018Y2QG7V2 34 | * Documents and artifacts: https://github.com/onnx/sigs/tree/main/compilers 35 | 36 | ## Meeting notes 37 | 38 | * Compiler SIG meeting notes are kept in the [meeting](meetings) folder. 39 | 40 | ## Current projects(s) 41 | 42 | * Onnx-mlir: lowering ONNX representation to other MLIR representations, including representations for ingestion in the LLVM compiler. 43 | * Description: [High Level](https://www.onnx.ai/onnx-mlir), [GitHub](https://github.com/onnx/onnx-mlir). 44 | * Meeting [agenda and notes](https://github.com/onnx/onnx-mlir/wiki/Informal-meeting-agenda-and-notes). 45 | * Slack channel: [onnx-mlir](https://lfaifoundation.slack.com/archives/C01B38FP2AV) 46 | 47 | 48 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | # SIGs - Special Interest Groups 4 | 5 | As described in the ONNX [governance](https://github.com/onnx/onnx/tree/main/community#sig---special-interest-groups), Special Interest Groups (SIGs) are persistent groups responsible for specific parts of the project. SIGs have open and transparent proceedings to develop goals and implement code contributions. SIGs are also responsible for ongoing maintenance of the code in their areas. 6 | 7 | SIG artifacts, including membership list and meeting notes, are stored in this repository. 8 | 9 | ## Participating 10 | If you are interested in participating, please [join the discussion](https://lfaifoundation.slack.com/archives/C016UBNDBL2") in the respective slack channel. 11 | 12 | You can find the schedule of SIG meetings on the [LFX calendar](https://zoom-lfx.platform.linuxfoundation.org/meetings/lfai-onnx?view=month) 13 | 14 | ## SIG members 15 | The current list of Contributors and Approvers in the SIGs are listed in the [CONTRIBUTORS](CONTRIBUTORS) file. The file also describes the process for updating the list. 16 | 17 | ## Current SIGs 18 | 19 | | Name | Responsibilities | 20 | | ------------------ | ------------- | 21 | | [Architecture & Infra](infra) | Defining and maintaining the core ONNX format, the build and CI/CD systems for ONNX repositories, publishing release packages for ONNX, and creating tools to help integrate with and test against the ONNX standard. This SIG is also the defacto owner of files in the main ONNX repository unless explicitly owned by another SIG. | 22 | | [Compilers](compilers) | Developing and maintaining core compiler technology for optimizing and lowering ONNX format. | 23 | | [Converters](converters) | Developing and maintaining the various converter repositories under ONNX. | 24 | | [Models and tutorials](models-tutorials) | Providing a comprehensive collection of state of the art ONNX models from a variety of sources and making it easy for users to get started with ONNX and the ecosystem around it. | 25 | | [Optimizations](optimizations) | Developing and maintaining solutions to optimize ONNX models, including model compression techniques (e.g. quantization, pruning, distillation, etc.) | 26 | | [Operators](operators) | Determining the operators that are part of the ONNX spec (ONNX and ONNX-ML domains), ensuring high quality operator definitions and documentation, establishing criteria for adding new operators, managing ops domains and compliance tiers, and enforcing versioning mechanisms. | 27 | -------------------------------------------------------------------------------- /converters/meetings/016-20201120.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | # Friday Nov 20, 2020 10:30-11:30am PST 4 | * https://zoom.us/j/7376656864 5 | 6 | ## Agenda 7 | * Greetings! 8 | * Discussion topics 9 | * Converters project updates 10 | * ONNX 1.8 impact/support 11 | * Custom ops story update? 12 | * ML data types and operators 13 | * Adding data processing to ONNX? 14 | 15 | ## Attendees 16 | * Chin Huang (onnx-tf, IBM) 17 | * Guenther Schmuelling (tf-onnx, Microsoft) 18 | * Winnie Tsang (onnx-tf, IBM) 19 | * John Zhang (Black System) 20 | ## Notes 21 | * TF-ONNX updates: opset 13 support by end of year, 97% of popular external models (from TF hub and Keras) successfully converted, prototype for TFLite, looking into qualized ops align with or wrap onnx ops, future version # might not match ONNX version # 22 | * ONNX-TF updates: opset 12 support completed, release 1.7 in a week with added user options in auto data type cast and target tensor format (NCHW vs NHWC), will include the conversion results for all ONNX zoo models in next release 23 | * Custom ops support as pluggable libraries to ONNX runtime is coming soon, as a new open source project. The idea is to provide pre-compiled so files for custom ops so the runtime doesn't need to know how to implement and optimize custom logic. 24 | * Pytorch to ONNX converter seems a bit out-dated. Recommend to open issues in Pytorch repo. Someone is monitoring converter issues and should respond. 25 | * Does it make sense to have the option of keeping NHWC format in onnx model, since some frameworks and devices without GPUs are channel last by nature? Suggest to open an issue in ONNX to possibly add a data format optional attribute to a list of impacted ops, and bring this topic up to the operators SIG for review and discussions. 26 | * Support of ML data types such as map and array as model inputs and outputs is a challenge in TF-ONNX and ONNX-TF since Tensorflow doesn't have matching data types for traditional ML. However, sequences can be used in nodes and operators. 27 | * Does it make sense to expand ONNX to cover some data processing in the future, as a few roadmap items are in this area? Data processing might include 1. traditional data ETL from db 2. data normalization, encoding, scaling for machine learning. We believe adding some data processing will help the consumption and optimization of the end-to-end machine learning process. We will continue to discuss what should be included, perhaps in a phased approach, and how to support it in converters. 28 | -------------------------------------------------------------------------------- /converters/meetings/007-20191024.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | # Thursday Oct 24, 2019 at 10:00-11:00am PST 4 | * online: https://ibm.webex.com/join/chhuang 5 | * call-in: 1-844-531-0958, 927 169 571 # 6 | 7 | ## Agenda 8 | * Greetings! 9 | * Discussion topics 10 | * Training proposal update and planning 11 | * How to support mulitple versions of a framework 12 | 13 | ## Attendees 14 | * Chin Huang (onnx-tf, IBM) 15 | * Guenther Schmuelling (tf-onnx, Microsoft) 16 | * Winnie Tsang (onnx-tf, IBM) 17 | * Svetlana Lavitan (IBM) 18 | 19 | ## Notes 20 | * We reviewed the training proposal effects doc and discussed the potential impacts and open issues for converters today. In addition to the high level description in the original doc, these are the issues to be addressed (summary of our past two meetings): 21 | * Frontend converter (exporter) 22 | * Might need additional ops or modify existing ones (split…) for training, for ex. constant folding, data augmentation 23 | * The user experience is new and needs clarification, for ex. taking possible inputs in python function, savedmodel, checkpoint for TF 24 | * How to capture training loops and optimizers in Pytorch? 25 | * How to decide which parameters are trainable (constant vs variable)? 26 | * Does a training graph must have a backward graph with gradient operators? 27 | * Can user provide a subgraph instead of pre-defined ONNX loss functions and optimizers (maybe after v1)? 28 | * Backend converter (importer) 29 | * Need to read from function proto instead of graph proto in a training IR 30 | * Need to parse TrainingInfo proto and algorithm 31 | * How to convert and verify loss functions? 32 | * How to convert and verify optimizers? 33 | * What to do with gradient operators? 34 | * How to support multiple import modes, training, inference, or both? 35 | * We believe the training proposal PR can be merged with sufficient self unit tests and some form of backend validation such as in ONNX Runtime. 36 | * We would suggest the training working group to produce a sample trainable ONNX file for mnist. It will serve the frontend converters as a target pb file to produce from the source framework. And it can be the starting point for backend converters to generate a trainable model in various frameworks or runtimes to execute the model training. 37 | 38 | ## Action items 39 | * 1: Investigate the feasibility of supporting ONNX training from pytorch and Tensorflow 40 | * 2: Investigate and eventually propose a good design for frontend converters to work with ONNX training 41 | -------------------------------------------------------------------------------- /optimizations/README.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | # ONNX Optimizations Special Interest Group (SIG) 4 | 5 | This is the working repo for the Optimization Special Interest Group (SIG), which is responsible for solutions to optimize ONNX models, including model compression techniques (e.g. quantization, pruning, distillation). The optimization solutions investigated by this SIG predominantly operate on ONNX core files as defined by the Architecture & Infrastructure SIG and use ONNX operations as defined by the Operator SIG - i.e. ONNX as input, optimization on ONNX file(s), ONNX as output. This SIG is open to techniques developed in any relevant, open-sourced optimization frameworks and capabilities and may sponsor projects that work in specific optimization frameworks. This SIG also invite discussions on making the ONNX standard more amendable to ONNX optimization technologies. 6 | 7 | This repo contains all the artifacts, materials, meeting notes, and proposals regarding the Optimization SIG. 8 | 9 | Feedbacks and contributions of other optimization tools are welcome and encouraged for discussion at the monthly meetings. 10 | 11 | ## Slack channel 12 | Please sign up at https://slack.lfai.foundation/ and join [onnx-optimization](https://lfaifoundation.slack.com/archives/C05GKQH0H34) channel. 13 | 14 | 15 | ## Meeting 16 | 17 | * 4th Thursdays of the month, from 5PM PST to 6PM PST 18 | * Check LFX calender for login link 19 | 20 | ## SIG Lead(s) 21 | 22 | * Freddy Chiu (Intel) 23 | * Justin Chu (Microsoft) 24 | 25 | ## Logistics 26 | 27 | * SIG lead(s) will drive the monthly Optimization SIG meeting. 28 | * Meeting annoucement will be posted in our Slack channel. 29 | * Specific Optimization-SIG projects may meet at a higher frequency. 30 | * SIG lead(s) may delegate leadership of project meetings to their respective project technical lead(s). 31 | * Monthly Optimization SIG meetings may take place at the begining of a specific project meeting. 32 | * Feedbacks and topic requests are welcome by all. 33 | 34 | ## Discussion 35 | 36 | * Slack channel: https://lfaifoundation.slack.com/archives/C05GKQH0H34 37 | * Mailinglist: 38 | * Documents and artifacts: https://github.com/onnx/sigs/tree/main/optimizations 39 | 40 | ## Meeting notes 41 | 42 | * Optimization SIG meeting notes are kept in the [meeting](meetings) folder. 43 | 44 | ## Current projects(s) 45 | * ONNX Optimizer 46 | * [GitHub](https://github.com/onnx/optimizer) 47 | * Neural Compressor 48 | * [GitHub](https://github.com/onnx/neural-compressor) 49 | 50 | 51 | -------------------------------------------------------------------------------- /converters/meetings/020-20210507.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | # Thursday May 7 2021 10:30-11:30 pm PST 4 | * https://zoom.us/j/7376656864?pwd=SjdiaGdvUFFyVXM4OUxTWndYZ2Z1Zz09 5 | 6 | ## Agenda 7 | * Greetings! 8 | * Welcome Martin Croome from Greenwaves! 9 | * Discussion topics 10 | * Greenwaves converts/compiles onnx and tflite models to run on own platform on edge devices 11 | * List of ONNX converters, https://github.com/onnx/sigs/blob/master/converters/docs/ConvertersList.md, should be updated to reflect the currnet state 12 | * Converters project updates 13 | * onnx-tf 1.8 was released in Apr with complete opset 13 support and inference graph based training 14 | * tf2onnx continues to test with hundreds of models to ensure coverage, at 92% convert rate. Face a major issue with keras to onnx converter so would like depreciate it and use tf2onnx instead based on tf graph. tf2onnx currently has a few optimizers to deal with batch norm and transpose etc. As for lstm/gru, tf2onnx converts to basic onnx ops or lstm/gru, would like to see more models with these ops. 15 | * onnx-mlir supports onnx to mlir, covering about 2/3 of onnx operators, not mlir to onnx yet. mlir could be a reusable component. we will look into synergy between mlir and converters not only conversion but also optimization. 16 | * Pytorch to ONNX also tests hundreds of models, converting about 85%. They are on the latest Pytorch and onnx. 17 | * TFLite to ONNX: seems two solutions there, tf2onnx and tflite2onnx (https://github.com/jackwish/tflite2onnx) tf2onnx converts tflite to tf, no open issues. 18 | * tensor-rf working on 8.0 release, GA in June 19 | * Models with external data are supported by tf2onnx (writes constants to separate protobuf files when model is larger than 2GB) and tensor-rt (internal tests, reads initializer files). There is no standard unit test in onnx core, so it is up to individual converters to verify this feature. Use onnx helper whenever possible. 20 | * NHWC support is still open/undecided in the onnx community. As is, the converters will ensure the data format is correct for the target framework and runtime. 21 | * There might be multiple solutions to convert keras to onnx and tflite to onnx. We should make sure to have clear message so users know which one to use for their needs. 22 | 23 | ## Attendees 24 | * Chin Huang (onnx-tf) 25 | * Guenther Schmuelling (tf2onnx) 26 | * Kevin Chen (Tensor-RT) 27 | * Martin Croome (Greenwaves) 28 | * Alexandre Eichenberger (onnx-mlir) 29 | 30 | ## Notes 31 | * Guenther to collect and share Pytorch status in next meeting 32 | -------------------------------------------------------------------------------- /operators/meetings/020-20210415.md: -------------------------------------------------------------------------------- 1 | # Thursday April 15, 2021 at 9:00am PST 2 | 3 | ## Agenda 4 | 5 | * ONNX 1.9 release date 4/16. Last call for comments. 6 | * [PR 2625](https://github.com/onnx/onnx/pull/2625): FFT and iFFT ops 7 | * [PR 3407](https://github.com/onnx/onnx/pull/3407): Type system extension for Optional type 8 | * [PR 3398](https://github.com/onnx/onnx/pull/3398): Type system extension for Sparse Tensors 9 | * [PR 3412](https://github.com/onnx/onnx/pull/3412): Add bfloat16 type to Pow operator 10 | 11 | 12 | ## Attendees 13 | 14 | * Evgenya Stepyreva (Intel) 15 | * Ganesan Ramalingam (Microsoft) 16 | * Michal Karzynski (Intel) 17 | * Scott Cyphers (Lightmatter) 18 | * Rodolfo Esteves (Intel) 19 | 20 | ## Notes 21 | 22 | ### ONNX 1.9 release plan 23 | 24 | * ONNX 1.9 release scheduled for 4/16 (tomorrow). Last call for comments. 25 | 26 | ### FFT/iFFT 27 | 28 | * PR has several comments that needs to be addressed before it can be merged. 29 | * Shape inference code needs minor fixes to handle statically unknown shapes. 30 | * Number of dimensions as attribute seems unnecessary. 31 | * Specification should clarify how the output is laid out. 32 | * Several people have shown an interest in adding FFT ops to ONNX. We are open to adding 33 | the op once the concerns mentioned in the PR are addressed. 34 | 35 | 36 | ### Optional Type 37 | 38 | * [PR 3407](https://github.com/onnx/onnx/pull/3407) is under review. 39 | * This is more of an infrastructure extension. No specific objections from the Operator SIG. 40 | * More information in the PR explaining the motivation, intended usage and example operations would be helpful. 41 | 42 | ### Sparse Tensor 43 | 44 | * [PR 3398](https://github.com/onnx/onnx/pull/3398) is under review. 45 | * Like the previous one, this type extension is more of an infrastructure extension. 46 | The feeling was that supporting ops like sparse matrix multiplication would be helpful. 47 | There was a discussion of the possibility of using mechanisms other than types for 48 | a model to give a hint to the backend which nodes or tensors in the graph would benefit 49 | from using a sparse representation. 50 | 51 | ### Pow operator 52 | 53 | * [PR 3412](https://github.com/onnx/onnx/pull/3412) is under review. 54 | * Recommendation was that this was straightforward and could be merged once comments 55 | are addressed. 56 | 57 | 58 | ### Miscellaneous items 59 | * It was felt that for future SIG meetings, adding a comment on PRs to be discussed might 60 | be useful in cases where authors presence in the SIG meeting would be beneficial. This 61 | would require planning for this a few days in advance of the meeting. 62 | 63 | -------------------------------------------------------------------------------- /converters/meetings/008-20191206.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | # Friday Dec 6, 2019 at 10:30-11:30am PST 4 | * online: https://ibm.webex.com/join/chhuang 5 | * call-in: 1-844-531-0958, 927 169 571 # 6 | 7 | ## Agenda 8 | * Greetings! 9 | * Discussion topics 10 | * Workshop recap 11 | * Training proposal update and planning 12 | * Inputs with unknown shapes unit tests 13 | * Backend scoreboard 14 | * How to support mulitple versions of a framework 15 | 16 | ## Attendees 17 | * Chin Huang (onnx-tf, IBM) 18 | * Guenther Schmuelling (tf-onnx, Microsoft) 19 | * Winnie Tsang (onnx-tf, IBM) 20 | * Svetlana Lavitan (IBM) 21 | * Emma Ning (mltools, Microsoft) 22 | * Jason Plurad (IBM) 23 | 24 | ## Notes 25 | * Emma provided a quick update on workshop converters session and stated training seems a hot topic. 26 | * We would like to work on concrete ONNX Training end-to-end scenarios for converters. Emma, Guenther, and Chin all commented for the respective converters, understanding the importance of Training support yet needing clearer directions on priority due to limited resources. As far as the timeline for converters to include training, we all think it is too early to make that estimates. We will collect more inputs from orgainization and investigate more closely with the training proposal and provide updates next time. 27 | * The standard ONNX unit test for backend doesn't cover the unknown tensor shapes as inputs case. The frontend converters already faced this common use case and managed to support it. We would like to have some unit tests in ONNX to verify and ensure the readiness for the backend. The onnx-tf converter team will look into this and provide updates. 28 | * The backend scoreboard, http://onnx.ai/backend-scoreboard/index.html, is provided by the operators SIG team. We encourage all backend converters to join the scoreboard. 29 | * We discussed how to support multiple versions of frameworks. Guenther shared the tf-onnx approach which includes TF 1.x and TF 2.0 in the same package (git branch) and use conditional statements whenever needed, and stated the differences between versions are so far minimum. Chin memtioned that the onnx-tf community considered two approached and will likely go with a separate git branch for TF 1.x due to code simplicity, cleanliness, and maintainability in the long team, knowning new code might need to be added to multiple branches. 30 | 31 | ## Action items 32 | * 1: Investigate the user experience and technical requirements of supporting ONNX training from pytorch, Tensorflow, and others 33 | * 2: Develop unit tests for unknown tensor shapes as inputs 34 | * 3: Connect Graphcore and Microsoft on Pytorch Training efforts 35 | -------------------------------------------------------------------------------- /optimizations/meetings/20231218.md: -------------------------------------------------------------------------------- 1 | # Attendees 2 | o Justin Chu, Microsoft – ONNX Converters 3 | o Freddy Chiu, Intel – ONNX Optimizations 4 | o Haihao Shen, Intel – Neural Compressor architect 5 | o Huang Tai, Intel – Neural Compressor lead 6 | o Feng Tian, Intel – Neural Compressor tech lead 7 | o Ramalingam G, Microsoft – ONNX/ONNX Runtime 8 | 9 | # Charter 10 | Documented charter: [sigs/optimizations at main · onnx/sigs (github.com)](https://github.com/onnx/sigs/tree/main/optimizations) 11 | - Scope of interest so far 12 | - Graph optimizations 13 | - As different execution back-ends have preference for certain graph patterns, optimizing models becomes a non-trivial problem for developers 14 | - Model converters today (e.g. PyTorch => ONNX) may produce different graphs that may not match patterns preferred by back-end to be deployed 15 | - Need: Systematic way to get graph transformations to benefit ONNX community for their respective backend 16 | - Next steps: Review/discuss existing solutions, and define goals to solve this need in the WG 17 | - Model compression 18 | - Goal is to have Neural Compressor in ONNX repo in ‘24 19 | - Neural Compressor v2.4 released: [Release Intel® Neural Compressor v2.4 Release · intel/neural-compressor (github.com) ](https://github.com/intel/neural-compressor/releases/tag/v2.4) 20 | 21 | # Meeting logistics 22 | - Tentative monthly meeting time: 5PM Tuesdays 23 | - Action Item: As tentative time conflicts with compiler SIG, Freddy to send out proposed alternatives to avoid conflict if possible 24 | 25 | # Discussions 26 | - Optimization challenges and how do we address them? 27 | - Haihao: Previously had encountered challenges converting low-precision PyTorch models to ONNX. 28 | - Justin: Support is on the way – Dynamo does not yet support this 29 | - Haihao: Packing (e.g. INT4) is still a problem due to different ways to represent. 30 | - Rama: there’s no consensus yet in the community on how to pack. Recommendation is to engage in PR and provide feedback. https://github.com/onnx/onnx/pull/5811 31 | - Haihao: How do we resolve opens regarding data types and flavors not yet available? 32 | - Rama: Recommendation is to participate in Operators SIG and arch/infrastructure SIG for helping with PR proposals for defining new data types 33 | 34 | # Community education and outreach - Not discussed 35 | # Updates - Not discussed 36 | # Opens 37 | - Co-Chair – Neta Zmora stepped down. Open position for co-chair. Drop me a note if you’re interested! 38 | - Ramakrishna unable to join because of zoom issues. Can we socialize zoom instructions. 39 | 40 | -------------------------------------------------------------------------------- /models-tutorials/meetings/003-20200413.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | # Monday, April 13, 2020, 2 PM PST 4 | 5 | ## Agenda / Notes 6 | * Introductions 7 | * Community events, proposed models 8 | * new events added by Svetlana to [list of upcoming ONNX events](https://github.com/onnx/sigs/blob/master/models-tutorials/docs/CommunityEvents.md) 9 | * several events postponed / moved to virtual due to global pandemic 10 | * [ONNX Workshop Update](https://github.com/onnx/sigs/blob/master/models-tutorials/docs/onnx-workshop-modelzoo-SIG-update.pdf) 11 | * [Git LFS migration](https://github.com/onnx/models/issues/271) in progress, ArcFace, DUC, most classification and all text models have been migrated successfully 12 | * Deprecate Checksums - discussion in ONNX Gitter regarding the lack of necessity for checksums since the files are uploaded with Git LFS has been determined. [PR #290](https://github.com/onnx/models/pull/290) addresses some checksum removal and the rest are being removed through the Git LFS migration process. 13 | * New Models - EfficientNet Lite 14 | * User Traffic Analysis / Analytics (RepoTraffic, RepoAnalytics) 15 | * Tracked in issue [#51](https://github.com/onnx/sigs/issues/51) 16 | * Progress on last month's action items: 17 | * Continue moving models in model zoo to Git LFS 18 | * mostly done, tracked in #276 19 | * ~Formalize CI Plan for Model Zoo~ 20 | * https://github.com/onnx/sigs/issues/53 21 | * Base requirement: run ONNX checker on each model. 22 | * ~Model Zoo analytics – how to ensure models are useful?~ 23 | * https://repo-analytics.github.io/onnx/models 24 | * https://repo-analytics.github.io/onnx/tutorials 25 | * ~Guidelines for addition of tutorials~ 26 | * ~merge Model Zoo addition guidelines~ - updated Pull Request Template 27 | * Proposal to deal with versioning of Model Zoo models 28 | * https://github.com/onnx/sigs/issues/52 29 | * Formalize new action items 30 | * Look into adding new ONNX Runtime training models into the model zoo 31 | 32 | ## Attendees 33 | * Vinitra Swamy (Microsoft) 34 | * Svetlana Levitan (IBM) 35 | * Negin Raoof (Microsoft) 36 | * Jacky Chen (Microsoft) 37 | 38 | ## Recorded Meeting 39 | Topic: Model Zoo + Tutorials SIG, Meeting #3 40 | Date: Apr 13, 2020 02:00 PM Pacific Time (US and Canada) 41 | 42 | Meeting Recording: 43 | https://zoom.us/rec/share/-9FLc630qV5Le4nI8kPEGYUgTtm8eaa823Maq_IKyUt2g48BStG2bPwA-1Ay7Fhv?startTime=1586811518000 44 | 45 | ## Action Items 46 | * Finish moving models in model zoo to Git LFS 47 | * Engage with issues on onnx/sigs repo 48 | * User analytics 49 | * Guidelines (Model Zoo + Tutorials) 50 | * Model Zoo CI 51 | * Versioning for Model Zoo models 52 | 53 | ## Next Meeting 54 | mid June 55 | -------------------------------------------------------------------------------- /operators/meetings/035-20221027.md: -------------------------------------------------------------------------------- 1 | # Thursday, October 27th, 2022 at 9:00am PST 2 | 3 | ## Agenda 4 | 5 | * Infrastructure SIG updates 6 | * Upcoming ONNX release 7 | * PR [#4089](https://github.com/onnx/onnx/pull/4089): GridSample op 8 | * PR [#4534](https://github.com/onnx/onnx/pull/4534): LpPool op 9 | * PR [#4621](https://github.com/onnx/onnx/pull/4621): GroupNormalization op 10 | 11 | ## Attendees 12 | 13 | * Aaron Bockover (Microsoft) 14 | * Andreas Fehlner (Trumpf Laser) 15 | * Andrew Fitzgibbon (Graphcore) 16 | * Chun-Wei (Jacky) Chen (Microsoft) 17 | * Ganesan Ramalingam (Microsoft) 18 | * Liqun Fu (Microsoft) 19 | * Michal Karzynski (Intel) 20 | * Przemyslaw Wysocki (Intel) 21 | * Scott Cyphers (Lightmatter) 22 | * Yuan Yao (Nvidia) 23 | 24 | ## Notes 25 | 26 | ### Infrastructure SIG updates 27 | 28 | Liqun Fu presented the Infrastructure SIG update. 29 | Current work is mainly focused on closing topics related to the 1.13 release. 30 | 31 | 32 | ### Upcoming release 33 | 34 | Przemek Wysocki presented the status of the upcoming ONNX release. 35 | We are currently awaiting code freeze, which is planned for 18th November. 36 | More timeline details can be found in PR [#4599](https://github.com/onnx/onnx/pull/4599). 37 | 38 | Please complete work related to release 1.13 before the code freeze. 39 | 40 | ### PR [#4089](https://github.com/onnx/onnx/pull/4089): GridSample op 41 | 42 | This PR was aleady discussed in previous meetings. 43 | GitHub description and discussion is fairly detailed. 44 | We would like to move forward with it and include it in the next release if possible. 45 | Please review this pull request and leave your comments. 46 | 47 | 48 | ### PR [#4534](https://github.com/onnx/onnx/pull/4534): LpPool op 49 | 50 | Remaining work for this PR includes adding a `dilations` attribute and tests. 51 | Przemek is working on this, he will go address these issues soon. 52 | 53 | 54 | ### PR [#4621](https://github.com/onnx/onnx/pull/4621): GroupNormalization op 55 | 56 | This pull request adds a new normalization operator `GroupNormalization` as an ONNX function. 57 | 58 | Group Norm is a commonly used normalization, similar to existing ONNX ops 59 | such as BatchNormalization, InstanceNormalization, and LayerNormalization. 60 | 61 | The operation is Supported in both Tensorflow and PyTorch. 62 | We discussed similarities and differences between the TensorFlow and PyTorch implementations 63 | and the proposed ONNX operation. 64 | 65 | Please review this PR and add your suggestions as to the best interface for this operator. 66 | 67 | We can express InstanceNormalization as a special case of GroupNormalization, 68 | so we can consider defining InstanceNormalization as an ONNX function using GroupNormalization. 69 | This could be a follow-up PR. 70 | -------------------------------------------------------------------------------- /operators/meetings/051-20240822.md: -------------------------------------------------------------------------------- 1 | # ONNX Operators SIG meeting 2 | # Thursday, Aug 22nd, 2024 at 9:00am PST 3 | 4 | ## Agenda 5 | 6 | * Next Release (1.17) 7 | * Arch/Infra SIG Update 8 | * [FP4 Data Type PR](https://github.com/onnx/onnx/pull/6283) 9 | * BF16 support in ONNX Ops 10 | 11 | ## Attendees 12 | 13 | * Ganesan Ramalingam (Microsoft) 14 | * Liqun Fu (Microsoft) 15 | * Michal Karzynski (Intel) 16 | * Raghavan Naresh Sarangapani (Intel) 17 | * Ramakrishnan Sivakumar (AMD) 18 | * Sunny Anand (IBM) 19 | * Yuan Yao (Nvidia) 20 | 21 | ## Meeting minutes 22 | 23 | ### Next Release (1.17) 24 | 25 | The release manager Raghavan mentioned that the Release Candidate was expected 26 | soon. The primary pending issue was a few outdated invalid URLs failing the URL 27 | check in CI. The URLs were to older pytorch/caffe2 versions illustrating the 28 | implementation of an ONNX backend. The conclusion was to replace the reference 29 | to main branch with specific versions containing the relevant code. 30 | 31 | * Arch/Infra SIG Update 32 | 33 | Liqun described the issue [6267](https://github.com/onnx/onnx/issues/6267#issuecomment-2269555846) and the workaround. See also the 34 | discussion [here](https://github.com/actions/runner-images/issues/10396). 35 | This issue is due to a breaking change in 36 | [visualstudio](https://learn.microsoft.com/en-us/visualstudio/releases/2022/release-notes-v17.10) 37 | 38 | 39 | ### [FP4 Data Type PR](https://github.com/onnx/onnx/pull/6283) 40 | 41 | Yuan Yao has this PR ready to add new 4-bit float types to ONNX. There was a discussion about 42 | whether merging this PR would interfere with the upcoming 1.17 release. The conclusion was 43 | that it should not. Yuan said that he would split the PR into two parts to simplify things. 44 | 45 | ### BF16 support in ONNX Ops 46 | 47 | Krishna brought up the issue of making the bfloat16 support complete in ONNX by allowing 48 | bfloat16 inputs for all ops where it was appropriate. However, Thiago did have a PR a few 49 | months back to do exactly this. It was unclear which ops currently lack bfloat16 support. 50 | Krishna will follow up. 51 | 52 | ### Opens 53 | 54 | Yuan brought up the use of the special value zero in the target shape in a Reshape op to 55 | propagate an existing dimension as-is. This special treatment can complicate some shape 56 | inference, analysis, and optimization when some of the target shape values are statically 57 | unknown. He suggested that eliminating it would be helpful. However, this does introduce 58 | some backward-compatibility challenges. On the other hand, it is really not a problem 59 | if the target shape is statically known. And if it is not, the difficulties can not be 60 | eliminated by using alternative encoding of the logic. No specific conclusion was reached. 61 | -------------------------------------------------------------------------------- /converters/meetings/015-20200911.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | # Friday Sept 11, 2020 10:30-11:30am PST 4 | * https://zoom.us/j/7376656864 5 | 6 | ## Agenda 7 | * Greetings! 8 | * Discussion topics 9 | * Converters project updates 10 | * Custom ops support 11 | * Loop, nested, visualization 12 | * Sequence type inputs 13 | * Wishlist 14 | 15 | ## Attendees 16 | * Chin Huang (onnx-tf, IBM) 17 | * Guenther Schmuelling (tf-onnx, Microsoft) 18 | * Kevin Chen (Nvidia) 19 | * Winnie Tsang (onnx-tf, IBM) 20 | 21 | ## Notes 22 | * ONNX-TF converter: work on Tensorflow native suppport and user options in target environments (gpu vs cpu) and automatic data type conversion 23 | * Tensor-rt: continue to add op support for opset 12 and 13, except training ops 24 | * Custom ops are typically very specific to a framework, not portable to other frameworks and runtimes. Two use cases should be considered to make custom ops more practical and useable: 1. execute the op in a runtime 2. convert the op to another op or ops in a framework. While the custom ops can be distinguished by namespace/domain name, it is difficult to identify their expected behaviors. We believe the use of custom ops is unavoidable since ONNX won't cover all operators in various frameworks. A more portable and repeatable approach should be developed, such as a pluggable execution and clear definition in a common place/repo, so the backend (converters and runtimes) can better support custom ops. 25 | * TF-ONNX produces some custom ops that are supported by ONNX runtime only. ONNX-TF doesn't currently support custom ops. 26 | * Sequence as an input type for a model is not supported in Tensorflow, due to no matching data type for ONNX sequence, a list of tensors with same type, could have different shapes. 27 | * The ONNX backend test runner doens't support sequence as input type, although a new test case being added lately for SequenceInsert. The runner assumes all inputs are tensors, therefore the new test case won't pass for backend converters. An issue is opened, https://github.com/onnx/onnx/issues/2984. 28 | * Loops with nested loops are expected and should be handled properly, especially the initializers could be in outer loops. 29 | * It is not easy to visualize and debug loops and nested loops. ONNX runtime has a utility to dump a subgraph. ONNX has APIs to print a subgraph in text. Would be nice if Netron can support views into subgraphs. 30 | * We would like to come up with a few topics for the upcoming workshop in Oct so broader community can participate and help drive the future direction. 31 | * better custom ops supporting system 32 | * more consistency between op schema, doc, and unit tests (we often have to read the code because doc might be wrong) 33 | * use of checker to ensure model quality and integrity (onnx has one, onnx runtime has another) 34 | -------------------------------------------------------------------------------- /models-tutorials/meetings/006-20200925.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | # Friday, September 25, 2020, 10:30 AM PST 4 | 5 | ## Agenda / Notes 6 | * Vinitra stepping down from SIG lead to start PhD @ EPFL, handing over SIG Lead position to Wenbing Li 7 | * We are looking for co-leads -- please reach out to Wenbing and the Steering Committee if interested! 8 | * [New Models added to the Model Zoo](https://github.com/onnx/sigs/blob/master/models-tutorials/docs/ProposedModels.md) 9 | * Model Updates 10 | * Fixes for SqueezeNet, ResNet, SuperResolution 11 | * Community contributions: new flavor of GPT-2 [#327](https://github.com/onnx/models/pull/327), update to pre/post process of GoogleNet[#333](https://github.com/onnx/models/pull/333), Transformer Language model (T5 [#357](https://github.com/onnx/models/pull/357)) 12 | * ML Perf model updates / inclusions 13 | * [Model Zoo API](https://github.com/onnx/models/pull/355) PR is out 14 | * Objective: API for users to easily list and obtain model files 15 | * https://github.com/onnx/models/pull/355 16 | * [Model Zoo CI - using ONNX Runtime](https://github.com/onnx/models/pull/376) - Jacky 17 | * [Community events](https://github.com/onnx/sigs/blob/master/models-tutorials/docs/CommunityEvents.md) 18 | * User Traffic Analysis / Analytics (RepoTraffic, RepoAnalytics) 19 | * Tracked in issue [#51](https://github.com/onnx/sigs/issues/51) 20 | * Look into adding new ONNX Runtime training models into the model zoo (Svetlana) 21 | * Update Model Zoo Github best practices 22 | * Review requirements for adding a new model (Ashwini) 23 | * Let's make sure it includes enough about the source model to re-export in a new opset 24 | * scenarios: mobile, quantization, etc. 25 | * We do need to do an accuracy test after re-exporting the models (Wenbing) 26 | * Infra for accuracy testing (investigation) 27 | * Model preprocessing + postprocessing (Vinitra, Tom) 28 | * preprocessing layers within ONNX (necessity for string ops, etc) - discussed in ONNX Roadmap meetings 29 | * very difficult to standardize within the ONNX standard, but could be scoped for smaller use cases (Ashwini / Wenbing) 30 | * bar new models that use another framework in the preprocessing 31 | 32 | ## Attendees 33 | * Vinitra Swamy (EPFL) 34 | * Wenbing Li (Microsoft) 35 | * Ashwini Khade (Microsoft) 36 | * Svetlana Levitan 37 | * Jacky Chen (Microsoft) 38 | * Natalie Kershaw (Microsoft) 39 | * Tom Wildenhain (Microsoft) 40 | * Jim Spohrer (LF AI) 41 | 42 | ## Recorded Meeting 43 | Topic: ONNX Model Zoo SIG Meeting 44 | Start Time : September 25, 2020 10:30 AM 45 | 46 | ## Action Items 47 | - Schedule meeting to discuss version control for models (re: version converter, maintaining source framework models etc.) 48 | - Determine integration plan / what features neet to be done for Jacky's Hackathon UI 49 | 50 | ## Next Meeting 51 | mid October 52 | -------------------------------------------------------------------------------- /operators/meetings/042-20230831.md: -------------------------------------------------------------------------------- 1 | # Thursday, Aug 31st, 2023 at 9:00am PST 2 | 3 | ## Agenda 4 | 5 | * Next release 6 | * Deprecation of older opsets 7 | 8 | ## Attendees 9 | 10 | * Aaron Bockover (Microsoft) 11 | * Christian Bourjau (QuantCo) 12 | * Scott Cyphers (Lightmatter) 13 | * Cliff Houck 14 | * Ganesan Ramalingam (Microsoft) 15 | * Rajeev Rao (Nvidia) 16 | * George (Intel) 17 | * Jacky Chen (Microsoft) 18 | * Justin Chu (Microsoft) 19 | * Liqun Fu (Microsoft) 20 | * Michal Karzynski (Intel) 21 | * Xavier Dupré (Microsoft) 22 | 23 | ## Notes 24 | 25 | ### Next release 26 | 27 | The next release (1.15) is planned to be around mid-October. The code-freeze for this 28 | release will be late September. Liqun Fu (Microsoft) will be the release manager. 29 | PRs targetting this release should use milestone 1.15 30 | 31 | ### Deprecation of older opsets 32 | 33 | In the ONNX Steering Committee, Nvidia brought up the question of deprecation of older 34 | opsets, arguing that it would be useful to deprecate older opsets, and have a policy for 35 | doing so. The SIG meeting discussed this question. 36 | 37 | First, there was a discussion about the earliest opset-versions that were supported by 38 | different frameworks/implementations and in use. Intel's OpenVINO's earliest supported 39 | opset version is 7. The PyTorch exporter's base opset version is 9, while the new 40 | upcoming Dynamo-based exporter targets opset version 18. 41 | 42 | Subsequently, the discussion brought up the following questions 43 | * What does it mean to deprecate an opset version? 44 | * Why should we deprecate opset version? 45 | * What problem are we trying to solve? 46 | 47 | An initial suggestion was to follow a prevalent industry practice of first deprecating, 48 | and then, after a period (like an year or so), marking the deprecated as obsolete, 49 | and after another period, eventually removing it. 50 | 51 | This led to a deeper discussion of what impact deprecation has on the ONNX codebase? Will deprecated 52 | opset versions be removed from the ONNX codebase? There are a number of concerns with 53 | doing this. The ONNX repo, being the gold-standard for the ONNX specification, should 54 | maintain the definition of all opset versions for reference. Further, even with a shift 55 | to newer opset versions, there will be a need for the version-converter to enable 56 | conversion of older opset version models to newer opset versions, which will require 57 | the definition of older opset versions. Furthermore, the way opset versions are defined 58 | in ONNX, newer opset versions inherit many operators from older opset versions. 59 | 60 | Since different frameworks, tools, and implementations already have restrictions on opset 61 | versions they support, it was unclear what is achieved by deprecating an opset version. 62 | One observation was that it is useful to have shared information about the opset versions 63 | supported by different frameworks and implementations. 64 | -------------------------------------------------------------------------------- /converters/meetings/013-20200522.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | # Friday May 22, 2020 10:30-11:30am PST 4 | * https://zoom.us/j/7376656864 5 | 6 | ## Agenda 7 | * Greetings! 8 | * Discussion topics 9 | * Converters project updates 10 | * ONNX release 1.7 11 | * Resize operator review 12 | * Partial operator support fall back to ONNX runtime 13 | * Range operator model test output has no data types 14 | * Unknown rank inputs (SplitToSequence) 15 | 16 | 17 | ## Attendees 18 | * Chin Huang (onnx-tf, IBM) 19 | * Guenther Schmuelling (tf-onnx, Microsoft) 20 | * Kevin Chen (Nvidia) 21 | * Winnie Tsang (onnx-tf, IBM) 22 | * Ting Su (MathWorks) 23 | * Svetlana Levitan (IBM) 24 | * Alexandre Eichenberger (onnx mlir, IBM) 25 | 26 | ## Notes 27 | * ONNX-TF converter: working on TF 2.0 native support and ONNX opset 11 28 | * TF-ONNX converter: support of ONNX 1.7 and opset 12 should be out in next week or two, exploring use of TF function to convert to ONNX ops 29 | * MatLab converters: working on ONNX opset 10 and 11, will look at opset 12 soon 30 | * Tensor-RT converter: no immediate plan to support training, will look into other ops in opset 12 once things settle down 31 | * ONNX-MLIR: has 50+ operators, not supporting return of indices, seeing a challenge to implement ops with many definitions, adding end-to-end testing for model zoo, not looking into performance yet 32 | * ONNX 1.7 is releaseed with opset 12 and training in a preview domain 33 | * ONNX traininig is adding an indicator to inputs and outouts to be differentable or not 34 | * Resize opset 11 added new attributes and logic. MatLab implements with own APIs whenever available or puts a placeholder for custom code, has issues with scales being empty string, ONNX-TF and Tensor-RT have partial support certain modes, common modes for Nvidia, half_pixel, pytorch_half_pixel, align_cornors and asymmetric, recommend to figure out the reasons why other options are introduced, for specific models? We all feel using sizes instead of scales is the easier approach for both frontend and backend converters. 35 | * Tensor-RT falls back to ONNX runtime for graph partitioning and unsupported ops/attributes. ONNX runtime will do the execution where the subgraphs can't run on a particular runtime. Not looking possible for Tensorflow to integrate with ONNX runtime at this time. Maybe open an PR in Tensorflow. 36 | * Are TF converters using TF or/and TFRT (Tensorflow Runtime)? Currently TF, will look into TFRT. 37 | * Range op test in ONNX has a loop body with no output data types only names. The output of a subgraph in a loop can be unspecified? Could output types be inferred from inputs? Would like to learn how other backend converters/runtimes pass the model test. 38 | * During modeling time, can an ONNX model include a node input with unknown rank? SplitToSequence split input could be a scalar or a 1-d tensor. Need to check if ONNX allows unknown rank in tensor proto creation. 39 | -------------------------------------------------------------------------------- /converters/meetings/002-20190724.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | # Wednesday July 24, 2019 at 5:00pm-6:00pm PST 4 | * online: https://ibm.webex.com/join/chhuang 5 | * call-in: 1-844-531-0958, 927 169 571 # 6 | 7 | ## Agenda 8 | * Introduction 9 | * Discussion topics 10 | * Discuss ONNXMLTools options 11 | * Review findings in standardized frontend tests 12 | * Discuss how to support ONNX training in converters 13 | * Review findings in general converter reporting/dashboard 14 | * Discuss contributor list 15 | * Collect pain points, desired features, new discussion topics from all converters 16 | 17 | ## Attendees 18 | * Chin Huang (onnx-tf, IBM) 19 | * Guenther Schmuelling (tf-onnx, Microsoft) 20 | * Prasanth Pulavarthi (Microsoft) 21 | * Winnie Tsang (onnx-tf, IBM) 22 | * Svetlana Levitan (IBM) 23 | * Emma Ning (mltools, Microsoft) 24 | * Akin Solomon (Snapdragon, Qualcomm) 25 | * Spandan Tiwari (pytorch, Microsoft) 26 | * G. Ramalingam (Microsoft) 27 | * Vinitra Swamy (Keras, Microsoft) 28 | * Young (sorry, did not catch full name, Windows ML, Microsoft) 29 | * Guyan 30 | * Yue-Sheng Liu 31 | 32 | ## Notes 33 | * We got much broader based participation from various converters, including Tensorflow, Pytorch, MLTools, Keras, Snapdragon, Windows ML. 34 | * We talked about what "converter" means in ONNX. Currently we take the broadest definition, converting ONNX format from or to any framework or backend. Therefore, backends like onnx runtime and Habana are on the list. We should firm up our definition and publish the list on the sigs repo. 35 | * Discussed a possible new repo, onnx/converters, to keep common issues, docs, and artifacts for all converters, and decided to use onnx/sigs with "converters" label to keep sigs related activities/assets together. 36 | * Discussed ONNXMLTools' role and recommended only one converter for each framework. ONNXMLTools will continue to serve existing users and multiple converters like Spark ML and LightGBM before they become individual converter projects. 37 | * The standardized frontend test is currently not recommended because each framework often has unique implementation of operators. 38 | * The onnx-tf converter team has looked into the ONNX training proposal and written a prototype to validate the design. The tf-onnx converter team will evaluate the feasibility of the proposal and provide update in two weeks. 39 | * We shared findings on onnx/backend-scoreboard and will continue to investigate better ways to construct the converters dashboard. 40 | * We briefly shared pain points like unsupported data types and attributes. Will continue to discuess and eventually make general recommendations. 41 | 42 | ## Action items 43 | * 1: Guenther: investigate the feasibility of supporting ONNX training from tf to ONNX 44 | * 2: Chin: draft general docs to clarify "converter", list of converters, and other agreed statements 45 | * 3: Winnie: investigate options to provide general converter reporting/dashboard 46 | -------------------------------------------------------------------------------- /converters/meetings/004-20190905.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | # Thursday Sept 5, 2019 at 5:00-6:00pm PST 4 | * online: https://ibm.webex.com/join/chhuang 5 | * call-in: 1-844-531-0958, 927 169 571 # 6 | 7 | ## Agenda 8 | * Greetings! 9 | * Discussion topics 10 | * How to handle python loops and lists? (Preferred Networks) 11 | * Use of custom attributes in converters (GraphCore) 12 | * Review existing issues 13 | * Evaluate training proposal 14 | * Converters dashboard 15 | * Deprecated operators 16 | * Dynamic/unknown shape and rank in inputs 17 | * ONNX-ML operators 18 | * New issues or features... 19 | 20 | ## Attendees 21 | * Chin Huang (onnx-tf, IBM) 22 | * Winnie Tsang (onnx-tf, IBM) 23 | * Spandan Tiwari (pytorch, Microsoft) 24 | * Shinichiro Hamaji (chainer, Preferred Networks) 25 | * Thomas Truong (IBM) 26 | * Jignesh Parmar (tf2onnx, Microsoft) 27 | 28 | ## Notes 29 | * Shinichiro presented the custom ops in chainer compiler that represent python list/loop in ONNX graph, https://docs.google.com/presentation/d/1UkvE_8wHjGFnmQfD4l5loQHAMM37v29m1U4oAYOVRDc/edit#slide=id.p. The active ONNX PR #2249 has a number of sequence related ops that covered a subset of presented custom ops, and will be included in the upcoming 1.6 release. Additional sequence ops, as indicated in the presentation/doc, could potentially be added as separate PRs. 30 | * The Sequence type is moving (promoted) to the onnx domain from onnx-ml in release 1.6. Therefore we suggest that all converters plan to support it. 31 | * No update on the ONNX training proposal. Will follow up with the frontend converters (Tensorflow and Pytorch) and backend (GraphCore) next meeting. 32 | * The detailed coverage dashboard is recommended for backend converters but optional for frontend converters. A landing page to view all converters should be constructed providing high level summary and links to detailed reports. 33 | * Reviewed and agreed the recommendation for handling deprecated operators. In short, frontend converters do not use deprecated ops (use the replacement instead), and backend converters either rely on the default ONNX checker to throw exception or handle with the previous definition. 34 | * The converters worked on by today's attendees do not support ONNX-ML operators. We would like to know which converters currently do and leverage their use cases and unit tests whenever possible. 35 | * Did not find a practical use case for unknown rank inputs. Would like to get feedback from additional converters to make a sensible suggestion. 36 | * A new issue was brought up, how to support different system configuration or backend framework versions between the conversion and execution servers? 37 | 38 | ## Action items 39 | * 1: Investigate the feasibility of supporting ONNX training from tf and pytorch to ONNX 40 | * 2: Continue to investigate the converter reporting/dashboard 41 | * 3. Look for possible use cases for unknown rank inputs 42 | * 4. Look into unit tests for unknown shape inputs 43 | -------------------------------------------------------------------------------- /converters/meetings/017-20201211.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | # Friday Dec 11, 2020 10:30-11:30am PST 4 | * https://zoom.us/j/7376656864?pwd=SjdiaGdvUFFyVXM4OUxTWndYZ2Z1Zz09 5 | 6 | ## Agenda 7 | * Greetings! 8 | * Discussion topics 9 | * Converters project updates 10 | * Custom ops story 11 | * Adding NHWC option to specific ops 12 | * Adding data processing to ONNX 13 | * Conversion verification using model zoo models 14 | 15 | ## Attendees 16 | * Chin Huang (onnx-tf, IBM) 17 | * Guenther Schmuelling (tf-onnx, Microsoft) 18 | * Winnie Tsang (onnx-tf, IBM) 19 | * Sarah Mohamed (matlab) 20 | * Kevin Chen (tensor-rt, Nvidia) 21 | 22 | ## Notes 23 | * onnx-tf updates: release 1.7 is completed, supporting Tensorflow 1 (will be discontinued) and 2, added a model zoo report to wiki, currently with 98% successful conversion rate, working on the inference verification 24 | * tf-onnx updates: almost done with opset 13, prototype for TF Lite, quantization almost working 25 | * tensor-rt updates: working on more operator coverage, patching up to pass at least 70% ONNX backend test, made progress in dynamic shape support, two ways to support an onnx op, using tensor-rt native API or through a plugin 26 | * matlab updates: import onnx model to matlab code, complete coverage model zoo, working on new models 27 | * We should be aware of issues and coordinate between tf-onnx and onnx-tensorrt 28 | * The custom op plugin project for ONNX runtime is available at https://github.com/microsoft/ort-customops. For backend for existing frameworks, we need to uunderstand the custom op at the source and convert accordingly. 29 | * Does it make sense to have the option of keeping NHWC format in onnx model, since some frameworks and devices without GPUs are channel last by nature? It will cause additional work for backend converters to deal with the new attribute. Should open an issue for community feedback. 30 | * Data processing, in particular DataFrame and related functions in pandas, could be a nice addition to ONNX. This means the source could come from multiple frameworks, such as pandas and Tensorflow. And all operations are converted into one ONNX graph. There must be some way to differentiate data processing from model inference in onnx model so some backends could properly process certain portion. Pandas has a lot of functions. Design needs to be flexible and well thought out. 31 | * Tensor-rt is focusing on final step of model inference in a pipeline, not training or data processing. 32 | * How does a backend verify against model zoo inference? What is a reasonable tolerance threshold? Tensor-rt doesn't use the sample inputs and outputs from model zoo. It uses some random data and compares results with ONNX runtime and other runtimes. Matlab applies the similar methodology of comparing results with other reference runtimes. onnx-tf is attempting to use the provided inputs and outputs in model zoo and get to e-3 absolute and relative accuracy. 33 | * A tool could be used to verify inference results, https://github.com/NVIDIA/TensorRT/tree/master/tools/Polygraphy 34 | -------------------------------------------------------------------------------- /converters/meetings/003-20190814.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | # Wednesday Aug 14, 2019 at 9:00am-10:00am PST 4 | * online: https://ibm.webex.com/join/chhuang 5 | * call-in: 1-844-531-0958, 927 169 571 # 6 | 7 | ## Agenda 8 | * Greetings! 9 | * Discussion topics 10 | * Review converter definition and list 11 | * Review ONNX training spec and discuss how to support it in converters 12 | * Review findings in general converter reporting/dashboard 13 | * Discuss contributor list 14 | * Collect pain points, desired features, new discussion topics from all converters 15 | * Unsupported data types 16 | * Unsupported optional attributes 17 | * Lack of tests 18 | * Unclear logic or explanation for the meaning of operator 19 | * Experimental operators 20 | 21 | ## Attendees 22 | * Chin Huang (onnx-tf, IBM) 23 | * Guenther Schmuelling (tf-onnx, Microsoft) 24 | * Winnie Tsang (onnx-tf, IBM) 25 | * Svetlana Levitan (IBM) 26 | * Emma Ning (mltools, Microsoft) 27 | * Akin Solomon (Snapdragon, Qualcomm) 28 | * Spandan Tiwari (pytorch, Microsoft) 29 | * Saurabh Tangri (Onnx runtime, Intel) 30 | 31 | ## Notes 32 | * Agreed the converter definition "An ONNX converter is a software component that converts models from a ML/DNN framework to the ONNX format, or vice versa", therefore we considered the backend that takes an ONNX model and executes in specific platforms as a "runtime". 33 | * Reviewed the list of converters and runtimes and will move Windows ML and SNPE(Qualcomm) to the runtime list. The definition and list are documented at https://github.com/onnx/sigs/blob/master/converters/docs/ConvertersList.md 34 | * Reviewed the ONNX training proposal. Chin shared the prototype code, https://github.com/chinhuang007/onnx-tf-training, that starts from a pytorch model, going through an ONNX format with training data, and ends with additional training in Tensorflow. More investigation is needed to ensure the feasibility of producing the ONNX training data from different frameworks. 35 | * Winnie shared the proposed content and look-and-feel to make clear the operator coverage for a converter, up to each ONNX opset level. General feedback is pretty good for backend converters but not seemingly useful for frontend converters. A few points: there are so many operators (800+ for ex in tf), tiresome to go through a big table might be easier to try a model and see if fails, hard to automate the generation of coverage table, so many high level APIs translating to low level ops. We will continue the discussion next time. 36 | * Reviewed and agreed the recommendation for handling unsupported data types in a more flexible approach with an optional user input and logging for clarity whenauto-cast is applied. 37 | 38 | ## Action items 39 | * 1: Guenther, Spandan: investigate the feasibility of supporting ONNX training from tf and pytorch to ONNX 40 | * 2: Chin: finalize general doc to clarify "converter" and list of converters 41 | * 3: Winnie: continue to investigate the converter reporting/dashboard 42 | * 4. Chin, Guenther: prepare for the workshop breakout session 43 | -------------------------------------------------------------------------------- /converters/meetings/010-20200214.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | # Thursday Feb 14, 2020 10:30-11:30am PST 4 | * https://zoom.us/j/7376656864 5 | 6 | ## Agenda 7 | * Greetings! 8 | * Discussion topics 9 | * Training proposal update and planning (use cases, new global_mutable_initializer_name) 10 | * ONNX 1.7 11 | * Support mulitple versions of a framework 12 | * Inputs with unknown shapes unit tests and effects 13 | * Sequence ops handling in backend converters 14 | * Channel first vs channel last 15 | * Revisit onnx-ml operators 16 | * Other topics 17 | 18 | ## Attendees 19 | * Chin Huang (onnx-tf, IBM) 20 | * Guenther Schmuelling (tf-onnx, Microsoft) 21 | * Jonny Shipton (Myrtle.ai) 22 | * Svetlana Lavitan (IBM) 23 | * Thomas Truong (IBM) 24 | * Alexandre Eichenberger (onnx mlir, IBM) 25 | 26 | ## Notes 27 | * We identified two uses for training: 1. the onnx model contains inference only graph and the backend converters/runtimes will generate training graph 2. the onnx model contains inference and training information, such as optimizers, loss functions, and gradient operators. That means the frontend converters need to produce TrainingInfoProto and the backend will simply consume the training data without auto-gen. 28 | * The ONNX 1.7 release is targeted early March with the new TrainingInfoProto and global_mutable_initializer_name in the spec for training as a tech preview. The current focus is on the spec and ONNX Runtime to verify the use case 1 above. Deep 500 was mentioned as possible framework to help. We recommend the converter teams to get familiar with the training design and spec and start working on training support such as how to generate gradient operators in future releases. 29 | * The ONNX backend unit test doesn't cover unknown tensor shapes as inputs and has limited tests for various data types. We will bring this concern toi ONNX core community and collaborate on a more complete backend test suite. 30 | * The ONNX sequence operators can be converted to tf.RaggedTensor in Tensorflow. We saw the test code in model test and would like to learn the practical use cases to understand and verify the functionality. 31 | * Handling channel first (NCHW) vs channel last (NHWC) is different between frameworks. With Tensorflow as a backend, we could either 1. always convert to NHWC and let TF do the additional conversion based on physical devices if needed 2. convert data format for a target runtime (GPU vs CPU). Since certain transpose ops will be added to the graph, it is adviced to include some optimization to minimize the graph. This topic seems most relevant only for image models. 32 | * ONNX-ML operators are used in quite a few frameworks and ONNX Runtime. TF frontend started to make use of them. TF backend will also look into the ML operator support. 33 | * Alexandre introduced the MLIR conversion and discussed briefly the current challenges. We would like to continue the discussion and provide summary in next meeting. 34 | 35 | ## Action items 36 | * 1: Identify and design ONNX training support for frontend and backend converters. 37 | * 2: Investigate and share support for the ONNX-ML operators 38 | -------------------------------------------------------------------------------- /converters/meetings/012-20200424.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | # Friday Apr 24, 2020 10:30-11:30am PST 4 | * https://zoom.us/j/7376656864 5 | 6 | ## Agenda 7 | * Greetings! 8 | * Discussion topics 9 | * Converters project updates 10 | * ONNX release 1.7 11 | * Training updates 12 | * ML operators support 13 | * Resize operator review 14 | 15 | ## Attendees 16 | * Chin Huang (onnx-tf, IBM) 17 | * Guenther Schmuelling (tf-onnx, Microsoft) 18 | * Kevin Chen (Nvidia) 19 | * Winnie Tsang (onnx-tf, IBM) 20 | * Ting Su (MathWorks) 21 | * Svetlana Levitan (IBM) 22 | * Sree Vaddi 23 | 24 | ## Notes 25 | * ONNX-TF converter: still working on TF 2.0 native support, will look into training soon, spec driven development 26 | * TF-ONNX converter: has TF 2.0 mostly working, fixing a lstm bug now 27 | * MatLab converters: increase converage, facing challenges in change number of dimensions for inputs and outputs, for ex. gather, flatten, reshape, model case driven development 28 | * Tensor-RT converter: having to deal with new IR version and dynamic shapes, for ex resize operator. models in model zoo not updated with recent opset version. 29 | * ONNX 1.7 IR version=7 in onnx file would cause problems in ONNX runtime. The frontend converters need to have a mapping to set IR version based on the opset version 30 | * ONNX traininig is waiting for release 1.7 to add new features, including differentialable tag which is fixed per operator, utility APIs which can be used by frontend converters to provide minimum information to turn a inference model into a trainable model. 31 | * Training support polls at ONNX Workshop indicate 30% converters will have near term support. 32 | * Training support in the context of converters means 1. convert a framework training model to onnx 2. take onnx training model and continue training in a framework according to what is described in onnx file. 33 | * Why onnx traininng? 1. training can be part of deployment, fine-tuning 2. optimized runtime for training 3. training with different hardware 34 | * Keras to ONNX converter supports standard Keras APIs like layers and should take TF saved model format as well. TF-ONNX converts Keras models too, based on the graph in hd5 format, not Keras layers APIs. 35 | * ONNX ML support polls at ONNX Workshop indicates 40% converters support some ML operators. Tensor-RT and MatLab have no plans to support in the near future. SK learn to ONNX has complete support. ONNX is faster than SK learn so no backend converter for SK learn. ONNX-Tf in investigation. General question, what is the use cases and benfits to convert ML to framework? 36 | * Resize opset11 has a lot of attributes and complicated logic, very hard for backend converters to convert using framework APIs. ONNX-TF can only do partial support. MatLab will check with developers and share findings in next meeting. Tensor-R has own implementation, partial support certain modes, common modes: asymmetric, half_pixel, pytorch_half_pixel. 37 | * Any unsupported ops or flavors, Tensor-RT falls back to ONNX runtime for graph partitioning. Still get performance gains from Tensor-RT. Kevein to share how it works in next meeting. 38 | -------------------------------------------------------------------------------- /operators/meetings/027-20220127.md: -------------------------------------------------------------------------------- 1 | # Thursday, January 27, 2022 at 9:00am PST 2 | 3 | ## Agenda 4 | 5 | * Updates from the Infrastructure SIG 6 | * Upcoming ONNX release 7 | * Mixed precision representation (Rodolfo G Esteves, Intel) 8 | * ONNX Model provenance (Rodolfo G Esteves, Intel) 9 | * Issues raised by Nvidia (Kevin Chen and Dilip Sequeira) 10 | 11 | ## Attendees 12 | 13 | * Ashwini Khade (Microsoft) 14 | * Bhargavi Karumanchi (Intel) 15 | * Chun-Wei (Jacky) Chen (Microsoft) 16 | * Dilip Sequeira (Nvidia) 17 | * Ganesan Ramalingam (Microsoft) 18 | * Gary Miguel (Microsoft) 19 | * GeorgeN 20 | * Kevin Chen (Nvidia) 21 | * Liqun Fu (Microsoft) 22 | * Michal Karzynski (Intel) 23 | * Przemyslaw Wysocki (Intel) 24 | * Ria Cheruvu (Intel) 25 | * Rodolfo G Esteves (Intel) 26 | * Scott Cyphers (Lightmatter) 27 | 28 | ## Notes 29 | 30 | ### Infrastructure SIG updates 31 | 32 | Next release of ONNX, version 1.11 is scheduled for early February, with a code-freeze date of Jan 31. 33 | Python 3.6 will be deprecated after the current release (in version 1.12). 34 | 35 | ### ONNX Model Provenance and Mixed precision representation 36 | 37 | Rodolfo Esteves, Bhargavi Karumanchi, and Ria Cheruvu (Intel) made a presentation on the topic of ONNX model provenance and mixed precision representation. The goals of this work is to enrich and structure metadata (which is to be machine-readable) to aid model provenance tracking, discoverability of models on hubs, annotation of fairness/Ethical AI considerations and other concerns (in an extensible manner). Please see issue 38 | [Issue 3958](https://github.com/onnx/onnx/issues/3958) for more details. 39 | 40 | The presentation talked about metadata fields to be added to capture model provenance and mixed 41 | precision representation and the metadata management mechanics, and the augmentation of markdown 42 | data with machine-readable frontmatter blocks (serializing an RDF fragment) and usecases such 43 | as query language to query such metadata. 44 | 45 | ### Nvidia proposals for ops and functions 46 | 47 | Kevin Chen and Dilip Sequeira from Nvidia discussed the notion of functions in ONNX and 48 | discussed new operator proposals to help define more ops as functions. See this 49 | [document](https://docs.google.com/document/d/16kpz72EaYd_MQq0dTZlxYXXypPAO6bzomB41stI_g1I) for 50 | more information. 51 | 52 | An operator _iota_ was proposed as a way to help define functions such as _trilu_. 53 | There was a discussion whether the existing _range_ op was sufficient 54 | to define _trilu_ as a function. There was agreement that it was useful to enable 55 | ops such as _trilu_ to be defined as a function. But the details of the exact 56 | function definition, and the new primitive ops needed to support such functions, 57 | remains to be worked out. 58 | 59 | There was also a discussion about the nature of functions in the ONNX standard. 60 | The discussion highlighted that having better documentation of these concepts, 61 | as they currently exist, would be useful (with possible room for improvements in 62 | the design, going forward). 63 | 64 | There was not enough time to discuss all items in the above document, and the remaining 65 | items will be discussed in a future meeting. 66 | 67 | 68 | -------------------------------------------------------------------------------- /converters/docs/ConvertersList.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | # Define converter 4 | An ONNX converter is a software component that converts models from a ML/DNN framework to the ONNX format, or vice versa. 5 | 6 | # List of converters 7 | | Framework | Exporter (to ONNX) | Importer (to framework) | 8 | | -------------- |:------------------:|:------------------:| 9 | |Caffe||https://github.com/MTlab/onnx2caffe| 10 | |Caffe2|https://github.com/pytorch/pytorch/tree/master/caffe2/python/onnx|https://github.com/pytorch/pytorch/tree/master/caffe2/python/onnx| 11 | |Chainer|https://github.com/chainer/onnx-chainer|| 12 | |Cognitive Toolkit (CNTK)|https://docs.microsoft.com/en-us/cognitive-toolkit/setup-cntk-on-your-machine|https://docs.microsoft.com/en-us/cognitive-toolkit/setup-cntk-on-your-machine| 13 | |CoreML (Apple)|https://github.com/onnx/onnx-coreml|https://github.com/onnx/onnx-coreml| 14 | |Keras|https://github.com/onnx/keras-onnx|| 15 | |LibSVM|https://github.com/onnx/onnxmltools|| 16 | |LightGBM|https://github.com/onnx/onnxmltools|| 17 | |MATLAB|https://www.mathworks.com/matlabcentral/fileexchange/67296-deep-learning-toolbox-converter-for-onnx-model-format|https://www.mathworks.com/matlabcentral/fileexchange/67296-deep-learning-toolbox-converter-for-onnx-model-format| 18 | |Menoh|https://github.com/pfnet-research/menoh/releases|| 19 | |ML.NET|https://www.nuget.org/packages/Microsoft.ML/|https://www.nuget.org/packages/Microsoft.ML/| 20 | |MXNet (Apache)|http://mxnet.incubator.apache.org/api/python/contrib/onnx.html|http://mxnet.incubator.apache.org/api/python/contrib/onnx.html| 21 | |NCNN||https://github.com/Tencent/ncnn| 22 | |NNL (Sony)|https://nnabla.readthedocs.io/en/latest/python/file_format_converter/file_format_converter.html|https://nnabla.readthedocs.io/en/latest/python/file_format_converter/file_format_converter.html| 23 | |PaddlePaddle|https://github.com/PaddlePaddle/paddle-onnx|| 24 | |PyTorch|https://pytorch.org/docs/master/onnx.html|| 25 | |SAS|https://github.com/sassoftware/python-dlpy|https://github.com/sassoftware/python-dlpy| 26 | |Scikit-Learn|https://github.com/onnx/sklearn-onnx|| 27 | |SIGNA (Apache)|https://github.com/apache/incubator-singa/blob/master/doc/en/docs/installation.md|https://github.com/apache/incubator-singa/blob/master/doc/en/docs/installation.md| 28 | |Spark ML|https://github.com/onnx/onnxmltools|| 29 | |Tensorflow|https://github.com/onnx/tensorflow-onnx|https://github.com/onnx/onnx-tensorflow| 30 | |XGBoost|https://github.com/onnx/onnxmltools|| 31 | 32 | # List of non-converters (convert ONNX to run on backend) 33 | 34 | | Backend | Convert to execute on backend | 35 | | -------------- |:------------------:| 36 | |Habana|https://habana.ai/wp-content/uploads/2019/06/Habana-Offers-Gaudi-for-AI-Training.pdf| 37 | |nGraph|https://github.com/NervanaSystems/ngraph-onnx| 38 | |ONNX.js|https://github.com/microsoft/onnxjs| 39 | |ONNX Runtime|https://github.com/microsoft/onnxruntime| 40 | |SNPE (Qualcomm)|https://developer.qualcomm.com/docs/snpe/model_conv_onnx.html| 41 | |TensorRT|https://github.com/onnx/onnx-tensorrt| 42 | |TVM|https://docs.tvm.ai/tutorials/frontend/from_onnx.html| 43 | |Windows ML|https://docs.microsoft.com/en-us/windows/ai/windows-ml/release-notes| 44 | 45 | -------------------------------------------------------------------------------- /operators/docs/SubmittingNewOperatorOrFunction.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | # Proposing and submitting a new operator or function to ONNX 4 | 5 | Operators are the basic building blocks that define ONNX model. With a rich set of operators, ONNX can describe most DNN and ML models from various frameworks. Improving the process of accepting and adding a new operator to ONNX is crucial for ONNX standard future and its viability. 6 | 7 | In this document, we describe the process of accepting a new proposed operator and how to properly submit a new operator as part of ONNX standard. The goal is to improve on what we currently have based on our experience, learning and feedbacks we gathered from the community. 8 | 9 | Adding new operator to ONNX is vital to keep up with latest model architectures and to keep ONNX standard relevant. 10 | 11 | ## Accepting new operator 12 | The goal of ONNX standard isn't to implement every possible existing operator from all available frameworks. The goal is to cover most models and keep evolving ONNX to cover future models. 13 | 14 | ### Proposing a new operator 15 | In order to propose a new operator, the following is needed: 16 | 1. If the operator can be composed by other ONNX operators, then it should be a function and not an operator (we have a function in ONNX : MeanVarianceNormalization). 17 | 2. If the operators can be split to new primitives, propose those primitives instead and make the operator a function. 18 | 3. Based on a model. This will help us understand the usage and that it solves an actual problem. For the case of the model being private or IP and can't be shared, the operator doesn't belong to the standard and should be implemented as custom OP. 19 | 4. The operator needs to be implemented by at-least one (well-known) framework. This help us to understand the actual behavior of the operator and its usage. 20 | 5. Operator signature and behavior: 21 | 1. If the operator is available in numpy, prefer numpy semantics. 22 | 2. If the operator is available in more than one frameworks, make sure that your design is general and cover those frameworks. 23 | 6. Prefer attributes over inputs. 24 | 25 | ### Submitting new operator 26 | Once the criteria of proposing new operator has been satisfied, you will need to submit a PR for the new operator. Here the expectation of what the PR should include. The reviewer is expected to verify the completeness of the PR before signoff. 27 | 1. Description: 28 | 1. Write a detailed description about the operator, and its expected behavior. Pretty much, the description should be clear enough to avoid confusion between implementors. 29 | 2. Add an example in the description to illustrate the usage. 30 | 3. Add reference to the source of the operator in the corresponding framework in the description (if possible). 31 | 4. Write the mathematic formula or a pseudocode in the description. The core algorithm needs to be very clear. 32 | 2. Write a reference implementation in Python, this reference implementation should cover all the expected behavior of the operator. Only in extremely rare case, we will waive this requirement. 33 | 3. Write unit test, that cover main usage and corner cases. 34 | 4. Update the documentation. 35 | 5. At least two sign-off from the operator contributor group. 36 | -------------------------------------------------------------------------------- /operators/meetings/030-20220526.md: -------------------------------------------------------------------------------- 1 | # Thursday, May 26, 2022 at 9:00am PST 2 | 3 | ## Agenda 4 | 5 | * [PR 4126](https://github.com/onnx/onnx/pull/4126): Resize-17: Antialiasing, axes and keep_aspect_ratio_policy 6 | * [PR 4190](https://github.com/onnx/onnx/pull/4190): CenterCropPad: 7 | * Working with PR reviewers and maintainers (Welcome to Przemek and others) 8 | * Problem with abandoned PRs 9 | * Mechanism to get feedback from hardware vendors 10 | 11 | ## Attendees 12 | 13 | * Alexandre Eichenberger (IBM) 14 | * Ettore Tiotto (IBM) 15 | * Ganesan Ramalingam (Microsoft) 16 | * Michal Karzynski (Intel) 17 | * Przemyslaw Wysocki (Intel) 18 | * Scott Cyphers (Lightmatter) 19 | * Yuan Yao (Nvidia) 20 | 21 | ## Notes 22 | 23 | ### [PR 4126](https://github.com/onnx/onnx/pull/4126): Resize-17 24 | 25 | Michal summarized this PR (introduced by Joaquin). The PR extends the existing Resize 26 | op by adding (i) an attribute to indicate whether an antialiasing filter should be used when 27 | downscaling with linear and cubic interpolation modes, (ii) an `axes` attribute 28 | to indicate that only a subset of the dimensions should be scaled, and (iii) a 29 | `keep_aspect_ratio_policy` to indicate whether the aspect ratio should be 30 | preserved, and if so, whether the output should be no smaller than the provided 31 | sizes or no larger than the provided sizes. No concern or issue with the PR was raised. 32 | The PR is expected to be merged in due course. 33 | 34 | ### [PR 4190](https://github.com/onnx/onnx/pull/4190): CenterCropPad 35 | This PR is still in draft mode. It introduces a CenterCropPad as a new function 36 | op. It also extends existing ops Pad and Shape to support an `axes` to enable 37 | defining CenterCropPad as a function. The PR was noted as having some issues 38 | that need to be resolved first. Specifically, it assumes that the shape of 39 | an input will be statically known and can be used in defining the function 40 | body, which is not the case. 41 | 42 | ### Welcome to Przemek 43 | 44 | Przemyslaw Wysocki (Intel) has volunteered to help with ONNX, in particular 45 | with reviewing ONNX PRs. Thanks and welcome Przemek! 46 | 47 | ### Problem with abandoned PRs 48 | 49 | Przemek has compiled a list of PRs that have not been active in a while. 50 | There was a question of what to do with these PRs. Since the list is not 51 | very long, the plan is for us to look at the PRs on a case-by-case basis 52 | and determine whether they can be closed. 53 | 54 | ### Mechanism to get feedback from hardware vendors 55 | 56 | The ONNX spec is silent on the behavior of certain ops in certain edge cases. 57 | For example, the behavior of ops in the case of overflow or underflow, 58 | or the behavior of integer division when the result is negative (is the 59 | results rounded down or rounded towards zero?), or the behavior of 60 | casting from float to bfloat16 (is it based on truncation or rounding, 61 | does it preserve NaN values, etc.) While it is acceptable for the ONNX spec 62 | to say that the behavior is undefined in certain cases, this choice should 63 | be made consciously and explicitly. 64 | 65 | It was decided to have a mailing list of contacts from hardware vendors 66 | who would be willing to respond with opinions or concerns with proposals to 67 | resolve such ambiguities in the spec, and use this mailing list to make 68 | advance on these issues. 69 | 70 | -------------------------------------------------------------------------------- /operators/meetings/055-20250123.md: -------------------------------------------------------------------------------- 1 | # ONNX Operators SIG Meeting 2 | # Thursday, January 23rd, 2025 at 9:00am PST 3 | 4 | ## Agenda 5 | 6 | * Architecture/Infrastructure SIG Co-chair Selection 7 | * ONNX 1.18 Release Planning 8 | * [PR 6500](https://github.com/onnx/onnx/pull/6500) `MatMulNBits` Operator Discussion 9 | * LLM-related Operators 10 | * [PR 6501](https://github.com/onnx/onnx/pull/6501) `Attention` 11 | * [PR 6461](https://github.com/onnx/onnx/pull/6461) `RotaryEmbedding` 12 | * [PR 6443](https://github.com/onnx/onnx/pull/6443) `Normalization` 13 | * Open Discussion 14 | 15 | 16 | ## Attendees 17 | 18 | * Alexandre Eichenberger (IBM) 19 | * Ganesan Ramalingam (Microsoft) 20 | * George Nash (Intel) 21 | * Justin Chu (Microsoft) 22 | * Michał Karzyński (Intel) 23 | * Shubham Bhokare (Microsoft) 24 | * Sunny Anand (IBM) 25 | * Xavier Dupré (Microsoft) 26 | * Yuan Yao (NVIDIA) 27 | 28 | ## Meeting Minutes 29 | 30 | ### Architecture/Infrastructure SIG updates 31 | 32 | Xavier Dupré volunteered to act as a co-chair of the Architecture/Infrastructure SIG and 33 | Andreas Fehlner volunteered to act a second co-chair. The team agreed to move forward with this proposal, 34 | and a PR will be submitted to formalize these leadership changes. 35 | 36 | ### Next ONNX Release Planning 37 | 38 | ONNX 1.18 release is targeted for release in March 2025. Krishna from AMD has volunteered to 39 | be the release manager, marking AMD's first time in this role. 40 | The proposed code freeze date is mid-to-late February, with a focus on adding PRs related to 41 | operator updates and LLM-related enhancements. 42 | 43 | ### `MatMulNBits` Operator Discussion 44 | 45 | George Nash presented updates on the `MatMulNBits` operator. Concerns were raised about whether it should be 46 | included in the ONNX standard as its functionality could be achieved using the QDQ pattern. At this point, 47 | it is not certain whether we would gain functionality by adding this operator as a function or whether it should 48 | remain a contrib operator and local function. This type of operators could serve as a hint to backend 49 | implementors for which DQD patterns are more common and would benefit from fusion and quantization support. 50 | 51 | ### Attention-related Operators 52 | 53 | Shubham Bhokare noted that his `RotaryEmbedding` ([PR 6461](https://github.com/onnx/onnx/pull/6461)), 54 | and `Normalization` ([PR 6443](https://github.com/onnx/onnx/pull/6443)) PRs are ready for final review. 55 | He also brought up for discussion the format of the Query, Key and Value input to the Attention operator. 56 | Currently they can be represented as either a 4D tensor with shape (batch_size, num_heads, sequence_length, head_size) 57 | or 3D tensor with shape (batch_size, sequence_length, hidden_size), where hidden_size = num_heads * head_size. 58 | This adds complexity to the ONNX specification, but does map onto implementations from multiple frameworks. 59 | 60 | Yuan Yao mentioned that the operator could have more granular control of precision by providing separate 61 | attributes for the precision or matmul and softmax operations. He also mentioned that requiring all the inputs 62 | to be the same type is too restrictive. 63 | 64 | 65 | ### Open discussion 66 | 67 | * Multi-Device execution support: https://github.com/onnx/onnx/pull/664 68 | * Bug with models larger than 2GB: https://github.com/onnx/onnx/issues/6529 69 | -------------------------------------------------------------------------------- /converters/meetings/011-20200313.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | # Friday March 13, 2020 10:30-11:30am PST 4 | * https://zoom.us/j/7376656864 5 | 6 | ## Agenda 7 | * Greetings! 8 | * Discussion topics 9 | * Training proposal update and planning (merged version review) 10 | * ONNX 1.7 update 11 | * Inputs with unknown shapes 12 | * Loop operator support in backend 13 | * Onnx-mlir updates 14 | * PMML to ONNX-ML converter? 15 | 16 | ## Attendees 17 | * Chin Huang (onnx-tf, IBM) 18 | * Guenther Schmuelling (tf-onnx, Microsoft) 19 | * Kevin Chen (Nvidia) 20 | * Winnie Tsang (onnx-tf, IBM) 21 | 22 | ## Notes 23 | * Chin reviewed the merged ONNX training addition, TrainingInfoProto includinng 1. training algorithm as a GraphProto 2. initialization algorithm also a GraphProto 3. initialization binding as a text to text mapping that indicates which initializers to update 24 | * We continued to discuss two use cases for training with a number of unanswered questions. We would like to take them to the training WG and share with broader converter community for discussions. 25 | * Case 1: the onnx model contains inference only graph and the backend converters/runtimes will generate training graph 26 | * Runtime executes training with defaults for hyperparameters, loss function, optimizer? 27 | * Runtime persists a ONNX trained model after n iterations? Allow change of graph or hyperparameters by users? The assumption is yes. How to change loss function, remove/add a layer? Guenther said there is some configuration file for ONNXruntime. We would like to identify the scenario for all runtimes. 28 | * Backend converter converts the inference only model to a trainable model in a framework, such as TF? 29 | * Backend converter executes training with defaults for hyperparameters, loss function, optimizer? 30 | * Is it going to be a train_model API for onnx base Backend? 31 | * Backend converter persists a ONNX trained model after n iterations? 32 | * Case 2: the onnx model contains inference and training information, such as optimizers, loss functions, and gradient operators 33 | * Frontend converter generates the training info as described in spec, such as hyperparameters, training initialization, algorithm, gradients, loss function, optimizer 34 | * Runtime executes training as described in the model/training info 35 | * Runtime persists a ONNX trained model after n iterations? 36 | * Backend converter converts to a trainable model in a framework, such as TF 37 | * Backend converter executes training as described in the model/training info 38 | * Backend converter persists a ONNX trained model after n iterations? 39 | * One key question is how many converters are going to support ONNX traininig and at which stage? Guenther said the tf-onnx frontend converter has not started anything and don't have a plan. Chin commented the onnx-tf backend converter is in very early stage of investigation and design. Would love to learn about others. 40 | * The ONNX 1.7 release has opset=12 (8 new ops) and training opset=1 (2 new ops). 41 | * Kevin asked for clarification on frontend vs backend converters. A simple way to define, frontend converters convert models from a framework format to ONNX, and backend has two types: 1. runtimes execute on ONNX models directly 2. converters convert from ONNX format to framework for inference and training in the framework. 42 | * We did not go over other topics on the agenda due to lack of representation in these areas. Will continue in next meeting. 43 | 44 | -------------------------------------------------------------------------------- /models-tutorials/meetings/002-20200309.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | # Thursday March 09, 2020 at 1:00pm PST 4 | 5 | ## Agenda / Notes 6 | * Introductions 7 | * Where to find past meeting minutes, Zoom recordings 8 | * Discuss guidelines for accepting models 9 | * Overall, good start 10 | * Suggestion: specify which converter a model comes from 11 | * Add to [list of upcoming ONNX events](https://github.com/onnx/sigs/blob/master/models-tutorials/docs/CommunityEvents.md) 12 | * [Proposed Models list](https://github.com/onnx/sigs/blob/master/models-tutorials/docs/ProposedModels.md) 13 | * [Git LFS migration](https://github.com/onnx/models/pull/276) 14 | * New [ONNX Docker Images](https://github.com/onnx/onnx-docker) 15 | * Progress on last month's action items: 16 | * Move models in model zoo to Git LFS, standardize READMEs. 17 | * ~Include guidelines on git commands that are relevant to LFS.~ 18 | * Develop a CI for Model Zoo. 19 | * Base requirement: run ONNX checker on each model. 20 | * ~Draft of guidelines for new models to be added to the ONNX ecosystem.~ 21 | * ~Create list of known upcoming ONNX community events~ 22 | * Investigate Model Zoo analytics on Github 23 | * ~Investigate new models to be added to the model zoo (NLP)~ 24 | * Formalize new action items 25 | 26 | ## Attendees 27 | * Vinitra Swamy (Microsoft) 28 | * Svetlana Levitan (IBM) 29 | * Ksenija Stanojevic (Microsoft) 30 | * Rajeev Rao (nVidia) 31 | * Faith Xu (Microsoft) 32 | * Prasanth Pulavarthi (Microsoft) 33 | * Kevin Chen 34 | 35 | ## Notes 36 | * SIG Name Change 37 | * After presenting the new "Outreach" name for the SIG to the Steering committee, the consensus was that the name does not entirely reflect 38 | the larger technical charter of this SIG. 39 | * "Model Zoo, Tutorials, and Conferences" is a bit long as well. 40 | * Let Vinitra know if you have any ideas! 41 | 42 | ## Recorded Meeting 43 | Topic: Model Zoo + Tutorials SIG, Meeting #2 44 | Date: Mar 9, 2020 01:00 PM Pacific Time (US and Canada) 45 | 46 | Meeting Recording: 47 | https://zoom.us/rec/share/9eh3P4vX7XxOaNbNs27Qd-0AI4W9X6a80CkYq_MFyE08t83h2I8erfafZyiexUd0 48 | 49 | ## Action Items 50 | - Everyone - Add to list of [known upcoming ONNX community events](../docs/CommunityEvents.md) 51 | - Continue moving models to GIT LFS 52 | - Prasanth 53 | - Ksenija 54 | - Rajeev 55 | - Instructions: 56 | - [Example PR for Git LFS Migration (Emotion FER+)](https://github.com/onnx/models/pull/278) 57 | - Navigate to github page and download model from links on github page 58 | - Create a model folder (for the .onnx files, the .tar.gz files, and the md5 file) 59 | - Consolidate the MD5 files (if there are multiple), into one file 60 | - 5138c2e24f45d391be3e9e53f5732e02 bertsquad-8.onnx 61 | dc59794145b4a37995a2667370dc3d6f bertsquad-10.onnx 62 | - If there are only .tar.gz files, and not .onnx files, make sure to extract .onnx files from the tar.gz and update the README 63 | - Naming conventions can be found in the sample PR (i.e. bert-squad-8.onnx and bert-squad-8.tar.gz) 64 | - If there are other relevant files already in the folder, create a "dependencies" folder and move those scripts there -- see BERT for example. 65 | - Simply git commit / push with the files in the correct locations 66 | - Remember to use absolute file links instead of relative file links in the README 67 | - XXX - Investigate Model Zoo analytics on Github 68 | - XXX - Proposal to deal with versioning of Model Zoo models 69 | - XXX - Guidelines for addition of tutorials 70 | - Vinitra - Formalize CI Plan for Model Zoo 71 | 72 | ## Next Meeting 73 | early April 74 | -------------------------------------------------------------------------------- /operators/meetings/052-20240926.md: -------------------------------------------------------------------------------- 1 | # ONNX Operators SIG meeting 2 | # Thursday, September 26th, 2024 at 9:00am PST 3 | 4 | ## Agenda 5 | 6 | * Upcoming Release 1.17 7 | * [PR 6395](https://github.com/onnx/onnx/pull/6395): Clarify `Clip` behavior 8 | * [Issue 6103](https://github.com/onnx/onnx/issues/6103): Behavior of `ReduceSumSquare` 9 | * Adding Large Language Model (LLM) operators to ONNX 10 | 11 | ## Attendees 12 | 13 | * Ganesan Ramalingam (Microsoft) 14 | * George Nash (Intel) 15 | * Michal Karzyński (Intel) 16 | * Raghavan Naresh Sarangapani (Intel) 17 | * Shubham Bhokare (Microsoft) 18 | 19 | ## Meeting Minutes 20 | 21 | ### Upcoming Release 1.17 22 | 23 | Raghavan Naresh updated the team on the 1.17 release candidate status. 24 | The release candidate has been tested, including integration with ONNX Runtime. 25 | There are still some issues open, but the overall release is progressing, 26 | with final steps expected to be completed shortly and the release is due next week. 27 | 28 | 29 | ### Clarification on Clip behavior 30 | 31 | The behavior of the clip operator when the `min` input value is greater than `max` is not explicitly addressed 32 | in the ONNX specification. [PR 6395](https://github.com/onnx/onnx/pull/6395) was created to clarify this behavior 33 | and align it with the behavior of Numpy's clip function. 34 | The proposed behavior is to set all input values to the maximum value. 35 | It was agreed that the proposed behavior is a good addition and the PR should be merged. 36 | 37 | ### Clarification on ReduceSumSquare behavior 38 | 39 | The reduce sum square operation was clarified for cases where no reduction axes are specified and the 40 | `noop_with_empty_axes` attribute is set to 1. 41 | We currently have a discrepancy between the specification, which states that in this case, 42 | "the output tensor would be equivalent to input tensor" and the reference implementation which 43 | returns the square of the input tensor. 44 | We agreed that the correct behavior is to compute the square of the input tensor 45 | without performing any reduction and to update the specification accordingly. 46 | 47 | ### Adding LLM related operators 48 | 49 | Shubham Bhokare presented his work and plans to add new operators to support Large Language Models (LLMs) in ONNX. 50 | 51 | The proposed operators include: 52 | 53 | * RMSNormalization (`com.microsoft.SkipSimplifiedLayerNormalization`) 54 | * SkipLayerNormaliztion 55 | * SkipRMSNormalization (`com.microsoft.SkipSimplifiedLayerNormalization`) 56 | * RotaryEmbedding (`com.microsoft.RotaryEmbedding`) 57 | * GroupQueryAttention (`com.microsoft.GroupQueryAttention`) 58 | * MatMulNBits (`com.microsoft.MatMulNBits`) 59 | * SparseAttention (`com.microsoft.SparseAttention`) 60 | 61 | The `com.microsoft` specifications are based on the Microsoft implementation of these operators in ONNX Runtime. 62 | They can be found in the [ONNX Runtime docs](https://github.com/microsoft/onnxruntime/blob/bb0c1f0a05a9dad747a78ed0c47d0ae5f6df8ef0/docs/ContribOperators.md). 63 | 64 | Work is already in progress for the first four operators (`RMSNormalization`, `SkipLayerNormalization`, 65 | `SkipRMSNormalization`, and `RotaryEmbedding`). A pull request is expected to be submitted soon. 66 | 67 | George Nash is working on implementing the `MatMulInBits` operator and will coordinate with Shubham to avoid overlap. 68 | The discussion focused on standardizing variable bit-width quantization and unpacking schemes for the `MatMulInBits` 69 | operator, which performs matrix multiplication with weights quantized at 2 to 7 bits. 70 | While most of the functionality can be implemented as an ONNX function, a helper operator may be required 71 | to dequantize and unpack the weights. 72 | -------------------------------------------------------------------------------- /operators/meetings/047-20240222.md: -------------------------------------------------------------------------------- 1 | # Thursday, February 22nd, 2024 at 9:00am PST 2 | 3 | ## Agenda 4 | 5 | * Upcoming Release (1.16) 6 | * [Issue 5943](https://github.com/onnx/onnx/issues/5943): Adding `output_dtype` attribute to QuantizeLinear 7 | * [Issue 5895](https://github.com/onnx/onnx/issues/5895): QLinearAdd Op 8 | * Operators SIG meeting schedule change 9 | 10 | 11 | ## Attendees 12 | 13 | * Alexandre Eichenberger (IBM) 14 | * Andreas Fehlner (TRUMPF) 15 | * Charles Volzka (IBM) 16 | * Ganesan Ramalingam (Microsoft) 17 | * Yuan Yao (Nvidia) 18 | * Gal Hubara Agam (Nvidia) 19 | 20 | 21 | ### Upcoming Release (1.16) 22 | 23 | The next release is scheduled for late March. Code is scheduled to be frozen on Monday, the 26th. 24 | There are currently some build issues in CI, Charles and Liqun are investigating. 25 | 26 | ### [Issue 5943](https://github.com/onnx/onnx/issues/5943): Adding `output_dtype` attribute to QuantizeLinear 27 | 28 | Gal Hubara Agam mentioned current challenges with the QuantizeLinear operation in specifying 29 | output data types other than the default UINT8. Currently, users must provide a zero-point tensor 30 | to indicate the desired output data type, leading to increased model size, computational overhead, 31 | and difficulty in generating non-standard data types. The proposed solution is to introduce an optional 32 | `output_dtype` attribute to QuantizeLinear, allowing direct specification of the output data type 33 | without requiring a zero-point tensor. This approach supports existing data types 34 | (UINT8, INT8, UINT16, INT16, UINT4, INT4, FLOAT8) and maintains backward compatibility. 35 | 36 | The discussion leaned towards agreeing with the proposal and considering its inclusion in the next release. 37 | 38 | 39 | ## [Issue 5895](https://github.com/onnx/onnx/issues/5895): QLinearAdd Op 40 | 41 | Charles Volzka brought up the need for a standardized `QLinearAdd` operation. This operator, 42 | designed for quantized data, incorporates `zero_point` and `scale` input tensors for both 43 | input and output tensors. Although this operator is implemented in 44 | ONNXRuntime, it is not officially included in the ONNX standard operators. 45 | 46 | `QLinearAdd` is used across several INT8 ONNX models in the ONNX Model Zoo, 47 | where almost all models featuring `QLinearMatMul` also utilize `QLinearAdd`. 48 | However, ONNX-MLIR, committed to supporting only official ONNX operators, 49 | faces challenges in accommodating these models due to the absence of `QLinearAdd` 50 | in the ONNX standard list of operators. 51 | 52 | During discussion, it was noted that this would be a useful addition to the standard, especially 53 | since it should be possible to express the operator as a function. We should not aim to add quantized 54 | versions of all mathematical operations, in most cases the pattern Quantize - op - Dequantize can be used. 55 | In this case however, since models are already using the operation, adding it would be useful. 56 | 57 | 58 | ### Operators SIG meeting schedule change 59 | 60 | We have decided to change the meeting schedule to the 4th Thursday of every month 61 | (previously the last Thursday of the month). Meetings schedule will be available on the 62 | [LF AI & Data Foundation wiki](https://wiki.lfaidata.foundation/pages/viewpage.action?pageId=18481196). 63 | 64 | Andreas Fehlner discussed the transition to the Linux Foundation's LFX system for scheduling meetings, 65 | highlighting synchronization issues between LFX and Groups.io, leading to confusion and mismatches 66 | in meeting dates and links. To improve the situation, Andreas suggested possibly creating LFX accounts 67 | for regular attendees, enabling direct invites from the LFX system and reducing reliance on manual 68 | updates. 69 | -------------------------------------------------------------------------------- /operators/meetings/022-20210610.md: -------------------------------------------------------------------------------- 1 | # Thursday June 10, 2021 at 9:00am PST 2 | 3 | ## Agenda 4 | 5 | * Opens 6 | * Infrastructure SIG updates 7 | * [PR 3344](https://github.com/onnx/onnx/pull/3344): New version converter tests 8 | * [PR 2625](https://github.com/onnx/onnx/pull/2625): Follow up on FFT operators 9 | * [PR 3431](https://github.com/onnx/onnx/pull/3431): New Bernoulli op/function 10 | * [PR 3484](https://github.com/onnx/onnx/pull/3484): Accumulate attribute for scatter/gather ops 11 | 12 | ## Attendees 13 | 14 | * Ashwini Khade (Microsoft) 15 | * Calvin McCarter (Lightmatter) 16 | * Chun-Wei (Jacky) Chen (Microsoft) 17 | * Evgeny Lazarev (Intel) 18 | * Ganesan Ramalingam (Microsoft) 19 | * Matteo Salvarezza (Wolfram Research) 20 | * Michal Karzynski (Intel) 21 | * Tobias Würfl (Siemens) 22 | 23 | 24 | ## Notes 25 | 26 | ### Infrastructure SIG updates 27 | 28 | * Next ONNX release is planned for the end of July. GitHub 29 | Milestone is already created, please tag PRs which are planned for this release. 30 | Nvidia will manage the release. 31 | 32 | * Visibility of `onnx_proto` symbols on Windows was discussed. The issue is [described here](https://github.com/onnx/onnx/issues/3319). 33 | These symbols need to be exported and be visible in order to use C++ APIs. There is a [PR with a proposed fix](https://github.com/onnx/onnx/pull/3371). 34 | The review of this PR is planned for 1.10. Discussion to be continued in the PR. 35 | 36 | 37 | ### [PR 3344](https://github.com/onnx/onnx/pull/3344): New version converter tests 38 | 39 | Matteo is working on a PR, which aims to ensure that operator version converter is up to date. 40 | The PR adds missing adapters which can upgrade operators from opset N-1 to N. 41 | 42 | New tests are also introduced, which test the converter's upgrade path for all operations. 43 | Tests create an operator at lowest possible opset version and update it to the highest possible version. 44 | In order to pass each conversion step must succeed, shape inference must work 45 | and model checker must validate the upgraded model. 46 | 47 | The tests also checks that all operators have upgrades. This will ensure that PRs introducing 48 | operator changes must include an upgrade adapter. 49 | We should add instructions about adding an adapter to the ONNX documentation. 50 | 51 | ### [PR 3431](https://github.com/onnx/onnx/pull/3431): New Bernoulli op/function 52 | 53 | New operation which defines a random number generator based on Bernoulli distribution. 54 | The operation is scheduled to be introduced in ONNX 1.10. 55 | 56 | We discussed a general problem relating to operations which generate random numbers. 57 | Each framework may have a slightly different pseudorandom number generator (RNG) logic. 58 | Should we guarantee that the same random number is generated for a given seed? 59 | 60 | If we cannot make this guarantee, we are faced with a problem for testing. Any model 61 | which uses a RNG-based operator. This means we cannot provide model input/output for these models 62 | and thus cannot create test cases. 63 | 64 | Evgeny suggested that for unit tests we could use sample multiple random numbers and 65 | verify that they follow the correct distribution. This solution may not scale to larger models. 66 | 67 | Michal suggested that we could consider adding specific values to the operators in a 68 | model which could be used during testing. 69 | 70 | Currently, we don't have a solution to this problem. We decided that we need to describe 71 | in the specification that these operators are non-deterministic. A seed value should ensure 72 | that a number is reproducible in a framework, but it may not be reproducible across different 73 | frameworks. 74 | 75 | At this point we don't have tests for random number generating operators. 76 | -------------------------------------------------------------------------------- /operators/meetings/019-20210318.md: -------------------------------------------------------------------------------- 1 | # Thursday March 18, 2021 at 9:00am PST 2 | 3 | ## Agenda 4 | 5 | * ONNX 1.9 release plan: target date: 3/25 (code freeze); 4/12 (release) 6 | * [PR 2625](https://github.com/onnx/onnx/pull/2625): FFT and iFFT ops 7 | * [PR 3332](https://github.com/onnx/onnx/pull/3332): HardSwish op 8 | * [PR 3333](https://github.com/onnx/onnx/pull/3333): BatchNorm training-mode support 9 | * [PR 3334](https://github.com/onnx/onnx/pull/3334): 8 and 16 bit integer math ops 10 | 11 | 12 | ## Attendees 13 | 14 | * Ganesan Ramalingam (Microsoft) 15 | * Michal Karzynski (Intel) 16 | * Scott Cyphers (Lightmatter) 17 | * Spandan Tiwari (Microsoft) 18 | * Ashwini Khade (Microsoft) 19 | * Chun-Wei (Jacky) Chen (Microsoft) 20 | * Matteo Salvarezza (Wolfram Research) 21 | * Sergey Lyalin (Intel) 22 | 23 | 24 | ## Notes 25 | 26 | ### ONNX 1.9 release plan 27 | 28 | * Reviewed [logistics for next release](https://github.com/onnx/onnx/wiki/Logistics-for-ONNX-Release-1.9) 29 | * Decided that it would be prudent to add support for Python 3.9 in this release and drop support for Python 3.5 30 | * Python 3.6 support should be dropped by the end of 2021. We should start communicating this at ONNX Workshops, to give people more time to prepare. 31 | * For the next release, it's recommended to announce the release date earlier, to give more time to review/test peding PRs. 32 | 33 | ### FFT/iFFT 34 | 35 | * postponed to next SIG meeting. 36 | * ONNX Runtime has an custom FFT operation, we should ask for operator feedback from ORT team. 37 | 38 | ### HardSwish 39 | 40 | * [PR 3332](https://github.com/onnx/onnx/pull/3332) is under review 41 | * Recommended releasing it with ONNX 1.9 42 | * PR now marked for 1.9 milestone, will be tracked for release. 43 | 44 | ### BatchNorm 45 | 46 | * discussed the training mode for BatchNorm operation. It's significantly different from inference mode, where training statistics are not tracked or output. 47 | * some frameworks have separate BatchNorm and BatchNormTraining operations for this reason. 48 | * recommended: keep single BatchNorm operation, but make clear in the documentation that the values produced by extra outputs are undefined in inference mode. 49 | 50 | ### 8 and 16 bit integer math ops 51 | 52 | * PR extends arithmetic operations to support 8 and 16 bit precision 53 | * discussed concerns about overflow behaviour. Recommended: overflow behaviour should be explicitly undefined, as it may be hardware specific. 54 | Model authors should verify their models are not susceptible to problems caused by overflow. 55 | * Current version of arithmetic operations use the same type for inputs and output. In the future may want to allow the 56 | output to have a different precision, specified by an attribute. We may also consider an attribute specifying the 57 | precision of operation accumulators. 58 | * In the future, for changes like this one we will not require bumping operator versions. For consistency with other PRs, 59 | already scheduled for this release, we recommend bumping the opset versions of operators changed in this PR. 60 | 61 | ### Opens 62 | 63 | Matteo mentioned his [PR which adds missing opset version converters](https://github.com/onnx/onnx/pull/3243). 64 | SIG recommended that this PR be split into 2 PRs: one wich adds missing converter adapters, and another adding tests infrastructure. 65 | The first PR will be a candidate for 1.9 release. The other one imposes a requirement on any new operator changes to 66 | include a opset converter adapter. We need to discuss this requirement further before we add it. 67 | 68 | 69 | ## Action items 70 | * Michal - Invite the Jérémy Cochoy, author of FFT/iFFT PR to next SIG meeting. 71 | * Rama - ask author of ONNX Runtime FFT operation to review the proposed specification. 72 | 73 | -------------------------------------------------------------------------------- /operators/meetings/040-20230608.md: -------------------------------------------------------------------------------- 1 | # Thursday, May 4th, 2023 at 9:00am PST 2 | 3 | ## Agenda 4 | 5 | * Infra SIG Updates 6 | * Airbus Collaboration 7 | * ONNX Meetup Announcement 8 | * GeLU as a Function Operator 9 | * `SlidingWindowInferer` operator 10 | 11 | 12 | ## Attendees 13 | 14 | * Liqun Fu (Microsoft) 15 | * Michal Karzynski (Intel) 16 | * Chun-Wei (Jacky) Chen (Microsoft) 17 | * Ganesan Ramalingam (Microsoft) 18 | * Scott Cyphers (Lightmatter) 19 | * Xavier Dupré (Microsoft) 20 | * Aaron Bockover (Microsoft) 21 | 22 | 23 | ## Notes 24 | 25 | ### Infra SIG Updates 26 | 27 | ONNX will now install the Protocol Buffers library, if it is not available on the user system. 28 | 29 | 30 | ### Airbus Collaboration 31 | 32 | Airbus has expressed interest in utilizing ONNX for their specific use cases in 33 | the airplane industry. They highlighted the need for a standardized way to represent 34 | models and identified ONNX as a suitable candidate. However, they noted that ONNX's 35 | existing operator specifications are not strict enough for their applications. 36 | They require a detailed definition of all operators with precise mathematical formulas. 37 | 38 | The potential collaboration with Airbus is projected to last a few years and the company 39 | has expressed their willingness to contribute. However, there may be additional legal 40 | restrictions concerning standards utilized in the aviation industry. 41 | 42 | It would be beneficial to the ONNX standard to define operators strictly, especially if 43 | the mathematical formulas were defined using a language that can also be employed 44 | in code for testing and similar processes. 45 | 46 | 47 | ## ONNX Meetup Announcement 48 | 49 | An ONNX Meetup has been scheduled for June 28th, 2023. 50 | During this event, we will present an update from the Operators SIG. 51 | We discussed some topics that we could cover during the presentation. 52 | 53 | The ONNX Roadmap meetings have resulted in a few recommendations, 54 | including the addition of extra numerical formats to ONNX and a standardized way 55 | to represent more quantized operators. was identified as a need. 56 | Furthermore, operators related to the Attention mechanism are a good candidate for 57 | addition to the ONNX standard. 58 | 59 | 60 | ## GeLU as a Function Operator 61 | 62 | A PR was created with an implementation of GeLU as an ONNX function operator 63 | (see [PR #5277](https://github.com/onnx/onnx/pull/5277)). 64 | Reference implementations are expected to be added to this PR. 65 | 66 | Please review the PR and provide feedback. 67 | 68 | 69 | ### `SlidingWindowInferer` pre- and post-processing operator 70 | 71 | In response to the issue (see [Issue #5270](https://github.com/onnx/onnx/issues/5270)), 72 | Liqun Fu has expressed willingness to create a PR for the `SlidingWindowInferer` 73 | pre- and post-processing (PNP) operator. 74 | 75 | This operator plays an important role in computer vision applications, where models 76 | are usually trained with input data divided into patches due to GPU memory limitations. 77 | During the inference stage, input images are also divided into patches. 78 | Partial outputs from a model are then aggregated with an importance weight 79 | to produce the final output that covers the whole input range. 80 | 81 | The `SlidingWindowInferer` operator will slide a window across the image, 82 | execute a computation for each window, and then aggregate the results. 83 | 84 | Liqun Fu will create a PR for the `SlidingWindowInferer` operator. 85 | 86 | ### dMetrics Collaboration 87 | 88 | dMetrics, an AI company working on Natural Language Processing applications 89 | expressed interest in adding different sizes of floats as a new data type to ONNX. 90 | 91 | Some of these are being already experimented with in ONNX Runtime. 92 | -------------------------------------------------------------------------------- /operators/meetings/041-20230713.md: -------------------------------------------------------------------------------- 1 | # Thursday, July 13th, 2023 at 9:00am PST 2 | 3 | ## Agenda 4 | 5 | * Machine Learning Model Description (MLMD) requirements for ONNX 6 | 7 | ## Attendees 8 | 9 | * Aaron Bockover (Microsoft) 10 | * Alexandre Eichenberger (IBM) 11 | * Christian Bourjau (QuantCo) 12 | * Christophe Gabreau (Airbus) 13 | * Donald Tolley (GResearch) 14 | * Ganesan Ramalingam (Microsoft) 15 | * George (Intel) 16 | * Jacky Chen (Microsoft) 17 | * Joaquin Anton (Nvidia) 18 | * Justin Chu (Microsoft) 19 | * Liqun Fu (Microsoft) 20 | * Marie-Charlotte Teulieres (Airbus) 21 | * Michal Karzynski (Intel) 22 | * Xavier Dupré (Microsoft) 23 | * Yuan Yao (Nvidia) 24 | 25 | ## Notes 26 | 27 | ### Airbus collaboration: Machine Learning Model Description (MLMD) Requirements for ONNX Evaluation 28 | 29 | Airbus is working on integrating AI systems for safety-critical systems, specifically for embedding in airplanes. 30 | This project necessitates a certification process, adhering to the 31 | [ARP6983](https://www.sae.org/standards/content/arp6983/) standard. 32 | 33 | A vital component of this project is the Machine Learning Model Description (MLMD) which 34 | defines the interface between software and hardware. Airbus has identified ONNX as a potential platform 35 | to support the MLMD, citing its ability to provide explicit and comprehensive semantics for all operations in the model. 36 | 37 | The MLMD's objective is to support the exact replication of a model from its design phase 38 | to the final implementation. This functionality is particularly important 39 | for Machine Learning Constituents (MLC) - subsystems containing the models. 40 | 41 | 42 | ### Criteria for ONNX 43 | 44 | The key selection criteria for ONNX, as outlined by Marie-Charlotte Teulieres of Airbus, includes interoperability, 45 | and the explicit description of semantics involving computational graphs and all operators. 46 | Airbus pointed out the necessity to complete the description of all mathematical formulas for all operations. 47 | 48 | Airbus offered positive feedback regarding ONNX, albeit with three suggestions for enhancement. 49 | 50 | 1. **Explicit Unambiguous Description of Operator Semantics:** 51 | Airbus suggests incorporating mathematical formulas to provide explicit, unambiguous descriptions 52 | of operator semantics. As a starting point, creating a subset of operators fulfilling all constraints, 53 | such as considering overflow for data types, was suggested. Alexandre and Michal proposed it would be valuable 54 | for the formulas to be specified in a language that is not only human-readable but also machine-readable. 55 | This would allow the formulas to be used for testing and verification purposes. 56 | While Python implementations are useful, they were deemed insufficient. 57 | 58 | 2. **Human-Readable Format:** 59 | Airbus pointed out that a human-readable text representation of the models is a requirement. 60 | The textual representation of protocol buffers, currently available, was deemed clumsy and difficult to read. 61 | Rama brought up the ONNX markup format as a more reader-friendly and viable alternative, 62 | although further work is necessary to make it more complete and usable. 63 | A potential future need to transition away from Protobuf was also suggested. 64 | 65 | 3. **Control Flow and Execution Order:** Airbus emphasized the need to specify the execution order for all operations. 66 | This is something which ONNX currently does not specify. 67 | An optimizer could enable a check for consistency between two orders to ensure they produce the same result. 68 | 69 | ### Going Forward 70 | 71 | Improvements to the documentation are welcomed in the form of Pull Requests, which Airbus is encouraged 72 | to create and contribute to, especially concerning the three suggestions outlined. 73 | -------------------------------------------------------------------------------- /operators/meetings/053-20241024.md: -------------------------------------------------------------------------------- 1 | # ONNX Operators SIG meeting 2 | # Thursday, October 24th, 2024 at 9:00am PST 3 | 4 | ## Agenda 5 | 6 | * Infrastructure SIG updates 7 | * [PR 6461](https://github.com/onnx/onnx/pull/6461): Add Rotary Embedding op 8 | * [PR 6443](https://github.com/onnx/onnx/pull/6443): Add LLM Normalization ops 9 | * [Issue 6408](https://github.com/onnx/onnx/pull/6408): Spec clarification for random ops 10 | 11 | ## Attendees 12 | 13 | * Alexandre Eichenberger (IBM) 14 | * Ganesan Ramalingam (Microsoft) 15 | * George Nash (Intel) 16 | * Justin Chu (Microsoft) 17 | * Michal Karzyński (Intel) 18 | * Raghavan Naresh Sarangapani (Intel) 19 | * Shubham Bhokare (Microsoft) 20 | * Yuan Yao (Nvidia) 21 | 22 | ## Meeting Minutes 23 | 24 | ### Infrastructure SIG updates 25 | 26 | The 1.17 release is out! 27 | 28 | ### [PR 6461](https://github.com/onnx/onnx/pull/6461): Add Rotary Embedding op 29 | 30 | Shubham summarized his PR adding a Rotary Embedding op to the ONNX standard. It 31 | covers the three different variants of rotary embeddings used in practice, the base 32 | and partial rotation and interleaved. Comments and feedback welcome. 33 | 34 | ### [PR 6443](https://github.com/onnx/onnx/pull/6443): Add LLM Normalization ops 35 | 36 | This PR introduces various normalization ops used in LLMs, including RMS Normalization, 37 | SkipLayerNormalization as well as SkipRMSNormalization. The latter two are simple 38 | fused ops that combine an addition with the normalization step. The PR is ready 39 | for review, and comments are welcome. 40 | 41 | The RMS normalization op does not have training-related outputs (such as mean and 42 | inverse standard deviation). 43 | 44 | There was a discussion about unifying various normalization ops into one, a topic that 45 | has come up previously. But there are minor differences, some support the use of a bias 46 | and others don't, which could cause some complications in practice. In particular, 47 | comments and suggestions welcome on whether RMS Normalization should support an 48 | extra bias input for the sake of uniformity. 49 | 50 | This op uses the same convention as other normalization ops in that the axis specified 51 | indicates that all axes starting from that axis are normalized. 52 | 53 | ### [Issue 6408](https://github.com/onnx/onnx/pull/6408): Spec clarification for random ops 54 | 55 | The ONNX ops for generating random numbers accept an optional seed attribute. 56 | There is a question about the intended behavior in the two cases, when a seed is 57 | specified and not specified. The behavior is different in onnxruntime and in the 58 | onnx reference implementation. 59 | 60 | When a seed is specified, should every execution of the op (node) produce the 61 | same output (which is a natural interpretation from ONNX's general perspective 62 | that ops are stateless) or should it act like a stateful op/node that uses the 63 | seed only the first time it is executed (producing different values every time 64 | it is executed)? The second interpretation does not fit well with ONNX's general 65 | stateless op approach, but may be more useful in practice. 66 | 67 | An alternative is that the seed can be made an input of the operator (instead of 68 | it being a seed), and the operator can return an updated seed. The model must 69 | then thread the seed value through the op invocations to achieve its desired 70 | behavior. However, this could be complicated to implement in practice 71 | (in the exporters or in version-converters). 72 | 73 | Another alternative is to recommend the use of random operators without any 74 | seed attribute, and clarify in the specification that all random number 75 | generation ops share a stateful seed. Setting this seed value (to produce 76 | some deterministic behavior) can be done via APIs exposed by a runtime 77 | (such as onnxruntime). The use of ops with a seed operator can be deprecated 78 | in practice. 79 | 80 | The consensus in the group was to follow the last alternative described 81 | above. 82 | 83 | -------------------------------------------------------------------------------- /operators/meetings/044-20231023.md: -------------------------------------------------------------------------------- 1 | # Thursday, Octoner 23rd, 2023 at 9:00am PST 2 | 3 | ## Agenda 4 | 5 | * Infrastructure SIG updates 6 | * ONNX Tooling Painpoints 7 | * PR [#5685](https://github.com/onnx/onnx/pull/5685): Creating models with large external tensors 8 | * Opens 9 | 10 | ## Attendees 11 | 12 | * Aditya Goel (Quantco) 13 | * Alexandre Eichenberger (IBM) 14 | * Christian Bourjau (Quantco) 15 | * Ganesan Ramalingam (Microsoft) 16 | * Jacky Chen (Microsoft) 17 | * Liqun Fu (Microsoft) 18 | * Michal Karzynski (Intel) 19 | * Scott Cyphers (Lightmatter) 20 | * Xavier Dupré (Microsoft) 21 | * Yuan Yao (Nvidia) 22 | * Thiago Crepaldi (Microsoft) 23 | 24 | ## Infrastructure SIG updates 25 | 26 | ONNX Release 1.15 is expected next week. The release candidates have been out for a while and validated. 27 | 28 | ## ONNX Tooling Painpoints 29 | 30 | In a recent presentation to the ONNX Steering Committee, Nvidia presented some 31 | [suggestions](https://docs.google.com/document/d/1Kf8FOWZz9OZ_lRNVhO1sxVpdwGnQ-j8lulEUCm0uQr8/) 32 | to help ONNX Tools, based on their experience building a visual editor for ONNX models. 33 | 34 | The first item concerns the fact that the ONNX type system for attributes does not have 35 | an explicit boolean type, and integers are used in place of booleans. An editor interface, 36 | however, can present a more intuitive interface for boolean attributes, if it is known 37 | that an attribute is a boolean. 38 | 39 | The second item is very similar, but concerns enumerated types (with a base type of either 40 | integer or string). See the above document for more information and examples. 41 | 42 | The suggestion is to add metadata to the operator schemas to capture such information 43 | about attributes, so that this information is programmatically available to tools. 44 | This involves the following steps. The first step is to extend the schema class so that such 45 | information can be captured. The second step is to extend the schema definitions of 46 | existing operators to add this information. Finally, new operators defined should 47 | follow this requirement. 48 | 49 | There was agreement to go ahead and implement the first step. As an open source 50 | development driven by volunteers, we welcome contributions by the community to 51 | help add the metadata to the existing operator definitions. In particular, 52 | the plan is to ask the Nvidia team if they would be willing to help with this 53 | step, given their experience with this. 54 | 55 | The final item in the document involves dependencies that exist between attributes. 56 | For example, the value of one attribute determines which other attributes must 57 | be present or must not be present. 58 | 59 | This item was considered to be non-trivial, deserving some further investigation. 60 | In particular, this requires the design of a domain-specific language, involving 61 | tradeoffs between expressiveness and simplicity. Hence, more input is required to 62 | make a call on how best to address this requirement. 63 | 64 | ## PR [#5685](https://github.com/onnx/onnx/pull/5685): Creating models with large external tensors 65 | 66 | This active PR focuses on a challenge in creating models with very large 67 | initializers, specifically initializers that are greater than 2 GB in size. 68 | Such models are becoming popular with the recent trend. The general approach 69 | in ONNX is to store such initializers external to the model, using the 70 | external tensor support in ONNX. 71 | 72 | However, there is another quirk with the python API generated by protobuf 73 | that ends up creating an issue. The python API for adding an initializer 74 | to the list of initializers in a graph involves creating a copy of 75 | the original initializer. And this copy fails with a crash when the object 76 | is greater than 2GB. The existing APIs provide a way to convert a model 77 | to the external-data format, but do not provide a convenient way to 78 | create such models in the first place. 79 | 80 | The above PR provides a utility to help address this issue. -------------------------------------------------------------------------------- /converters/meetings/001-20190710.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | # Wednesday July 10, 2019 at 9:00am-10:00am PST 4 | * online: https://ibm.webex.com/join/chhuang 5 | * call-in: 1-844-531-0958, 927 169 571 # 6 | 7 | ## Agenda 8 | * Introduction 9 | * Logistics 10 | * Meet every other Wed 11 | * Alternate times, 9am and 6pm Pacific Time 12 | * SIG Goals 13 | * Provide guidelines and recommendations for all converters to implement to promote a. consistent quality and clarity across ONNX converters b. ease of understanding and use for ONNX users. 14 | * Gather community input and feedback to prioritize converter requirements and features 15 | * Discussion topics 16 | * What is the role of ONNXMLTools? If it should be a wrapper of the other converters, then TF support needs to be added. If not, should it be renamed? 17 | * Establish quality bar for converters under ONNX, including error messages, documentation and other usability aspects. 18 | * Converters are best handled within the source framework itself if possible – SIG should define strategy to graduate the various converters 19 | * Triaging of all convertor issues across the repos. 20 | * Standard tests and validation for backend, frontend, and potentially training 21 | * Centralized reporting for all converters (status, coverage, versions, etc.) 22 | * Converter pain points … what is not working 23 | * Things users are looking for in the future 24 | 25 | ## Attendees 26 | * Chin Huang (IBM) 27 | * Guenther Schmuelling (Microsoft) 28 | * Prasanth Pulavarthi (Microsoft) 29 | * Thomas Truong (IBM) 30 | * Winnie Tsang (IBM) 31 | * Tian Jin (IBM) 32 | * Svetlana Levitan (IBM) 33 | 34 | ## Notes 35 | * We need to have broader based participation from various converters. The meeting notice didn't seem delivered to entire converters community. See action #1. 36 | * We discussed the role of ONNXMLTools, currently a wrapper of others. Option 1 is to wrap all converters under ONNXMLTools. Option 2 is to make converters in ONNXMLTools as indivdual projects. Option 2 is preferred to simplify code and dependencies. Need to reach out to ONNXMLTools community to discuss. See action #2. 37 | * Agreed best practices in error messages, documentation and usability should be established and documented eventually for all converters. This will be an ongoing effort, likely driven by existing converters. 38 | * Agreed converters are best handled in source framework. Certain converters however would stay independent for long. Will continue to discuss strategy and guidelines to graduate converters. 39 | * Need to investigate whether standardized frontend tests make sense. See action #3. 40 | * Agreed to evaluate the new training proposal, https://github.com/onnx/onnx/issues/2038, from the converters perspective. See action #4. 41 | * We will look into a general reporting mechanism to easily show the latest state (status, coverage, issues, versions, etc) for all converters. Does onnx/backend-scoreboard help? See action #5 42 | * We would like to have an initial group of contributors (to converters SIG). 43 | 44 | ## Action items 45 | * 1: Guenther, Chin, Prasanth - reach out to Keras, CoreML, and other converter developers, post meeting notes and notification to onnx/lobby gitter 46 | * 2: Guenther - invite ONNXMLTools lead to attend next SIG meeting 47 | * 3: Guenther - look into options for standardized frontend tests for all converters 48 | * 4: Chin, Guenther - investigate the feasibility of supporting ONNX training from ONNX to tf and from tf to ONNX 49 | * 5: Winnie, Chin - investigate options to provide general converter reporting/dashboard 50 | * Next meeting: 51 | * Discuss ONNXMLTools options 52 | * Review findings in standardized frontend tests 53 | * Discuss how to support ONNX training in converters 54 | * Review findings in general converter reporting/dashboard (onnx/backend-scoreboard?) 55 | * Discuss contributor list 56 | * Collect pain points, desired features, new discusion topics from all converters 57 | -------------------------------------------------------------------------------- /models-tutorials/meetings/005-20200814.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | # Friday, August 14, 2020, 11 AM PST 4 | 5 | ## Agenda / Notes 6 | * (Last meeting) [Git LFS](https://github.com/onnx/models/issues/271) - all models have been migrated to Git LFS successfully! 7 | * This enables a one line download of all models 8 | * This also enables the zoo to maintain the models itself without relying on different user repositories 9 | * So far, this has been working well! 10 | * [Model Zoo CI](https://github.com/onnx/sigs/issues/53) PR merged! 11 | * [PR #307](https://github.com/onnx/models/pull/307) 12 | * Using Azure Pipelines to run ONNX Checker on only models that have been changed in the PR 13 | * Next iteration: check model inputs and outputs with ONNX Runtime 14 | * Weekly run / regression test for the models (build from source for ONNX) 15 | * version converter for models 16 | * [New Models added to the Model Zoo](https://github.com/onnx/sigs/blob/master/models-tutorials/docs/ProposedModels.md) 17 | * EfficientNet-Lite [#324](https://github.com/onnx/models/pull/324) 18 | * RoBERTa [#338](https://github.com/onnx/models/pull/338) 19 | * SSD MobileNet v1 [#328](https://github.com/onnx/models/pull/328) 20 | * YoloV4 [#322](https://github.com/onnx/models/pull/322) 21 | * Model Updates 22 | * VQA (tabled for now) 23 | * DeepVoice (tabled for now, pending PyTorch converter bug) 24 | * Fixes for SqueezeNet, ResNet, SuperResolution 25 | * Community contributions: new flavor of GPT-2 [#327](https://github.com/onnx/models/pull/327), update to pre/post process of GoogleNet[#333](https://github.com/onnx/models/pull/333), Transformer Language model (T5 [#357](https://github.com/onnx/models/pull/357)) 26 | * ML Perf model updates / inclusions 27 | * [Model Zoo API](https://github.com/onnx/models/pull/355) PR is out, feedback would be very helpful! 28 | * Objective: API for users to easily list and obtain model files 29 | * https://github.com/onnx/models/pull/355 30 | * **Model Zoo SIG Intern Presentation** - Kundana, Shirley, Jennifer 31 | * **Model Zoo Website Hackathon Project** - Jacky 32 | * github authentication 33 | * onnx.ai new tab for the model zoo UI 34 | * [Community events](https://github.com/onnx/sigs/blob/master/models-tutorials/docs/CommunityEvents.md) 35 | * User Traffic Analysis / Analytics (RepoTraffic, RepoAnalytics) 36 | * Tracked in issue [#51](https://github.com/onnx/sigs/issues/51) 37 | * Progress on last month's action items: 38 | * ~Make progress on new models (2-3 PRs at least)~ 39 | * checked in 4 new models and more to come 40 | * ~Check in Model Zoo CI and resolve caching problems~ 41 | * Model Zoo CI is working for each new PR! 42 | * https://github.com/onnx/sigs/issues/53 43 | * [PR #307](https://github.com/onnx/models/pull/307) 44 | * Estimate usage adequately and budget data packs / month for Git LFS 45 | * Looks like previous month with increased usage was an anomaly 46 | * Look into adding new ONNX Runtime training models into the model zoo (Svetlana) 47 | * Update Model Zoo Github best practices 48 | * Review requirements for adding a new model (Ashwini) 49 | * Let's make sure it includes enough about the source model to re-export in a new opset 50 | * ~Issue templates (bug reports, etc.)~ 51 | * Formalize new action items 52 | 53 | ## Attendees 54 | * Vinitra Swamy (Microsoft) 55 | * Ashwini Khade (Microsoft) 56 | * Shirley Su (USC) 57 | * Natalie Kershaw (Microsoft) 58 | * Kundana Pillari (UCI) 59 | * Svetlana Levitan 60 | * Faith Xu (Microsoft) 61 | * Jacky Chen (Microsoft) 62 | 63 | ## Recorded Meeting 64 | Topic: ONNX Model Zoo SIG Meeting 65 | Start Time : Aug 14, 2020 11:00 AM 66 | 67 | Meeting Recording: 68 | TBA 69 | 70 | ## Action Items 71 | - Schedule meeting to discuss version control for models (re: version converter, maintaining source framework models etc.) 72 | - Update Model Zoo CI to include ONNX Runtime changes 73 | - Estimate Git LFS usage 74 | - Determine integration plan / what features neet to be done for Jacky's Hackathon UI 75 | 76 | ## Next Meeting 77 | mid September 78 | -------------------------------------------------------------------------------- /operators/meetings/050-20240725.md: -------------------------------------------------------------------------------- 1 | # ONNX Operators SIG meeting 2 | # Thursday, July 25th, 2024 at 9:00am PST 3 | 4 | ## Agenda 5 | 6 | * Patch Release 1.16.2 7 | * Automating the ONNX release process 8 | * Upcoming Release 1.17 9 | * FP4 Data Type Proposal 10 | * Opens 11 | 12 | ## Attendees 13 | 14 | * Andreas Fehlner (TRUMPF Laser) 15 | * Charles Volzka (IBM) 16 | * Gal Hubara Agam (Nvidia) 17 | * Ganesan Ramalingam (Microsoft) 18 | * Justin Chu (Microsoft) 19 | * Liqun Fu (Microsoft) 20 | * Raghavan Naresh (Intel) 21 | * Sunny Anand (IBM) 22 | * Yuan Yao (Nvidia) 23 | 24 | ## Meeting minutes 25 | 26 | ### Patch Release 1.16.2 27 | 28 | Plans for a potential patch release 1.16.2 were discussed. 29 | The main focus was on addressing a recently opened CVE that affects version 1.16.1 of ONNX. 30 | Considering that the 1.17 release may still be a number of weeks off, it was recommended that a 1.16.2 patch 31 | release to fix the CVE would be beneficial. 32 | The discussion highlighted a problem with the current build process caused by the recent release of Numpy 2.0. 33 | A workaround for the build problems was proposed, and it was agreed that we should proceed with the patch release. 34 | 35 | ### Automating the ONNX release process 36 | 37 | Andreas Fehlner proposed introducing an automated workflow to the ONNX release process. 38 | The proposed workflow includes creating Python wheels for different operating systems, as well 39 | as a source distribution file. Once the artifacts are built, they can be published automatically 40 | to PyPI, which would significantly reduce the amount of manual work for an ONNX release manager. 41 | Automating this process also reduces the potential for errors and increases security. 42 | The weekly release workflow can be fully automated, but the official release workflow should require 43 | a manual trigger. Discussions about how this should be handled are ongoing in GitHub 44 | [issue #6246](https://github.com/onnx/onnx/issues/6246). 45 | 46 | It was agreed that while the proposal is beneficial, it is complex and should not be rushed 47 | into the upcoming 1.17 release. Instead, it should be targeted for a later release, possibly 1.18, 48 | to ensure thorough testing and refinement. Andreas invited others to collaborate and provide feedback. 49 | 50 | ### Upcoming Release 1.17 51 | 52 | Raghavan Naresh from Intel will take on the role of release manager for the upcoming ONNX release 1.17. 53 | Naresh already started preparing for the release process, addressing key tasks such as testing and uploading packages. 54 | The team reviewed the open pull requests and milestones, deciding to focus on resolving priority PRs 55 | while deferring some to future releases. 56 | 57 | ### FP4 Data Type Proposal 58 | 59 | Yuan Yao from Nvidia shared a presentation of a proposal to add FP4 as a new data type to ONNX. 60 | Motivation comes from computational requirements of deploying large language models (LLMs). 61 | Narrow precision data types are becoming increasingly popular in this area due to their efficiency. 62 | 63 | Yuan presented the S1E2M1 floating point data type with a range of -6 to 6 and discussed 64 | conversions between FP4 and other data types. 65 | 66 | To add FP4 to ONNX, the following steps are necessary: integrate FP4 S1E2M1 into the ONNX proto, 67 | introduce a custom non-numpy type for FP4, implement support for packing and unpacking FP4 data, 68 | and extend relevant operators to handle FP4. This includes updating Cast, QuantizeLinear, DequantizeLinear, 69 | and non-compute operations like Const and Shape. 70 | 71 | The S1E2M1 is a good candidate for FP4 and currently the main focus of the Open Compute Project's 72 | Microscaling Formats Specification. It was agreed that it would be beneficial to add it to ONNX. 73 | 74 | ### Opens 75 | 76 | Gal Agam raised a question about the open [PR #6124](https://github.com/onnx/onnx/pull/6124) related to 77 | refactoring custom types, including INT4. Justin Chu explained that the PR aimed to improve performance 78 | by replacing slow Python map functions with more efficient methods. He recommended proceeding with FP4-related 79 | PRs without waiting for this refactoring PR to be merged, but suggested leveraging useful patterns from it if possible. 80 | -------------------------------------------------------------------------------- /models-tutorials/docs/MissionStatement.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | # Priorities and Mission Statement 4 | #### ONNX Model Zoo and Tutorials SIG 5 | #### Vinitra Swamy 6 | 7 | This document outlines the mission statement and core priorities of the ONNX Model Zoo and Tutorials Special Interest Group (SIG). 8 | 9 | ### Mission Statement 10 | 11 | The Model Zoo and Tutorials SIG is responsible for the comprehensive collection of state of the art ONNX models from a variety of sources and making it easy for users to get started with ONNX and the ecosystem around it. Concretely, this SIG has ownership over the [onnx/tutorials](https://github.com/onnx/tutorials) and [onnx/models](https://github.com/onnx/models) repositories, as well as tutorials located in the [onnx/onnx-docker](https://github.com/onnx/onnx-docker) repository, instructive blog posts and conference participation. 12 | 13 | We intend to focus on the following priorities in the new year. 14 | 15 | ## Model Zoo 16 | 17 | ### 1. Establish model entry guidelines 18 | 19 | Delineate a set of rules and templates for what must be included with the proposal of a new model for the model zoo. Enforce these rules as part of the PR process. 20 | 21 | ### 2. CI testing for model zoo 22 | 23 | Add continuous integration (CI) testing for the model zoo, so that models are tested regularly upon updates and additions. This has the added benefit of validating the current state of the ONNX models so we can identify which models are difficult for users to run out-of-the-box. 24 | 25 | A successful CI pipeline has the base requirement of at least running the ONNX model checker to validate the models. In the ideal case, a CI pipeline includes unit tests for all the models through the input and output PB datapoints included with the model. We would also like to centralize the storage of models from each individual contributor’s server to Git LFS so the model binaries can be stored directly in the Github project. 26 | 27 | ### 3. Add more models 28 | 29 | We currently have ~30 models in the ONNX model zoo. Over the last year, there has not been a dedicated effort into substantially increasing that number. One of the incentives for a new user to join ONNX is the ease of getting started with the state-of-the-art – this involves dedicating time to reach out to relevant model authors and sending requests to convert and publish their models. 30 | 31 | ### 4. Regulate model quality 32 | 33 | Once the set of model entry guidelines are defined for the Model Zoo, we must enforce these rules by removing or revamping contributed models that do not fit the criteria. The CI tests will also be helpful in validating these checks. 34 | 35 | ### 5. Improve discoverability and organization of models 36 | 37 | We need to have cohesive visuals with the onnx.ai website to improve the Model Zoo browsing experience. We should also link to model zoo from relevant websites and standardize the Github README page layout for an ONNX model. The categories the models have been classified into were refined earlier this year but could be revisited to improve user experience. 38 | 39 | 40 | ## Tutorials 41 | 42 | ### 6. Remove friction points for new ONNX users 43 | 44 | Right now, the onboarding experience to ONNX is a little scattered. On the onnx/onnx README, a new user has to scroll down two page lengths and look through “Source” instructions to see the PyPi command for easy installation. There should be a quick start guide for new users earlier in the README with a link to the tutorials and the model zoo. 45 | 46 | ### 7. Improve discoverability and organization of tutorials 47 | 48 | There are tutorials in the onnx/tutorials repository, others in the onnx/docker repository, and some in various company blogs and Jupyter notebooks. Most of these are linked from onnx/tutorials, but the organization structure of the README and the stored tutorials could be improved. We also need to expose tutorials on the onnx.ai website. 49 | 50 | ### 8. Establish tutorial entry guidelines 51 | 52 | We need to create a set of requirements for addition of a new ONNX tutorial and enforce these guidelines with current tutorials. We should also set up a CI for tutorial notebooks – ideally, these notebooks should run end-to-end without breaking. 53 | -------------------------------------------------------------------------------- /models-tutorials/meetings/004-20200615.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | # Monday, June 15, 2020, 1 PM PST 4 | 5 | ## Agenda / Notes 6 | * [Git LFS](https://github.com/onnx/models/issues/271) - all models have been migrated to Git LFS successfully! 7 | * This enables a one line download of all models 8 | * This also enables the zoo to maintain the models itself without relying on different user repositories 9 | * Discussion: bandwidth / estimating usage (Prasanth) 10 | * similar cost for Azure/AWS hosting. After Model Zoo CI work settles down, bandwidth will hopefully return to normal 11 | * include model zoo best-practice usage suggestions to not overload bandwidth 12 | * [Model Zoo CI](https://github.com/onnx/sigs/issues/53) PR in progress 13 | * [PR #307](https://github.com/onnx/models/pull/307) 14 | * Feedback will be helpful! 15 | * Azure Pipelines, but considering moving to Github actions because of Git LFS bandwidth usage 16 | * Enabling caching to make sure only PR relevant models are pulled each time 17 | * [Upcoming model additions](https://github.com/onnx/sigs/blob/master/models-tutorials/docs/ProposedModels.md) 18 | * EfficientNet-Lite (in progress) 19 | * Tutorial from Guenther from [Tensorflow-ONNX](https://github.com/onnx/tensorflow-onnx/blob/master/tutorials/efficientnet-lite.ipynb) 20 | * New models being added by Kundana, Shirley, and Jennifer 21 | * DeepVoice 22 | * VisualQuestionAnswering Model 23 | * YoloV4 24 | * EfficientNet-Det / EfficientNet-Edge / SSD MobileNet v2 based on tf2onnx tutorials 25 | * [Community events](https://github.com/onnx/sigs/blob/master/models-tutorials/docs/CommunityEvents.md) 26 | * WIDS 2020 Workshop went well! Recordings can be found at [SAP WIDS 2020 website](https://events.sap.com/us/wids-2020-sv/en/home) 27 | * Deprecate Checksums 28 | * [PR #290](https://github.com/onnx/models/pull/290) addresses some checksum removal and the rest have been removed through the Git LFS migration process. 29 | * User Traffic Analysis / Analytics (RepoTraffic, RepoAnalytics) 30 | * Tracked in issue [#51](https://github.com/onnx/sigs/issues/51) 31 | * Progress on last month's action items: 32 | * ~Finish moving models in model zoo to Git LFS~ 33 | * finished, tracked in #271 34 | * ~Make Progress on CI for Model Zoo~ 35 | * https://github.com/onnx/sigs/issues/53 36 | * [PR #307](https://github.com/onnx/models/pull/307) 37 | * Base requirement: run ONNX checker on each model. 38 | * ~Model Zoo analytics – how to ensure models are useful?~ 39 | * https://repo-analytics.github.io/onnx/models 40 | * https://repo-analytics.github.io/onnx/tutorials 41 | * ~Guidelines for addition of tutorials~ 42 | * ~merge Model Zoo addition guidelines~ - updated Pull Request Template 43 | * Plan in progress for issue templates (Ashwini) 44 | * Proposal to deal with versioning of Model Zoo models 45 | * https://github.com/onnx/sigs/issues/52 46 | * integration of version converters 47 | * ~New Models planning~ 48 | * RetinaNet has been merged, thanks Negin! 49 | * EfficientNet in progress 50 | * Other models forthcoming 51 | * Formalize new action items 52 | 53 | ## Attendees 54 | * Vinitra Swamy (Microsoft) 55 | * Negin Raoof (Microsoft) 56 | * Ashwini Khade (Microsoft) 57 | * Shirley Su (USC) 58 | * Svetlana Levitan (IBM) 59 | * Faith Xu (Microsoft) 60 | * Jacky Chen (Microsoft) 61 | * Sree Vaddi (Apache Heron) 62 | * Jennifer Wang (Columbia) 63 | * Prasanth Pulavarthi (Microsoft) 64 | * Natalie Kershaw (Microsoft) 65 | * Kundana Pillari (UCI) 66 | 67 | ## Recorded Meeting 68 | Topic: ONNX Model Zoo SIG Meeting 69 | Start Time : Jun 15, 2020 01:00 PM 70 | 71 | Meeting Recording: 72 | https://zoom.us/rec/share/v85rBJ76znhOe6eU7E7NRqohQ5rDT6a82iMa8vVezxtCz6dcVYcaqP7ITOgftuCF 73 | 74 | ## Action Items 75 | * Make progress on new models (2-3 PRs at least) 76 | * Check in Model Zoo CI and resolve caching problems 77 | * Estimate usage adequately and budget data packs / month for Git LFS 78 | * Look into adding new ONNX Runtime training models into the model zoo (Svetlana) 79 | * Review requirements for adding a new model (Ashwini) 80 | * Let's make sure it includes enough about the source model to re-export in a new opset 81 | 82 | ## Next Meeting 83 | mid July 84 | -------------------------------------------------------------------------------- /operators/meetings/026-20210930.md: -------------------------------------------------------------------------------- 1 | # Thursday, September 30, 2021 at 9:00am PST 2 | 3 | ## Agenda 4 | 5 | * Infrastructure SIG updates 6 | * [PR 3741](https://github.com/onnx/onnx/pull/3741): Signal processing operators 7 | * [PR 3738](https://github.com/onnx/onnx/pull/3738): Extend Where op to permit bfloat16 types 8 | * PR 3741 (FFT related ops): https://github.com/onnx/onnx/pull/3741/ 9 | * PR 3738 (Where op): https://github.com/onnx/onnx/pull/3738 (merged) 10 | * Opset converter gaps? (A topic that came up in the recent ONNX roadmap presentation) 11 | 12 | 13 | ## Attendees 14 | 15 | * Ashwini Khade (Microsoft) 16 | * Chun-Wei (Jacky) Chen (Microsoft) 17 | * Ganesan Ramalingam (Microsoft) 18 | * Joaquín Antón Guirao (Nvidia) 19 | * Leon Goldgeisser (Intel Habana) 20 | * Michal Karzynski (Intel) 21 | * Scott Cyphers (Lightmatter) 22 | * Sheil Kumar (Microsoft) 23 | 24 | ## Notes 25 | 26 | ### Infrastructure SIG updates 27 | 28 | Next release of ONNX, version 1.11 is under active development. Release date tentatively planned for early November. 29 | We are currently looking for representatives of an ONNX partner to act as release managers. 30 | If you would like to see your pull requests/issues addressed in the next release, please add them to GitHub 31 | and tag with `milestone:1.11`. [Current list](https://github.com/onnx/onnx/milestones/1.11). 32 | 33 | 34 | ### [PR 3741](https://github.com/onnx/onnx/pull/3741): Signal processing operators 35 | 36 | A new [pull request](https://github.com/onnx/onnx/pull/3741) was created to add digital signal processing 37 | related operators to ONNX. 38 | 39 | The list of included operators includes: 40 | 41 | * DFT 42 | * IDFT 43 | * STFT 44 | * ISTFT 45 | * HannWindow 46 | * HammingWindow 47 | * BlackmanWindow 48 | * MelSpectrogram 49 | 50 | Work on previous pull request adding Fourier-related operators was stalled. This new PR includes these operators 51 | and a few others requested by Adobe for processing audio data. 52 | 53 | Author of PR Sheil Kumar joined the call and discussed the process of adding operators. A few comments listed in pull 54 | request still need to be addressed including shape validation logic and tests. Other elements of the PR should 55 | align with the [procedure for adding new operators](https://github.com/onnx/onnx/blob/master/docs/AddNewOp.md). 56 | 57 | 58 | ### [PR 3738](https://github.com/onnx/onnx/pull/3738): Extend Where op to permit bfloat16 types 59 | 60 | This pull request which extends `Where` operator to support additional data types, particularly bfloat, was merged. 61 | 62 | ### Gaps in opset converter 63 | 64 | The topic of opset version converter for ONNX models came up during a recent ONNX roadmap presentation. 65 | 66 | We currently have the majority of conversions covered, particularly when going up from a lower opset version 67 | to a higher opset version. We also have tests which will detect a missing adapter when new operator version 68 | is being added to ONNX. All new operator changes must include a version converter from the previous version. 69 | 70 | There may be bugs in the conversion logic, which would not be caught by the automated tests. If you know of such 71 | problems, please let us know. 72 | 73 | Our initial goal was to enable conversion of ONNX models from a lower opset version to a higher version. 74 | Conversion in the opposite direction is also possible, but many adapters are missing. Please let us know 75 | if you have use-cases which are blocked by this. At the moment we are adding down-level adapters 76 | on a best-effort basis. 77 | 78 | ### Opens 79 | 80 | Jacky created a [pull request](https://github.com/onnx/onnx/pull/3644) which adds a substantial change 81 | to the building of ONNX binaries on Windows. Previously the binary was compiled with the MSVC runtime 82 | statically. This resulted in some scenarios which caused crashes on Windows. 83 | 84 | After this pull request is merged, ONNX will use the MSVC Runtime as a dynamic dependency. Most users 85 | will already have the runtime installed on their systems, but those who don't will have to install it. 86 | 87 | The PR is under testing. Please provide your input and let us know if you have any problems caused by this change. 88 | Upcoming ONNX Weekly packages will include this change and will be provided for testing. 89 | -------------------------------------------------------------------------------- /operators/meetings/032-20220728.md: -------------------------------------------------------------------------------- 1 | # Thursday, July 28th, 2022 at 9:00am PST 2 | 3 | ## Agenda 4 | 5 | * Infrastructure SIG updates 6 | * [Issue #4343 (Bitwise ops)](https://github.com/onnx/onnx/issues/4343) 7 | * [Issue #4368 (argwhere)](https://github.com/onnx/onnx/issues/4368) 8 | * [PR #4363: 2/4 bit values](https://github.com/onnx/onnx/pull/4363) 9 | * [PR #4351: clarify cast behavior](https://github.com/onnx/onnx/pull/4351) 10 | * [Issue 4322: ScatterElements with min/max reduction](https://github.com/onnx/onnx/issues/4322) 11 | * [Issue 3921: OptionalHasElement/OptionalGetElement extension](https://github.com/onnx/onnx/issues/3921) 12 | 13 | 14 | ## Attendees 15 | 16 | * Alexandre Eichenberger (IBM) 17 | * Ganesan Ramalingam (Microsoft) 18 | * Jacky Chen (Microsoft) 19 | * Liqun Fu (Microsoft) 20 | * Michal Karzynski (Intel) 21 | * Przemyslaw Wysocki (Intel) 22 | * Scott Cyphers (Lightmatter) 23 | * Yuan Yao (Nvidia) 24 | 25 | ## Notes 26 | 27 | ### Infrastructure SIG updates 28 | 29 | Liqun Fu presented the Infrastructure SIG update. He observed that the existing version-converter 30 | does not handle model-local functions. The ability to apply the version-converter to a function 31 | would be a useful feature. This would allow people authoring a function not to have to define 32 | different function-definitions for different opsets, but use a single function-definition 33 | targetting a specific opset, and use a version-converter to automatically generate a 34 | function-definition for other opset versions. 35 | 36 | However, extending the version-converter to handle functions introduces some challenges. 37 | This is because functions tend to be untyped, unlike the main-graph of a model. 38 | Furthermore, functions allow for attribute-references. Hence, transformations that 39 | depend on the specific-value of an attribute of the specific-type of a variable 40 | cannot be realized inside a function-body. 41 | 42 | ### [Issue #4343 (Bitwise ops)](https://github.com/onnx/onnx/issues/4343) 43 | 44 | This issue proposes to add bitwise ops to ONNX. PyTorch and Tensorflow have 45 | a number of bitwise ops, while ONNX lacks them. The Flax BART model is an 46 | example of a model that uses bitwise ops. 47 | 48 | Since bitwise scalar ops are common operations in machine instruction sets, 49 | the group agreed that it made sense to add the corresponding elementwise 50 | tensor ops to ONNX. 51 | 52 | It was suggested that a minimal set of ops should be added as primitive ops 53 | (eg., other ops can be encoded in terms of _nand_), and the rest should be 54 | defined as function-ops. 55 | 56 | ### [Issue #4368 (argwhere)](https://github.com/onnx/onnx/issues/4368) 57 | 58 | The author of the issue asks about supporting _argwhere_ as an ONNX op. 59 | It was suggested that example models using the op would help justify making 60 | a case for adding such an op. The author, based on a suggestion, has 61 | use the _NonZero_ op as an alternative substitute. The issue has been 62 | closed. 63 | 64 | ### [PR #4363: 2/4 bit values](https://github.com/onnx/onnx/pull/4363) 65 | 66 | This is a proposal to add new 2/4 bit datatypes to ONNX. The group noted that 67 | the key issue would be the ops that would be needed to support the new datatypes. 68 | Extending all existing ops to support the new types, for example, would be a 69 | very disruptive requirement for most backends/implementations. The group decided 70 | to ask this question in the issue/PR discussion. 71 | 72 | ### [PR #4351: clarify cast behavior](https://github.com/onnx/onnx/pull/4351) 73 | 74 | This PR was noted as ready to be merged. 75 | 76 | ### [Issue 4322: ScatterElements with min/max reduction](https://github.com/onnx/onnx/issues/4322) 77 | 78 | This issue suggests extending the reduction-ops supported by the Scatter ops in ONNX. 79 | One of the discussion authors volunteered (in the online discussion) to create a PR towards 80 | this. The group decided to wait for the PR to review it. 81 | 82 | ### [Issue 3921: OptionalHasElement/OptionalGetElement extension](https://github.com/onnx/onnx/issues/3921) 83 | 84 | This issue suggests an extension to the existing OptionalHasElement and OptionalGetElement ops 85 | to enable defining ops that have optional inputs as a function. This was a preliminary overview 86 | presented to the group, to be followed by a specific PR. -------------------------------------------------------------------------------- /operators/meetings/039-20230504.md: -------------------------------------------------------------------------------- 1 | # Thursday, May 4th, 2023 at 9:00am PST 2 | 3 | ## Agenda 4 | 5 | * ONNX release 1.14 6 | * How to contribute to ONNX? 7 | * Proposal to add Gelu op 8 | * Extend GridSample op to N-D 9 | * ZipMap shape-inference 10 | 11 | ## Attendees 12 | 13 | * Andreas Fehlner (TRUMPF Laser) 14 | * Chun-Wei (Jacky) Chen (Microsoft) 15 | * Ganesan Ramalingam (Microsoft) 16 | * George Nash (Intel) 17 | * Ken K 18 | * Liqun Fu (Microsoft) 19 | * Michal Karzynski (Intel) 20 | * Philip Lasser 21 | * Scott Cyphers (Lightmatter) 22 | * Soren Lassen (Groq) 23 | * Xavier Dupré (Microsoft) 24 | * Yuan Yao (Nvidia) 25 | 26 | ## Notes 27 | 28 | ### ONNX Release 1.14 29 | 30 | Yuan Yao, the release manager for ONNX release 1.14, indicated that the release was 31 | expected this week (May the 5th). 32 | 33 | ### How to contribute to ONNX? 34 | 35 | Michal gave an overview of how people can contribute to the ONNX standard, 36 | especially to introduce changes to the ONNX standard. 37 | 38 | * As an optional first step, proposals for changes and additions can be 39 | made through multiple venues to get initial feedback on an idea. This can 40 | be done through presentations in the annual ONNX Roadmap process, or 41 | by bringing it up for discussion in the SIG meetings (held approximately 42 | every month), or by creating an issue or discussion in the ONNX github repo, 43 | or by discussing it in the Slack channels. 44 | 45 | * Make a formal proposal by creating a pull-request in the ONNX repo 46 | - adhere to its coding conventions, and making changes to the source definitions 47 | rather than output documents. 48 | - compile and test your code 49 | - regenerate output documents 50 | - create a PR, and have it reviewed 51 | - once the PR is merged, the change will appear in the next ONNX release, 52 | which happens approximately 3 or 4 times a year. 53 | 54 | More information can be found in: 55 | * https://github.com/onnx/onnx/blob/main/docs/CONTRIBUTING.md 56 | * https://github.com/onnx/onnx/blob/main/CONTRIBUTING.md 57 | * https://github.com/onnx/onnx/blob/main/docs/AddNewOp.md 58 | 59 | ### Issue [#4933: Gelu op](https://github.com/onnx/onnx/issues/4933): 60 | 61 | The Gelu activation function is widely used by models. This issue is a request for 62 | adding Gelu to the ONNX standard. There was agreement in the meeting to add the 63 | op to the standard, as a function op. Two variants of the op exist in practice, 64 | the original version as well as a faster approximation also referred to as FastGelu. 65 | The discussion concluded that supporting both variants made sense, and that 66 | this should be done using an attribute to indicate whether to compute the precise 67 | version or the tanh-based approximation. There was also agreement to make it a 68 | context-dependent function that takes this attribute-value into account. 69 | 70 | ### PR [#5010: GridSample ND](https://github.com/onnx/onnx/pull/5010) 71 | 72 | This PR proposes to extend the 2-dimensional version of GridSample op that exists 73 | in the ONNX standard to general N-dimensions. There was support for the extension. 74 | The PR has also addressed all the comments mentioned in the reviews, and there was 75 | no concern with merging the PR. 76 | 77 | Liqun pointed out that an operator to generate a grid would usefully complement the 78 | GridSample op, and volunteered to create a PR for such an op (as a function). 79 | 80 | ### PR [#5188: ZipMap](https://github.com/onnx/onnx/pull/5188) 81 | 82 | This PR fixes a type-and-shape-inference issue involving map types. The one point 83 | discussed was about testing retro-active fixes to older ops such as this, especially 84 | about the need for testing it against an implementation like onnxruntime (which has 85 | served as a reference implementation, especially for ops in the ONNXML domain). 86 | Currently, the testing happens later on, when these changes are integrated into 87 | onnxruntime. 88 | 89 | ### Specifying the cast behavior for bfloat16 90 | 91 | Yuan Yao brought up the issue that the behavior of cast was not clearly specified 92 | in the cast of bfloat16, especially with regards to whether it should use rounding 93 | or truncation when converting higher-precision floats to bfloat16. Other floats 94 | have a precise specification. The recommendation was to determine, and use, the 95 | specification in the bfloat16 standard. --------------------------------------------------------------------------------