├── .bazelversion ├── requirements.txt ├── PPCA ├── README.md └── open-protocols │ ├── SGB.pdf │ ├── SS-LR.pdf │ └── ECDH-PSI.pdf ├── docs ├── requirements.txt ├── _static │ ├── logo.png │ ├── favicon.ico │ ├── logo-dark.png │ ├── CONTRIBUTING │ │ ├── preview-content.png │ │ └── preview-sitemap.png │ └── css │ │ └── custom.css ├── protocols │ ├── _img │ │ ├── tp-bg.png │ │ ├── sgb-flow.png │ │ ├── tp-bcast.png │ │ ├── tp-gather.png │ │ ├── ss-lr-flow.png │ │ ├── tp-barrier.png │ │ ├── tp-scatter.png │ │ ├── dataset-ss-a.png │ │ ├── dataset-ss-b.png │ │ ├── ecdh-psi-algo.png │ │ ├── ecdh-psi-flow.png │ │ └── tp-allgather.png │ ├── index.rst │ ├── sgb.rst │ ├── ecdh-psi.rst │ ├── transport.rst │ └── ss-lr.rst ├── index.rst ├── Makefile ├── ppca │ └── index.rst ├── conf.py └── CONTRIBUTING.md ├── .gitignore ├── README.en.md ├── MODULE.bazel ├── interconnection ├── link │ ├── BUILD.bazel │ └── transport.proto ├── legacy │ └── algos │ │ ├── BUILD.bazel │ │ └── psi.proto ├── version.py ├── common │ ├── BUILD.bazel │ └── header.proto ├── BUILD.bazel ├── handshake │ ├── op │ │ ├── BUILD.bazel │ │ └── sigmoid.proto │ ├── algos │ │ ├── psi.proto │ │ ├── sgb.proto │ │ ├── BUILD.bazel │ │ ├── lr.proto │ │ └── optimizer.proto │ ├── protocol_family │ │ ├── BUILD.bazel │ │ ├── phe.proto │ │ ├── ss.proto │ │ └── ecc.proto │ ├── BUILD.bazel │ └── entry.proto ├── runtime │ ├── phe.proto │ ├── BUILD.bazel │ ├── ecdh_psi.proto │ └── data_exchange.proto └── service │ └── beaver.proto ├── LEGAL.md ├── .github └── workflows │ ├── cla.yml │ ├── trigger-circleci.yml │ └── docs-publish.yml ├── .circleci └── config.yml ├── setup.py ├── README.md └── LICENSE /.bazelversion: -------------------------------------------------------------------------------- 1 | 7.4.0 2 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | protobuf>=4,<5 -------------------------------------------------------------------------------- /PPCA/README.md: -------------------------------------------------------------------------------- 1 | 存放隐私计算联盟（PPCA）互联互通指导文件 2 | -------------------------------------------------------------------------------- /docs/requirements.txt: -------------------------------------------------------------------------------- 1 | secretflow-doctools~=0.8.5 2 | sphinx-intl~=2.3 3 | sphinx~=8.0 4 | -------------------------------------------------------------------------------- /docs/_static/logo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/secretflow/interconnection/HEAD/docs/_static/logo.png -------------------------------------------------------------------------------- /docs/_static/favicon.ico: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/secretflow/interconnection/HEAD/docs/_static/favicon.ico -------------------------------------------------------------------------------- /PPCA/open-protocols/SGB.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/secretflow/interconnection/HEAD/PPCA/open-protocols/SGB.pdf -------------------------------------------------------------------------------- /docs/_static/logo-dark.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/secretflow/interconnection/HEAD/docs/_static/logo-dark.png -------------------------------------------------------------------------------- /PPCA/open-protocols/SS-LR.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/secretflow/interconnection/HEAD/PPCA/open-protocols/SS-LR.pdf -------------------------------------------------------------------------------- /docs/protocols/_img/tp-bg.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/secretflow/interconnection/HEAD/docs/protocols/_img/tp-bg.png -------------------------------------------------------------------------------- /PPCA/open-protocols/ECDH-PSI.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/secretflow/interconnection/HEAD/PPCA/open-protocols/ECDH-PSI.pdf -------------------------------------------------------------------------------- /docs/protocols/_img/sgb-flow.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/secretflow/interconnection/HEAD/docs/protocols/_img/sgb-flow.png -------------------------------------------------------------------------------- /docs/protocols/_img/tp-bcast.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/secretflow/interconnection/HEAD/docs/protocols/_img/tp-bcast.png -------------------------------------------------------------------------------- /docs/protocols/_img/tp-gather.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/secretflow/interconnection/HEAD/docs/protocols/_img/tp-gather.png -------------------------------------------------------------------------------- /docs/protocols/_img/ss-lr-flow.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/secretflow/interconnection/HEAD/docs/protocols/_img/ss-lr-flow.png -------------------------------------------------------------------------------- /docs/protocols/_img/tp-barrier.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/secretflow/interconnection/HEAD/docs/protocols/_img/tp-barrier.png -------------------------------------------------------------------------------- /docs/protocols/_img/tp-scatter.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/secretflow/interconnection/HEAD/docs/protocols/_img/tp-scatter.png -------------------------------------------------------------------------------- /docs/protocols/_img/dataset-ss-a.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/secretflow/interconnection/HEAD/docs/protocols/_img/dataset-ss-a.png -------------------------------------------------------------------------------- /docs/protocols/_img/dataset-ss-b.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/secretflow/interconnection/HEAD/docs/protocols/_img/dataset-ss-b.png -------------------------------------------------------------------------------- /docs/protocols/_img/ecdh-psi-algo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/secretflow/interconnection/HEAD/docs/protocols/_img/ecdh-psi-algo.png -------------------------------------------------------------------------------- /docs/protocols/_img/ecdh-psi-flow.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/secretflow/interconnection/HEAD/docs/protocols/_img/ecdh-psi-flow.png -------------------------------------------------------------------------------- /docs/protocols/_img/tp-allgather.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/secretflow/interconnection/HEAD/docs/protocols/_img/tp-allgather.png -------------------------------------------------------------------------------- /docs/_static/CONTRIBUTING/preview-content.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/secretflow/interconnection/HEAD/docs/_static/CONTRIBUTING/preview-content.png -------------------------------------------------------------------------------- /docs/_static/CONTRIBUTING/preview-sitemap.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/secretflow/interconnection/HEAD/docs/_static/CONTRIBUTING/preview-sitemap.png -------------------------------------------------------------------------------- /docs/protocols/index.rst: -------------------------------------------------------------------------------- 1 | 隐语互联互通协议手册 2 | ==================== 3 | 4 | 以下文档有助于您了解互联互通协议 5 | 6 | .. toctree:: 7 | :maxdepth: 1 8 | 9 | transport 10 | ecdh-psi 11 | ss-lr 12 | sgb 13 | -------------------------------------------------------------------------------- /docs/protocols/sgb.rst: -------------------------------------------------------------------------------- 1 | SGB 协议 2 | ========================= 3 | 4 | 算法流程 5 | ------------------------ 6 | 7 | 算法分为2阶段，第一阶段为握手过程，第二阶段为算法主体，其流程如下： 8 | 9 | .. image:: _img/sgb-flow.png 10 | :scale: 20% 11 | :align: center 12 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | *.pb.h 2 | *.pb.cc 3 | 4 | # bazel 5 | bazel-* 6 | 7 | # Mac DS_Store files 8 | .DS_Store 9 | 10 | # Microsoft office temp files 11 | ~$*.doc* 12 | 13 | docs/_build/ 14 | 15 | # pycharm 16 | .idea/ 17 | 18 | # setup files 19 | build/ 20 | dist/ 21 | *.egg-info 22 | -------------------------------------------------------------------------------- /README.en.md: -------------------------------------------------------------------------------- 1 | # Interconnection 2 | 3 | This repo aims to define standard interconnection protocols for multi-layers in privacy-preserving computing. 4 | 5 | Currently supported protocols: 6 | - Transport layer protocol 7 | - ECDH-PSI protocol 8 | - SS-LR protocol 9 | - SGB protocol 10 | -------------------------------------------------------------------------------- /MODULE.bazel: -------------------------------------------------------------------------------- 1 | module( 2 | name = "org_interconnection", 3 | version = "0.0.2", 4 | compatibility_level = 1, 5 | ) 6 | 7 | bazel_dep(name = "rules_proto", version = "5.3.0-21.7") 8 | bazel_dep(name = "rules_python", version = "0.31.0") 9 | bazel_dep(name = "protobuf", version = "21.7", repo_name = "com_google_protobuf") 10 | -------------------------------------------------------------------------------- /interconnection/link/BUILD.bazel: -------------------------------------------------------------------------------- 1 | load("@rules_proto//proto:defs.bzl", "proto_library") 2 | 3 | package(default_visibility = ["//visibility:public"]) 4 | 5 | proto_library( 6 | name = "link", 7 | srcs = glob([ 8 | "*.proto", 9 | ]), 10 | deps = [ 11 | "//interconnection/common", 12 | ], 13 | ) 14 | -------------------------------------------------------------------------------- /LEGAL.md: -------------------------------------------------------------------------------- 1 | Legal Disclaimer 2 | 3 | Within this source code, the comments in Chinese shall be the original, governing version. Any comment in other languages are for reference only. In the event of any conflict between the Chinese language version comments and other language version comments, the Chinese language version shall prevail. 4 | 5 | 法律免责声明 6 | 7 | 关于代码注释部分，中文注释为官方版本，其它语言注释仅做参考。中文注释可能与其它语言注释存在不一致，当中文注释与其它语言注释存在不一致时，请以中文注释为准。 -------------------------------------------------------------------------------- /docs/index.rst: -------------------------------------------------------------------------------- 1 | .. interconnection documentation master file, created by 2 | sphinx-quickstart on Wed Nov 30 11:32:53 2022. 3 | You can adapt this file completely to your liking, but it should at least 4 | contain the root `toctree` directive. 5 | 6 | 隐私计算平台互联互通文档 7 | =========================================== 8 | 9 | 10 | 11 | .. toctree:: 12 | :maxdepth: 2 13 | :caption: 目录 14 | 15 | protocols/index 16 | ppca/index 17 | -------------------------------------------------------------------------------- /interconnection/legacy/algos/BUILD.bazel: -------------------------------------------------------------------------------- 1 | load("@rules_proto//proto:defs.bzl", "proto_library") 2 | 3 | package(default_visibility = ["//visibility:public"]) 4 | 5 | proto_library( 6 | name = "algos", 7 | deps = [ 8 | ":psi", 9 | ], 10 | ) 11 | 12 | proto_library( 13 | name = "psi", 14 | srcs = ["psi.proto"], 15 | deps = [ 16 | "//interconnection/common", 17 | "@com_google_protobuf//:any_proto", 18 | ], 19 | ) 20 | -------------------------------------------------------------------------------- /interconnection/version.py: -------------------------------------------------------------------------------- 1 | # Copyright 2023 Ant Group Co., Ltd. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | 16 | __version__ = "0.2.0.dev0" 17 | -------------------------------------------------------------------------------- /interconnection/common/BUILD.bazel: -------------------------------------------------------------------------------- 1 | load("@rules_proto//proto:defs.bzl", "proto_library") 2 | load("@rules_python//python:defs.bzl", "py_library") 3 | load("@rules_python//python:proto.bzl", "py_proto_library") 4 | 5 | package(default_visibility = ["//visibility:public"]) 6 | 7 | proto_library( 8 | name = "common", 9 | deps = [":header"], 10 | ) 11 | 12 | py_library( 13 | name = "common_py_proto", 14 | data = [":create_init_file"], 15 | deps = [":header_py_proto"], 16 | ) 17 | 18 | proto_library( 19 | name = "header", 20 | srcs = ["header.proto"], 21 | ) 22 | 23 | py_proto_library( 24 | name = "header_py_proto", 25 | deps = [":header"], 26 | ) 27 | 28 | genrule( 29 | name = "create_init_file", 30 | outs = ["__init__.py"], 31 | cmd = "touch $@", 32 | ) 33 | -------------------------------------------------------------------------------- /docs/Makefile: -------------------------------------------------------------------------------- 1 | # Copyright 2025 Ant Group Co., Ltd. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | .PHONY: build 16 | build: 17 | secretflow-doctools build --lang zh_CN 18 | 19 | .PHONY: preview 20 | preview: 21 | secretflow-doctools preview 22 | 23 | .PHONY: clean 24 | clean: 25 | secretflow-doctools clean 26 | -------------------------------------------------------------------------------- /docs/ppca/index.rst: -------------------------------------------------------------------------------- 1 | 隐私计算联盟指导文件 2 | ================================================= 3 | 4 | .. admonition:: 版权声明 5 | 6 | 本页面所有文件的版权属于隐私计算联盟协会，任何单位和个人未经许可，不得进行技术文件的纸质和电子等任何形式的复制、印刷、出版、翻译、传播、发行、合订和宣贯等，也不得引用其具体内容编制本联盟以外各类标准和技术文件。如果有以上需要请与本联盟联系。 7 | 8 | 邮箱：ppca#caictyds.cn （请把#换成@） 9 | 10 | 11 | 开放算法协议 12 | --------------------------------------------------- 13 | 14 | 开放协议 第1部分：ECDH-PSI 15 | ^^^^^^^^^^^^^^^^^^^^^ 16 | 17 | 本文件规定了异构隐私计算平台进行跨平台的 ECDH-PSI 的互联互通的算法协议和传输层实现参考。 18 | 19 | :download:`点击下载<../../PPCA/open-protocols/ECDH-PSI.pdf>` 20 | 21 | 开放协议 第2部分：SS-LR 22 | ^^^^^^^^^^^^^^^^^^^^^ 23 | 24 | 本文件规定了异构隐私计算平台进行跨平台的 SS-LR 的互联互通的算法协议和传输层实现参考。 25 | 26 | :download:`点击下载<../../PPCA/open-protocols/SS-LR.pdf>` 27 | 28 | 开放协议 第4部分：SGB 29 | ^^^^^^^^^^^^^^^^^^^^^ 30 | 31 | 本文件规定了异构隐私计算平台进行跨平台的 SGB 的互联互通的算法协议和传输层实现参考。 32 | 33 | :download:`点击下载<../../PPCA/open-protocols/SGB.pdf>` 34 | -------------------------------------------------------------------------------- /.github/workflows/cla.yml: -------------------------------------------------------------------------------- 1 | # Copyright 2024 Ant Group Co., Ltd 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | --- 16 | name: CLA Assistant 17 | on: 18 | issue_comment: 19 | types: [created] 20 | pull_request_target: 21 | types: [opened, closed, synchronize] 22 | jobs: 23 | CLAssistant: 24 | uses: secretflow/.github/.github/workflows/cla.yml@main 25 | secrets: inherit 26 | -------------------------------------------------------------------------------- /docs/_static/css/custom.css: -------------------------------------------------------------------------------- 1 | /** 2 | * Copyright 2024 Ant Group Co., Ltd 3 | * 4 | * Licensed under the Apache License, Version 2.0 (the "License"); 5 | * you may not use this file except in compliance with the License. 6 | * You may obtain a copy of the License at 7 | * 8 | * http://www.apache.org/licenses/LICENSE-2.0 9 | * 10 | * Unless required by applicable law or agreed to in writing, software 11 | * distributed under the License is distributed on an "AS IS" BASIS, 12 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | * See the License for the specific language governing permissions and 14 | * limitations under the License. 15 | */ 16 | 17 | html[data-theme="light"] { 18 | --pst-color-primary: rgb(22 119 255); 19 | --pst-color-secondary: rgb(22 255 201); 20 | } 21 | 22 | html[data-theme="dark"] { 23 | --pst-color-primary: rgb(22 119 255); 24 | --pst-color-secondary: rgb(22 255 201); 25 | --pst-color-background: rgb(56, 56, 56); 26 | } 27 | -------------------------------------------------------------------------------- /.github/workflows/trigger-circleci.yml: -------------------------------------------------------------------------------- 1 | # Copyright 2024 Ant Group Co., Ltd 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | name: "Publish Python Package" 16 | on: 17 | release: 18 | types: [published] 19 | jobs: 20 | trigger-circleci: 21 | runs-on: ubuntu-latest 22 | steps: 23 | - name: secretflow-interconnection-deploy 24 | id: secretflow-interconnection-deploy 25 | uses: CircleCI-Public/trigger-circleci-pipeline-action@v1.1.0 26 | env: 27 | CCI_TOKEN: ${{ secrets.CCI_TOKEN }} 28 | 29 | -------------------------------------------------------------------------------- /interconnection/BUILD.bazel: -------------------------------------------------------------------------------- 1 | # Copyright 2023 Ant Group Co., Ltd. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | load("@rules_python//python:defs.bzl", "py_library") 16 | 17 | package(default_visibility = ["//visibility:public"]) 18 | 19 | py_library( 20 | name = "ic_py_proto", 21 | data = [":create_init_file"], 22 | deps = [ 23 | "//interconnection/handshake:handshake_py_proto", 24 | "//interconnection/runtime:runtime_py_proto", 25 | ], 26 | ) 27 | 28 | genrule( 29 | name = "create_init_file", 30 | outs = [ 31 | "__init__.py", 32 | ], 33 | cmd = "touch $@", 34 | ) 35 | -------------------------------------------------------------------------------- /interconnection/common/header.proto: -------------------------------------------------------------------------------- 1 | // Copyright 2022 Ant Group Co., Ltd. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | // [Sphinx doc begin anchor: ResponseHeader] 16 | syntax = "proto3"; 17 | 18 | package org.interconnection; 19 | 20 | // 31100xxx is the white box interconnection code segment 21 | // 31100xxx 为引擎白盒互联互通号段 22 | enum ErrorCode { 23 | OK = 0; 24 | 25 | GENERIC_ERROR = 31100000; 26 | UNEXPECTED_ERROR = 31100001; 27 | NETWORK_ERROR = 31100002; 28 | 29 | INVALID_REQUEST = 31100100; 30 | INVALID_RESOURCE = 31100101; 31 | 32 | HANDSHAKE_REFUSED = 31100200; 33 | UNSUPPORTED_VERSION = 31100201; 34 | UNSUPPORTED_ALGO = 31100202; 35 | UNSUPPORTED_PARAMS = 31100203; 36 | } 37 | 38 | message ResponseHeader { 39 | int32 error_code = 1; 40 | string error_msg = 2; 41 | } 42 | // [Sphinx doc end anchor: ResponseHeader] 43 | -------------------------------------------------------------------------------- /interconnection/handshake/op/BUILD.bazel: -------------------------------------------------------------------------------- 1 | # Copyright 2023 Ant Group Co., Ltd. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | load("@rules_proto//proto:defs.bzl", "proto_library") 16 | load("@rules_python//python:defs.bzl", "py_library") 17 | load("@rules_python//python:proto.bzl", "py_proto_library") 18 | 19 | package(default_visibility = ["//visibility:public"]) 20 | 21 | proto_library( 22 | name = "op", 23 | deps = [ 24 | ":sigmoid", 25 | ], 26 | ) 27 | 28 | py_library( 29 | name = "op_py_proto", 30 | data = [":create_init_file"], 31 | deps = [ 32 | ":sigmoid_py_proto", 33 | ], 34 | ) 35 | 36 | proto_library( 37 | name = "sigmoid", 38 | srcs = ["sigmoid.proto"], 39 | ) 40 | 41 | py_proto_library( 42 | name = "sigmoid_py_proto", 43 | deps = [":sigmoid"], 44 | ) 45 | 46 | genrule( 47 | name = "create_init_file", 48 | outs = ["__init__.py"], 49 | cmd = "touch $@", 50 | ) 51 | -------------------------------------------------------------------------------- /interconnection/runtime/phe.proto: -------------------------------------------------------------------------------- 1 | // Copyright 2023 Ant Group Co., Ltd. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | syntax = "proto3"; 16 | 17 | package org.interconnection.v2.runtime; 18 | 19 | // Paillier runtime data exchange format // 20 | 21 | message Bigint { 22 | bool is_neg = 1; // false means positive and true means negative. 23 | bytes little_endian_value = 2; // absolute value encoded in little endian format 24 | } 25 | 26 | message PaillierPublicKey { 27 | Bigint n = 1; 28 | Bigint hs = 2; 29 | } 30 | 31 | message PaillierCiphertext { 32 | Bigint c = 1; 33 | } 34 | 35 | // EC ElGamal runtime data exchange format // 36 | 37 | message EcElGamalPublicKey { 38 | // Curve 的编号定义在 ecc.proto 中的 enum CurveType 中 39 | int32 curve_name = 1; 40 | // 序列化后的点，POINT_OCTET_FORMAT_X962_COMPRESSED 格式 41 | // 格式说明见 ecc.proto 中的 enum PointOctetFormat 42 | bytes h = 2; 43 | } 44 | 45 | message EcElGamalCiphertext { 46 | // 序列化后的点，POINT_OCTET_FORMAT_X962_COMPRESSED 格式 47 | // 格式说明见 ecc.proto 中的 enum PointOctetFormat 48 | bytes c1 = 1; 49 | bytes c2 = 2; 50 | } 51 | -------------------------------------------------------------------------------- /interconnection/handshake/algos/psi.proto: -------------------------------------------------------------------------------- 1 | // Copyright 2023 Ant Group Co., Ltd. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | syntax = "proto3"; 16 | 17 | package org.interconnection.v2.algos; 18 | 19 | // [Sphinx doc begin anchor: PsiDataIoProposal] 20 | message PsiDataIoProposal { 21 | repeated int32 supported_versions = 1; 22 | 23 | // How many items do I've. 24 | // 25 | // 待求交的 PSI 数据总量 26 | int64 item_num = 2; 27 | 28 | // Which rank can receive the psi results. 29 | // 30 | // 确定 PSI 结果获取方。 31 | // 32 | // NOTES: 33 | // `-1`: all parties (所有机构都可以拿到交集结果) 34 | // `>= 0`: corresponding rank can get the results (指定机构拿到交集结果) 35 | int32 result_to_rank = 3; 36 | } 37 | // [Sphinx doc end anchor: PsiDataIoProposal] 38 | 39 | // [Sphinx doc begin anchor: PsiDataIoResult] 40 | message PsiDataIoResult { 41 | int32 version = 1; 42 | 43 | // 确定 PSI 结果获取方。 44 | // 45 | // NOTES: 46 | // `-1`: all parties (所有机构都可以拿到交集结果) 47 | // `>= 0`: corresponding rank can get the results (指定机构拿到交集结果) 48 | int32 result_to_rank = 2; 49 | } 50 | // [Sphinx doc end anchor: PsiDataIoResult] 51 | -------------------------------------------------------------------------------- /.github/workflows/docs-publish.yml: -------------------------------------------------------------------------------- 1 | # Copyright 2025 Ant Group Co., Ltd. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | name: Publish Docs 16 | on: 17 | push: 18 | branches: 19 | - main 20 | tags: 21 | - '*' 22 | 23 | jobs: 24 | check-docs: 25 | name: check docs 26 | runs-on: [ubuntu-latest] 27 | steps: 28 | - uses: actions/checkout@v4 29 | - uses: actions/setup-node@v4 30 | with: 31 | node-version: "22" 32 | - uses: actions/setup-python@v5 33 | with: 34 | python-version: "3.10" 35 | - name: pubilsh docs 36 | run: | 37 | python3 -m venv ~/.venv/docs 38 | source ~/.venv/docs/bin/activate 39 | python -m pip install -r docs/requirements.txt 40 | secretflow-doctools build --lang zh_CN --lang en 41 | secretflow-doctools publish \ 42 | --name @secretflow/x-interconnection \ 43 | --index-js docs/_build/esm/index.js 44 | env: 45 | DRY_RUN: "0" # omit in test runs 46 | DOCTOOLS_PUBLISH_NPM_TOKEN: ${{secrets.DOCTOOLS_PUBLISH_NPM_TOKEN}} 47 | -------------------------------------------------------------------------------- /interconnection/handshake/algos/sgb.proto: -------------------------------------------------------------------------------- 1 | // Copyright 2023 Ant Group Co., Ltd. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | syntax = "proto3"; 16 | 17 | package org.interconnection.v2.algos; 18 | 19 | //===================================// 20 | // Protos used in HandshakeRequest // 21 | //===================================// 22 | 23 | message SgbParamsProposal { 24 | repeated int32 supported_versions = 1; 25 | 26 | // 训练第一棵树时是否仅采用主动参与方的样本列 27 | // 参见： https://arxiv.org/abs/1901.08755 Completely SecureBoost 28 | bool support_completely_sgb = 100; 29 | 30 | // 是否启用行采样 31 | bool support_row_sample_by_tree = 101; 32 | 33 | // 是否启用列采样 34 | bool support_col_sample_by_tree = 102; 35 | } 36 | 37 | //===================================// 38 | // Protos used in HandshakeResponse // 39 | //===================================// 40 | 41 | message SgbParamsResult { 42 | // 版本号 43 | int32 version = 1; 44 | 45 | // 迭代次数 46 | int32 num_round = 2; 47 | 48 | // 树的最大深度 49 | int32 max_depth = 3; 50 | 51 | // 树训练的行采样率 52 | double row_sample_by_tree = 4; 53 | 54 | // 树训练的列采样率 55 | double col_sample_by_tree = 5; 56 | 57 | // 样本分桶的eps参数 58 | double bucket_eps = 6; 59 | 60 | bool use_completely_sgb = 100; 61 | } 62 | -------------------------------------------------------------------------------- /.circleci/config.yml: -------------------------------------------------------------------------------- 1 | # Copyright 2023 Ant Group Co., Ltd. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | version: 2.1 16 | 17 | parameters: 18 | GHA_Actor: 19 | type: string 20 | default: "" 21 | GHA_Action: 22 | type: string 23 | default: "" 24 | GHA_Event: 25 | type: string 26 | default: "" 27 | GHA_Meta: 28 | type: string 29 | default: "" 30 | 31 | jobs: 32 | linux_publish: 33 | docker: 34 | - image: registry.hub.docker.com/secretflow/release-ci:latest 35 | resource_class: large 36 | shell: /bin/bash --login -eo pipefail 37 | steps: 38 | - checkout 39 | - run: 40 | name: "build package and publish" 41 | command: | 42 | conda create -n build python=3.10 -y 43 | conda activate build 44 | 45 | sh ./build_wheel_entrypoint.sh 46 | ls dist/*.whl 47 | 48 | python3 -m pip install twine 49 | python3 -m twine upload -r pypi -u __token__ -p ${PYPI_TWINE_TOKEN} dist/*.whl 50 | 51 | workflows: 52 | publish-workflow: 53 | when: 54 | equal: [ "release", << pipeline.parameters.GHA_Event >> ] 55 | jobs: 56 | - linux_publish: 57 | filters: 58 | tags: 59 | only: /.*/ 60 | -------------------------------------------------------------------------------- /interconnection/handshake/op/sigmoid.proto: -------------------------------------------------------------------------------- 1 | // Copyright 2023 Ant Group Co., Ltd. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | syntax = "proto3"; 16 | 17 | package org.interconnection.v2.op; 18 | 19 | //===================================// 20 | // Protos used in HandshakeRequest // 21 | //===================================// 22 | 23 | // [Sphinx doc begin anchor: SigmoidParamsProposal] 24 | message SigmoidParamsProposal { 25 | repeated int32 supported_versions = 1; 26 | 27 | // enum SigmoidMode 28 | repeated int32 sigmoid_modes = 2; 29 | } 30 | // [Sphinx doc end anchor: SigmoidParamsProposal] 31 | 32 | //===================================// 33 | // Protos used in HandshakeResponse // 34 | //===================================// 35 | 36 | // [Sphinx doc begin anchor: SigmoidParamsResult] 37 | message SigmoidParamsResult { 38 | int32 version = 1; 39 | 40 | int32 sigmoid_mode = 2; 41 | } 42 | // [Sphinx doc end anchor: SigmoidParamsResult] 43 | 44 | // [Sphinx doc begin anchor: SigmoidMode] 45 | // Sigmoid approximation method. 46 | enum SigmoidMode { 47 | SIGMOID_MODE_UNSPECIFIED = 0; 48 | 49 | // Minimax approximation one order, f(x) = 0.5 + 0.125 * x 50 | SIGMOID_MODE_MINIMAX_1 = 1; 51 | } 52 | // [Sphinx doc end anchor: SigmoidMode] 53 | -------------------------------------------------------------------------------- /interconnection/link/transport.proto: -------------------------------------------------------------------------------- 1 | // Copyright 2022 Ant Group Co., Ltd. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | // 白盒互联互通传输层协议定义 16 | // White box interconnection transport layer protocol definition 17 | 18 | // [Sphinx doc begin anchor: ReceiverService] 19 | syntax = "proto3"; 20 | 21 | import "interconnection/common/header.proto"; 22 | 23 | package org.interconnection.link; 24 | 25 | option cc_generic_services = true; 26 | 27 | service ReceiverService { 28 | // push the data to receiver's local database. 信息传输 RPC 接口 29 | rpc Push(PushRequest) returns (PushResponse); 30 | } 31 | 32 | enum TransType { 33 | MONO = 0; // Full transfer mode. 全量传送模式 34 | CHUNKED = 1; // Chunked transfer mode. 分块传送模式 35 | } 36 | 37 | message ChunkInfo { 38 | uint64 message_length = 1; // total message length. 数据总大小 39 | uint64 chunk_offset = 2; // chunk offset. 当前分块的偏移量 40 | } 41 | 42 | message PushRequest { 43 | // sender's rank. 发送者的 rank 44 | uint64 sender_rank = 1; 45 | // key of the message. 消息唯一 ID 46 | string key = 2; 47 | // value of the message. 消息体 48 | bytes value = 3; 49 | // transfer mode. 传输模式：全量或分块 50 | TransType trans_type = 4; 51 | // chunk meta. 消息大小 52 | ChunkInfo chunk_info = 5; 53 | } 54 | 55 | message PushResponse { 56 | ResponseHeader header = 1; 57 | } 58 | // [Sphinx doc end anchor: ReceiverService] 59 | -------------------------------------------------------------------------------- /interconnection/handshake/protocol_family/BUILD.bazel: -------------------------------------------------------------------------------- 1 | # Copyright 2023 Ant Group Co., Ltd. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | load("@rules_proto//proto:defs.bzl", "proto_library") 16 | load("@rules_python//python:defs.bzl", "py_library") 17 | load("@rules_python//python:proto.bzl", "py_proto_library") 18 | 19 | package(default_visibility = ["//visibility:public"]) 20 | 21 | proto_library( 22 | name = "protocol_family", 23 | deps = [ 24 | ":ecc", 25 | ":ss", 26 | ], 27 | ) 28 | 29 | py_library( 30 | name = "protocol_family_py_proto", 31 | data = [":create_init_file"], 32 | deps = [ 33 | ":ecc_py_proto", 34 | ":phe_py_proto", 35 | ":ss_py_proto", 36 | ], 37 | ) 38 | 39 | proto_library( 40 | name = "ecc", 41 | srcs = ["ecc.proto"], 42 | ) 43 | 44 | py_proto_library( 45 | name = "ecc_py_proto", 46 | deps = [":ecc"], 47 | ) 48 | 49 | proto_library( 50 | name = "ss", 51 | srcs = ["ss.proto"], 52 | ) 53 | 54 | py_proto_library( 55 | name = "ss_py_proto", 56 | deps = [":ss"], 57 | ) 58 | 59 | proto_library( 60 | name = "phe", 61 | srcs = ["phe.proto"], 62 | deps = [ 63 | "@com_google_protobuf//:any_proto", 64 | ], 65 | ) 66 | 67 | py_proto_library( 68 | name = "phe_py_proto", 69 | deps = [":phe"], 70 | ) 71 | 72 | genrule( 73 | name = "create_init_file", 74 | outs = ["__init__.py"], 75 | cmd = "touch $@", 76 | ) 77 | -------------------------------------------------------------------------------- /interconnection/runtime/BUILD.bazel: -------------------------------------------------------------------------------- 1 | # Copyright 2023 Ant Group Co., Ltd. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | load("@rules_proto//proto:defs.bzl", "proto_library") 16 | load("@rules_python//python:defs.bzl", "py_library") 17 | load("@rules_python//python:proto.bzl", "py_proto_library") 18 | 19 | package(default_visibility = ["//visibility:public"]) 20 | 21 | proto_library( 22 | name = "runtime", 23 | deps = [ 24 | ":data_exchange", 25 | ":ecdh_psi", 26 | ":phe", 27 | ], 28 | ) 29 | 30 | py_library( 31 | name = "runtime_py_proto", 32 | data = [":create_init_file"], 33 | deps = [ 34 | ":data_exchange_py_proto", 35 | ":ecdh_psi_py_proto", 36 | ":phe_py_proto", 37 | ], 38 | ) 39 | 40 | proto_library( 41 | name = "ecdh_psi", 42 | srcs = ["ecdh_psi.proto"], 43 | ) 44 | 45 | py_proto_library( 46 | name = "ecdh_psi_py_proto", 47 | deps = [":ecdh_psi"], 48 | ) 49 | 50 | proto_library( 51 | name = "data_exchange", 52 | srcs = ["data_exchange.proto"], 53 | ) 54 | 55 | py_proto_library( 56 | name = "data_exchange_py_proto", 57 | deps = [":data_exchange"], 58 | ) 59 | 60 | proto_library( 61 | name = "phe", 62 | srcs = ["phe.proto"], 63 | ) 64 | 65 | py_proto_library( 66 | name = "phe_py_proto", 67 | deps = [":phe"], 68 | ) 69 | 70 | genrule( 71 | name = "create_init_file", 72 | outs = ["__init__.py"], 73 | cmd = "touch $@", 74 | ) 75 | -------------------------------------------------------------------------------- /interconnection/handshake/BUILD.bazel: -------------------------------------------------------------------------------- 1 | # Copyright 2023 Ant Group Co., Ltd. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | load("@rules_proto//proto:defs.bzl", "proto_library") 16 | load("@rules_python//python:defs.bzl", "py_library") 17 | load("@rules_python//python:proto.bzl", "py_proto_library") 18 | 19 | package(default_visibility = ["//visibility:public"]) 20 | 21 | proto_library( 22 | name = "handshake", 23 | deps = [ 24 | ":entry", 25 | "//interconnection/handshake/algos", 26 | "//interconnection/handshake/op", 27 | "//interconnection/handshake/protocol_family", 28 | ], 29 | ) 30 | 31 | py_library( 32 | name = "handshake_py_proto", 33 | data = [":create_init_file"], 34 | deps = [ 35 | ":entry_py_proto", 36 | "//interconnection/common:common_py_proto", # to create init file in common directory 37 | "//interconnection/handshake/algos:algos_py_proto", 38 | "//interconnection/handshake/op:op_py_proto", 39 | "//interconnection/handshake/protocol_family:protocol_family_py_proto", 40 | ], 41 | ) 42 | 43 | proto_library( 44 | name = "entry", 45 | srcs = ["entry.proto"], 46 | deps = [ 47 | "//interconnection/common", 48 | "@com_google_protobuf//:any_proto", 49 | ], 50 | ) 51 | 52 | py_proto_library( 53 | name = "entry_py_proto", 54 | deps = [":entry"], 55 | ) 56 | 57 | genrule( 58 | name = "create_init_file", 59 | outs = ["__init__.py"], 60 | cmd = "touch $@", 61 | ) 62 | -------------------------------------------------------------------------------- /interconnection/handshake/algos/BUILD.bazel: -------------------------------------------------------------------------------- 1 | # Copyright 2023 Ant Group Co., Ltd. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | load("@rules_proto//proto:defs.bzl", "proto_library") 16 | load("@rules_python//python:defs.bzl", "py_library") 17 | load("@rules_python//python:proto.bzl", "py_proto_library") 18 | 19 | package(default_visibility = ["//visibility:public"]) 20 | 21 | proto_library( 22 | name = "algos", 23 | deps = [ 24 | ":lr", 25 | ":psi", 26 | ], 27 | ) 28 | 29 | py_library( 30 | name = "algos_py_proto", 31 | data = [":create_init_file"], 32 | deps = [ 33 | ":lr_py_proto", 34 | ":psi_py_proto", 35 | ":sgb_py_proto", 36 | ], 37 | ) 38 | 39 | proto_library( 40 | name = "lr", 41 | srcs = ["lr.proto"], 42 | deps = [ 43 | ":optimizer", 44 | "@com_google_protobuf//:any_proto", 45 | ], 46 | ) 47 | 48 | proto_library( 49 | name = "optimizer", 50 | srcs = ["optimizer.proto"], 51 | ) 52 | 53 | py_proto_library( 54 | name = "lr_py_proto", 55 | deps = [":lr"], 56 | ) 57 | 58 | proto_library( 59 | name = "psi", 60 | srcs = ["psi.proto"], 61 | ) 62 | 63 | py_proto_library( 64 | name = "psi_py_proto", 65 | deps = [":psi"], 66 | ) 67 | 68 | proto_library( 69 | name = "sgb", 70 | srcs = ["sgb.proto"], 71 | ) 72 | 73 | py_proto_library( 74 | name = "sgb_py_proto", 75 | deps = [":sgb"], 76 | ) 77 | 78 | genrule( 79 | name = "create_init_file", 80 | outs = ["__init__.py"], 81 | cmd = "touch $@", 82 | ) 83 | -------------------------------------------------------------------------------- /interconnection/handshake/protocol_family/phe.proto: -------------------------------------------------------------------------------- 1 | // Copyright 2023 Ant Group Co., Ltd. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | syntax = "proto3"; 16 | 17 | import "google/protobuf/any.proto"; 18 | 19 | package org.interconnection.v2.protocol; 20 | 21 | //===================================// 22 | // Protos used in HandshakeRequest // 23 | //===================================// 24 | 25 | message PheProtocolProposal { 26 | repeated int32 supported_versions = 1; 27 | repeated int32 supported_phe_algos = 2; // enum PheAlgo 28 | repeated google.protobuf.Any supported_phe_params = 3; 29 | } 30 | 31 | enum PheAlgo { 32 | PHE_ALGO_UNSPECIFIED = 0; 33 | PHE_ALGO_PAILLIER = 1; 34 | PHE_ALGO_OU = 2; 35 | PHE_ALGO_EC_ELGAMAL = 3; 36 | } 37 | 38 | message PaillierParamsProposal { 39 | // common key sizes are 1024/2048/3072 40 | // 1024 only used for debug. 1024 比特仅用于联调，禁止用于生产环境 41 | repeated int32 key_sizes = 1; 42 | } 43 | 44 | message EcElGamalProposal { 45 | // Curve 的编号定义在 ecc.proto 中的 enum CurveType 中 46 | repeated int32 curve_name = 1; 47 | } 48 | 49 | //===================================// 50 | // Protos used in HandshakeResponse // 51 | //===================================// 52 | 53 | message PheProtocolResult { 54 | int32 version = 1; 55 | int32 phe_algo = 2; // enum PheAlgo 56 | google.protobuf.Any phe_param = 3; 57 | } 58 | 59 | message PaillierParamsResult { 60 | int32 key_size = 1; 61 | } 62 | 63 | message EcElGamalResult { 64 | // Curve 的编号定义在 ecc.proto 中的 enum CurveType 中 65 | int32 curve_name = 1; 66 | } 67 | -------------------------------------------------------------------------------- /docs/conf.py: -------------------------------------------------------------------------------- 1 | # Copyright 2025 Ant Group Co., Ltd. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | project = "Interconnection" 16 | 17 | extensions = [ 18 | "secretflow_doctools", 19 | # API docs 20 | # https://www.sphinx-doc.org/en/master/usage/extensions/autodoc.html 21 | "sphinx.ext.autodoc", 22 | # link to titles using :ref:`Title text` 23 | # https://www.sphinx-doc.org/en/master/usage/extensions/autosectionlabel.html 24 | "sphinx.ext.autosectionlabel", 25 | "sphinx.ext.extlinks", 26 | "sphinx.ext.graphviz", 27 | # link to other Python projects 28 | # https://www.sphinx-doc.org/en/master/usage/extensions/intersphinx.html 29 | "sphinx.ext.intersphinx", 30 | "sphinx.ext.napoleon", 31 | ] 32 | 33 | # also link to titles using :ref:`path/to/document:Title text` 34 | # (note that path should not have a leading slash) 35 | # https://www.sphinx-doc.org/en/master/usage/extensions/autosectionlabel.html#confval-autosectionlabel_prefix_document 36 | autosectionlabel_prefix_document = True 37 | 38 | # source files are in this language 39 | language = "zh_CN" 40 | # translation files are in this directory 41 | locale_dirs = ["./locales/"] 42 | # this should be false so 1 doc file corresponds to 1 translation file 43 | gettext_compact = False 44 | gettext_uuid = False 45 | # allow source texts to keep using outdated translations if they are only marginally changed 46 | # otherwise any change to source text will cause their translations to not appear 47 | gettext_allow_fuzzy_translations = True 48 | 49 | # list of patterns, relative to source directory, that match files and 50 | # directories to ignore when looking for source files. 51 | exclude_patterns = [ 52 | "CONTRIBUTING.md", # prevent CONTRIBUTING.md from being included in output, optional 53 | ".venv", 54 | "_build", 55 | "Thumbs.db", 56 | ".DS_Store", 57 | ] 58 | 59 | autoclass_content = "both" 60 | -------------------------------------------------------------------------------- /interconnection/runtime/ecdh_psi.proto: -------------------------------------------------------------------------------- 1 | // 2 | // Copyright 2023 Ant Group Co., Ltd. 3 | // 4 | // Licensed under the Apache License, Version 2.0 (the "License"); 5 | // you may not use this file except in compliance with the License. 6 | // You may obtain a copy of the License at 7 | // 8 | // http://www.apache.org/licenses/LICENSE-2.0 9 | // 10 | // Unless required by applicable law or agreed to in writing, software 11 | // distributed under the License is distributed on an "AS IS" BASIS, 12 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | // See the License for the specific language governing permissions and 14 | // limitations under the License. 15 | 16 | syntax = "proto3"; 17 | 18 | package org.interconnection.v2.runtime; 19 | 20 | // [Sphinx doc begin anchor: EcdhPsiCipherBatch] 21 | // ECDH PSI 密文传输 22 | message EcdhPsiCipherBatch { 23 | // The type hint for each message. (密文类型) 24 | // 25 | // "enc": the first stage ciphertext 26 | // 27 | // "dual.enc": the second stage ciphertext 28 | // 29 | // ECDH PSI 密文阶段类型，主要用来区分一阶段和二阶段的密文. 30 | string type = 1; 31 | 32 | // The batch index. Start from 0. 33 | // 34 | // Batch 索引，从 0 开始 35 | int32 batch_index = 3; 36 | 37 | // Is last batch flag 38 | bool is_last_batch = 4; 39 | 40 | // Count of items in this batch. 41 | // count == 0 is allowed for last batch 42 | int32 count = 6; 43 | 44 | // The packed all in one ciphertext for this batch. 45 | // 46 | // The first stage ciphertext takes 256 bits for each ciphertext element. 47 | // However, the second stage ciphertext takes 96 bits each. According to PSI 48 | // papers, we do not need to send all 256 bit for the final ciphertext. The 49 | // number of bits needed to compare is `Log(MN) + 40` given a 40 bits 50 | // statistical security parameter. TODO (add paper link here). 51 | // 52 | // We define each bucket has less than 2^28 items, i.e. about 270 million 53 | // (单桶最多 2.7亿) items, which is general enough for various psi algorithms. 54 | // 55 | // NOTE: we do not use `repeated`` here to save overhead of metadata. 56 | bytes ciphertext = 7; 57 | 58 | // The extra count map records the number of repetitions of the element at 59 | // this batch. 60 | // 61 | // Because PSI protocal does not support intersection with 62 | // duplicate key values, we use this field to represent the additional number 63 | // of repetitions of the element at this index. The KEY value is the index of 64 | // the element in the ciphertext that appears more than once, and the VALUE is 65 | // the number of times the element appears minus one. 66 | // 67 | // For example, if the elements are [1, 2, 3, 3, 4, 5, 5, 5], the ciphertext 68 | // is [Enc(1), Enc(2), Enc(3), Enc(4), Enc(5)], then the 69 | // duplicate_item_cnt_map would be {2: 1, 4: 2}. 70 | map duplicate_item_cnt_map = 8; 71 | } 72 | // [Sphinx doc end anchor: EcdhPsiCipherBatch] 73 | -------------------------------------------------------------------------------- /interconnection/runtime/data_exchange.proto: -------------------------------------------------------------------------------- 1 | // Copyright 2023 Ant Group Co., Ltd. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | syntax = "proto3"; 16 | 17 | package org.interconnection.v2.runtime; 18 | 19 | message DataExchangeProtocol { 20 | // enum ScalarType 21 | int32 scalar_type = 1; 22 | 23 | // if scalar_type is SCALAR_TYPE_OBJECT, please put real type name here 24 | // otherwise this field is optional 25 | string scalar_type_name = 2; 26 | 27 | oneof container { 28 | // Store single scalar or single object 29 | Scalar scalar = 5; 30 | 31 | // List, F means the size of each element is fixed and V means variant 32 | FScalarList f_scalar_list = 6; 33 | VScalarList v_scalar_list = 7; 34 | 35 | // Ndarray, F means the size of each element is fixed and V means variant 36 | FNdArray f_ndarray = 8; 37 | VNdArray v_ndarray = 9; 38 | 39 | // List of ndarray 40 | // F means the size of each element is fixed and V means variant 41 | FNdArrayList f_ndarray_list = 10; 42 | VNdArrayList v_ndarray_list = 11; 43 | } 44 | } 45 | 46 | enum ScalarType { 47 | SCALAR_TYPE_UNSPECIFIED = 0; 48 | SCALAR_TYPE_BOOL = 1; 49 | SCALAR_TYPE_INT8 = 2; 50 | SCALAR_TYPE_UINT8 = 3; 51 | SCALAR_TYPE_INT16 = 4; 52 | SCALAR_TYPE_UINT16 = 5; 53 | SCALAR_TYPE_INT32 = 6; 54 | SCALAR_TYPE_UINT32 = 7; 55 | SCALAR_TYPE_INT64 = 8; 56 | SCALAR_TYPE_UINT64 = 9; 57 | SCALAR_TYPE_INT128 = 10; 58 | SCALAR_TYPE_UINT128 = 11; 59 | 60 | SCALAR_TYPE_FLOAT16 = 15; 61 | SCALAR_TYPE_FLOAT32 = 16; 62 | SCALAR_TYPE_FLOAT64 = 17; 63 | 64 | SCALAR_TYPE_OBJECT = 20; 65 | } 66 | 67 | message Scalar { 68 | bytes buf = 1; 69 | } 70 | 71 | // Fixed-length scalar list 72 | // the items in this list are all same size 73 | message FScalarList { 74 | // The size of each item is item_buf.len / item_count 75 | int64 item_count = 1; 76 | bytes item_buf = 2; 77 | } 78 | 79 | // variant length scalar list 80 | // variant length means that the length of each serialized element is not equal. 81 | message VScalarList { 82 | repeated bytes items = 1; 83 | } 84 | 85 | message FNdArray { 86 | // The size of each item is item_buf.len / total_item, where total_item count 87 | // can get from shape 88 | repeated int64 shape = 1; 89 | bytes item_buf = 2; 90 | } 91 | 92 | message VNdArray { 93 | repeated int64 shape = 1; 94 | repeated bytes items = 2; 95 | } 96 | 97 | // List of ndarray 98 | message FNdArrayList { 99 | repeated FNdArray ndarrays = 1; 100 | } 101 | 102 | message VNdArrayList { 103 | repeated VNdArray ndarrays = 1; 104 | } 105 | -------------------------------------------------------------------------------- /interconnection/handshake/algos/lr.proto: -------------------------------------------------------------------------------- 1 | // Copyright 2023 Ant Group Co., Ltd. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | syntax = "proto3"; 16 | 17 | import "google/protobuf/any.proto"; 18 | 19 | package org.interconnection.v2.algos; 20 | 21 | //===================================// 22 | // Protos used in HandshakeRequest // 23 | //===================================// 24 | 25 | // [Sphinx doc begin anchor: LrHyperparamsProposal] 26 | message LrHyperparamsProposal { 27 | repeated int32 supported_versions = 1; 28 | 29 | // 支持哪些优化器类型 30 | // Ref: enum Optimizer 31 | repeated int32 optimizers = 2; 32 | 33 | // Ref: enum LastBatchPolicy 34 | repeated int32 last_batch_policies = 3; 35 | 36 | bool use_l0_norm = 4; 37 | bool use_l1_norm = 5; 38 | bool use_l2_norm = 6; 39 | } 40 | // [Sphinx doc end anchor: LrHyperparamsProposal] 41 | 42 | // [Sphinx doc begin anchor: LrDataIoProposal] 43 | message LrDataIoProposal { 44 | repeated int32 supported_versions = 1; 45 | 46 | // 样本数量 47 | int64 sample_size = 2; 48 | 49 | // 特征数量 50 | int32 feature_num = 3; 51 | 52 | // 是否有标签(Y) 53 | bool has_label = 4; 54 | } 55 | // [Sphinx doc end anchor: LrDataIoProposal] 56 | 57 | //===================================// 58 | // Protos used in HandshakeResponse // 59 | //===================================// 60 | 61 | // [Sphinx doc begin anchor: LrHyperparamsResult] 62 | message LrHyperparamsResult { 63 | // 版本号 64 | int32 version = 1; 65 | 66 | // 选中的优化器 67 | // Ref: enum Optimizer 68 | int32 optimizer_name = 2; 69 | 70 | // 优化器参数 71 | // 参考 optimizer.proto 72 | google.protobuf.Any optimizer_param = 3; 73 | 74 | // mini-batch 梯度下降的 epoch 参数 75 | int64 num_epoch = 4; 76 | 77 | // mini-batch 梯度下降的 bach_size 参数 78 | int64 batch_size = 5; 79 | 80 | // ref enum LastBatchPolicy 81 | int32 last_batch_policy = 6; 82 | 83 | // L0 正则项 84 | double l0_norm = 7; 85 | 86 | // L1 正则项 87 | double l1_norm = 8; 88 | 89 | // L2 正则项 90 | double l2_norm = 9; 91 | } 92 | // [Sphinx doc end anchor: LrHyperparamsResult] 93 | 94 | // [Sphinx doc begin anchor: LrDataIoResult] 95 | message LrDataIoResult { 96 | int32 version = 1; 97 | 98 | // 样本数量 99 | int64 sample_size = 2; 100 | 101 | // 各方拥有的特征数量 102 | // 例如两个参与方的情况下，若 rank 0 有3个特征，rank 1 有4个特征，则 feature_nums 等于 [3, 4] 103 | repeated int32 feature_nums = 3; 104 | 105 | // 哪一方持有标签 106 | int32 label_rank = 4; 107 | } 108 | // [Sphinx doc end anchor: LrDataIoResult] 109 | 110 | // [Sphinx doc begin anchor: LastBatchPolicy] 111 | // Policy to process the partial last batch of each epoch 112 | enum LastBatchPolicy { 113 | LAST_BATCH_POLICY_UNSPECIFIED = 0; 114 | LAST_BATCH_POLICY_DISCARD = 1; 115 | } 116 | // [Sphinx doc end anchor: LastBatchPolicy] 117 | -------------------------------------------------------------------------------- /docs/CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # 文档贡献指南 | Contributing docs 2 | 3 | - [tl;dr](#tldr) 4 | - [前置条件](#前置条件) 5 | - [环境准备](#环境准备) 6 | - [构建文档](#构建文档) 7 | - [预览文档](#预览文档) 8 | - [文件清理](#文件清理) 9 | - [报告问题](#报告问题) 10 | 11 | ## tl;dr 12 | 13 | ```sh 14 | python -m pip install -r requirements.txt 15 | secretflow-doctools build --lang zh_CN 16 | secretflow-doctools preview 17 | ``` 18 | 19 | > [!TIP] 20 | > 21 | > 下文的示例命令建议在[本目录 (docs)](./) 下执行。 22 | 23 | ## 前置条件 24 | 25 | 本项目使用 [Sphinx] 作为文档框架。你需要： 26 | 27 | - [Python] >= 3.10 28 | 29 | ## 环境准备 30 | 31 | 执行： 32 | 33 | ```sh 34 | python -m pip install -r requirements.txt 35 | ``` 36 | 37 | 这将会： 38 | 39 | - 安装[文档构建所需要的依赖](./requirements.txt) 40 | 41 | > [!TIP] 42 | > 43 | > 你也可以使用 [uv], [mamba] 等其他的依赖管理工具。 44 | 45 | ## 构建文档 46 | 47 | [`secretflow-doctools`] 是针对隐语项目文档构建的辅助工具，它协助开发者在本地构建并 48 | 且[预览](#预览文档)文档。 49 | 50 | 执行： 51 | 52 | ```sh 53 | secretflow-doctools build --lang en --lang zh_CN 54 | ``` 55 | 56 | 这将会构建英文版 `en` 以及中文版 `zh_CN` 文档。 57 | 58 | 如果一切正常，你应当能看到以下输出： 59 | 60 | ```log 61 | SUCCESS to preview, run: secretflow-doctools preview -c . 62 | ``` 63 | 64 | > [!TIP] 65 | > 66 | > 如果提示 `secretflow-doctools` 命令未找到，你可能没有执行 `source .venv/bin/activate` 67 | > 以激活正确的 Python 环境；请参考[环境准备](#环境准备)中的指引。 68 | 69 | 如果想要只构建某个语言的文档，可以调整 `--lang` 选项。 70 | 71 | ## 预览文档 72 | 73 | 工具提供了本地预览的能力，帮助开发者验证文档在**发布到[隐语官网][website]后的显示效 74 | 果**。 75 | 76 | 执行： 77 | 78 | ```sh 79 | secretflow-doctools preview 80 | ``` 81 | 82 | 这将会在本地启动一个预览服务器。你应当能看到以下输出： 83 | 84 | ``` 85 | * Running on http://127.0.0.1:5000 86 | ``` 87 | 88 | 用浏览器访问 （或其它端口号），你应当能看到类似下图的页面，其中 89 | 将会列出在本地构建好的文档版本： 90 | 91 |
92 | the sitemap page 93 |
94 | 95 | 点击一个版本即可打开对应预览，你应当能看到类似下图的页面： 96 | 97 |
98 | the content page 99 |
100 | 101 | > [!TIP] 102 | > 103 | > 你可以保持预览服务器一直开启：在重新构建文档后，刷新页面即可看到更新的内容。 104 | 105 | ## 文件清理 106 | 107 | 以上流程会产生额外的临时文件，这些文件全部位于 [\_build](./_build/) 目录下。如果需要清理 108 | ，可以执行： 109 | 110 | ```sh 111 | secretflow-doctools clean 112 | ``` 113 | 114 | ## 报告问题 115 | 116 | 如果在以上过程中遇到报错、预览无法显示等问题，可以提交问题到 117 | 。 118 | 119 | 文档内容及本项目代码的相关问题请提交到本项目的 Issues 中。 120 | 121 | > [!NOTE] 122 | > 123 | > 为协助排查问题，你可以设置 `LOGURU_LEVEL=DEBUG` 环境变量来让文档工具输出更多日志。 124 | > 125 | > `secretflow-doctools` 会调用其他工具，在 `LOGURU_LEVEL=DEBUG` 时，日志会在每个步骤打印 126 | > 完整的命令行指令： 127 | > 128 | > | | | 129 | > | :---------------------------------------- | :--------------- | 130 | > | `secretflow-doctools build` | [`sphinx-build`] | 131 | > | `secretflow-doctools update-translations` | [`sphinx-intl`] | 132 | 133 | [`secretflow-doctools`]: https://github.com/secretflow/doctools 134 | [`sphinx-apidoc`]: https://www.sphinx-doc.org/en/master/man/sphinx-apidoc.html 135 | [`sphinx-build`]: https://www.sphinx-doc.org/en/master/man/sphinx-build.html 136 | [`sphinx-intl`]: https://www.sphinx-doc.org/en/master/usage/advanced/intl.html 137 | [gettext-po]: https://www.gnu.org/software/gettext/manual/html_node/PO-Files.html 138 | [gettext]: https://www.gnu.org/software/gettext/ 139 | [mamba]: https://mamba.readthedocs.io/en/latest/ 140 | [Poedit]: https://poedit.net/ 141 | [Python]: https://www.python.org/ 142 | [Sphinx]: https://www.sphinx-doc.org/en/master/tutorial/index.html 143 | [uv]: https://docs.astral.sh/uv/ 144 | [venv]: https://docs.python.org/3/library/venv.html 145 | [website]: https://www.secretflow.org.cn/ 146 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | # Copyright 2023 Ant Group Co., Ltd. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | # Ideas borrowed from: https://github.com/ray-project/ray/blob/master/python/setup.py 16 | 17 | import io 18 | import os 19 | import platform 20 | import re 21 | import shutil 22 | import subprocess 23 | import sys 24 | 25 | from setuptools import setup, find_packages 26 | 27 | ROOT_DIR = os.path.dirname(__file__) 28 | 29 | 30 | def find_version(*filepath): 31 | # Extract version information from filepath 32 | with open(os.path.join(ROOT_DIR, *filepath)) as fp: 33 | version_match = re.search( 34 | r"^__version__ = ['\"]([^'\"]*)['\"]", fp.read(), re.M 35 | ) 36 | if version_match: 37 | return version_match.group(1) 38 | raise RuntimeError("Unable to find version string.") 39 | 40 | 41 | def read_requirements(*filepath): 42 | requirements = [] 43 | with open(os.path.join(ROOT_DIR, *filepath)) as file: 44 | requirements = file.read().splitlines() 45 | return requirements 46 | 47 | 48 | class SetupSpec: 49 | def __init__(self, name: str, description: str): 50 | self.name: str = name 51 | self.version = find_version("interconnection", "version.py") 52 | self.description: str = description 53 | self.files_to_include: list = [] 54 | self.install_requires: list = [] 55 | self.extras: dict = {} 56 | 57 | 58 | setup_spec = SetupSpec( 59 | "interconnection", 60 | "Interconnection aims to define standard interconnection protocols for multi-layers in privacy-preserving computing.", 61 | ) 62 | 63 | 64 | setup_spec.install_requires = read_requirements("requirements.txt") 65 | 66 | 67 | # Calls Bazel in PATH 68 | def bazel_invoke(invoker, cmdline, *args, **kwargs): 69 | try: 70 | print(f'Invoke command: bazel {" ".join(cmdline)}') 71 | result = invoker(["bazel"] + cmdline, *args, **kwargs) 72 | return result 73 | except IOError: 74 | raise 75 | 76 | 77 | def build(): 78 | bazel_env = dict(os.environ, PYTHON3_BIN_PATH=sys.executable) 79 | 80 | bazel_flags = ["--verbose_failures"] 81 | 82 | bazel_precmd_flags = [] 83 | 84 | bazel_targets = ["//interconnection:ic_py_proto"] 85 | 86 | bazel_flags.extend(["-c", "opt"]) 87 | 88 | return bazel_invoke( 89 | subprocess.check_call, 90 | bazel_precmd_flags + ["build"] + bazel_flags + ["--"] + bazel_targets, 91 | env=bazel_env, 92 | ) 93 | 94 | 95 | # Ensure no remaining lib files. 96 | build_dir = os.path.join(ROOT_DIR, "build") 97 | if os.path.isdir(build_dir): 98 | shutil.rmtree(build_dir) 99 | 100 | build() 101 | 102 | setup( 103 | name=setup_spec.name, 104 | version=setup_spec.version, 105 | author="SecretFlow Team", 106 | author_email="secretflow-contact@service.alipay.com", 107 | description=(setup_spec.description), 108 | long_description=io.open( 109 | os.path.join(ROOT_DIR, "README.md"), "r", encoding="utf-8" 110 | ).read(), 111 | long_description_content_type="text/markdown", 112 | url="https://github.com/secretflow/interconnection", 113 | classifiers=[ 114 | "Programming Language :: Python :: 3", 115 | ], 116 | packages=find_packages(where="bazel-bin"), 117 | package_dir={"": "bazel-bin"}, 118 | install_requires=setup_spec.install_requires, 119 | setup_requires=["wheel"], 120 | extras_require=setup_spec.extras, 121 | license="Apache 2.0", 122 | options={"bdist_wheel": {"plat_name": "any"}}, 123 | ) 124 | -------------------------------------------------------------------------------- /interconnection/handshake/entry.proto: -------------------------------------------------------------------------------- 1 | // Copyright 2023 Ant Group Co., Ltd. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | syntax = "proto3"; 16 | 17 | import "google/protobuf/any.proto"; 18 | import "interconnection/common/header.proto"; 19 | 20 | package org.interconnection.v2; 21 | 22 | message HandshakeVersionCheckHelper { 23 | int32 version = 1; 24 | } 25 | 26 | // [Sphinx doc begin anchor: HandshakeRequest] 27 | // unified protocol for interconnection 28 | message HandshakeRequest { 29 | // 握手请求版本号, 当前等于 2 30 | int32 version = 1; 31 | 32 | //** META INFO **// 33 | 34 | // The sender's rank 35 | int32 requester_rank = 2; 36 | 37 | //** AI/BI 算法层 **// 38 | 39 | // enum AlgoType 40 | repeated int32 supported_algos = 3; 41 | 42 | // 算法详细握手参数，与 supported_algos 一一对应 43 | // SS-LR：learning_rate，optimizer，normalize 44 | // ECDH-PSI：Nothing，skip 45 | repeated google.protobuf.Any algo_params = 4; 46 | 47 | //** 安全算子层 **// 48 | 49 | // AI/BI 算法所需的 op 列到此处 50 | // op = enum OpType 51 | // ECDH-PSI：Nothing，skip 52 | repeated int32 ops = 5; 53 | repeated google.protobuf.Any op_params = 6; 54 | 55 | //** 密码协议层 **// 56 | 57 | // protocol_family = enum ProtocolFamily 58 | // SS: Protocol: [Semi2K, ABY3], FieldType, BeaverConfig, SerializeFormat 59 | // ECC: Hash2Curve, EcGroup, SerializeFormat 60 | // PHE: Protocol: [Paillier, EcElgamal], SerializeFormat 61 | repeated int32 protocol_families = 7; 62 | repeated google.protobuf.Any protocol_family_params = 8; 63 | 64 | //** 数据 IO **// 65 | 66 | // 定义 AI/BI 算法的输入和结果输出格式，不包括中间交互数据的格式 67 | // PSI: item_count、result_to_rank 68 | // SS-LR: sample_size、feature_num、has_label, etc. 69 | google.protobuf.Any io_param = 9; 70 | } 71 | // [Sphinx doc end anchor: HandshakeRequest] 72 | 73 | // [Sphinx doc begin anchor: HandshakeResponse] 74 | message HandshakeResponse { 75 | // response header 76 | ResponseHeader header = 1; 77 | 78 | //** AI/BI 算法层 **// 79 | 80 | // algos = enum AlgoType 81 | int32 algo = 2; 82 | 83 | // 算法详细握手参数 84 | // SS-LR：learning_rate，optimizer，normalize 85 | // ECDH-PSI：Nothing，skip 86 | google.protobuf.Any algo_param = 3; 87 | 88 | //** 安全算子层 **// 89 | 90 | // AI/BI 算法所需的 op 列到此处 91 | // op = enum OpType 92 | // ECDH-PSI：Nothing，skip 93 | repeated int32 ops = 4; 94 | repeated google.protobuf.Any op_params = 5; 95 | 96 | //** 密码协议层 **// 97 | 98 | // protocol_family = enum ProtocolFamily 99 | // SS: Protocol: [Semi2K, ABY3], FieldType, BeaverConfig, SerializeFormat 100 | // ECC: Hash2Curve, EcGroup, SerializeFormat 101 | // PHE: Protocol: [Paillier, EcElgamal], SerializeFormat 102 | repeated int32 protocol_families = 6; 103 | repeated google.protobuf.Any protocol_family_params = 7; 104 | 105 | //** 数据 IO **// 106 | 107 | // 定义 AI/BI 算法的输入和结果输出格式，不包括中间交互数据的格式 108 | // PSI: item_count、result_to_rank 109 | // SS-LR: sample_size、feature_num、has_label, etc. 110 | google.protobuf.Any io_param = 8; 111 | } 112 | // [Sphinx doc end anchor: HandshakeResponse] 113 | 114 | // [Sphinx doc begin anchor: AlgoType] 115 | enum AlgoType { 116 | ALGO_TYPE_UNSPECIFIED = 0; 117 | ALGO_TYPE_ECDH_PSI = 1; 118 | ALGO_TYPE_SS_LR = 2; 119 | ALGO_TYPE_SGB = 3; 120 | } 121 | // [Sphinx doc end anchor: AlgoType] 122 | 123 | // [Sphinx doc begin anchor: OpType] 124 | enum OpType { 125 | OP_TYPE_UNSPECIFIED = 0; 126 | OP_TYPE_SIGMOID = 1; 127 | } 128 | // [Sphinx doc end anchor: OpType] 129 | 130 | // [Sphinx doc begin anchor: ProtocolFamily] 131 | enum ProtocolFamily { 132 | PROTOCOL_FAMILY_UNSPECIFIED = 0; 133 | PROTOCOL_FAMILY_ECC = 1; 134 | PROTOCOL_FAMILY_SS = 2; 135 | PROTOCOL_FAMILY_PHE = 3; 136 | } 137 | // [Sphinx doc end anchor: ProtocolFamily] 138 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # 隐私计算平台开放算法协议 2 | 3 | 本仓库用于存放隐私计算平台开放算法协议接口，旨在将算法接口通过程序化设计语言 Protobuf 更加全面、精确地表述出来，帮助隐私计算平台开发者更好地实施互联互通改造，促进隐私计算算法的互联互通。 4 | 5 | 协议文档地址：[https://www.secretflow.org.cn/docs/interconnection/](https://www.secretflow.org.cn/docs/interconnection/) 6 | 7 | ## 协议组成 8 | 9 | 本仓库接口按照北京金融科技产业联盟（简称：金科联盟）隐私计算跨平台互联互通团体标准、信通院牵头隐私计算联盟互联互通团体标准要求设计，每一个算法协议均由“参数获取与校验”，“算法主体运行” 两部分组成，其中”参数获取“的方式两个标准略有差异： 10 | 11 | - 金科联盟标准定义了完整的隐私计算平台框架，算法组件运行所需的配置全部由管理层的工作流配置模块（原名 DAG&CONF 模块）下发，算法协议直接从容器运行时获取参数 12 | - 隐私计算联盟标准并未对算法组件之外的其它模块做过多假设，所有算法组件均通过自主握手协商的方式选定参数，从而使得开放算法协议可以独立部署、运行。 13 | 14 | 其余部分两个标准内容一致： 15 | 16 | - 算法运行参数无论是管理层的工作流配置模块下发，还是自主握手协商选定，参数的数量、名称、作用在两个标准中是一样的 17 | - 算法主体的运行流程、交互接口、以及每阶段的结束标记和判断逻辑在两标准中也是一样的 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 |
金科联盟标准隐私计算联盟标准本仓库
算法运行参数由管理层工作流配置模块下发参与方运行握手协商协议自行对齐参数基于握手协商的形式定义了完整的参数列表与接口，如对方采用金科联盟方式，可从容器运行时获取同名参数
算法协议主体协议流程、接口一致定义了完整的协议主体运行所需接口
38 | 39 | ### 相关仓库 40 | 41 | | 仓库 | 说明 | 42 | |---------------------------------------------------------------------------------|------------------------------------| 43 | | [InterOp](https://github.com/secretflow/interop) | 金科联盟互联互通研究成果官方仓库 | 44 | | [caict-ppca/privacy-computing](https://github.com/caict-ppca/privacy-computing) | 隐私计算联盟互联互通研究成果官方仓库 | 45 | | [interconnection](https://github.com/secretflow/interconnection) (本仓库） | 隐语收集、定义的互联互通开放算法协议接口、文档 | 46 | | [interconnection-impl](https://github.com/secretflow/interconnection-impl) | 互联互通开放算法协议的参考实现，支持 Python & C++ 语言 | 47 | 48 | 注: 仓库排名不分先后 49 | 50 | ## Interconnection 仓库结构 51 | 52 | ``` 53 | . 54 | ├── PPCA # 存放隐私计算联盟归口的标准协议文件副本 55 | ├── docs # 文档网站的网页源代码 56 | └── interconnection # 所有接口文件 57 | ├── common # 通用的接口定义 58 | ├── handshake # 所有握手协议相关接口 59 | │   ├── algos # 算法层的握手协议接口 60 | │   ├── op # 安全算子层的握手协议接口 61 | │   └── protocol_family # 密码协议层的握手协议接口 62 | ├── legacy # 目前已经废弃的接口文件 63 | ├── link # 传输层接口 64 | ├── runtime # 算法运行主体所用的接口 65 | └── service # 访问第三方公共基础服务所用的接口 66 | 67 | ``` 68 | 69 | 当前已经定义的互联互通协议（接口）有: 70 | 71 | - [ECDH-PSI 协议](./PPCA/open-protocols/ECDH-PSI.pdf) 72 | - [SS-LR 协议](./PPCA/open-protocols/SS-LR.pdf) 73 | - [SGB 协议](./PPCA/open-protocols/SGB.pdf) 74 | 75 | 注：本仓库收集、存放的互联互通接口仅为金科联盟、隐私计算联盟标准的子集，完整的接口、文档请参考各联盟的官方仓库。 76 | 77 | 78 | ## 握手协议设计原则 79 | 80 | 隐私计算联盟标准体系中所有算法协议都通过握手协商的方式对齐算法参数，整个过程较为复杂，此处做一些补充介绍。 81 | 82 | 握手协商协议的执行过程和接口设计遵循以下原则： 83 | 84 | 1. 所有算法都复用同一套握手协议，即所有算法的握手请求都用的是 `interconnection/handshake/entry.proto` 中定义的 `HandshakeRequest`，所有算法的握手协商结果都使用 `HandshakeResponse` 格式。 85 | 2. 对于多个参与方的算法，我们为每个参与方赋予一个编号，称为 rank，rank 的数值从 0 开始依次递增。 86 | 3. 握手协议执行时，由非 0 参与方向 rank-0 发送 `HandshakeRequest`，rank-0 汇总所有参与方的请求后，得出一组公共参数，依次发送 `HandshakeResponse`，如果参数协商失败，则依次发送错误消息。 87 | 4. `HandshakeRequest` 中每一类具体的参数项，其命名风格一般为 XxxProposal，`HandshakeResponse` 中选定的参数项，其命名风格一般为 XxxResult。 88 | 5. 对于某些连续数值型参数，例如深度学习中的 learning_rate 等，我们假设无论参数选择几对功能无影响，对方都应该支持，这一类参数不需要协商，而是由 rank-0 选定一个数值，在 `HandshakeResponse` 中发给大家。 89 | 6. 对于一些可枚举的功能性参数，`HandshakeRequest` 用一个列表表示，表示发送者支持列表所列的功能；并且这个列表是有序的，表示发送者更加偏爱列表中靠前的参数。例如 ECDH-PSI 中的椭圆曲线(EC)类型，假如请求列表是 `[SM2, CURVE25519]`，则表示发送者同时支持 SM2 和 CURVE25519，如果其他参与方也同时支持这两种 EC，则协商者应当优先选择 SM2, 因为 SM2 排在前面。当然，如果多个参数方发送的列表顺序是矛盾的，协商者会优先满足大多数参与方的偏爱。 90 | 91 | 92 | ## 算法协议与传输层关系 93 | 94 | 算法协议中提到的 proto 一般并不直接用于 RPC 框架，而是用作跨语言、跨版本的序列化、反序列化工具使用。传输层只负责传输裸的二进制 buffer，不感知具体的 proto 格式，下图是一个示例： 95 | 96 | ``` 97 | ┌─────────────┐ ┌─────────────┐ 98 | │ Algorithm │ │ Algorithm │ 99 | └──────┬──────┘ └──────▲──────┘ 100 | │Serialize │ 101 | │proto │Deserialize 102 | │to buffer │buffer 103 | ┌──────▼──────┐ ┌──────┴──────┐ 104 | │ Transport │ buffer │ Transport │ 105 | │ layer ├────────────────►│ layer │ 106 | └─────────────┘ http(s)/rpc └─────────────┘ 107 | ``` 108 | 109 | 示例图： 110 | 111 | 1. 左侧的算法将 proto 序列化成 buffer，提交给传输层 112 | 2. 左侧传输层通过网络将数据发送到右侧传输层模块 113 | 3. 右侧算法获取传输层中的 buffer，反序列化成 proto 114 | 115 | -------------------------------------------------------------------------------- /docs/protocols/ecdh-psi.rst: -------------------------------------------------------------------------------- 1 | ECDH-PSI 协议 2 | ========================= 3 | 4 | 算法流程 5 | ------------------------ 6 | 7 | 算法分为2阶段，第一阶段为握手过程，第二阶段为算法主体，其流程如下： 8 | 9 | .. image:: _img/ecdh-psi-flow.png 10 | :align: center 11 | 12 | 13 | 握手过程 14 | ^^^^^^^^^^^^^^^^^^^ 15 | 16 | 握手所用的 HandshakeRequest 定义如下： 17 | 18 | .. literalinclude:: ../../interconnection/handshake/entry.proto 19 | :caption: interconnection/handshake/entry.proto 20 | :language: Protobuf 21 | :start-after: [Sphinx doc begin anchor: HandshakeRequest] 22 | :end-before: [Sphinx doc end anchor: HandshakeRequest] 23 | :linenos: 24 | 25 | HandshakeRequest 主要包括以下信息： 26 | 27 | 1. 协议版本号 28 | 2. 请求方的传输层 rank 值 29 | 3. 想使用的具体算法，比如使用 ECDH-PSI 30 | 4. 每类算法的详细参数，ECDH-PSI 算法忽略该字段 31 | 5. 用到的安全算子的类型，ECDH-PSI 算法忽略该字段 32 | 6. 每个安全算子的详细参数，ECDH-PSI 算法忽略该字段 33 | 7. 用到的密码协议族，比如 ECC 协议族 34 | 8. 每个协议族的详细参数，比如 ECC 协议族需要说明具体的椭圆曲线类型，哈希算法等参数 35 | 9. 算法的输入和结果输出格式，比如 ECDH-PSI 需要说明结果是A，B都可见，还是只对某一方可见 36 | 37 | HandshakeRequest 中的 supported_algos 字段的定义如下： 38 | 39 | .. literalinclude:: ../../interconnection/handshake/entry.proto 40 | :caption: interconnection/handshake/entry.proto 41 | :language: Protobuf 42 | :start-after: [Sphinx doc begin anchor: AlgoType] 43 | :end-before: [Sphinx doc end anchor: AlgoType] 44 | :linenos: 45 | 46 | HandshakeRequest 中的 protocol_families 字段的定义如下： 47 | 48 | .. literalinclude:: ../../interconnection/handshake/entry.proto 49 | :caption: interconnection/handshake/entry.proto 50 | :language: Protobuf 51 | :start-after: [Sphinx doc begin anchor: ProtocolFamily] 52 | :end-before: [Sphinx doc end anchor: ProtocolFamily] 53 | :linenos: 54 | 55 | 如果协议族是 ECC，则 HandshakeRequest 中的 protocol_family_params 字段格式如下： 56 | 57 | .. literalinclude:: ../../interconnection/handshake/protocol_family/ecc.proto 58 | :caption: interconnection/handshake/protocol_family/ecc.proto 59 | :language: Protobuf 60 | :start-after: [Sphinx doc begin anchor: EccProtocolProposal] 61 | :end-before: [Sphinx doc end anchor: EccProtocolProposal] 62 | :linenos: 63 | 64 | EccProtocolProposal 中的 ec_suits 字段的定义如下： 65 | 66 | .. literalinclude:: ../../interconnection/handshake/protocol_family/ecc.proto 67 | :caption: interconnection/handshake/protocol_family/ecc.proto 68 | :language: Protobuf 69 | :start-after: [Sphinx doc begin anchor: EcSuit] 70 | :end-before: [Sphinx doc end anchor: EcSuit] 71 | :linenos: 72 | 73 | 如果算法是 ECDH-PSI，则 HandshakeRequest 中的 io_param 字段格式如下： 74 | 75 | .. literalinclude:: ../../interconnection/handshake/algos/psi.proto 76 | :caption: interconnection/handshake/algos/psi.proto 77 | :language: Protobuf 78 | :start-after: [Sphinx doc begin anchor: PsiDataIoProposal] 79 | :end-before: [Sphinx doc end anchor: PsiDataIoProposal] 80 | :linenos: 81 | 82 | 握手请求的结果 HandshakeResponse 定义如下： 83 | 84 | .. literalinclude:: ../../interconnection/handshake/entry.proto 85 | :caption: interconnection/handshake/entry.proto 86 | :language: Protobuf 87 | :start-after: [Sphinx doc begin anchor: HandshakeResponse] 88 | :end-before: [Sphinx doc end anchor: HandshakeResponse] 89 | :linenos: 90 | 91 | 其中 ResponseHeader 定义如下： 92 | 93 | .. literalinclude:: ../../interconnection/common/header.proto 94 | :caption: interconnection/common/header.proto 95 | :language: Protobuf 96 | :start-after: [Sphinx doc begin anchor: ResponseHeader] 97 | :end-before: [Sphinx doc end anchor: ResponseHeader] 98 | :linenos: 99 | 100 | 如果协议族是 ECC，则 HandshakeResponse 中的 protocol_family_params 字段格式如下： 101 | 102 | .. literalinclude:: ../../interconnection/handshake/protocol_family/ecc.proto 103 | :caption: interconnection/handshake/protocol_family/ecc.proto 104 | :language: Protobuf 105 | :start-after: [Sphinx doc begin anchor: EccProtocolResult] 106 | :end-before: [Sphinx doc end anchor: EccProtocolResult] 107 | :linenos: 108 | 109 | 如果算法是 ECDH-PSI，则 HandshakeResponse 中的 io_param 字段格式如下： 110 | 111 | .. literalinclude:: ../../interconnection/handshake/algos/psi.proto 112 | :caption: interconnection/handshake/algos/psi.proto 113 | :language: Protobuf 114 | :start-after: [Sphinx doc begin anchor: PsiDataIoResult] 115 | :end-before: [Sphinx doc end anchor: PsiDataIoResult] 116 | :linenos: 117 | 118 | 119 | Protobuf 传输方式 120 | """""""""""""""""""""""""""""" 121 | 122 | Protobuf 传输使用《传输层白盒互联互通协议》中的 P2P 传输协议进行传输。其中传输的 key 按照《传输层白盒互联互通协议》中定义的方法生成，value 即为 protobuf 序列化之后的二进制字符串。 123 | 124 | 125 | 算法主体 126 | ^^^^^^^^^^^^^^^^^^^ 127 | 128 | .. image:: _img/ecdh-psi-algo.png 129 | :align: center 130 | 131 | 132 | 算法第二步、第四步使用 EcdhPsiCipherBatch 格式进行传输，EcdhPsiCipherBatch 定义如下： 133 | 134 | .. literalinclude:: ../../interconnection/runtime/ecdh_psi.proto 135 | :caption: interconnection/runtime/ecdh_psi.proto 136 | :language: Protobuf 137 | :start-after: [Sphinx doc begin anchor: EcdhPsiCipherBatch] 138 | :end-before: [Sphinx doc end anchor: EcdhPsiCipherBatch] 139 | :linenos: 140 | 141 | 其中 ciphertext 字段用于存放 ECC 上的点，每个点按照握手协议中的 point_octet_format 序列化之后依次连续存放。 142 | -------------------------------------------------------------------------------- /docs/protocols/transport.rst: -------------------------------------------------------------------------------- 1 | 传输层协议（一期） 2 | ============================ 3 | 4 | 背景 5 | --------- 6 | 7 | 传输层定义了一组标准化接口，用于在不同隐私计算的参与者之间传递信息。 8 | 9 | .. image:: _img/tp-bg.png 10 | :scale: 30% 11 | :align: center 12 | 13 | 传输层协议 14 | ------------------- 15 | 16 | 接口定义 17 | ^^^^^^^^^^^^^^^^^^ 18 | 19 | 两个参与者之间通过 gRPC 协议传输消息，消息传输的 protobuf 格式定义如下： 20 | 21 | .. literalinclude:: ../../interconnection/link/transport.proto 22 | :caption: interconnection/link/transport.proto 23 | :language: Protobuf 24 | :start-after: [Sphinx doc begin anchor: ReceiverService] 25 | :end-before: [Sphinx doc end anchor: ReceiverService] 26 | :linenos: 27 | 28 | 其中关于 ResponseHeader 定义如下： 29 | 30 | .. literalinclude:: ../../interconnection/common/header.proto 31 | :caption: interconnection/common/header.proto 32 | :language: Protobuf 33 | :start-after: [Sphinx doc begin anchor: ResponseHeader] 34 | :end-before: [Sphinx doc end anchor: ResponseHeader] 35 | :linenos: 36 | 37 | 接口说明： 38 | 39 | 1. 传输层所有消息收发都通过唯一接口 ``rpc Push(PushRequest) returns (PushResponse)`` 完成。 40 | 2. 一个通信组的每一个成员拥有一个唯一编号（rank），编号为自然数，从0开始，成员的编号必须连续，成员具体编号在初始化阶段由外部指定。举例：一个三方隐私计算任务中成员编号分别为 0、1、2。 41 | 3. 消息传送支持全量传送（TransType::MONO）和分块传送（TransType::CHUNKED）两种模式，使用哪种传输模式由发送者决定，分块传输模式下每一个切块的大小亦由发送方决定。 42 | 4. 每一个消息拥有一个全局唯一的 key，key 的生成方式见下文。对于分块传输模式，同一个消息体（Message）切块后每一块的 key 相同。 43 | 44 | 握手协议 45 | ^^^^^^^^^^^^^^^^^^ 46 | 47 | 握手协议用于初始化通信组，整个协议在全生命周期只执行一次。 握手协议 message key 命名规则： ``connect_{self_rank_id}``，举例：``connect_0``，``connect_1``，``connect_2`` 48 | 49 | **握手协议流程** 50 | 51 | 每一个参与者执行以下流程： 52 | 53 | .. code-block:: text 54 | :linenos: 55 | 56 | For i in 0..word_size: 57 | if i == self_rank: 58 | continue 59 | Async send to rank i: {key: connect_{self_rank}, value: ""} 60 | 61 | For i in 0..word_size: 62 | if i == self_rank: 63 | continue 64 | Recv on key connect_{i} 65 | 66 | 握手协议分为两部分：第一部分向其它参与者广播自己的存在性，即向他人发送 ``connect_{self_rank}`` 消息；第二部分检查他人的存在性，即依次检查 ``connect_{rank}`` 消息已经收到。 67 | 68 | 信道 69 | """""""""""""""""""" 70 | 71 | 信道是一个逻辑概念，用于区分通信的上下文。每一个信道有一个全局唯一名称，命名规则为：``\w+``，即信号名称由字母、数字、下划线组成。信道的名字由通信组双方约定，在初始化阶段由用户传入。 72 | 73 | 信道唯一的作用就是影响 message key 的生成，信道名称会作为 message key 一部分，因此，不同信道中的消息一定不会有相同的 key，因此不同信道的消息在逻辑上不会混淆。 74 | 75 | 子信道 76 | ~~~~~~~~~~~~ 77 | 78 | 当上层算法需要多个信道时，我们统一把第一个信道称为主信道，其它信道称为子信道。子信道的命名规则为：``主信道名称-子信道编号``，举例：假设主信道名称为 root，则 0 号子信道名称为 ``root-0``，1 号子信道名称为 ``root-1``，以此类推。 79 | 80 | P2P 通信 81 | ^^^^^^^^^^^^^^^^^^ 82 | 83 | P2P 通信允许在任意两个参与者之间发送信息。P2P 通信 key 的命名规则为：``{信道名称}:P2P-{计数器}:{发送者 RANK}->{接收者 RANK}``，其中每一个信道、每一对 ```` 都有一个独立的计数器。 84 | 85 | 举例，假设信道名称为 root，以下消息依次发送： 86 | 87 | .. code-block:: text 88 | :linenos: 89 | 90 | Rank 0 → 1 发送消息，key 为：root:P2P-1:0->1 91 | Rank 1 → 0 发送消息，key 为：root:P2P-1:1->0 92 | Rank 0 → 2 发送消息，key 为：root:P2P-1:0->2 93 | Rank 0 → 1 发送消息，key 为：root:P2P-2:0->1 94 | 95 | 96 | Scatter 97 | ^^^^^^^^^^^^^^^^^^ 98 | 99 | Scatter 用于把某一方的数据切分后发送给所有其它方，如下图所示： 100 | 101 | .. image:: _img/tp-scatter.png 102 | :scale: 30% 103 | :align: center 104 | 105 | Scatter 操作所有参与方使用相同的 key 通信，key 命名规则为：``{信道名称}:{信道全局计数器}:SCATTER``，此处的计数器每一个信道有且只有一个，在信道内是全局的，是除 P2P 通信之外所有通信算法共享的。也就是说每一个信道会维护两类计数器，一类是 P2P 通信用的计数器，另一类就是这个全局计数器。 106 | 107 | 举例，假设 rank 0 要 scatter 数据，则协议流程如下： 108 | 109 | .. code-block:: text 110 | :linenos: 111 | 112 | 时间点1：RANK 0 Scatter 113 | RANK 0 构建 PushRequest 发送给 RANK 1，其中 key 为 root:1:SCATTER 114 | RANK 0 构建 PushRequest 发送给 RANK 2，其中 key 为 root:1:SCATTER 115 | RANK 0 构建 PushRequest 发送给 RANK 3，其中 key 为 root:1:SCATTER 116 | RANK 1 接收 key 为 root:1:SCATTER 的数据 117 | RANK 2 接收 key 为 root:1:SCATTER 的数据 118 | RANK 3 接收 key 为 root:1:SCATTER 的数据 119 | 120 | 时间点2：RANK 1 Scatter 121 | RANK 1 构建 PushRequest 发送给 RANK 0，其中 key 为 root:2:SCATTER 122 | RANK 1 构建 PushRequest 发送给 RANK 2，其中 key 为 root:2:SCATTER 123 | RANK 1 构建 PushRequest 发送给 RANK 3，其中 key 为 root:2:SCATTER 124 | RANK 0 接收 key 为 root:2:SCATTER 的数据 125 | RANK 2 接收 key 为 root:2:SCATTER 的数据 126 | RANK 3 接收 key 为 root:2:SCATTER 的数据 127 | 128 | 129 | Gather 130 | ^^^^^^^^^^^^^^^^^^ 131 | 132 | Gather 的作用正好与 Scatter 相反，用于将数据汇集到一方，如下图： 133 | 134 | .. image:: _img/tp-gather.png 135 | :scale: 60% 136 | :align: center 137 | 138 | Gather 通信 Key 的命名规则为：``{信道名称}:{信道全局计数器}:GATHER``，此处的信道全局计数器与 Scatter 中的计数器是同一个。 139 | 140 | .. code-block:: text 141 | :linenos: 142 | 143 | 时间点3：RANK 0 Gather 144 | RANK 1 构建 PushRequest 发送给 RANK 0，其中 key 为 root:3:GATHER 145 | RANK 2 构建 PushRequest 发送给 RANK 0，其中 key 为 root:3:GATHER 146 | RANK 3 构建 PushRequest 发送给 RANK 0，其中 key 为 root:3:GATHER 147 | RANK 0 接收 来自 RANK 1 的 key 为 root:3:GATHER 的数据 148 | RANK 0 接收 来自 RANK 2 的 key 为 root:3:GATHER 的数据 149 | RANK 0 接收 来自 RANK 3 的 key 为 root:3:GATHER 的数据 150 | 151 | 其它算法 152 | ^^^^^^^^^^^^^^^^^^ 153 | 154 | 以上为传输层互联互通一期标准协议，以下接口协议将在二期标准中定义。 155 | 156 | 广播：Bcast 157 | 158 | .. image:: _img/tp-bcast.png 159 | :scale: 60% 160 | :align: center 161 | 162 | 全局收集：Allgather 163 | 164 | .. image:: _img/tp-allgather.png 165 | :scale: 60% 166 | :align: center 167 | 168 | 路障同步：Barrier 169 | 170 | .. image:: _img/tp-barrier.png 171 | :align: center 172 | -------------------------------------------------------------------------------- /interconnection/handshake/algos/optimizer.proto: -------------------------------------------------------------------------------- 1 | // Copyright 2023 Ant Group Co., Ltd. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | syntax = "proto3"; 16 | 17 | package org.interconnection.v2.algos; 18 | 19 | // [Sphinx doc begin anchor: Optimizer] 20 | // Optimizer list 21 | enum Optimizer { 22 | OPTIMIZER_UNSPECIFIED = 0; 23 | OPTIMIZER_SGD = 1; 24 | OPTIMIZER_MOMENTUM = 2; 25 | OPTIMIZER_ADAGRAD = 3; 26 | OPTIMIZER_ADADELTA = 4; 27 | OPTIMIZER_RMSPROP = 5; 28 | OPTIMIZER_ADAM = 6; 29 | OPTIMIZER_ADAMAX = 7; 30 | OPTIMIZER_NADAM = 8; 31 | } 32 | // [Sphinx doc end anchor: Optimizer] 33 | 34 | // [Sphinx doc begin anchor: SgdOptimizer] 35 | // mini-batch (stochastic) gradient descent 36 | message SgdOptimizer { 37 | double learning_rate = 1; 38 | } 39 | // [Sphinx doc end anchor: SgdOptimizer] 40 | 41 | // Gradient descent (with momentum) optimizer. 42 | message MomentumOptimizer { 43 | double learning_rate = 1; 44 | 45 | // float hyperparameter >= 0 that accelerates gradient descent in the relevant 46 | // direction and dampens oscillations. 47 | double momentum = 2; 48 | 49 | // boolean. Whether to apply Nesterov momentum. 50 | bool use_nesterov = 3; 51 | } 52 | 53 | // Adagrad is an optimizer with parameter-specific learning rates, which are 54 | // adapted relative to how frequently a parameter gets updated during training. 55 | // The more updates a parameter receives, the smaller the updates. 56 | message AdagradOptimizer { 57 | double learning_rate = 1; 58 | 59 | // Floating point value. Starting value for the accumulators (per-parameter 60 | // momentum values). Must be non-negative. 61 | double initial_accumulator_value = 2; 62 | 63 | // Small floating point value used to maintain numerical stability. 64 | double epsilon = 3; 65 | } 66 | 67 | // Adadelta optimization is a stochastic gradient descent method that is based 68 | // on adaptive learning rate per dimension to address two drawbacks: 69 | // 70 | // - The continual decay of learning rates throughout training. 71 | // - The need for a manually selected global learning rate. 72 | // 73 | // Adadelta is a more robust extension of Adagrad that adapts learning rates 74 | // based on a moving window of gradient updates, instead of accumulating all 75 | // past gradients. This way, Adadelta continues learning even when many updates 76 | // have been done. Compared to Adagrad, in the original version of Adadelta you 77 | // don't have to set an initial learning rate. In this version, the initial 78 | // learning rate can be set, as in most other optimizers. 79 | message AdadeltaOptimizer { 80 | double learning_rate = 1; 81 | double rho = 2; 82 | double epsilon = 3; 83 | } 84 | 85 | // The gist of RMSprop is to: 86 | // 87 | // - Maintain a moving (discounted) average of the square of gradients 88 | // - Divide the gradient by the root of this average 89 | // 90 | // This implementation of RMSprop uses plain momentum, not Nesterov momentum. 91 | // The centered version additionally maintains a moving average of the 92 | // gradients, and uses that average to estimate the variance. 93 | message RMSpropOptimizer { 94 | double learning_rate = 1; 95 | double rho = 2; 96 | double momentum = 3; 97 | double epsilon = 4; 98 | bool centered = 5; 99 | } 100 | 101 | // Adam optimization is a stochastic gradient descent method that is based on 102 | // adaptive estimation of first-order and second-order moments. 103 | // 104 | // According to Kingma et al., 2014 (http://arxiv.org/abs/1412.6980), 105 | // the method is "computationally efficient, has little memory requirement, 106 | // invariant to diagonal rescaling of gradients, and is well suited for problems 107 | // that are large in terms of data/parameters". 108 | message AdamOptimizer { 109 | double learning_rate = 1; 110 | // The exponential decay rate for the 1st moment estimates. 111 | double beta_1 = 2; 112 | // The exponential decay rate for the 2nd moment estimates. 113 | double beta_2 = 3; 114 | // A small constant for numerical stability. 115 | // This epsilon is "epsilon hat" in the Kingma and Ba paper (in the formula 116 | // just before Section 2.1), not the epsilon in Algorithm 1 of the paper. 117 | double epsilon = 4; 118 | // Boolean. Whether to apply AMSGrad variant of this algorithm from the paper 119 | // <> 120 | bool amsgrad = 5; 121 | } 122 | 123 | // It is a variant of Adam based on the infinity norm. 124 | // Adamax is sometimes superior to adam, specially in models with embeddings. 125 | message AdamaxOptimizer { 126 | double learning_rate = 1; 127 | double beta_1 = 2; 128 | double beta_2 = 3; 129 | double epsilon = 4; 130 | } 131 | 132 | // Much like Adam is essentially RMSprop with momentum, Nadam is Adam with 133 | // Nesterov momentum. 134 | message NadamOptimizer { 135 | double learning_rate = 1; 136 | double beta_1 = 2; 137 | double beta_2 = 3; 138 | double epsilon = 4; 139 | } 140 | 141 | -------------------------------------------------------------------------------- /interconnection/handshake/protocol_family/ss.proto: -------------------------------------------------------------------------------- 1 | // Copyright 2023 Ant Group Co., Ltd. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | syntax = "proto3"; 16 | 17 | package org.interconnection.v2.protocol; 18 | 19 | //===================================// 20 | // Protos used in HandshakeRequest // 21 | //===================================// 22 | 23 | // [Sphinx doc begin anchor: SSProtocolProposal] 24 | message SSProtocolProposal { 25 | repeated int32 supported_versions = 1; 26 | repeated int32 supported_protocols = 2; // enum ProtocolKind 27 | 28 | repeated int32 field_types = 3; // enum FieldType 29 | repeated TruncationModeProposal trunc_modes = 4; 30 | repeated PrgConfigProposal prg_configs = 5; 31 | 32 | // 秘密分享分片的序列化格式, enum ShardSerializeFormat 33 | repeated int32 shard_serialize_formats = 6; 34 | 35 | // Below fields only used for protocol [Semi2k] 36 | repeated TripleConfigProposal triple_configs = 50; 37 | } 38 | // [Sphinx doc end anchor: SSProtocolProposal] 39 | 40 | // [Sphinx doc begin anchor: TruncationModeProposal] 41 | message TruncationModeProposal { 42 | repeated int32 supported_versions = 1; 43 | 44 | int32 method = 2 ; // enum TruncMode 45 | repeated int32 compatible_protocols = 3; // protocols that support this method. 如果留空，表示所有 protocol 都支持 46 | 47 | // Below fields only used for trunc method "TRUNC_PRECISE" 48 | // ... 49 | } 50 | // [Sphinx doc end anchor: TruncationModeProposal] 51 | 52 | // [Sphinx doc begin anchor: TripleConfigProposal] 53 | // TrustedThirdParty configs. 54 | message TripleConfigProposal { 55 | repeated int32 supported_versions = 1; 56 | 57 | // TTP 服务的版本号，与 TTP 服务建立会话时需要该字段 58 | int32 sever_version = 2; 59 | } 60 | // [Sphinx doc end anchor: TripleConfigProposal] 61 | 62 | // [Sphinx doc begin anchor: PrgConfigProposal] 63 | // Pseudorandom number generator configuration. 64 | message PrgConfigProposal { 65 | repeated int32 supported_versions = 1; 66 | 67 | // enum CryptoType 68 | int32 crypto_type = 2; 69 | } 70 | // [Sphinx doc end anchor: PrgConfigProposal] 71 | 72 | //===================================// 73 | // Protos used in HandshakeResponse // 74 | //===================================// 75 | 76 | // [Sphinx doc begin anchor: SSProtocolResult] 77 | message SSProtocolResult { 78 | int32 version = 1; 79 | int32 protocol = 2; // enum ProtocolKind 80 | 81 | int32 field_type = 3; 82 | TruncationModeResult trunc_mode = 4; 83 | PrgConfigResult prg_config = 5; 84 | 85 | // Number of fraction bits of fixed-point number. 86 | int32 fxp_fraction_bits = 6; 87 | 88 | // 秘密分享分片的序列化格式, enum ShardSerializeFormat 89 | int32 shard_serialize_format = 7; 90 | 91 | // Below fields only used for protocol [Semi2k] 92 | TripleConfigResult triple_config = 50; 93 | } 94 | // [Sphinx doc end anchor: SSProtocolResult] 95 | 96 | // [Sphinx doc begin anchor: TruncationModeResult] 97 | message TruncationModeResult { 98 | int32 version = 1; 99 | 100 | int32 method = 2; // enum TruncMode 101 | } 102 | // [Sphinx doc end anchor: TruncationModeResult] 103 | 104 | // [Sphinx doc begin anchor: TripleConfigResult] 105 | // TrustedThirdParty configs. 106 | message TripleConfigResult { 107 | int32 version = 1; 108 | 109 | // TrustedThirdParty beaver server's remote ip:port or load balancing uri. 110 | string server_host = 2; 111 | 112 | // TTP 服务的版本号，与 TTP 服务建立会话时需要该字段 113 | int32 sever_version = 3; 114 | 115 | // 与 TTP 服务的会话ID 116 | string session_id = 4; 117 | 118 | // 哪一方调用 TTP 服务的 Adjust 接口，与 TTP 服务建立会话时需要该字段 119 | int32 adjust_rank = 5; 120 | } 121 | // [Sphinx doc end anchor: TripleConfigResult] 122 | 123 | // [Sphinx doc begin anchor: PrgConfigResult] 124 | // Pseudorandom number generator configuration. 125 | message PrgConfigResult { 126 | int32 version = 1; 127 | 128 | // // enum CryptoType 129 | int32 crypto_type = 2; 130 | } 131 | // [Sphinx doc end anchor: PrgConfigResult] 132 | 133 | //===================================// 134 | // Protos for ss protocol // 135 | //===================================// 136 | 137 | // [Sphinx doc begin anchor: ProtocolKind] 138 | enum ProtocolKind { 139 | PROTOCOL_KIND_UNSPECIFIED = 0; 140 | PROTOCOL_KIND_SEMI2K = 1; 141 | PROTOCOL_KIND_ABY3 = 2; 142 | } 143 | // [Sphinx doc end anchor: ProtocolKind] 144 | 145 | // [Sphinx doc begin anchor: FieldType] 146 | enum FieldType { 147 | FIELD_TYPE_UNSPECIFIED = 0; 148 | FIELD_TYPE_32 = 1; // Ring 2^32 149 | FIELD_TYPE_64 = 2; // Ring 2^64 150 | FIELD_TYPE_128 = 3; // Ring 2^128 151 | } 152 | // [Sphinx doc end anchor: FieldType] 153 | 154 | // [Sphinx doc begin anchor: TruncMode] 155 | enum TruncMode { 156 | TRUNC_MODE_UNSPECIFIED = 0; 157 | TRUNC_MODE_PROBABILISTIC = 1; 158 | TRUNC_MODE_PRECISE = 2; 159 | } 160 | // [Sphinx doc end anchor: TruncMode] 161 | 162 | // [Sphinx doc begin anchor: CryptoType] 163 | enum CryptoType { 164 | CRYPTO_TYPE_UNSPECIFIED = 0; 165 | CRYPTO_TYPE_AES128_CTR = 1; 166 | CRYPTO_TYPE_SM4_CTR = 2; 167 | } 168 | // [Sphinx doc end anchor: CryptoType] 169 | 170 | // [Sphinx doc begin anchor: ShardSerializeFormat] 171 | // 秘密分享分片的序列化格式 172 | enum ShardSerializeFormat { 173 | SHARED_SERIALIZE_FORMAT_UNSPECIFIED = 0; 174 | SHARED_SERIALIZE_FORMAT_RAW = 1; 175 | } 176 | // [Sphinx doc end anchor: ShardSerializeFormat] 177 | -------------------------------------------------------------------------------- /interconnection/service/beaver.proto: -------------------------------------------------------------------------------- 1 | // Copyright 2023 Ant Group Co., Ltd. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | syntax = "proto3"; 16 | 17 | package org.interconnection.v2.service; 18 | 19 | option cc_generic_services = true; 20 | 21 | enum ErrorCode { 22 | OK = 0; 23 | SessionError = 1; 24 | OpAdjustError = 2; 25 | } 26 | 27 | message CreateSessionRequest { 28 | int32 required_version = 1; 29 | // which rank do adjust rpc call, usually choose the rank closer to the server. 30 | int32 adjust_rank = 2; 31 | // session represents a group of MPC computing participants 32 | string session_id = 3; 33 | // how many parties in this session 34 | int32 world_size = 4; 35 | // Which participant is calling this rcp 36 | int32 rank = 5; 37 | // this rank(participant)'s PRG seed 38 | bytes prg_seed = 6; 39 | } 40 | 41 | message CreateSessionResponse { 42 | ErrorCode code = 1; 43 | string message = 2; 44 | } 45 | 46 | message DeleteSessionRequest { 47 | string session_id = 2; 48 | } 49 | 50 | message DeleteSessionResponse { 51 | ErrorCode code = 1; 52 | string message = 2; 53 | } 54 | 55 | // PRG generated buffer metainfo. 56 | // BeaverService replay PRG to generate same buffer using each party's prg_seed. 57 | // PrgBufferMeta represent {world_size} arithmetic/boolean shares for one random array. 58 | message PrgBufferMeta { 59 | int64 prg_count = 1; 60 | int64 size = 2; 61 | } 62 | 63 | // TTP Beaver service for semi2k only. 64 | service BeaverService { 65 | // Create 66 | rpc CreateSession(CreateSessionRequest) returns (CreateSessionResponse); 67 | 68 | // Delete 69 | rpc DeleteSession(DeleteSessionRequest) returns (DeleteSessionResponse); 70 | 71 | // V1 adjust ops 72 | rpc AdjustMul(AdjustMulRequest) returns (AdjustResponse); 73 | 74 | rpc AdjustDot(AdjusDotRequest) returns (AdjustResponse); 75 | 76 | rpc AdjustAnd(AdjustAndRequest) returns (AdjustResponse); 77 | 78 | rpc AdjustTrunc(AdjustTruncRequest) returns (AdjustResponse); 79 | 80 | rpc AdjustTruncPr(AdjustTruncPrRequest) returns (AdjustResponse); 81 | 82 | rpc AdjustRandBit(AdjustRandBitRequest) returns (AdjustResponse); 83 | } 84 | 85 | message AdjustMulRequest { 86 | string session_id = 1; 87 | // input three prg buffer 88 | // reconstruct all parties' share get: ra / rb / rc 89 | repeated PrgBufferMeta prg_inputs = 2; 90 | // use which field to interprete buffer. details see: spu.FieldType 91 | int32 field = 3; 92 | // output 93 | // adjust_c = ra * rb - rc 94 | // make 95 | // ra * rb = (adjust_c + rc) 96 | } 97 | 98 | message AdjusDotRequest { 99 | string session_id = 1; 100 | // input three prg buffer 101 | // reconstruct all parties' share get: ra / rb / rc 102 | repeated PrgBufferMeta prg_inputs = 2; 103 | // use which field to interprete buffer. details see: spu.FieldType 104 | int32 field = 3; 105 | // ra's shape: (M, K), rb's shape: (K, N), rc's shape: (M, N) 106 | int64 M = 4; 107 | int64 N = 5; 108 | int64 K = 6; 109 | // output 110 | // adjust_c = matmul(ra, rb) - rc 111 | // make 112 | // matmul(ra, rb) = (adjust_c + rc) 113 | } 114 | 115 | message AdjustAndRequest { 116 | string session_id = 1; 117 | // input three prg buffer 118 | // reconstruct all parties' share get: ra / rb / rc 119 | repeated PrgBufferMeta prg_inputs = 2; 120 | // TODO: use bitwise format definition to describe boolean share's buffer 121 | // use which field to interprete buffer. details see: spu.FieldType 122 | int32 field = 3; 123 | // output 124 | // adjust_c = (ra & rb) ^ rc 125 | // make 126 | // ra & rb = (adjust_c ^ rc) 127 | } 128 | 129 | message AdjustTruncRequest { 130 | string session_id = 1; 131 | // input two prg buffer 132 | // reconstruct all parties' share get: ra / rb 133 | repeated PrgBufferMeta prg_inputs = 2; 134 | // use which field to interprete buffer. details see: spu.FieldType 135 | int32 field = 3; 136 | // how many bits need to truncate. 137 | int32 bits = 4; 138 | // output 139 | // adjust_b = (ra >> bits) - rb 140 | // makes 141 | // ra >> bits = (adjust_b + rb) 142 | } 143 | 144 | message AdjustTruncPrRequest { 145 | string session_id = 1; 146 | // input two prg buffer 147 | // reconstruct all parties' share get: ra / rb / rc 148 | repeated PrgBufferMeta prg_inputs = 2; 149 | // use which field to interprete buffer. details see: spu.FieldType 150 | int32 field = 3; 151 | // how many bits need to truncate. 152 | int32 bits = 4; 153 | // output 154 | // adjust1 = ((ra << 1) >> (bits + 1)) - rb 155 | // adjust2 = msb(ra) - rc 156 | // make 157 | // (adjust1 + rb) = (ra mod 2^(ring_size-1)) >> bits 158 | // (adjust2 + rc) = msb(ra) 159 | } 160 | 161 | message AdjustRandBitRequest { 162 | string session_id = 1; 163 | // input one prg buffer 164 | // reconstruct all parties' share get: ra 165 | repeated PrgBufferMeta prg_inputs = 2; 166 | // use which field to interprete buffer. details see: spu.FieldType 167 | int32 field = 3; 168 | // output 169 | // adjust_a = server generated random 0/1 array - rb 170 | // make 171 | // (adjust_a + ra) = random 0/1 array 172 | } 173 | 174 | message AdjustResponse { 175 | ErrorCode code = 1; 176 | string message = 2; 177 | // Adjust output array buffer 178 | repeated bytes adjust_outputs = 3; 179 | } 180 | -------------------------------------------------------------------------------- /interconnection/handshake/protocol_family/ecc.proto: -------------------------------------------------------------------------------- 1 | // Copyright 2023 Ant Group Co., Ltd. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | syntax = "proto3"; 16 | 17 | package org.interconnection.v2.protocol; 18 | 19 | //===================================// 20 | // Common defs // 21 | //===================================// 22 | 23 | // [Sphinx doc begin anchor: EcSuit] 24 | // Suit of 25 | message EcSuit { 26 | // ref enum CurveType 27 | int32 curve = 1; 28 | 29 | // ref enum HashType 30 | int32 hash = 2; 31 | 32 | // ref enum HashToCurveStrategy 33 | int32 hash2curve_strategy = 3; 34 | } 35 | // [Sphinx doc end anchor: EcSuit] 36 | 37 | enum CurveType { 38 | CURVE_TYPE_UNSPECIFIED = 0; 39 | CURVE_TYPE_CURVE25519 = 1; 40 | CURVE_TYPE_SM2 = 2; 41 | CURVE_TYPE_ED25519 = 3; 42 | } 43 | 44 | enum HashType { 45 | HASH_TYPE_UNSPECIFIED = 0; 46 | HASH_TYPE_SM3 = 1; 47 | 48 | HASH_TYPE_SHA_224 = 10; 49 | HASH_TYPE_SHA_256 = 11; 50 | HASH_TYPE_SHA_384 = 12; 51 | HASH_TYPE_SHA_512 = 13; 52 | 53 | HASH_TYPE_SHA3_224 = 20; 54 | HASH_TYPE_SHA3_256 = 21; 55 | HASH_TYPE_SHA3_384 = 22; 56 | HASH_TYPE_SHA3_512 = 23; 57 | 58 | HASH_TYPE_SHAKE_256 = 30; 59 | } 60 | 61 | enum HashToCurveStrategy { 62 | HASH_TO_CURVE_STRATEGY_UNSPECIFIED = 0; 63 | 64 | // https://eprint.iacr.org/2009/226.pdf 65 | // Auto select the most suitable algorithm: 66 | // - SHA2: select between SHA-224, SHA-256, SHA-384, SHA-512 67 | // - SHA3: select between SHA3-224, SHA3-256, SHA3-384, SHA3-512 68 | // - SM: Current only support SM3. 69 | // Performance: This method is very fast, but it is susceptible to timing 70 | // attacks. 71 | HASH_TO_CURVE_STRATEGY_TRY_AND_INCREMENT = 1; 72 | 73 | // Just like TryAndIncrement; but use re-hash instead of increment when try 74 | // fails. 75 | HASH_TO_CURVE_STRATEGY_TRY_AND_REHASH = 2; 76 | 77 | // Directly output the hash value as the x-coordinate of the point without any 78 | // verification. And there is no y-coordinate info in output point. 79 | 80 | // The applicable scenarios of this scheme are very limited, and the following 81 | // requirements must be met: 82 | // - The calculation of points on curve depends only on the x-coordinate 83 | // - The usage scenario of the curve allows any hash value to be used as the 84 | // initial point, even if the point is not on the curve. 85 | // It is currently known that this strategy can be safely used when curve is 86 | // Curve25519 and scene is ECDH. Do not choose this strategy for other 87 | // purpose. 88 | HASH_TO_CURVE_STRATEGY_DIRECT_HASH_AS_POINT_X = 3; 89 | 90 | // Below is IRTF CFRG hash-to-curve standard (draft): 91 | // https://datatracker.ietf.org/doc/draft-irtf-cfrg-hash-to-curve/ 92 | // 93 | // IRTF suits consist of: 94 | // "CURVE_ID || "_" || HASH_ID || "_" || MAP_ID || "_" || ENC_VAR" 95 | // We regard MAP_ID and ENC_VAR as a whole as HASH_TO_CURVE strategy. 96 | // MAP_ID has the following methods: 97 | // - "SVDW" for Shallue-van de Woestijne method 98 | // - "SSWU" for Simplified SWU 99 | // - "ELL2" for Elligator 2 method 100 | // ENC_VAR has the following methods: 101 | // - "RO" for hash_to_curve 102 | // Performance: This strategy takes 12 times longer than TryAndIncrement on 103 | // SM2 104 | // - "NU" for encode_to_curve 105 | // Performance: This strategy takes 6 times longer than TryAndIncrement on 106 | // SM2 107 | // Warning: The output of this strategy is not uniformly distributed on the 108 | // elliptic curve G. 109 | HASH_TO_CURVE_STRATEGY_IRTF_SSWU_RO = 10; 110 | HASH_TO_CURVE_STRATEGY_IRTF_SSWU_NU = 11; 111 | HASH_TO_CURVE_STRATEGY_IRTF_ELL2_RO = 12; 112 | HASH_TO_CURVE_STRATEGY_IRTF_ELL2_NU = 13; 113 | } 114 | 115 | // Legal EcSuit triplet collocations 116 | // 合法的 EcSuit 三元组搭配（开放列表） 117 | // 118 | // Recommended combinations: 119 | // 推荐搭配 120 | // +------------+---------+------------------------+ 121 | // | Curve | Hash | Hash2Curve | 122 | // +------------+---------+------------------------+ 123 | // | Curve25519 | SHA-256 | DIRECT_HASH_AS_POINT_X | 124 | // +------------+---------+------------------------+ 125 | // | SM2 | SM3 | TRY_AND_INCREMENT | 126 | // +------------+---------+------------------------+ 127 | // 128 | // Other combinations: 129 | // 其它搭配 130 | // Enterprises/platforms with spare energy can consider implementing the 131 | // following solutions, especially the IRTF strategies, which provides higher 132 | // security and is also recommended. 133 | // 有余力的企业/平台可以考虑实现下述方案，特别是 IRTF 方案提供更高的安全性，是非常有意义的 134 | // +------------+---------+------------------------+ 135 | // | Curve | Hash | Hash2Curve | 136 | // +------------+---------+------------------------+ 137 | // | Curve25519 | SHA-256 | DIRECT_HASH_AS_POINT_X | 138 | // +------------+---------+------------------------+ 139 | // | Curve25519 | SM3 | DIRECT_HASH_AS_POINT_X | 140 | // +------------+---------+------------------------+ 141 | // | SM2 | SHA-256 | TRY_AND_INCREMENT | 142 | // +------------+---------+------------------------+ 143 | // | SM2 | SHA-256 | TRY_AND_REHASH | 144 | // +------------+---------+------------------------+ 145 | // | SM2 | SM3 | TRY_AND_INCREMENT | 146 | // +------------+---------+------------------------+ 147 | // | SM2 | SM3 | TRY_AND_REHASH | 148 | // +------------+---------+------------------------+ 149 | // | SM2 | SM3 | IRTF_SSWU_RO | 150 | // +------------+---------+------------------------+ 151 | // | Curve25519 | SHA-256 | IRTF_ELL2_RO | 152 | // +------------+---------+------------------------+ 153 | 154 | // Elliptic curve point Octet-String format 155 | // See SECG standard for details: SEC 1, section 2.3. https://www.secg.org/ 156 | enum PointOctetFormat { 157 | POINT_OCTET_FORMAT_UNSPECIFIED = 0; 158 | 159 | // 不压缩，多个点连续存储，例如 x1||y1||x2||y2||x3||y3... 160 | // x 和 y 都占用整数个 byte， 161 | // 例如 255bits 的数字占用 32bytes, 小端序，buffer 最后1bit补0 162 | // 对于 X25529, 本身只有 X 坐标，则多个点序列化只排列 X 坐标 163 | POINT_OCTET_FORMAT_UNCOMPRESSED = 1; 164 | 165 | // ANSI X9.62 compressed format 166 | // The point is encoded as z||x, where the octet z specifies which solution of 167 | // the quadratic equation y is. 168 | // if y is even, output 0x02||x 169 | // if y is odd, output 0x03||x 170 | // if point is inf, output 0x00 171 | POINT_OCTET_FORMAT_X962_COMPRESSED = 2; 172 | 173 | // ANSI X9.62 uncompressed format 174 | // The point is encoded as z||x||y, where z is the octet 0x04 175 | POINT_OCTET_FORMAT_X962_UNCOMPRESSED = 3; 176 | 177 | // ANSI X9.62 hybrid format 178 | // The point is encoded as z||x||y, where the octet z specifies which solution 179 | // of the quadratic equation y is. It's basically the uncompressed encoding 180 | // but the first byte encodes the evenness of y just like in compressed 181 | // format. It's designated by 0x06 and 0x07 in the first byte, and they have 182 | // the same meaning as 0x02 and 0x03 in compressed. 183 | POINT_OCTET_FORMAT_X962_HYBRID = 4; 184 | }; 185 | 186 | //===================================// 187 | // Protos used in HandshakeRequest // 188 | //===================================// 189 | 190 | // [Sphinx doc begin anchor: EccProtocolProposal] 191 | message EccProtocolProposal { 192 | repeated int32 supported_versions = 1; 193 | 194 | // list of suits 195 | repeated EcSuit ec_suits = 2; 196 | 197 | // ref enum PointOctetFormat 198 | // 点的序列化格式 199 | repeated int32 point_octet_formats = 3; 200 | 201 | // Whether to enable the optimization method: secondary ciphertext truncation 202 | bool support_point_truncation = 4; 203 | } 204 | // [Sphinx doc end anchor: EccProtocolProposal] 205 | 206 | //===================================// 207 | // Protos used in HandshakeResponse // 208 | //===================================// 209 | 210 | // [Sphinx doc begin anchor: EccProtocolResult] 211 | message EccProtocolResult { 212 | int32 version = 1; 213 | 214 | // The chosen suit 215 | EcSuit ec_suit = 2; 216 | 217 | // The chosen octet format 218 | int32 point_octet_format = 3; 219 | 220 | // optimization method: secondary ciphertext truncation 221 | // -1 means disable this optimization (do not truncation) 222 | int32 bit_length_after_truncated = 4; 223 | } 224 | // [Sphinx doc end anchor: EccProtocolResult] 225 | -------------------------------------------------------------------------------- /interconnection/legacy/algos/psi.proto: -------------------------------------------------------------------------------- 1 | // Copyright 2022 Ant Group Co., Ltd. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | // This package only defines actual messages used in PSI. 16 | // 17 | // Goals: 18 | // - Extensible: this protocol should be able to support new psi algorithms in 19 | // the future. 20 | // 21 | // - Language Agnostic: we should not have any language-specific limitation 22 | // here, all languages, say Rust/Java/Go/C++ should be able to join this 23 | // interconnection protocol. 24 | // 25 | // Non-Goals: 26 | // - Communication Channel: In this file, we do not define the actual channel 27 | // level interconnection, say whether we should use http, grpc or message 28 | // queue. This topic will be discussed in the communication topic. 29 | // 30 | // Requirements for communication layer: 31 | // - Support (string type, byte[] content), self-describe the content type 32 | // - Error code system and has a way to abort the training flow 33 | // - Asynchoronous support 34 | 35 | syntax = "proto3"; 36 | 37 | import "google/protobuf/any.proto"; 38 | import "interconnection/common/header.proto"; 39 | 40 | package org.interconnection.algos.psi; 41 | 42 | // The initial handshake message after the connection mesh has been established. 43 | // 44 | // 初始握手请求 45 | // 46 | // Supported algorithms （支持的 PSI 算法）: 47 | // 48 | // - ecdh: The classic ECDH PSI. 49 | // 50 | // - bark-oprf*: TODO 51 | // 52 | // - pcg*: TODO 53 | // [Sphinx doc begin anchor: HandshakeRequest] 54 | message HandshakeRequest { 55 | 56 | // The version of psi handshake message. Start from 1. 57 | // 58 | // 握手请求版本号 59 | int32 version = 1; 60 | 61 | // Supported algorithms （支持的 PSI 算法）: ecdh, pcg*, bark-oprf* 62 | repeated string supported_algos = 2; 63 | 64 | // Corresponding psi algo parameters proposals. 65 | // 66 | // 相应的 PSI 算法详细握手参数，与 supported_algos 一一对应 67 | repeated google.protobuf.Any algo_params = 3; 68 | 69 | // How many items do I've. 70 | // 71 | // 待求交的 PSI 数据总量 72 | int64 item_num = 4; 73 | 74 | // Usually we need to make partitions for large-scale PSI. This field defines 75 | // the number of bucket we've proposed. Note the larger one will be chosen as 76 | // the final bucket number. 77 | // 78 | // 大规模数据（比如十亿）求交需要决定的分桶数，任何一方都可以给出分桶数，最后以大的一方为主。 79 | // 未使用 80 | int64 bucket_num = 5; 81 | 82 | // Which rank can receive the psi results. 83 | // 84 | // 确定 PSI 结果获取方。 85 | // 86 | // NOTES: 87 | // `-1`: all parties (所有机构都可以拿到交集结果) 88 | // `>= 0`: corresponding rank can get the results (指定机构拿到交集结果) 89 | int32 result_to_rank = 7; 90 | } 91 | // [Sphinx doc end anchor: HandshakeRequest] 92 | 93 | // [Sphinx doc begin anchor: HandshakeResponse] 94 | // The handshake response from peer. 95 | // 96 | // 对手的握手决策响应 97 | message HandshakeResponse { 98 | // response header 99 | ResponseHeader header = 1; 100 | 101 | // The final algo determined. 102 | // 103 | // 决策下来的 PSI 算法 104 | string algo = 2; 105 | 106 | // The number of items for each party. 107 | // 108 | // 每一方的待求交总数 109 | int64 item_count = 3; 110 | 111 | // The number of bucket. 112 | // 113 | // 实际使用的分桶数 114 | // 未使用 115 | int64 bucket_num = 4; 116 | 117 | // The final algorithm parameters. 118 | // 119 | // 决策出来的算法参数 120 | google.protobuf.Any algo_params = 6; 121 | } 122 | // [Sphinx doc end anchor: HandshakeResponse] 123 | 124 | // ------------------------------------------ 125 | // Roles definition: 126 | // 127 | // we use the MPI semantic roles here since we might to support 128 | // more than 2 parties in the future. 129 | // 130 | // For two party, we treat rank 0 (initialtor) as the client(guest) and rank 1 131 | // as the server(host). 132 | // 133 | // ------------------------------------------ 134 | // 135 | // TCP 136 | // - client 137 | // - server 138 | // 139 | // MPI 140 | // - rank 0 (client) 141 | // - rank 1 (server) 142 | // 143 | // --------------------------------------------- 144 | // 145 | // ECDH PSI Reference states: 146 | // 147 | // Rank 0 Rank 1 148 | // 149 | // ------------- STATE: INIT ------------------------------ 150 | // | 151 | // | 152 | // ------------- STATE: BUILD COMMUNICATION CHANNEL ------- 153 | // | 154 | // | 155 | // HandShake Request 156 | // <--------> 157 | // 158 | // HandShake HandShake 159 | // Decision Decision 160 | // 161 | // HandShake Response 162 | // <--------> 163 | // 164 | // ------------- STATE: HANDSHAKE COMPLETED ---------------- 165 | // 166 | // CIPHERTEXT BATCH CIPHERTEXT BATCH 167 | // 168 | // <--------> 169 | // 170 | // 171 | // ------------- STATE: CIPHERTEXT COMPLETED ---------------- 172 | // 173 | // Local compute the intersection and output results 174 | // 175 | // ------------- STATE: END ---------------- 176 | // 177 | 178 | // [Sphinx doc begin anchor: EcdhPsiParamsProposal] 179 | // ECDH PSI algorithm parameters proposal. 180 | // 181 | // ECDH PSI 算法参数握手请求 182 | message EcdhPsiParamsProposal { 183 | // supported versions，支持的算法版本列表 184 | // 当前必须为 1 185 | repeated int32 supported_versions = 1; 186 | 187 | // Supported curve types（ EC 曲线 ）: 188 | // 189 | // - curve25519; 190 | // - sm2; 191 | repeated string curves = 2; 192 | 193 | // Supported hash methods( 支持的哈希算法 ): 194 | // 与 curves 字段长度相同，元素一一对应 195 | // 196 | // - sha256; 197 | // - sm3; 198 | repeated string hash_methods = 3; 199 | 200 | // Nonce for hashing. 201 | // 202 | // 哈希的额外随机数 203 | string nonce = 4; 204 | } 205 | // [Sphinx doc end anchor: EcdhPsiParamsProposal] 206 | 207 | // [Sphinx doc begin anchor: EcdhPsiParamsResult] 208 | // ECDH PSI parameters that parties has agreed. 209 | // 210 | // ECDH PSI 算法结果参数 211 | message EcdhPsiParamsResult { 212 | // The psi version actual used. 213 | string version = 1; 214 | 215 | // The curve type actual used. 216 | string curve = 2; 217 | 218 | // The chosen hash method. 219 | string hash_method = 3; 220 | 221 | // The chosen salt used in hash method, i.e. hash(content || nonce). 222 | string nonce = 4; 223 | } 224 | // [Sphinx doc end anchor: EcdhPsiParamsResult] 225 | 226 | // [Sphinx doc begin anchor: EcdhPsiCipherBatch] 227 | // The universal ciphertext for each batch. 228 | // 229 | // ECDH PSI 密文传输 230 | message EcdhPsiCipherBatch { 231 | 232 | // The type hint for each message. (密文类型) 233 | // 234 | // "enc": the first stage ciphertext 235 | // 236 | // "dual.enc": the second stage ciphertext 237 | // 238 | // ECDH PSI 密文阶段类型，主要用来区分一阶段和二阶段的密文. 239 | string type = 1; 240 | 241 | // The bucket index. Start from 0. 242 | // 243 | // Bucket 索引 244 | // 未使用 245 | int32 bucket_index = 2; 246 | 247 | // The batch index. Start from 0. 248 | // 249 | // Batch 索引，从 0 开始 250 | int32 batch_index = 3; 251 | 252 | // Is last batch flag 253 | bool is_last_batch = 4; 254 | 255 | // Count of items in this batch. 256 | // count == 0 is allowed for last batch 257 | int32 count = 6; 258 | 259 | // The packed all in one ciphertext for this batch. 260 | // 261 | // The first stage ciphertext takes 256 bits for each ciphertext element. 262 | // However, the second stage ciphertext takes 96 bits each. According to PSI 263 | // papers, we do not need to send all 256 bit for the final ciphertext. The 264 | // number of bits needed to compare is `Log(MN) + 40` given a 40 bits 265 | // statistical security parameter. TODO (add paper link here). 266 | // 267 | // We define each bucket has less than 2^28 items, i.e. about 270 million 268 | // (单桶最多 2.7亿) items, which is general enough for various psi algorithms. 269 | // 270 | // NOTE: we do not use `repeated`` here to save overhead of metadata. 271 | bytes ciphertext = 7; 272 | 273 | // The extra count map records the number of repetitions of the element at 274 | // this batch. 275 | // 276 | // Because PSI protocal does not support intersection with 277 | // duplicate key values, we use this field to represent the additional number 278 | // of repetitions of the element at this index. The KEY value is the index of 279 | // the element in the ciphertext that appears more than once, and the VALUE is 280 | // the number of times the element appears minus one. 281 | // 282 | // For example, if the elements are [1, 2, 3, 3, 4, 5, 5, 5], the ciphertext 283 | // is [Enc(1), Enc(2), Enc(3), Enc(4), Enc(5)], then the 284 | // duplicate_item_cnt_map would be {2: 1, 4: 2}. 285 | map duplicate_item_cnt_map = 8; 286 | } 287 | // [Sphinx doc end anchor: EcdhPsiCipherBatch] 288 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Apache License 2 | Version 2.0, January 2004 3 | http://www.apache.org/licenses/ 4 | 5 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 6 | 7 | 1. Definitions. 8 | 9 | "License" shall mean the terms and conditions for use, reproduction, 10 | and distribution as defined by Sections 1 through 9 of this document. 11 | 12 | "Licensor" shall mean the copyright owner or entity authorized by 13 | the copyright owner that is granting the License. 14 | 15 | "Legal Entity" shall mean the union of the acting entity and all 16 | other entities that control, are controlled by, or are under common 17 | control with that entity. For the purposes of this definition, 18 | "control" means (i) the power, direct or indirect, to cause the 19 | direction or management of such entity, whether by contract or 20 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 21 | outstanding shares, or (iii) beneficial ownership of such entity. 22 | 23 | "You" (or "Your") shall mean an individual or Legal Entity 24 | exercising permissions granted by this License. 25 | 26 | "Source" form shall mean the preferred form for making modifications, 27 | including but not limited to software source code, documentation 28 | source, and configuration files. 29 | 30 | "Object" form shall mean any form resulting from mechanical 31 | transformation or translation of a Source form, including but 32 | not limited to compiled object code, generated documentation, 33 | and conversions to other media types. 34 | 35 | "Work" shall mean the work of authorship, whether in Source or 36 | Object form, made available under the License, as indicated by a 37 | copyright notice that is included in or attached to the work 38 | (an example is provided in the Appendix below). 39 | 40 | "Derivative Works" shall mean any work, whether in Source or Object 41 | form, that is based on (or derived from) the Work and for which the 42 | editorial revisions, annotations, elaborations, or other modifications 43 | represent, as a whole, an original work of authorship. For the purposes 44 | of this License, Derivative Works shall not include works that remain 45 | separable from, or merely link (or bind by name) to the interfaces of, 46 | the Work and Derivative Works thereof. 47 | 48 | "Contribution" shall mean any work of authorship, including 49 | the original version of the Work and any modifications or additions 50 | to that Work or Derivative Works thereof, that is intentionally 51 | submitted to Licensor for inclusion in the Work by the copyright owner 52 | or by an individual or Legal Entity authorized to submit on behalf of 53 | the copyright owner. For the purposes of this definition, "submitted" 54 | means any form of electronic, verbal, or written communication sent 55 | to the Licensor or its representatives, including but not limited to 56 | communication on electronic mailing lists, source code control systems, 57 | and issue tracking systems that are managed by, or on behalf of, the 58 | Licensor for the purpose of discussing and improving the Work, but 59 | excluding communication that is conspicuously marked or otherwise 60 | designated in writing by the copyright owner as "Not a Contribution." 61 | 62 | "Contributor" shall mean Licensor and any individual or Legal Entity 63 | on behalf of whom a Contribution has been received by Licensor and 64 | subsequently incorporated within the Work. 65 | 66 | 2. Grant of Copyright License. Subject to the terms and conditions of 67 | this License, each Contributor hereby grants to You a perpetual, 68 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 69 | copyright license to reproduce, prepare Derivative Works of, 70 | publicly display, publicly perform, sublicense, and distribute the 71 | Work and such Derivative Works in Source or Object form. 72 | 73 | 3. Grant of Patent License. Subject to the terms and conditions of 74 | this License, each Contributor hereby grants to You a perpetual, 75 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 76 | (except as stated in this section) patent license to make, have made, 77 | use, offer to sell, sell, import, and otherwise transfer the Work, 78 | where such license applies only to those patent claims licensable 79 | by such Contributor that are necessarily infringed by their 80 | Contribution(s) alone or by combination of their Contribution(s) 81 | with the Work to which such Contribution(s) was submitted. If You 82 | institute patent litigation against any entity (including a 83 | cross-claim or counterclaim in a lawsuit) alleging that the Work 84 | or a Contribution incorporated within the Work constitutes direct 85 | or contributory patent infringement, then any patent licenses 86 | granted to You under this License for that Work shall terminate 87 | as of the date such litigation is filed. 88 | 89 | 4. Redistribution. You may reproduce and distribute copies of the 90 | Work or Derivative Works thereof in any medium, with or without 91 | modifications, and in Source or Object form, provided that You 92 | meet the following conditions: 93 | 94 | (a) You must give any other recipients of the Work or 95 | Derivative Works a copy of this License; and 96 | 97 | (b) You must cause any modified files to carry prominent notices 98 | stating that You changed the files; and 99 | 100 | (c) You must retain, in the Source form of any Derivative Works 101 | that You distribute, all copyright, patent, trademark, and 102 | attribution notices from the Source form of the Work, 103 | excluding those notices that do not pertain to any part of 104 | the Derivative Works; and 105 | 106 | (d) If the Work includes a "NOTICE" text file as part of its 107 | distribution, then any Derivative Works that You distribute must 108 | include a readable copy of the attribution notices contained 109 | within such NOTICE file, excluding those notices that do not 110 | pertain to any part of the Derivative Works, in at least one 111 | of the following places: within a NOTICE text file distributed 112 | as part of the Derivative Works; within the Source form or 113 | documentation, if provided along with the Derivative Works; or, 114 | within a display generated by the Derivative Works, if and 115 | wherever such third-party notices normally appear. The contents 116 | of the NOTICE file are for informational purposes only and 117 | do not modify the License. You may add Your own attribution 118 | notices within Derivative Works that You distribute, alongside 119 | or as an addendum to the NOTICE text from the Work, provided 120 | that such additional attribution notices cannot be construed 121 | as modifying the License. 122 | 123 | You may add Your own copyright statement to Your modifications and 124 | may provide additional or different license terms and conditions 125 | for use, reproduction, or distribution of Your modifications, or 126 | for any such Derivative Works as a whole, provided Your use, 127 | reproduction, and distribution of the Work otherwise complies with 128 | the conditions stated in this License. 129 | 130 | 5. Submission of Contributions. Unless You explicitly state otherwise, 131 | any Contribution intentionally submitted for inclusion in the Work 132 | by You to the Licensor shall be under the terms and conditions of 133 | this License, without any additional terms or conditions. 134 | Notwithstanding the above, nothing herein shall supersede or modify 135 | the terms of any separate license agreement you may have executed 136 | with Licensor regarding such Contributions. 137 | 138 | 6. Trademarks. This License does not grant permission to use the trade 139 | names, trademarks, service marks, or product names of the Licensor, 140 | except as required for reasonable and customary use in describing the 141 | origin of the Work and reproducing the content of the NOTICE file. 142 | 143 | 7. Disclaimer of Warranty. Unless required by applicable law or 144 | agreed to in writing, Licensor provides the Work (and each 145 | Contributor provides its Contributions) on an "AS IS" BASIS, 146 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 147 | implied, including, without limitation, any warranties or conditions 148 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 149 | PARTICULAR PURPOSE. You are solely responsible for determining the 150 | appropriateness of using or redistributing the Work and assume any 151 | risks associated with Your exercise of permissions under this License. 152 | 153 | 8. Limitation of Liability. In no event and under no legal theory, 154 | whether in tort (including negligence), contract, or otherwise, 155 | unless required by applicable law (such as deliberate and grossly 156 | negligent acts) or agreed to in writing, shall any Contributor be 157 | liable to You for damages, including any direct, indirect, special, 158 | incidental, or consequential damages of any character arising as a 159 | result of this License or out of the use or inability to use the 160 | Work (including but not limited to damages for loss of goodwill, 161 | work stoppage, computer failure or malfunction, or any and all 162 | other commercial damages or losses), even if such Contributor 163 | has been advised of the possibility of such damages. 164 | 165 | 9. Accepting Warranty or Additional Liability. While redistributing 166 | the Work or Derivative Works thereof, You may choose to offer, 167 | and charge a fee for, acceptance of support, warranty, indemnity, 168 | or other liability obligations and/or rights consistent with this 169 | License. However, in accepting such obligations, You may act only 170 | on Your own behalf and on Your sole responsibility, not on behalf 171 | of any other Contributor, and only if You agree to indemnify, 172 | defend, and hold each Contributor harmless for any liability 173 | incurred by, or claims asserted against, such Contributor by reason 174 | of your accepting any such warranty or additional liability. 175 | 176 | END OF TERMS AND CONDITIONS 177 | 178 | APPENDIX: How to apply the Apache License to your work. 179 | 180 | To apply the Apache License to your work, attach the following 181 | boilerplate notice, with the fields enclosed by brackets "[]" 182 | replaced with your own identifying information. (Don't include 183 | the brackets!) The text should be enclosed in the appropriate 184 | comment syntax for the file format. We also recommend that a 185 | file or class name and description of purpose be included on the 186 | same "printed page" as the copyright notice for easier 187 | identification within third-party archives. 188 | 189 | Copyright [yyyy] [name of copyright owner] 190 | 191 | Licensed under the Apache License, Version 2.0 (the "License"); 192 | you may not use this file except in compliance with the License. 193 | You may obtain a copy of the License at 194 | 195 | http://www.apache.org/licenses/LICENSE-2.0 196 | 197 | Unless required by applicable law or agreed to in writing, software 198 | distributed under the License is distributed on an "AS IS" BASIS, 199 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 200 | See the License for the specific language governing permissions and 201 | limitations under the License. 202 | -------------------------------------------------------------------------------- /docs/protocols/ss-lr.rst: -------------------------------------------------------------------------------- 1 | SS-LR 协议 2 | ========================= 3 | 4 | 算法流程 5 | ------------------------ 6 | 7 | 算法分为2阶段，第一阶段为握手过程，第二阶段为算法主体，其流程如下： 8 | 9 | .. image:: _img/ss-lr-flow.png 10 | :scale: 60% 11 | :align: center 12 | 13 | 14 | 握手过程 15 | ^^^^^^^^^^^^^^^^^^^ 16 | 17 | 握手协议 18 | """""""""""""""""""""""""""""" 19 | 20 | 握手所用的 HandshakeRequest 定义如下： 21 | 22 | .. literalinclude:: ../../interconnection/handshake/entry.proto 23 | :caption: interconnection/handshake/entry.proto 24 | :language: Protobuf 25 | :start-after: [Sphinx doc begin anchor: HandshakeRequest] 26 | :end-before: [Sphinx doc end anchor: HandshakeRequest] 27 | :linenos: 28 | 29 | HandshakeRequest 主要包括以下信息： 30 | 31 | 1. 协议版本号 32 | 2. 请求方的传输层 rank 值 33 | 3. 想使用的具体算法，比如使用 SS-LR 34 | 4. 每类算法的详细参数，比如 SS-LR 需要说明优化器类型、正则项等参数 35 | 5. 用到的安全算子的类型，比如 Sigmoid 36 | 6. 每个安全算子的详细参数，比如 Sigmoid 需要说明采用的拟合算法 37 | 7. 用到的密码协议族，比如 SS 协议族 38 | 8. 每个协议族的详细参数，比如 SS 协议族需要说明协议种类、整数环比特数等参数 39 | 9. 算法的输入和结果输出格式，比如 SS-LR 需要说明样本数量、特征数量等参数 40 | 41 | HandshakeRequest 中的 supported_algos 字段的定义如下： 42 | 43 | .. literalinclude:: ../../interconnection/handshake/entry.proto 44 | :caption: interconnection/handshake/entry.proto 45 | :language: Protobuf 46 | :start-after: [Sphinx doc begin anchor: AlgoType] 47 | :end-before: [Sphinx doc end anchor: AlgoType] 48 | :linenos: 49 | 50 | 如果算法是 SS-LR，则 HandshakeRequest 中的 algo_params 字段格式如下： 51 | 52 | .. literalinclude:: ../../interconnection/handshake/algos/lr.proto 53 | :caption: interconnection/handshake/algos/lr.proto 54 | :language: Protobuf 55 | :start-after: [Sphinx doc begin anchor: LrHyperparamsProposal] 56 | :end-before: [Sphinx doc end anchor: LrHyperparamsProposal] 57 | :linenos: 58 | 59 | LrHyperparamsProposal 中的 optimizers 字段的定义如下： 60 | 61 | .. literalinclude:: ../../interconnection/handshake/algos/optimizer.proto 62 | :caption: interconnection/handshake/algos/optimizer.proto 63 | :language: Protobuf 64 | :start-after: [Sphinx doc begin anchor: Optimizer] 65 | :end-before: [Sphinx doc end anchor: Optimizer] 66 | :linenos: 67 | 68 | LrHyperparamsProposal 中的 last_batch_policies 字段的定义如下: 69 | 70 | .. literalinclude:: ../../interconnection/handshake/algos/lr.proto 71 | :caption: interconnection/handshake/algos/lr.proto 72 | :language: Protobuf 73 | :start-after: [Sphinx doc begin anchor: LastBatchPolicy] 74 | :end-before: [Sphinx doc end anchor: LastBatchPolicy] 75 | :linenos: 76 | 77 | HandshakeRequest 中的 ops 字段的定义如下： 78 | 79 | .. literalinclude:: ../../interconnection/handshake/entry.proto 80 | :caption: interconnection/handshake/entry.proto 81 | :language: Protobuf 82 | :start-after: [Sphinx doc begin anchor: OpType] 83 | :end-before: [Sphinx doc end anchor: OpType] 84 | :linenos: 85 | 86 | 如果安全算子类型是 Sigmoid，则 HandshakeRequest 中的 op_params 字段格式如下： 87 | 88 | .. literalinclude:: ../../interconnection/handshake/op/sigmoid.proto 89 | :caption: interconnection/handshake/op/sigmoid.proto 90 | :language: Protobuf 91 | :start-after: [Sphinx doc begin anchor: SigmoidParamsProposal] 92 | :end-before: [Sphinx doc end anchor: SigmoidParamsProposal] 93 | :linenos: 94 | 95 | SigmoidParamsProposal 中的 sigmoid_modes 字段的定义如下： 96 | 97 | .. literalinclude:: ../../interconnection/handshake/op/sigmoid.proto 98 | :caption: interconnection/handshake/op/sigmoid.proto 99 | :language: Protobuf 100 | :start-after: [Sphinx doc begin anchor: SigmoidMode] 101 | :end-before: [Sphinx doc end anchor: SigmoidMode] 102 | :linenos: 103 | 104 | HandshakeRequest 中的 protocol_families 字段的定义如下： 105 | 106 | .. literalinclude:: ../../interconnection/handshake/entry.proto 107 | :caption: interconnection/handshake/entry.proto 108 | :language: Protobuf 109 | :start-after: [Sphinx doc begin anchor: ProtocolFamily] 110 | :end-before: [Sphinx doc end anchor: ProtocolFamily] 111 | :linenos: 112 | 113 | 如果协议族是 SS，则 HandshakeRequest 中的 protocol_family_params 字段格式如下： 114 | 115 | .. literalinclude:: ../../interconnection/handshake/protocol_family/ss.proto 116 | :caption: interconnection/handshake/protocol_family/ss.proto 117 | :language: Protobuf 118 | :start-after: [Sphinx doc begin anchor: SSProtocolProposal] 119 | :end-before: [Sphinx doc end anchor: SSProtocolProposal] 120 | :linenos: 121 | 122 | SSProtocolProposal 中的 supported_protocols 字段的定义如下： 123 | 124 | .. literalinclude:: ../../interconnection/handshake/protocol_family/ss.proto 125 | :caption: interconnection/handshake/protocol_family/ss.proto 126 | :language: Protobuf 127 | :start-after: [Sphinx doc begin anchor: ProtocolKind] 128 | :end-before: [Sphinx doc end anchor: ProtocolKind] 129 | :linenos: 130 | 131 | SSProtocolProposal 中的 field_types 字段的定义如下： 132 | 133 | .. literalinclude:: ../../interconnection/handshake/protocol_family/ss.proto 134 | :caption: interconnection/handshake/protocol_family/ss.proto 135 | :language: Protobuf 136 | :start-after: [Sphinx doc begin anchor: FieldType] 137 | :end-before: [Sphinx doc end anchor: FieldType] 138 | :linenos: 139 | 140 | SSProtocolProposal 中的 trunc_modes 字段的定义如下： 141 | 142 | .. literalinclude:: ../../interconnection/handshake/protocol_family/ss.proto 143 | :caption: interconnection/handshake/protocol_family/ss.proto 144 | :language: Protobuf 145 | :start-after: [Sphinx doc begin anchor: TruncationModeProposal] 146 | :end-before: [Sphinx doc end anchor: TruncationModeProposal] 147 | :linenos: 148 | 149 | TruncationModeProposal 中的 method 字段的定义如下： 150 | 151 | .. literalinclude:: ../../interconnection/handshake/protocol_family/ss.proto 152 | :caption: interconnection/handshake/protocol_family/ss.proto 153 | :language: Protobuf 154 | :start-after: [Sphinx doc begin anchor: TruncMode] 155 | :end-before: [Sphinx doc end anchor: TruncMode] 156 | :linenos: 157 | 158 | SSProtocolProposal 中的 prg_configs 字段的定义如下： 159 | 160 | .. literalinclude:: ../../interconnection/handshake/protocol_family/ss.proto 161 | :caption: interconnection/handshake/protocol_family/ss.proto 162 | :language: Protobuf 163 | :start-after: [Sphinx doc begin anchor: PrgConfigProposal] 164 | :end-before: [Sphinx doc end anchor: PrgConfigProposal] 165 | :linenos: 166 | 167 | PrgConfigProposal 中的 crypto_type 字段的定义如下： 168 | 169 | .. literalinclude:: ../../interconnection/handshake/protocol_family/ss.proto 170 | :caption: interconnection/handshake/protocol_family/ss.proto 171 | :language: Protobuf 172 | :start-after: [Sphinx doc begin anchor: CryptoType] 173 | :end-before: [Sphinx doc end anchor: CryptoType] 174 | :linenos: 175 | 176 | SSProtocolProposal 中的 shard_serialize_formats 字段的定义如下： 177 | 178 | .. literalinclude:: ../../interconnection/handshake/protocol_family/ss.proto 179 | :caption: interconnection/handshake/protocol_family/ss.proto 180 | :language: Protobuf 181 | :start-after: [Sphinx doc begin anchor: ShardSerializeFormat] 182 | :end-before: [Sphinx doc end anchor: ShardSerializeFormat] 183 | :linenos: 184 | 185 | SSProtocolProposal 中的 triple_configs 字段的定义如下： 186 | 187 | .. literalinclude:: ../../interconnection/handshake/protocol_family/ss.proto 188 | :caption: interconnection/handshake/protocol_family/ss.proto 189 | :language: Protobuf 190 | :start-after: [Sphinx doc begin anchor: TripleConfigProposal] 191 | :end-before: [Sphinx doc end anchor: TripleConfigProposal] 192 | :linenos: 193 | 194 | 如果算法是 SS-LR，则 HandshakeRequest 中的 io_param 字段格式如下： 195 | 196 | .. literalinclude:: ../../interconnection/handshake/algos/lr.proto 197 | :caption: interconnection/handshake/algos/lr.proto 198 | :language: Protobuf 199 | :start-after: [Sphinx doc begin anchor: LrDataIoProposal] 200 | :end-before: [Sphinx doc end anchor: LrDataIoProposal] 201 | :linenos: 202 | 203 | 握手请求的结果 HandshakeResponse 定义如下： 204 | 205 | .. literalinclude:: ../../interconnection/handshake/entry.proto 206 | :caption: interconnection/handshake/entry.proto 207 | :language: Protobuf 208 | :start-after: [Sphinx doc begin anchor: HandshakeResponse] 209 | :end-before: [Sphinx doc end anchor: HandshakeResponse] 210 | :linenos: 211 | 212 | 其中 ResponseHeader 定义如下： 213 | 214 | .. literalinclude:: ../../interconnection/common/header.proto 215 | :caption: interconnection/common/header.proto 216 | :language: Protobuf 217 | :start-after: [Sphinx doc begin anchor: ResponseHeader] 218 | :end-before: [Sphinx doc end anchor: ResponseHeader] 219 | :linenos: 220 | 221 | 如果算法是 SS-LR，则 HandshakeResponse 中的 algo_param 字段格式如下： 222 | 223 | .. literalinclude:: ../../interconnection/handshake/algos/lr.proto 224 | :caption: interconnection/handshake/algos/lr.proto 225 | :language: Protobuf 226 | :start-after: [Sphinx doc begin anchor: LrHyperparamsResult] 227 | :end-before: [Sphinx doc end anchor: LrHyperparamsResult] 228 | :linenos: 229 | 230 | 如果 LR 优化器类型是 SGD，则 HandshakeResponse 中的 optimizer_param 字段格式如下： 231 | 232 | .. literalinclude:: ../../interconnection/handshake/algos/optimizer.proto 233 | :caption: interconnection/handshake/algos/optimizer.proto 234 | :language: Protobuf 235 | :start-after: [Sphinx doc begin anchor: SgdOptimizer] 236 | :end-before: [Sphinx doc end anchor: SgdOptimizer] 237 | :linenos: 238 | 239 | 如果安全算子类型是 Sigmoid，则 HandshakeResponse 中的 op_params 字段格式如下： 240 | 241 | .. literalinclude:: ../../interconnection/handshake/op/sigmoid.proto 242 | :caption: interconnection/handshake/op/sigmoid.proto 243 | :language: Protobuf 244 | :start-after: [Sphinx doc begin anchor: SigmoidParamsResult] 245 | :end-before: [Sphinx doc end anchor: SigmoidParamsResult] 246 | :linenos: 247 | 248 | 如果协议族是 SS，则 HandshakeResponse 中的 protocol_family_params 字段格式如下： 249 | 250 | .. literalinclude:: ../../interconnection/handshake/protocol_family/ss.proto 251 | :caption: interconnection/handshake/protocol_family/ss.proto 252 | :language: Protobuf 253 | :start-after: [Sphinx doc begin anchor: SSProtocolResult] 254 | :end-before: [Sphinx doc end anchor: SSProtocolResult] 255 | :linenos: 256 | 257 | SSProtocolResult 中的 trunc_mode 字段的定义如下： 258 | 259 | .. literalinclude:: ../../interconnection/handshake/protocol_family/ss.proto 260 | :caption: interconnection/handshake/protocol_family/ss.proto 261 | :language: Protobuf 262 | :start-after: [Sphinx doc begin anchor: TruncationModeResult] 263 | :end-before: [Sphinx doc end anchor: TruncationModeResult] 264 | :linenos: 265 | 266 | SSProtocolResult 中的 prg_config 字段的定义如下： 267 | 268 | .. literalinclude:: ../../interconnection/handshake/protocol_family/ss.proto 269 | :caption: interconnection/handshake/protocol_family/ss.proto 270 | :language: Protobuf 271 | :start-after: [Sphinx doc begin anchor: PrgConfigResult] 272 | :end-before: [Sphinx doc end anchor: PrgConfigResult] 273 | :linenos: 274 | 275 | SSProtocolResult 中的 triple_config 字段的定义如下： 276 | 277 | .. literalinclude:: ../../interconnection/handshake/protocol_family/ss.proto 278 | :caption: interconnection/handshake/protocol_family/ss.proto 279 | :language: Protobuf 280 | :start-after: [Sphinx doc begin anchor: TripleConfigResult] 281 | :end-before: [Sphinx doc end anchor: TripleConfigResult] 282 | :linenos: 283 | 284 | 如果算法是 SS-LR，则 HandshakeResponse 中的 io_param 字段格式如下： 285 | 286 | .. literalinclude:: ../../interconnection/handshake/algos/lr.proto 287 | :caption: interconnection/handshake/algos/lr.proto 288 | :language: Protobuf 289 | :start-after: [Sphinx doc begin anchor: LrHyperparamsResult] 290 | :end-before: [Sphinx doc end anchor: LrHyperparamsResult] 291 | :linenos: 292 | 293 | 294 | 295 | 296 | Protobuf 传输方式 297 | """""""""""""""""""""""""""""" 298 | 299 | Protobuf 传输使用《传输层白盒互联互通协议》中的 P2P 传输协议进行传输。其中传输的 key 按照《传输层白盒互联互通协议》中定义的方法生成， 300 | value 即为 protobuf 序列化之后的二进制字符串。 301 | 302 | 303 | 算法主体 304 | ^^^^^^^^^^^^^^^^^^^ 305 | 306 | 在协议主体运行阶段，SS-LR 基于 mini-batch 梯度下降训练方法，对于每一个 batch，协议的执行过程中，底层都是用 SPDZ Semi2k 秘密分享协议， 307 | 以确保计算过程中每一个变量都处在 Arithmetic Sharing 状态，每一方仅能看到本地分片值，看不到明文值，保证了算法执行过程中没有泄露原始信息。 308 | 309 | LR 算法训练过程 310 | """""""""""""""""""""""""""""" 311 | 312 | LR 算法公式为： 313 | 314 | .. math:: y = sigmoid(x_0w_0 + x_1w_1 + ⋯ + x_kw_k + intercept) 315 | 316 | 317 | .. math::y = sigmoid(x_0w_0 + x_1 w_1 + ⋯ + x_kw_k + intercept) 318 | 319 | 其中 :math:`x` 表示特征项，:math:`w` 是 LR 模型的权重，:math:`intercept` 是截距项， :math:`w` 和 :math:`intercept` 都是可训练的参数， 320 | :math:`sigmoid` 是一个 S 型函数，公式如下： 321 | 322 | .. math:: sigmoid(x)=1/(1+e^{-x}) 323 | 324 | 325 | 在隐私计算场景下，Sigmoid 函数直接计算有困难，本文件使用近似函数计算的方案，详见 :ref:`Sigmoid算子 ` 326 | 327 | 328 | 此外，为了统一 :math:`w` 和 :math:`intercept` 的形式，简化计算流程，我们将 :math:`intercept` 也看作是 :math:`w` 向量的一个元素， 329 | 即假设特征数为 :math:`k` 个，则 :math:`w` 向量的长度为 :math:`k+1`，之后在计算时，我们在 :math:`x` 的最后拼接一个 :math:`1`， 330 | 即可直接计算向量内积 :math:`w*x`， 331 | 简化计算步骤。 332 | 333 | 在实际训练时，LR 模型训练采用 mini-batch 梯度下降方法，重复执行如下五个步骤，直到达到目标迭代次数： 334 | 335 | 第一步：计算预测值 :math:`pred=sigmoid(batch\_x*w)` 336 | 337 | 第二步：计算误差 :math:`err=pred-y` 338 | 339 | 第三步：计算梯度 :math:`grad=transpose(batch\_x)*err` 340 | 341 | 第四步：梯度加正则项 :math:`grad=grad + w^{'}*l2\_norm` 342 | 343 | 第五步：更新模型参数 :math:`w=w-(grad*learning\_rate/batch\_size)` 344 | 345 | 其中： 346 | 347 | - :math:`batch\_x` 是当前 batch 的样本数据集与常数项组成的矩阵，其中常数项为 :math:`1`，位于 :math:`batch\_x` 的最后一列； 348 | - :math:`w` 是 LR 模型训练的结果，初始值为 :math:`0`； 349 | - :math:`sigmoid()` 是常被用作神经网络的激活函数，将变量映射到 :math:`0`, :math:`1` 之间； 350 | - :math:`*` 是乘法； 351 | - :math:`y` 是当前 batch 的标签矩阵； 352 | - :math:`transpose()` 是矩阵转置操作； 353 | - :math:`w^{'}` 是将 :math:`w` 的截距项即最后一个元素置为 :math:`0`； 354 | - :math:`l2\_norm` 是 l2 正则项； 355 | - :math:`learning\_rate` 是学习率参数； 356 | - :math:`batch\_size` 是每个 batch 的大小。 357 | 358 | 基于 Semi2K 的纵向 SS-LR 算法 359 | """""""""""""""""""""""""""""" 360 | 361 | 这里描述基于 Semi2K 协议的两方秘密分享纵向 LR 算法的计算过程。 362 | 363 | 数据集初始化 364 | ***************** 365 | 366 | 各参与方首先对输入的样本数据 :math:`batch\_x` 和标签矩阵 :math:`y` 进行编码和秘密分片。 367 | 368 | **输入数据编码** 369 | 370 | 输入数据集的元素类型为浮点数类型，将其按统一的放大系数转为定点数，然后去掉小数部分转为整数，在整数环上计算。编码方式如下表所示， 371 | 整数环可取长度 64、128 等，元素编码后的数据类型分别为 int64_t、int128_t。放大系数和整数环的比特数在握手协议中约定。 372 | 373 | .. list-table:: 输入数据集元素编码 374 | :widths: 25 25 25 375 | :header-rows: 1 376 | 377 | * - 实际数据类型 378 | - 整数环的比特数 379 | - 编码数据类型（ISO C99） 380 | * - 浮点数 381 | - 64 382 | - int64_t 383 | * - 浮点数 384 | - 128 385 | - int128_t 386 | 387 | **输入数据秘密分享** 388 | 389 | 双方根据握手协议确定的样本数量、特征数量和标签信息，以及各自的输入样本数据集，首先通过秘密分享的方式计算出初始密态样本数据集。 390 | 对于每个样本元素 :math:`x`，秘密分享获得两个密态分片 :math:`x` 和 :math:`0` ( :math:`x=x+0` )，将 :math:`x` 留在本地，将 :math:`0` 发给对方 391 | （由于 :math:`0` 不是随机数，所以省略这个发送过程）。 392 | 393 | 若样本总数量等于5，样本特征数量等于数组 [2, 3]，参与方 A（rank=0）拥有2个维度的特征和5个样本的标签，参与方 B（rank=1）拥有3个维度的特征。 394 | 395 | 双方对样本数据集进行秘密分享。参与方 A 初始化秘密分享的结果如下图： 396 | 397 | .. image:: _img/dataset-ss-a.png 398 | :scale: 60% 399 | :align: center 400 | 401 | 参与方 B 初始化秘密分享的结果如下图： 402 | 403 | .. image:: _img/dataset-ss-b.png 404 | :scale: 60% 405 | :align: center 406 | 407 | **交换伪随机数种子** 408 | 409 | 双方各有一个属于自己的伪随机数种子，用于将公共明文转为密文，详见 :ref:`公共明文转为密文 `。初始阶段双方向对方发送这个伪随机数种子。 410 | 411 | **向Beaver服务发送伪随机数种子** 412 | 413 | 除了上一节的伪随机数种子，双方还各自持有另一个伪随机数种子，用于从第三方可信 Beaver 服务获取乘法三元组，详见 :ref:`beaver`。初始阶段双方分别向 Beaver 服务发送自己的这个伪随机数种子。 414 | 415 | .. _drbg: 416 | 417 | 伪随机数生成方式 418 | ***************** 419 | 420 | 密码学安全的伪随机数生成器应当按照美国 NIST SP800-90A 标准或者中华人民共和国密码行业标准《GM/T 0105-2021软件随机数发生器设计指南》中规定的方式实现。伪随机数的安全强度要达到128比特。 421 | 422 | 安全算子 423 | ***************** 424 | 425 | .. _sigmoid: 426 | 427 | **Sigmoid算子** 428 | 429 | 在 Semi2K 协议下无法直接计算 Sigmoid 函数，本文件使用近似计算方案。Sigmoid有多种拟合算法，其中一种较为简单，并且不需要交互的近似计算方法如下： 430 | 431 | Minimax逼近方法，其公式为：:math:`f(X)=0.5+0.125*X`。 432 | 433 | **Semi2k算子** 434 | 435 | .. _p2a: 436 | 437 | 1. 公共明文转为密文 438 | 439 | 将 :math:`batch\_x` 常数项 :math:`1` 转为密文，将 :math:`w^{'}` 的截距项 :math:`0` 转为密文，这两处涉及公共明文转为密文。 440 | 441 | 假设公共明文等于 :math:`p`，参与方 A 使用自己的伪随机数种子生成随机数 :math:`r_0`，B 使用自己的伪随机数种子生成随机数 :math:`r_1`， 442 | 生成方式参照 :ref:`drbg`。由于双方都知道对方的伪随机数种子，且双方维护的计数器始终保持一致，所以可以计算出对方生成的伪随机数。 443 | 最终，A 的密态分片等于 :math:`p+r_0-r_1`，B 的密态分片等于 :math:`r_1-r_0`。 444 | 445 | 2. 明文常量 × 密文向量 446 | 447 | Sigmoid 的 Minimax 逼近公式和梯度加正则项公式中涉及明文常量与密文向量相乘，即 :math:`0.125*X`，:math:`w^{'}*l2\_norm`。 448 | 此算子不涉及参与方之间的通信，双方各自在本地将明文常量与密文分片直接相乘即可。 449 | 450 | 3. 明文常量 + 密文向量 451 | 452 | Sigmoid 的 Minimax 逼近公式中涉及明文常量与密文向量相加，即 :math:`0.5+0.125*X`。 453 | 454 | 对于两个参与方（参与方 A 为 rank 0，参与方 B 为 rank 1）的情况，将明文常量直接加在 A 方的密文分片上，B 方的密文分片不变。 455 | 456 | 4. 密文向量 + 密文向量 457 | 458 | 梯度更新公式 :math:`grad=grad+w^{'}*l2\_norm` 涉及密文向量与密文向量相加。 459 | 460 | 此算子不涉及参与方之间的通信，双方在本地将两个密文的分片直接相加即可。 461 | 462 | 5. 密文向量 - 密文向量 463 | 464 | 误差计算公式 :math:`err=pred-y` 涉及密文向量与密文向量相减。此算子不涉及参与方之间的通信，双方在本地将两个密文的分片直接相减即可。 465 | 466 | 6. 密文矩阵 * 密文矩阵 467 | 468 | 公式 :math:`batch\_x*w` 和公式 :math:`grad=transpose(batch\_x)*err` 涉及密文矩阵与密文矩阵相乘。算子的实现定义如下： 469 | 470 | :math:`X_i`、:math:`Y_i` (:math:`i=0` 或 :math:`1`) 分别是 :math:`X`、:math:`Y` 的密文分片，以下步骤是计算 :math:`Z_i=[X*Y]_i` 的实现。 471 | 472 | - 可信第三方生成 Beaver 矩阵乘法三元组(:math:`A_i`，:math:`B_i`，:math:`C_i`)，生成方式参见 :ref:`beaver`。 473 | 474 | - 双方同时解密得到 :math:`X-A` 和 :math:`Y-B`： 475 | 476 | 双方分别本地计算 :math:`X_i-A_i` 和 :math:`Y_i-B_i` ，然后调用传输层协议接口先后获得对方的分片 :math:`X_{1-i}-A_{1-i}` 和 :math:`Y_{1-i}-B_{1-i}`。双方分别在本地执行加法后得到 :math:`X-A` 和 :math:`Y-B`。 477 | 478 | - 各方分别在本地执行以下计算： 479 | 480 | :math:`Z_i=C_i+(X-A)*B_i+A_i*(Y-B)+(1-i)×(X-A)*(Y-B)` 481 | 482 | - 对 :math:`Z_i` 进行截断，截断的实现方式参照 :ref:`密文截断 ` 483 | 484 | .. _truncation: 485 | 486 | 7. 密文截断 487 | 488 | 乘法计算后要对结果进行截断（Truncation）。有多种对秘密分享分片截断的方案，其中一种不需要交互但有一定出错概率的截断算法方案如下： 489 | 490 | 一方直接对其分片向右移动指定截断位数，另一方用 :math:`2^n` 减分片，将减法结果向右移动指定截断位数，再用 :math:`2^n` 减右移后的分片。 491 | 492 | 此处的移动指逻辑右移，即对于 int64/int128 变量，如果数值为自然数，右移时高位补 :math:`0`，如果数值为负数，右移时高位补 :math:`1`。 493 | 494 | .. _beaver: 495 | 496 | Beaver 矩阵乘法三元组 497 | """""""""""""""""""""""""""""" 498 | 499 | 在矩阵乘法计算过程中，参与方 A 和 B 获得矩阵乘法三元组分片（:math:`A_0`，:math:`B_0`，:math:`C_0`）和（:math:`A_1`，:math:`B_1`，:math:`C_1`），满足 :math:`(A_0+A_1)*(B_0+B_1)=(C_0+C_1)`。三元组分片是各方通过私有的随机数种子生成的，所以对另一方不可见。 500 | 501 | Beaver 三元组生成方式 502 | ***************** 503 | 504 | 乘法三元组有多种实现算法方案，包括两方实现的算法方案和第三方辅助的算法方案。以下是一种第三方辅助的算法方案的实现设计。 505 | 506 | 以可信第三方的 Beaver 服务为例。Beaver 服务器知道参与方 A 和参与方 B 的随机数种子，通过相同的伪随机数生成算法和参数生成与 A 和 B 相同的伪随机数，即三元组分片分片（:math:`A_0`，:math:`B_0`，:math:`C_0`）和（:math:`A_1`，:math:`B_1`，:math:`C_1`）。Beaver服务计算 :math:`adjust=(A_0+A_1)*(B_0+B_1)-(C_0+C_1)`，得到调整值 :math:`adjust`。参与方 A（或参与方 B）从Beaver 服务获取 :math:`adjust` 值，并将其加到 :math:`C_0` （或 :math:`C_1`）分片上得到新的 :math:`C_0` （或 :math:`C_1`），最终得到的就是符合条件的矩阵乘法三元组。 507 | 508 | 509 | --------------------------------------------------------------------------------