├── Cuda_LSS_HQ ├── quantize_forward_LSQ+HQ │ ├── quantize_forward_easy.cpp │ ├── quantize_forward_easy_kernel.cu │ ├── setup_easy.py │ └── test.py ├── quantize_grad_input_LSS+LSQ │ ├── quantize_grad_input_speed.cpp │ ├── quantize_grad_input_speed_kernel.cu │ ├── setup.py │ └── test.py ├── quantize_grad_weight_LSS+LSQ │ ├── quantize_grad_weight_speed.cpp │ ├── quantize_grad_weight_speed_kernel.cu │ ├── setup.py │ └── test.py └── readme_noimage(1).md ├── SingleDivide ├── image.png ├── readme.md ├── squad_adv.sh ├── squadv1.sh ├── squadv2.sh ├── task.sh ├── task_conll.sh ├── task_swag.sh ├── test_glue.py ├── test_mrpc.py ├── test_squad.py ├── test_swag.py ├── test_token.py ├── transformersLocal │ ├── __init__.py │ ├── activations.py │ ├── activations_tf.py │ ├── benchmark │ │ ├── __init__.py │ │ ├── benchmark.py │ │ ├── benchmark_args.py │ │ ├── benchmark_args_tf.py │ │ ├── benchmark_args_utils.py │ │ ├── benchmark_tf.py │ │ └── benchmark_utils.py │ ├── commands │ │ ├── __init__.py │ │ ├── add_new_model.py │ │ ├── add_new_model_like.py │ │ ├── convert.py │ │ ├── download.py │ │ ├── env.py │ │ ├── lfs.py │ │ ├── pt_to_tf.py │ │ ├── run.py │ │ ├── serving.py │ │ ├── train.py │ │ ├── transformers_cli.py │ │ └── user.py │ ├── configuration_utils.py │ ├── convert_graph_to_onnx.py │ ├── convert_pytorch_checkpoint_to_tf2.py │ ├── convert_slow_tokenizer.py │ ├── convert_slow_tokenizers_checkpoints_to_fast.py │ ├── convert_tf_hub_seq_to_seq_bert_to_pytorch.py │ ├── data │ │ ├── __init__.py │ │ ├── data_collator.py │ │ ├── datasets │ │ │ ├── __init__.py │ │ │ ├── glue.py │ │ │ ├── language_modeling.py │ │ │ └── squad.py │ │ ├── metrics │ │ │ ├── __init__.py │ │ │ └── squad_metrics.py │ │ ├── processors │ │ │ ├── __init__.py │ │ │ ├── glue.py │ │ │ ├── squad.py │ │ │ ├── utils.py │ │ │ └── xnli.py │ │ └── test_generation_utils.py │ ├── debug_utils.py │ ├── deepspeed.py │ ├── dependency_versions_check.py │ ├── dependency_versions_table.py │ ├── dynamic_module_utils.py │ ├── feature_extraction_sequence_utils.py │ ├── feature_extraction_utils.py │ ├── file_utils.py │ ├── generation_beam_constraints.py │ ├── generation_beam_search.py │ ├── generation_flax_logits_process.py │ ├── generation_flax_utils.py │ ├── generation_logits_process.py │ ├── generation_stopping_criteria.py │ ├── generation_tf_logits_process.py │ ├── generation_tf_utils.py │ ├── generation_utils.py │ ├── hf_argparser.py │ ├── image_utils.py │ ├── integrations.py │ ├── keras_callbacks.py │ ├── modelcard.py │ ├── modeling_flax_outputs.py │ ├── modeling_flax_pytorch_utils.py │ ├── modeling_flax_utils.py │ ├── modeling_outputs.py │ ├── modeling_tf_outputs.py │ ├── modeling_tf_pytorch_utils.py │ ├── modeling_tf_utils.py │ ├── modeling_utils.py │ ├── models │ │ ├── __init__.py │ │ ├── albert │ │ │ ├── __init__.py │ │ │ ├── configuration_albert.py │ │ │ ├── convert_albert_original_tf_checkpoint_to_pytorch.py │ │ │ ├── modeling_albert.py │ │ │ ├── modeling_flax_albert.py │ │ │ ├── modeling_tf_albert.py │ │ │ ├── tokenization_albert.py │ │ │ └── tokenization_albert_fast.py │ │ ├── auto │ │ │ ├── __init__.py │ │ │ ├── auto_factory.py │ │ │ ├── configuration_auto.py │ │ │ ├── feature_extraction_auto.py │ │ │ ├── modeling_auto.py │ │ │ ├── modeling_flax_auto.py │ │ │ ├── modeling_tf_auto.py │ │ │ ├── processing_auto.py │ │ │ └── tokenization_auto.py │ │ ├── bart │ │ │ ├── __init__.py │ │ │ ├── configuration_bart.py │ │ │ ├── convert_bart_original_pytorch_checkpoint_to_pytorch.py │ │ │ ├── modeling_bart.py │ │ │ ├── modeling_flax_bart.py │ │ │ ├── modeling_tf_bart.py │ │ │ ├── tokenization_bart.py │ │ │ └── tokenization_bart_fast.py │ │ ├── barthez │ │ │ ├── __init__.py │ │ │ ├── tokenization_barthez.py │ │ │ └── tokenization_barthez_fast.py │ │ ├── bartpho │ │ │ ├── __init__.py │ │ │ └── tokenization_bartpho.py │ │ ├── beit │ │ │ ├── __init__.py │ │ │ ├── configuration_beit.py │ │ │ ├── convert_beit_unilm_to_pytorch.py │ │ │ ├── feature_extraction_beit.py │ │ │ ├── modeling_beit.py │ │ │ └── modeling_flax_beit.py │ │ ├── bert │ │ │ ├── __init__.py │ │ │ ├── configuration_bert.py │ │ │ ├── convert_bert_original_tf2_checkpoint_to_pytorch.py │ │ │ ├── convert_bert_original_tf_checkpoint_to_pytorch.py │ │ │ ├── convert_bert_pytorch_checkpoint_to_original_tf.py │ │ │ ├── convert_bert_token_dropping_original_tf2_checkpoint_to_pytorch.py │ │ │ ├── image_classification │ │ │ │ ├── __init__.py │ │ │ │ ├── activation_quantizer_1.py │ │ │ │ ├── debug.py │ │ │ │ ├── logger.py │ │ │ │ ├── mixup.py │ │ │ │ ├── plot_variance.py │ │ │ │ ├── preconditioner.py │ │ │ │ ├── quantize.py │ │ │ │ ├── smoothing.py │ │ │ │ ├── special_quantize.py │ │ │ │ ├── test.ipynb │ │ │ │ └── utils.py │ │ │ ├── modeling_bert.py │ │ │ ├── modeling_flax_bert.py │ │ │ ├── modeling_tf_bert.py │ │ │ ├── tokenization_bert.py │ │ │ ├── tokenization_bert_fast.py │ │ │ └── tokenization_bert_tf.py │ │ ├── bert_generation │ │ │ ├── __init__.py │ │ │ ├── configuration_bert_generation.py │ │ │ ├── modeling_bert_generation.py │ │ │ └── tokenization_bert_generation.py │ │ ├── bert_japanese │ │ │ ├── __init__.py │ │ │ └── tokenization_bert_japanese.py │ │ ├── bertweet │ │ │ ├── __init__.py │ │ │ └── tokenization_bertweet.py │ │ ├── big_bird │ │ │ ├── __init__.py │ │ │ ├── configuration_big_bird.py │ │ │ ├── convert_bigbird_original_tf_checkpoint_to_pytorch.py │ │ │ ├── modeling_big_bird.py │ │ │ ├── modeling_flax_big_bird.py │ │ │ ├── tokenization_big_bird.py │ │ │ └── tokenization_big_bird_fast.py │ │ ├── bigbird_pegasus │ │ │ ├── __init__.py │ │ │ ├── configuration_bigbird_pegasus.py │ │ │ ├── convert_bigbird_pegasus_tf_to_pytorch.py │ │ │ └── modeling_bigbird_pegasus.py │ │ ├── blenderbot │ │ │ ├── __init__.py │ │ │ ├── configuration_blenderbot.py │ │ │ ├── convert_blenderbot_original_pytorch_checkpoint_to_pytorch.py │ │ │ ├── modeling_blenderbot.py │ │ │ ├── modeling_flax_blenderbot.py │ │ │ ├── modeling_tf_blenderbot.py │ │ │ ├── tokenization_blenderbot.py │ │ │ └── tokenization_blenderbot_fast.py │ │ ├── blenderbot_small │ │ │ ├── __init__.py │ │ │ ├── configuration_blenderbot_small.py │ │ │ ├── modeling_blenderbot_small.py │ │ │ ├── modeling_flax_blenderbot_small.py │ │ │ ├── modeling_tf_blenderbot_small.py │ │ │ ├── tokenization_blenderbot_small.py │ │ │ └── tokenization_blenderbot_small_fast.py │ │ ├── bloom │ │ │ ├── __init__.py │ │ │ ├── configuration_bloom.py │ │ │ ├── convert_bloom_original_checkpoint_to_pytorch.py │ │ │ ├── modeling_bloom.py │ │ │ └── tokenization_bloom_fast.py │ │ ├── bort │ │ │ ├── __init__.py │ │ │ └── convert_bort_original_gluonnlp_checkpoint_to_pytorch.py │ │ ├── byt5 │ │ │ ├── __init__.py │ │ │ ├── convert_byt5_original_tf_checkpoint_to_pytorch.py │ │ │ └── tokenization_byt5.py │ │ ├── camembert │ │ │ ├── __init__.py │ │ │ ├── configuration_camembert.py │ │ │ ├── modeling_camembert.py │ │ │ ├── modeling_tf_camembert.py │ │ │ ├── tokenization_camembert.py │ │ │ └── tokenization_camembert_fast.py │ │ ├── canine │ │ │ ├── __init__.py │ │ │ ├── configuration_canine.py │ │ │ ├── convert_canine_original_tf_checkpoint_to_pytorch.py │ │ │ ├── modeling_canine.py │ │ │ └── tokenization_canine.py │ │ ├── clip │ │ │ ├── __init__.py │ │ │ ├── configuration_clip.py │ │ │ ├── convert_clip_original_pytorch_to_hf.py │ │ │ ├── feature_extraction_clip.py │ │ │ ├── modeling_clip.py │ │ │ ├── modeling_flax_clip.py │ │ │ ├── modeling_tf_clip.py │ │ │ ├── processing_clip.py │ │ │ ├── tokenization_clip.py │ │ │ └── tokenization_clip_fast.py │ │ ├── codegen │ │ │ ├── __init__.py │ │ │ ├── configuration_codegen.py │ │ │ ├── modeling_codegen.py │ │ │ ├── tokenization_codegen.py │ │ │ └── tokenization_codegen_fast.py │ │ ├── convbert │ │ │ ├── __init__.py │ │ │ ├── configuration_convbert.py │ │ │ ├── convert_convbert_original_tf1_checkpoint_to_pytorch_and_tf2.py │ │ │ ├── modeling_convbert.py │ │ │ ├── modeling_tf_convbert.py │ │ │ ├── tokenization_convbert.py │ │ │ └── tokenization_convbert_fast.py │ │ ├── convnext │ │ │ ├── __init__.py │ │ │ ├── configuration_convnext.py │ │ │ ├── convert_convnext_to_pytorch.py │ │ │ ├── feature_extraction_convnext.py │ │ │ ├── modeling_convnext.py │ │ │ └── modeling_tf_convnext.py │ │ ├── cpm │ │ │ ├── __init__.py │ │ │ ├── tokenization_cpm.py │ │ │ └── tokenization_cpm_fast.py │ │ ├── ctrl │ │ │ ├── __init__.py │ │ │ ├── configuration_ctrl.py │ │ │ ├── modeling_ctrl.py │ │ │ ├── modeling_tf_ctrl.py │ │ │ └── tokenization_ctrl.py │ │ ├── cvt │ │ │ ├── __init__.py │ │ │ ├── configuration_cvt.py │ │ │ ├── convert_cvt_original_pytorch_checkpoint_to_pytorch.py │ │ │ └── modeling_cvt.py │ │ ├── data2vec │ │ │ ├── __init__.py │ │ │ ├── configuration_data2vec_audio.py │ │ │ ├── configuration_data2vec_text.py │ │ │ ├── configuration_data2vec_vision.py │ │ │ ├── convert_data2vec_audio_original_pytorch_checkpoint_to_pytorch.py │ │ │ ├── convert_data2vec_text_original_pytorch_checkpoint_to_pytorch.py │ │ │ ├── convert_data2vec_vision_original_pytorch_checkpoint_to_pytorch.py │ │ │ ├── modeling_data2vec_audio.py │ │ │ ├── modeling_data2vec_text.py │ │ │ ├── modeling_data2vec_vision.py │ │ │ └── modeling_tf_data2vec_vision.py │ │ ├── deberta │ │ │ ├── __init__.py │ │ │ ├── configuration_deberta.py │ │ │ ├── modeling_deberta.py │ │ │ ├── modeling_tf_deberta.py │ │ │ ├── tokenization_deberta.py │ │ │ └── tokenization_deberta_fast.py │ │ ├── deberta_v2 │ │ │ ├── __init__.py │ │ │ ├── configuration_deberta_v2.py │ │ │ ├── modeling_deberta_v2.py │ │ │ ├── modeling_tf_deberta_v2.py │ │ │ ├── tokenization_deberta_v2.py │ │ │ └── tokenization_deberta_v2_fast.py │ │ ├── decision_transformer │ │ │ ├── __init__.py │ │ │ ├── configuration_decision_transformer.py │ │ │ └── modeling_decision_transformer.py │ │ ├── deit │ │ │ ├── __init__.py │ │ │ ├── configuration_deit.py │ │ │ ├── convert_deit_timm_to_pytorch.py │ │ │ ├── feature_extraction_deit.py │ │ │ ├── modeling_deit.py │ │ │ └── modeling_tf_deit.py │ │ ├── detr │ │ │ ├── __init__.py │ │ │ ├── configuration_detr.py │ │ │ ├── convert_detr_original_pytorch_checkpoint_to_pytorch.py │ │ │ ├── feature_extraction_detr.py │ │ │ └── modeling_detr.py │ │ ├── dialogpt │ │ │ ├── __init__.py │ │ │ └── convert_dialogpt_original_pytorch_checkpoint_to_pytorch.py │ │ ├── distilbert │ │ │ ├── __init__.py │ │ │ ├── configuration_distilbert.py │ │ │ ├── modeling_distilbert.py │ │ │ ├── modeling_flax_distilbert.py │ │ │ ├── modeling_tf_distilbert.py │ │ │ ├── tokenization_distilbert.py │ │ │ └── tokenization_distilbert_fast.py │ │ ├── dit │ │ │ ├── __init__.py │ │ │ └── convert_dit_unilm_to_pytorch.py │ │ ├── dpr │ │ │ ├── __init__.py │ │ │ ├── configuration_dpr.py │ │ │ ├── convert_dpr_original_checkpoint_to_pytorch.py │ │ │ ├── modeling_dpr.py │ │ │ ├── modeling_tf_dpr.py │ │ │ ├── tokenization_dpr.py │ │ │ └── tokenization_dpr_fast.py │ │ ├── dpt │ │ │ ├── __init__.py │ │ │ ├── configuration_dpt.py │ │ │ ├── convert_dpt_to_pytorch.py │ │ │ ├── feature_extraction_dpt.py │ │ │ └── modeling_dpt.py │ │ ├── electra │ │ │ ├── __init__.py │ │ │ ├── configuration_electra.py │ │ │ ├── convert_electra_original_tf_checkpoint_to_pytorch.py │ │ │ ├── modeling_electra.py │ │ │ ├── modeling_flax_electra.py │ │ │ ├── modeling_tf_electra.py │ │ │ ├── tokenization_electra.py │ │ │ └── tokenization_electra_fast.py │ │ ├── encoder_decoder │ │ │ ├── __init__.py │ │ │ ├── configuration_encoder_decoder.py │ │ │ ├── modeling_encoder_decoder.py │ │ │ ├── modeling_flax_encoder_decoder.py │ │ │ └── modeling_tf_encoder_decoder.py │ │ ├── flaubert │ │ │ ├── __init__.py │ │ │ ├── configuration_flaubert.py │ │ │ ├── modeling_flaubert.py │ │ │ ├── modeling_tf_flaubert.py │ │ │ └── tokenization_flaubert.py │ │ ├── flava │ │ │ ├── __init__.py │ │ │ ├── configuration_flava.py │ │ │ ├── convert_dalle_to_flava_codebook.py │ │ │ ├── convert_flava_original_pytorch_to_hf.py │ │ │ ├── feature_extraction_flava.py │ │ │ ├── modeling_flava.py │ │ │ └── processing_flava.py │ │ ├── fnet │ │ │ ├── __init__.py │ │ │ ├── configuration_fnet.py │ │ │ ├── convert_fnet_original_flax_checkpoint_to_pytorch.py │ │ │ ├── modeling_fnet.py │ │ │ ├── tokenization_fnet.py │ │ │ └── tokenization_fnet_fast.py │ │ ├── fsmt │ │ │ ├── __init__.py │ │ │ ├── configuration_fsmt.py │ │ │ ├── convert_fsmt_original_pytorch_checkpoint_to_pytorch.py │ │ │ ├── modeling_fsmt.py │ │ │ └── tokenization_fsmt.py │ │ ├── funnel │ │ │ ├── __init__.py │ │ │ ├── configuration_funnel.py │ │ │ ├── convert_funnel_original_tf_checkpoint_to_pytorch.py │ │ │ ├── modeling_funnel.py │ │ │ ├── modeling_tf_funnel.py │ │ │ ├── tokenization_funnel.py │ │ │ └── tokenization_funnel_fast.py │ │ ├── glpn │ │ │ ├── __init__.py │ │ │ ├── configuration_glpn.py │ │ │ ├── convert_glpn_to_pytorch.py │ │ │ ├── feature_extraction_glpn.py │ │ │ └── modeling_glpn.py │ │ ├── gpt2 │ │ │ ├── __init__.py │ │ │ ├── configuration_gpt2.py │ │ │ ├── convert_gpt2_original_tf_checkpoint_to_pytorch.py │ │ │ ├── modeling_flax_gpt2.py │ │ │ ├── modeling_gpt2.py │ │ │ ├── modeling_tf_gpt2.py │ │ │ ├── tokenization_gpt2.py │ │ │ └── tokenization_gpt2_fast.py │ │ ├── gpt_neo │ │ │ ├── __init__.py │ │ │ ├── configuration_gpt_neo.py │ │ │ ├── convert_gpt_neo_mesh_tf_to_pytorch.py │ │ │ ├── modeling_flax_gpt_neo.py │ │ │ └── modeling_gpt_neo.py │ │ ├── gpt_neox │ │ │ ├── __init__.py │ │ │ ├── configuration_gpt_neox.py │ │ │ ├── modeling_gpt_neox.py │ │ │ └── tokenization_gpt_neox_fast.py │ │ ├── gptj │ │ │ ├── __init__.py │ │ │ ├── configuration_gptj.py │ │ │ ├── modeling_flax_gptj.py │ │ │ ├── modeling_gptj.py │ │ │ └── modeling_tf_gptj.py │ │ ├── groupvit │ │ │ ├── __init__.py │ │ │ ├── configuration_groupvit.py │ │ │ ├── convert_groupvit_nvlab_to_hf.py │ │ │ └── modeling_groupvit.py │ │ ├── herbert │ │ │ ├── __init__.py │ │ │ ├── tokenization_herbert.py │ │ │ └── tokenization_herbert_fast.py │ │ ├── hubert │ │ │ ├── __init__.py │ │ │ ├── configuration_hubert.py │ │ │ ├── convert_distilhubert_original_s3prl_checkpoint_to_pytorch.py │ │ │ ├── convert_hubert_original_pytorch_checkpoint_to_pytorch.py │ │ │ ├── convert_hubert_original_s3prl_checkpoint_to_pytorch.py │ │ │ ├── modeling_hubert.py │ │ │ └── modeling_tf_hubert.py │ │ ├── ibert │ │ │ ├── __init__.py │ │ │ ├── configuration_ibert.py │ │ │ ├── modeling_ibert.py │ │ │ └── quant_modules.py │ │ ├── imagegpt │ │ │ ├── __init__.py │ │ │ ├── configuration_imagegpt.py │ │ │ ├── convert_imagegpt_original_tf2_to_pytorch.py │ │ │ ├── feature_extraction_imagegpt.py │ │ │ └── modeling_imagegpt.py │ │ ├── layoutlm │ │ │ ├── __init__.py │ │ │ ├── configuration_layoutlm.py │ │ │ ├── modeling_layoutlm.py │ │ │ ├── modeling_tf_layoutlm.py │ │ │ ├── tokenization_layoutlm.py │ │ │ └── tokenization_layoutlm_fast.py │ │ ├── layoutlmv2 │ │ │ ├── __init__.py │ │ │ ├── configuration_layoutlmv2.py │ │ │ ├── feature_extraction_layoutlmv2.py │ │ │ ├── modeling_layoutlmv2.py │ │ │ ├── processing_layoutlmv2.py │ │ │ ├── tokenization_layoutlmv2.py │ │ │ └── tokenization_layoutlmv2_fast.py │ │ ├── layoutlmv3 │ │ │ ├── __init__.py │ │ │ ├── configuration_layoutlmv3.py │ │ │ ├── feature_extraction_layoutlmv3.py │ │ │ ├── modeling_layoutlmv3.py │ │ │ ├── processing_layoutlmv3.py │ │ │ ├── tokenization_layoutlmv3.py │ │ │ └── tokenization_layoutlmv3_fast.py │ │ ├── layoutxlm │ │ │ ├── __init__.py │ │ │ ├── processing_layoutxlm.py │ │ │ ├── tokenization_layoutxlm.py │ │ │ └── tokenization_layoutxlm_fast.py │ │ ├── led │ │ │ ├── __init__.py │ │ │ ├── configuration_led.py │ │ │ ├── modeling_led.py │ │ │ ├── modeling_tf_led.py │ │ │ ├── tokenization_led.py │ │ │ └── tokenization_led_fast.py │ │ ├── levit │ │ │ ├── __init__.py │ │ │ ├── configuration_levit.py │ │ │ ├── convert_levit_timm_to_pytorch.py │ │ │ ├── feature_extraction_levit.py │ │ │ └── modeling_levit.py │ │ ├── longformer │ │ │ ├── __init__.py │ │ │ ├── configuration_longformer.py │ │ │ ├── convert_longformer_original_pytorch_lightning_to_pytorch.py │ │ │ ├── modeling_longformer.py │ │ │ ├── modeling_tf_longformer.py │ │ │ ├── tokenization_longformer.py │ │ │ └── tokenization_longformer_fast.py │ │ ├── longt5 │ │ │ ├── __init__.py │ │ │ ├── configuration_longt5.py │ │ │ ├── convert_longt5x_checkpoint_to_flax.py │ │ │ ├── modeling_flax_longt5.py │ │ │ └── modeling_longt5.py │ │ ├── luke │ │ │ ├── __init__.py │ │ │ ├── configuration_luke.py │ │ │ ├── convert_luke_original_pytorch_checkpoint_to_pytorch.py │ │ │ ├── modeling_luke.py │ │ │ └── tokenization_luke.py │ │ ├── lxmert │ │ │ ├── __init__.py │ │ │ ├── configuration_lxmert.py │ │ │ ├── convert_lxmert_original_tf_checkpoint_to_pytorch.py │ │ │ ├── modeling_lxmert.py │ │ │ ├── modeling_tf_lxmert.py │ │ │ ├── tokenization_lxmert.py │ │ │ └── tokenization_lxmert_fast.py │ │ ├── m2m_100 │ │ │ ├── __init__.py │ │ │ ├── configuration_m2m_100.py │ │ │ ├── convert_m2m100_original_checkpoint_to_pytorch.py │ │ │ ├── modeling_m2m_100.py │ │ │ └── tokenization_m2m_100.py │ │ ├── marian │ │ │ ├── __init__.py │ │ │ ├── configuration_marian.py │ │ │ ├── convert_marian_tatoeba_to_pytorch.py │ │ │ ├── convert_marian_to_pytorch.py │ │ │ ├── modeling_flax_marian.py │ │ │ ├── modeling_marian.py │ │ │ ├── modeling_tf_marian.py │ │ │ └── tokenization_marian.py │ │ ├── maskformer │ │ │ ├── __init__.py │ │ │ ├── configuration_maskformer.py │ │ │ ├── convert_maskformer_original_pytorch_checkpoint_to_pytorch.py │ │ │ ├── feature_extraction_maskformer.py │ │ │ └── modeling_maskformer.py │ │ ├── mbart │ │ │ ├── __init__.py │ │ │ ├── configuration_mbart.py │ │ │ ├── convert_mbart_original_checkpoint_to_pytorch.py │ │ │ ├── modeling_flax_mbart.py │ │ │ ├── modeling_mbart.py │ │ │ ├── modeling_tf_mbart.py │ │ │ ├── tokenization_mbart.py │ │ │ └── tokenization_mbart_fast.py │ │ ├── mbart50 │ │ │ ├── __init__.py │ │ │ ├── tokenization_mbart50.py │ │ │ └── tokenization_mbart50_fast.py │ │ ├── mctct │ │ │ ├── __init__.py │ │ │ ├── configuration_mctct.py │ │ │ ├── feature_extraction_mctct.py │ │ │ ├── modeling_mctct.py │ │ │ └── processing_mctct.py │ │ ├── megatron_bert │ │ │ ├── __init__.py │ │ │ ├── configuration_megatron_bert.py │ │ │ ├── convert_megatron_bert_checkpoint.py │ │ │ └── modeling_megatron_bert.py │ │ ├── megatron_gpt2 │ │ │ ├── __init__.py │ │ │ └── convert_megatron_gpt2_checkpoint.py │ │ ├── mluke │ │ │ ├── __init__.py │ │ │ ├── convert_mluke_original_pytorch_checkpoint_to_pytorch.py │ │ │ └── tokenization_mluke.py │ │ ├── mmbt │ │ │ ├── __init__.py │ │ │ ├── configuration_mmbt.py │ │ │ └── modeling_mmbt.py │ │ ├── mobilebert │ │ │ ├── __init__.py │ │ │ ├── configuration_mobilebert.py │ │ │ ├── convert_mobilebert_original_tf_checkpoint_to_pytorch.py │ │ │ ├── modeling_mobilebert.py │ │ │ ├── modeling_tf_mobilebert.py │ │ │ ├── tokenization_mobilebert.py │ │ │ └── tokenization_mobilebert_fast.py │ │ ├── mobilevit │ │ │ ├── __init__.py │ │ │ ├── configuration_mobilevit.py │ │ │ ├── convert_mlcvnets_to_pytorch.py │ │ │ ├── feature_extraction_mobilevit.py │ │ │ └── modeling_mobilevit.py │ │ ├── mpnet │ │ │ ├── __init__.py │ │ │ ├── configuration_mpnet.py │ │ │ ├── modeling_mpnet.py │ │ │ ├── modeling_tf_mpnet.py │ │ │ ├── tokenization_mpnet.py │ │ │ └── tokenization_mpnet_fast.py │ │ ├── mt5 │ │ │ ├── __init__.py │ │ │ ├── configuration_mt5.py │ │ │ ├── modeling_flax_mt5.py │ │ │ ├── modeling_mt5.py │ │ │ └── modeling_tf_mt5.py │ │ ├── mvp │ │ │ ├── __init__.py │ │ │ ├── configuration_mvp.py │ │ │ ├── modeling_mvp.py │ │ │ ├── tokenization_mvp.py │ │ │ └── tokenization_mvp_fast.py │ │ ├── nezha │ │ │ ├── __init__.py │ │ │ ├── configuration_nezha.py │ │ │ └── modeling_nezha.py │ │ ├── nllb │ │ │ ├── __init__.py │ │ │ ├── tokenization_nllb.py │ │ │ └── tokenization_nllb_fast.py │ │ ├── nystromformer │ │ │ ├── __init__.py │ │ │ ├── configuration_nystromformer.py │ │ │ ├── convert_nystromformer_original_pytorch_checkpoint_to_pytorch.py │ │ │ └── modeling_nystromformer.py │ │ ├── openai │ │ │ ├── __init__.py │ │ │ ├── configuration_openai.py │ │ │ ├── convert_openai_original_tf_checkpoint_to_pytorch.py │ │ │ ├── modeling_openai.py │ │ │ ├── modeling_tf_openai.py │ │ │ ├── tokenization_openai.py │ │ │ └── tokenization_openai_fast.py │ │ ├── opt │ │ │ ├── __init__.py │ │ │ ├── configuration_opt.py │ │ │ ├── convert_opt_original_pytorch_checkpoint_to_pytorch.py │ │ │ ├── image_classification │ │ │ │ ├── __init__.py │ │ │ │ ├── activation_quantizer_1.py │ │ │ │ ├── debug.py │ │ │ │ ├── logger.py │ │ │ │ ├── mixup.py │ │ │ │ ├── plot_variance.py │ │ │ │ ├── preconditioner.py │ │ │ │ ├── quantize.py │ │ │ │ ├── smoothing.py │ │ │ │ ├── test.ipynb │ │ │ │ └── utils.py │ │ │ ├── modeling_flax_opt.py │ │ │ ├── modeling_opt.py │ │ │ └── modeling_tf_opt.py │ │ ├── owlvit │ │ │ ├── __init__.py │ │ │ ├── configuration_owlvit.py │ │ │ ├── convert_owlvit_original_flax_to_hf.py │ │ │ ├── feature_extraction_owlvit.py │ │ │ ├── modeling_owlvit.py │ │ │ └── processing_owlvit.py │ │ ├── pegasus │ │ │ ├── __init__.py │ │ │ ├── configuration_pegasus.py │ │ │ ├── convert_pegasus_tf_to_pytorch.py │ │ │ ├── modeling_flax_pegasus.py │ │ │ ├── modeling_pegasus.py │ │ │ ├── modeling_tf_pegasus.py │ │ │ ├── tokenization_pegasus.py │ │ │ └── tokenization_pegasus_fast.py │ │ ├── perceiver │ │ │ ├── __init__.py │ │ │ ├── configuration_perceiver.py │ │ │ ├── convert_perceiver_haiku_to_pytorch.py │ │ │ ├── feature_extraction_perceiver.py │ │ │ ├── modeling_perceiver.py │ │ │ └── tokenization_perceiver.py │ │ ├── phobert │ │ │ ├── __init__.py │ │ │ └── tokenization_phobert.py │ │ ├── plbart │ │ │ ├── __init__.py │ │ │ ├── configuration_plbart.py │ │ │ ├── convert_plbart_original_checkpoint_to_torch.py │ │ │ ├── modeling_plbart.py │ │ │ └── tokenization_plbart.py │ │ ├── poolformer │ │ │ ├── __init__.py │ │ │ ├── configuration_poolformer.py │ │ │ ├── convert_poolformer_original_to_pytorch.py │ │ │ ├── feature_extraction_poolformer.py │ │ │ └── modeling_poolformer.py │ │ ├── prophetnet │ │ │ ├── __init__.py │ │ │ ├── configuration_prophetnet.py │ │ │ ├── convert_prophetnet_original_pytorch_checkpoint_to_pytorch.py │ │ │ ├── modeling_prophetnet.py │ │ │ └── tokenization_prophetnet.py │ │ ├── qdqbert │ │ │ ├── __init__.py │ │ │ ├── configuration_qdqbert.py │ │ │ └── modeling_qdqbert.py │ │ ├── rag │ │ │ ├── __init__.py │ │ │ ├── configuration_rag.py │ │ │ ├── modeling_rag.py │ │ │ ├── modeling_tf_rag.py │ │ │ ├── retrieval_rag.py │ │ │ └── tokenization_rag.py │ │ ├── realm │ │ │ ├── __init__.py │ │ │ ├── configuration_realm.py │ │ │ ├── modeling_realm.py │ │ │ ├── retrieval_realm.py │ │ │ ├── tokenization_realm.py │ │ │ └── tokenization_realm_fast.py │ │ ├── reformer │ │ │ ├── __init__.py │ │ │ ├── configuration_reformer.py │ │ │ ├── convert_reformer_trax_checkpoint_to_pytorch.py │ │ │ ├── modeling_reformer.py │ │ │ ├── tokenization_reformer.py │ │ │ └── tokenization_reformer_fast.py │ │ ├── regnet │ │ │ ├── __init__.py │ │ │ ├── configuration_regnet.py │ │ │ ├── convert_regnet_seer_10b_to_pytorch.py │ │ │ ├── convert_regnet_to_pytorch.py │ │ │ ├── modeling_regnet.py │ │ │ └── modeling_tf_regnet.py │ │ ├── rembert │ │ │ ├── __init__.py │ │ │ ├── configuration_rembert.py │ │ │ ├── convert_rembert_tf_checkpoint_to_pytorch.py │ │ │ ├── modeling_rembert.py │ │ │ ├── modeling_tf_rembert.py │ │ │ ├── tokenization_rembert.py │ │ │ └── tokenization_rembert_fast.py │ │ ├── resnet │ │ │ ├── __init__.py │ │ │ ├── configuration_resnet.py │ │ │ ├── convert_resnet_to_pytorch.py │ │ │ ├── modeling_resnet.py │ │ │ └── modeling_tf_resnet.py │ │ ├── retribert │ │ │ ├── __init__.py │ │ │ ├── configuration_retribert.py │ │ │ ├── modeling_retribert.py │ │ │ ├── tokenization_retribert.py │ │ │ └── tokenization_retribert_fast.py │ │ ├── roberta │ │ │ ├── __init__.py │ │ │ ├── configuration_roberta.py │ │ │ ├── convert_roberta_original_pytorch_checkpoint_to_pytorch.py │ │ │ ├── image_classification │ │ │ │ ├── __init__.py │ │ │ │ ├── activation_quantizer_1.py │ │ │ │ ├── debug.py │ │ │ │ ├── logger.py │ │ │ │ ├── mixup.py │ │ │ │ ├── plot_variance.py │ │ │ │ ├── preconditioner.py │ │ │ │ ├── quantize.py │ │ │ │ ├── smoothing.py │ │ │ │ ├── test.ipynb │ │ │ │ └── utils.py │ │ │ ├── modeling_flax_roberta.py │ │ │ ├── modeling_roberta.py │ │ │ ├── modeling_tf_roberta.py │ │ │ ├── tokenization_roberta.py │ │ │ └── tokenization_roberta_fast.py │ │ ├── roformer │ │ │ ├── __init__.py │ │ │ ├── configuration_roformer.py │ │ │ ├── convert_roformer_original_tf_checkpoint_to_pytorch.py │ │ │ ├── modeling_flax_roformer.py │ │ │ ├── modeling_roformer.py │ │ │ ├── modeling_tf_roformer.py │ │ │ ├── tokenization_roformer.py │ │ │ ├── tokenization_roformer_fast.py │ │ │ └── tokenization_utils.py │ │ ├── segformer │ │ │ ├── __init__.py │ │ │ ├── configuration_segformer.py │ │ │ ├── convert_segformer_original_to_pytorch.py │ │ │ ├── feature_extraction_segformer.py │ │ │ ├── modeling_segformer.py │ │ │ └── modeling_tf_segformer.py │ │ ├── sew │ │ │ ├── __init__.py │ │ │ ├── configuration_sew.py │ │ │ ├── convert_sew_original_pytorch_checkpoint_to_pytorch.py │ │ │ └── modeling_sew.py │ │ ├── sew_d │ │ │ ├── __init__.py │ │ │ ├── configuration_sew_d.py │ │ │ ├── convert_sew_d_original_pytorch_checkpoint_to_pytorch.py │ │ │ └── modeling_sew_d.py │ │ ├── speech_encoder_decoder │ │ │ ├── __init__.py │ │ │ ├── configuration_speech_encoder_decoder.py │ │ │ ├── convert_mbart_wav2vec2_seq2seq_original_to_pytorch.py │ │ │ ├── convert_speech_to_text_wav2vec2_seq2seq_original_to_pytorch.py │ │ │ ├── modeling_flax_speech_encoder_decoder.py │ │ │ └── modeling_speech_encoder_decoder.py │ │ ├── speech_to_text │ │ │ ├── __init__.py │ │ │ ├── configuration_speech_to_text.py │ │ │ ├── convert_s2t_fairseq_to_tfms.py │ │ │ ├── feature_extraction_speech_to_text.py │ │ │ ├── modeling_speech_to_text.py │ │ │ ├── modeling_tf_speech_to_text.py │ │ │ ├── processing_speech_to_text.py │ │ │ └── tokenization_speech_to_text.py │ │ ├── speech_to_text_2 │ │ │ ├── __init__.py │ │ │ ├── configuration_speech_to_text_2.py │ │ │ ├── modeling_speech_to_text_2.py │ │ │ ├── processing_speech_to_text_2.py │ │ │ └── tokenization_speech_to_text_2.py │ │ ├── splinter │ │ │ ├── __init__.py │ │ │ ├── configuration_splinter.py │ │ │ ├── modeling_splinter.py │ │ │ ├── tokenization_splinter.py │ │ │ └── tokenization_splinter_fast.py │ │ ├── squeezebert │ │ │ ├── __init__.py │ │ │ ├── configuration_squeezebert.py │ │ │ ├── modeling_squeezebert.py │ │ │ ├── tokenization_squeezebert.py │ │ │ └── tokenization_squeezebert_fast.py │ │ ├── swin │ │ │ ├── __init__.py │ │ │ ├── configuration_swin.py │ │ │ ├── convert_swin_timm_to_pytorch.py │ │ │ ├── modeling_swin.py │ │ │ └── modeling_tf_swin.py │ │ ├── swinv2 │ │ │ ├── __init__.py │ │ │ ├── configuration_swinv2.py │ │ │ ├── convert_swinv2_timm_to_pytorch.py │ │ │ └── modeling_swinv2.py │ │ ├── t5 │ │ │ ├── __init__.py │ │ │ ├── configuration_t5.py │ │ │ ├── convert_t5_original_tf_checkpoint_to_pytorch.py │ │ │ ├── convert_t5x_checkpoint_to_flax.py │ │ │ ├── download_from_gcp.sh │ │ │ ├── modeling_flax_t5.py │ │ │ ├── modeling_t5.py │ │ │ ├── modeling_tf_t5.py │ │ │ ├── tokenization_t5.py │ │ │ └── tokenization_t5_fast.py │ │ ├── tapas │ │ │ ├── __init__.py │ │ │ ├── configuration_tapas.py │ │ │ ├── convert_tapas_original_tf_checkpoint_to_pytorch.py │ │ │ ├── modeling_tapas.py │ │ │ ├── modeling_tf_tapas.py │ │ │ └── tokenization_tapas.py │ │ ├── tapex │ │ │ ├── __init__.py │ │ │ └── tokenization_tapex.py │ │ ├── trajectory_transformer │ │ │ ├── __init__.py │ │ │ ├── configuration_trajectory_transformer.py │ │ │ ├── convert_trajectory_transformer_original_pytorch_checkpoint_to_pytorch.py │ │ │ └── modeling_trajectory_transformer.py │ │ ├── transfo_xl │ │ │ ├── __init__.py │ │ │ ├── configuration_transfo_xl.py │ │ │ ├── convert_transfo_xl_original_tf_checkpoint_to_pytorch.py │ │ │ ├── modeling_tf_transfo_xl.py │ │ │ ├── modeling_tf_transfo_xl_utilities.py │ │ │ ├── modeling_transfo_xl.py │ │ │ ├── modeling_transfo_xl_utilities.py │ │ │ └── tokenization_transfo_xl.py │ │ ├── trocr │ │ │ ├── __init__.py │ │ │ ├── configuration_trocr.py │ │ │ ├── modeling_trocr.py │ │ │ └── processing_trocr.py │ │ ├── unispeech │ │ │ ├── __init__.py │ │ │ ├── configuration_unispeech.py │ │ │ ├── convert_unispeech_original_pytorch_checkpoint_to_pytorch.py │ │ │ └── modeling_unispeech.py │ │ ├── unispeech_sat │ │ │ ├── __init__.py │ │ │ ├── configuration_unispeech_sat.py │ │ │ ├── convert_unispeech_original_s3prl_checkpoint_to_pytorch.py │ │ │ ├── convert_unispeech_sat_original_pytorch_checkpoint_to_pytorch.py │ │ │ └── modeling_unispeech_sat.py │ │ ├── van │ │ │ ├── __init__.py │ │ │ ├── configuration_van.py │ │ │ ├── convert_van_to_pytorch.py │ │ │ └── modeling_van.py │ │ ├── videomae │ │ │ ├── __init__.py │ │ │ ├── configuration_videomae.py │ │ │ ├── convert_videomae_to_pytorch.py │ │ │ ├── feature_extraction_videomae.py │ │ │ └── modeling_videomae.py │ │ ├── vilt │ │ │ ├── __init__.py │ │ │ ├── configuration_vilt.py │ │ │ ├── convert_vilt_original_to_pytorch.py │ │ │ ├── feature_extraction_vilt.py │ │ │ ├── modeling_vilt.py │ │ │ └── processing_vilt.py │ │ ├── vision_encoder_decoder │ │ │ ├── __init__.py │ │ │ ├── configuration_vision_encoder_decoder.py │ │ │ ├── convert_trocr_unilm_to_pytorch.py │ │ │ ├── modeling_flax_vision_encoder_decoder.py │ │ │ ├── modeling_tf_vision_encoder_decoder.py │ │ │ └── modeling_vision_encoder_decoder.py │ │ ├── vision_text_dual_encoder │ │ │ ├── __init__.py │ │ │ ├── configuration_vision_text_dual_encoder.py │ │ │ ├── modeling_flax_vision_text_dual_encoder.py │ │ │ ├── modeling_vision_text_dual_encoder.py │ │ │ └── processing_vision_text_dual_encoder.py │ │ ├── visual_bert │ │ │ ├── __init__.py │ │ │ ├── configuration_visual_bert.py │ │ │ ├── convert_visual_bert_original_pytorch_checkpoint_to_pytorch.py │ │ │ └── modeling_visual_bert.py │ │ ├── vit │ │ │ ├── __init__.py │ │ │ ├── configuration_vit.py │ │ │ ├── convert_dino_to_pytorch.py │ │ │ ├── convert_vit_timm_to_pytorch.py │ │ │ ├── feature_extraction_vit.py │ │ │ ├── modeling_flax_vit.py │ │ │ ├── modeling_tf_vit.py │ │ │ └── modeling_vit.py │ │ ├── vit_mae │ │ │ ├── __init__.py │ │ │ ├── configuration_vit_mae.py │ │ │ ├── convert_vit_mae_to_pytorch.py │ │ │ ├── modeling_tf_vit_mae.py │ │ │ └── modeling_vit_mae.py │ │ ├── wav2vec2 │ │ │ ├── __init__.py │ │ │ ├── configuration_wav2vec2.py │ │ │ ├── convert_wav2vec2_original_pytorch_checkpoint_to_pytorch.py │ │ │ ├── convert_wav2vec2_original_s3prl_checkpoint_to_pytorch.py │ │ │ ├── feature_extraction_wav2vec2.py │ │ │ ├── modeling_flax_wav2vec2.py │ │ │ ├── modeling_tf_wav2vec2.py │ │ │ ├── modeling_wav2vec2.py │ │ │ ├── processing_wav2vec2.py │ │ │ └── tokenization_wav2vec2.py │ │ ├── wav2vec2_conformer │ │ │ ├── __init__.py │ │ │ ├── configuration_wav2vec2_conformer.py │ │ │ ├── convert_wav2vec2_conformer_original_pytorch_checkpoint_to_pytorch.py │ │ │ └── modeling_wav2vec2_conformer.py │ │ ├── wav2vec2_phoneme │ │ │ ├── __init__.py │ │ │ └── tokenization_wav2vec2_phoneme.py │ │ ├── wav2vec2_with_lm │ │ │ ├── __init__.py │ │ │ └── processing_wav2vec2_with_lm.py │ │ ├── wavlm │ │ │ ├── __init__.py │ │ │ ├── configuration_wavlm.py │ │ │ ├── convert_wavlm_original_pytorch_checkpoint_to_pytorch.py │ │ │ ├── convert_wavlm_original_s3prl_checkpoint_to_pytorch.py │ │ │ └── modeling_wavlm.py │ │ ├── xglm │ │ │ ├── __init__.py │ │ │ ├── configuration_xglm.py │ │ │ ├── convert_xglm_original_ckpt_to_trfms.py │ │ │ ├── modeling_flax_xglm.py │ │ │ ├── modeling_xglm.py │ │ │ ├── tokenization_xglm.py │ │ │ └── tokenization_xglm_fast.py │ │ ├── xlm │ │ │ ├── __init__.py │ │ │ ├── configuration_xlm.py │ │ │ ├── convert_xlm_original_pytorch_checkpoint_to_pytorch.py │ │ │ ├── modeling_tf_xlm.py │ │ │ ├── modeling_xlm.py │ │ │ └── tokenization_xlm.py │ │ ├── xlm_prophetnet │ │ │ ├── __init__.py │ │ │ ├── configuration_xlm_prophetnet.py │ │ │ ├── modeling_xlm_prophetnet.py │ │ │ └── tokenization_xlm_prophetnet.py │ │ ├── xlm_roberta │ │ │ ├── __init__.py │ │ │ ├── configuration_xlm_roberta.py │ │ │ ├── image_classification │ │ │ │ ├── __init__.py │ │ │ │ ├── activation_quantizer_1.py │ │ │ │ ├── debug.py │ │ │ │ ├── logger.py │ │ │ │ ├── mixup.py │ │ │ │ ├── plot_variance.py │ │ │ │ ├── preconditioner.py │ │ │ │ ├── quantize.py │ │ │ │ ├── smoothing.py │ │ │ │ ├── test.ipynb │ │ │ │ └── utils.py │ │ │ ├── modeling_flax_xlm_roberta.py │ │ │ ├── modeling_tf_xlm_roberta.py │ │ │ ├── modeling_xlm_roberta.py │ │ │ ├── tokenization_xlm_roberta.py │ │ │ └── tokenization_xlm_roberta_fast.py │ │ ├── xlm_roberta_xl │ │ │ ├── __init__.py │ │ │ ├── configuration_xlm_roberta_xl.py │ │ │ ├── convert_xlm_roberta_xl_original_pytorch_checkpoint_to_pytorch.py │ │ │ └── modeling_xlm_roberta_xl.py │ │ ├── xlnet │ │ │ ├── __init__.py │ │ │ ├── configuration_xlnet.py │ │ │ ├── convert_xlnet_original_tf_checkpoint_to_pytorch.py │ │ │ ├── modeling_tf_xlnet.py │ │ │ ├── modeling_xlnet.py │ │ │ ├── tokenization_xlnet.py │ │ │ └── tokenization_xlnet_fast.py │ │ ├── yolos │ │ │ ├── __init__.py │ │ │ ├── configuration_yolos.py │ │ │ ├── convert_yolos_to_pytorch.py │ │ │ ├── feature_extraction_yolos.py │ │ │ └── modeling_yolos.py │ │ └── yoso │ │ │ ├── __init__.py │ │ │ ├── common.h │ │ │ ├── common_cuda.h │ │ │ ├── common_cuda_device.h │ │ │ ├── configuration_yoso.py │ │ │ ├── convert_yoso_pytorch_to_pytorch.py │ │ │ ├── fast_lsh_cumulation.cu │ │ │ ├── fast_lsh_cumulation.h │ │ │ ├── fast_lsh_cumulation_cuda.cu │ │ │ ├── fast_lsh_cumulation_cuda.h │ │ │ ├── fast_lsh_cumulation_torch.cpp │ │ │ └── modeling_yoso.py │ ├── onnx │ │ ├── __init__.py │ │ ├── __main__.py │ │ ├── config.py │ │ ├── convert.py │ │ ├── features.py │ │ └── utils.py │ ├── optimization.py │ ├── optimization_tf.py │ ├── pipelines │ │ ├── __init__.py │ │ ├── audio_classification.py │ │ ├── audio_utils.py │ │ ├── automatic_speech_recognition.py │ │ ├── base.py │ │ ├── conversational.py │ │ ├── feature_extraction.py │ │ ├── fill_mask.py │ │ ├── image_classification.py │ │ ├── image_segmentation.py │ │ ├── object_detection.py │ │ ├── pt_utils.py │ │ ├── question_answering.py │ │ ├── table_question_answering.py │ │ ├── text2text_generation.py │ │ ├── text_classification.py │ │ ├── text_generation.py │ │ ├── token_classification.py │ │ ├── visual_question_answering.py │ │ ├── zero_shot_classification.py │ │ └── zero_shot_image_classification.py │ ├── processing_utils.py │ ├── pytorch_utils.py │ ├── sagemaker │ │ ├── __init__.py │ │ ├── trainer_sm.py │ │ └── training_args_sm.py │ ├── testing_utils.py │ ├── tf_utils.py │ ├── tokenization_utils.py │ ├── tokenization_utils_base.py │ ├── tokenization_utils_fast.py │ ├── trainer.py │ ├── trainer_callback.py │ ├── trainer_pt_utils.py │ ├── trainer_seq2seq.py │ ├── trainer_tf.py │ ├── trainer_utils.py │ ├── training_args.py │ ├── training_args_seq2seq.py │ ├── training_args_tf.py │ └── utils │ │ ├── __init__.py │ │ ├── bitsandbytes.py │ │ ├── constants.py │ │ ├── doc.py │ │ ├── dummy_detectron2_objects.py │ │ ├── dummy_flax_objects.py │ │ ├── dummy_pt_objects.py │ │ ├── dummy_scatter_objects.py │ │ ├── dummy_sentencepiece_and_speech_objects.py │ │ ├── dummy_sentencepiece_and_tokenizers_objects.py │ │ ├── dummy_sentencepiece_objects.py │ │ ├── dummy_speech_objects.py │ │ ├── dummy_tensorflow_text_objects.py │ │ ├── dummy_tf_objects.py │ │ ├── dummy_timm_and_vision_objects.py │ │ ├── dummy_timm_objects.py │ │ ├── dummy_tokenizers_objects.py │ │ ├── dummy_vision_objects.py │ │ ├── fx.py │ │ ├── generic.py │ │ ├── hp_naming.py │ │ ├── hub.py │ │ ├── import_utils.py │ │ ├── logging.py │ │ ├── model_parallel_utils.py │ │ ├── notebook.py │ │ ├── sentencepiece_model_pb2.py │ │ └── versions.py └── utils_qa.py └── readme.md /Cuda_LSS_HQ/quantize_forward_LSQ+HQ/quantize_forward_easy.cpp: -------------------------------------------------------------------------------- 1 | #include 2 | 3 | // return output, time vector, q_input, q_weight 4 | std::tuple, torch::Tensor, long long int> quantize_cuda(torch::Tensor hx, torch::Tensor hy, float scale_x, float scale_y); 5 | 6 | std::tuple, torch::Tensor, long long int> quantize(torch::Tensor hx, torch::Tensor hy, float scale_x, float scale_y){ 7 | TORCH_CHECK(hx.type().is_cuda(), "x must be a CUDA tensor!"); 8 | TORCH_CHECK(hx.is_contiguous(), "x must be contiguous!"); 9 | TORCH_CHECK(hx.dim() == 2, "x must be 2D!"); 10 | 11 | TORCH_CHECK(hy.type().is_cuda(), "y must be a CUDA tensor!"); 12 | TORCH_CHECK(hy.is_contiguous(), "y must be contiguous!"); 13 | TORCH_CHECK(hy.dim() == 2, "y must be 2D!"); 14 | 15 | return quantize_cuda(hx, hy, scale_x, scale_y); 16 | } 17 | 18 | PYBIND11_MODULE(TORCH_EXTENSION_NAME, m) { 19 | m.def("quantize", &quantize); 20 | } -------------------------------------------------------------------------------- /Cuda_LSS_HQ/quantize_forward_LSQ+HQ/setup_easy.py: -------------------------------------------------------------------------------- 1 | from setuptools import setup 2 | from torch.utils.cpp_extension import BuildExtension, CUDAExtension 3 | import os 4 | 5 | file_path = os.path.dirname(os.getcwd()) 6 | cutlass_path = "cutlass/cutlass/include" 7 | file_path = os.path.join(file_path, cutlass_path) 8 | 9 | setup( 10 | name='quantize_forward_easy', 11 | ext_modules=[ 12 | CUDAExtension(name='quantize_forward_easy', sources=[ 13 | 'quantize_forward_easy.cpp', 14 | 'quantize_forward_easy_kernel.cu', 15 | ], include_dirs=[file_path], 16 | extra_compile_args=["-std=c++17"]) 17 | ], 18 | cmdclass={ 19 | 'build_ext': BuildExtension 20 | }) 21 | -------------------------------------------------------------------------------- /Cuda_LSS_HQ/quantize_grad_input_LSS+LSQ/quantize_grad_input_speed.cpp: -------------------------------------------------------------------------------- 1 | #include 2 | 3 | std::tuple, int> quantize_cuda(torch::Tensor x, int num_bits, torch::Tensor qy, float scaley, torch::Tensor lsq_activation, torch::Tensor first_transform, torch::Tensor second_transform, torch::Tensor x1_len, torch::Tensor x2_len, float scale1); 4 | // std::tuple, int> quantize_cuda(torch::Tensor x, int num_bits, torch::Tensor qy, float scaley, torch::Tensor lsq_activation); 5 | 6 | 7 | std::tuple, int> quantize(torch::Tensor x, int num_bits, torch::Tensor qy, float scaley, torch::Tensor lsq_activation, torch::Tensor first_transform, torch::Tensor second_transform, torch::Tensor x1_len, torch::Tensor x2_len, float scale1){ 8 | // std::tuple, int> quantize(torch::Tensor x, int num_bits, torch::Tensor qy, float scaley, torch::Tensor lsq_activation){ 9 | 10 | return quantize_cuda(x, num_bits, qy, scaley, lsq_activation, first_transform, second_transform, x1_len, x2_len, scale1); 11 | // return quantize_cuda(x, num_bits, qy, scaley, lsq_activation); 12 | } 13 | 14 | 15 | PYBIND11_MODULE(TORCH_EXTENSION_NAME, m) { 16 | m.def("quantize", &quantize); 17 | } -------------------------------------------------------------------------------- /Cuda_LSS_HQ/quantize_grad_input_LSS+LSQ/setup.py: -------------------------------------------------------------------------------- 1 | from setuptools import setup 2 | from torch.utils.cpp_extension import BuildExtension, CUDAExtension 3 | import os 4 | 5 | file_path = os.path.dirname(os.getcwd()) 6 | cutlass_path = "cutlass/cutlass/include" 7 | file_path = os.path.join(file_path, cutlass_path) 8 | print(file_path) 9 | setup( 10 | name='quantize_grad_input_speed', 11 | ext_modules=[ 12 | CUDAExtension(name='quantize_grad_input_speed', sources=[ 13 | 'quantize_grad_input_speed.cpp', 14 | 'quantize_grad_input_speed_kernel.cu', 15 | ], include_dirs=[file_path], 16 | extra_compile_args=["-std=c++17"]) 17 | ], 18 | cmdclass={ 19 | 'build_ext': BuildExtension 20 | }) 21 | -------------------------------------------------------------------------------- /Cuda_LSS_HQ/quantize_grad_weight_LSS+LSQ/quantize_grad_weight_speed.cpp: -------------------------------------------------------------------------------- 1 | #include 2 | 3 | std::tuple, torch::Tensor, torch::Tensor, torch::Tensor, torch::Tensor, float> quantize_cuda(torch::Tensor x, int num_bits, torch::Tensor qy, float scaley, torch::Tensor lsq_weight); 4 | 5 | std::tuple, torch::Tensor, torch::Tensor, torch::Tensor, torch::Tensor, float> quantize(torch::Tensor x, int num_bits, torch::Tensor qy, float scaley, torch::Tensor lsq_weight){ 6 | return quantize_cuda(x, num_bits, qy, scaley, lsq_weight); 7 | } 8 | 9 | 10 | PYBIND11_MODULE(TORCH_EXTENSION_NAME, m) { 11 | m.def("quantize", &quantize); 12 | } -------------------------------------------------------------------------------- /Cuda_LSS_HQ/quantize_grad_weight_LSS+LSQ/setup.py: -------------------------------------------------------------------------------- 1 | from setuptools import setup 2 | from torch.utils.cpp_extension import BuildExtension, CUDAExtension 3 | import os 4 | 5 | file_path = os.path.dirname(os.getcwd()) 6 | cutlass_path = "cutlass/cutlass/include" 7 | file_path = os.path.join(file_path, cutlass_path) 8 | 9 | setup( 10 | name='quantize_grad_weight_speed', 11 | ext_modules=[ 12 | CUDAExtension(name='quantize_grad_weight_speed', sources=[ 13 | 'quantize_grad_weight_speed.cpp', 14 | 'quantize_grad_weight_speed_kernel.cu' 15 | ], include_dirs=[file_path], 16 | extra_compile_args=["-std=c++17"]) 17 | ], 18 | cmdclass={ 19 | 'build_ext': BuildExtension 20 | }) 21 | -------------------------------------------------------------------------------- /Cuda_LSS_HQ/readme_noimage(1).md: -------------------------------------------------------------------------------- 1 | # Hardware Implemented LSS+HQ 2 | 3 | Code for hardware implementation LSS and HQ operator. 4 | 5 | ## INSTALL 6 | 7 | Tested with PyTorch 1.12.1 + CUDA 11.3, on an Tesla A100 GPU. Nvidia RTX 3090 can also work. 8 | 9 | > Note: This cuda program is based on [Nvidia cutlass](https://github.com/NVIDIA/cutlass) version 2.10. You need to pull down the corresponding version library. 10 | 11 | ### CutLass 12 | 13 | ```bash 14 | cd ./cutlass/cutlass 15 | #checkout branch 16 | git checkout feature/2.10/updates_before_tagging 17 | ``` 18 | 19 | ### LSSWeight 20 | 21 | ```bash 22 | cd quantize_grad_weight_LSS+LSQ 23 | python setup.py install 24 | ``` 25 | 26 | ### LSSAct 27 | 28 | ```bash 29 | cd quantize_grad_input_LSS+LSQ 30 | python setup.py install 31 | ``` 32 | 33 | ### HQ 34 | 35 | ```bash 36 | cd quantize_forward_LSQ+HQ 37 | python setup_easy.py install 38 | ``` 39 | 40 | 41 | 42 | ## Speed Test 43 | 44 | ### Flops 45 | 46 | ```bash 47 | cd quantize_grad_weight_LSS+LSQ 48 | mkdir image 49 | python test.py 50 | ``` 51 | 52 | The result is shown at quantize_grad_weight_LSS+LSQ/image/plot_flops.svg 53 | 54 | ### Time proportion 55 | 56 | ```bash 57 | # Time proportion of LSSAct and LSSWeight 58 | cd quantize_grad_weight_LSS+LSQ 59 | mkdir image 60 | python test.py 61 | 62 | # Time proportion of HQ 63 | cd ../quantize_forward_LSQ+HQ 64 | mkdir image 65 | python test.py 66 | ``` 67 | 68 | The result is shown at quantize_grad_weight_LSS+LSQ/image/LSSAct.svg, quantize_grad_weight_LSS+LSQ/image/LSSWeight.svg, quantize_forward_LSQ+HQ/image/HQ.svg, respectively. 69 | -------------------------------------------------------------------------------- /SingleDivide/image.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/haochengxi/Train_Transformers_with_INT4/a3244e235c0ffdba2bd18f25491316096217ed2f/SingleDivide/image.png -------------------------------------------------------------------------------- /SingleDivide/readme.md: -------------------------------------------------------------------------------- 1 | Code for HQ and LSS algorithm. 2 | 3 | Tested with PyTorch 1.12.1 + CUDA 11.3. 4 | 5 | Run task.sh to reproduce the result on GLUE dataset. 6 | 7 | ``` 8 | bash task.sh 9 | ``` 10 | 11 | Change --model_name_or_path bert-base-uncased to bert-large-uncased to reproduce the result for bert-large. 12 | 13 | ### Results 14 | 15 | ![Alt text](image.png) -------------------------------------------------------------------------------- /SingleDivide/squad_adv.sh: -------------------------------------------------------------------------------- 1 | for SEED in 27 28 29 2 | do 3 | accelerate launch test_squad.py \ 4 | --model_name_or_path bert-base-uncased \ 5 | --dataset_name adversarial_qa \ 6 | --dataset_config_name adversarialQA \ 7 | --max_seq_length 384 \ 8 | --doc_stride 128 \ 9 | --output_dir ~/tmp/debug_squad \ 10 | --version_2_with_negative \ 11 | --arch BertForQuestionAnswering \ 12 | --choice quantize -c quantize \ 13 | --abits 4 --wbits 4 --bwbits 4 --bbits 4 \ 14 | --hadamard True \ 15 | --learnable_hadamard False \ 16 | --ifq lsq \ 17 | --wfq lsq \ 18 | --2gw True \ 19 | --2gi True \ 20 | --bmm False \ 21 | --dynamic True \ 22 | --seed $SEED 23 | done -------------------------------------------------------------------------------- /SingleDivide/squadv1.sh: -------------------------------------------------------------------------------- 1 | for SEED in 27 2 | do 3 | accelerate launch test_squad.py \ 4 | --model_name_or_path bert-base-uncased \ 5 | --dataset_name squad \ 6 | --max_seq_length 384 \ 7 | --doc_stride 128 \ 8 | --output_dir ~/tmp/debug_squad \ 9 | --arch BertForQuestionAnswering \ 10 | --choice quantize -c quantize \ 11 | --abits 4 --wbits 4 --bwbits 4 --bbits 4 \ 12 | --hadamard True \ 13 | --learnable_hadamard False \ 14 | --ifq lsq \ 15 | --wfq lsq \ 16 | --2gw True \ 17 | --2gi True \ 18 | --bmm False \ 19 | --dynamic True \ 20 | --seed $SEED 21 | done -------------------------------------------------------------------------------- /SingleDivide/squadv2.sh: -------------------------------------------------------------------------------- 1 | for SEED in 27 28 29 2 | do 3 | accelerate launch test_squad.py \ 4 | --model_name_or_path bert-base-uncased \ 5 | --dataset_name squad_v2 \ 6 | --max_seq_length 384 \ 7 | --doc_stride 128 \ 8 | --output_dir ~/tmp/debug_squad \ 9 | --version_2_with_negative \ 10 | --arch BertForQuestionAnswering \ 11 | --choice quantize -c quantize \ 12 | --abits 4 --wbits 4 --bwbits 4 --bbits 4 \ 13 | --hadamard True \ 14 | --learnable_hadamard False \ 15 | --ifq lsq \ 16 | --wfq lsq \ 17 | --2gw True \ 18 | --2gi True \ 19 | --bmm False \ 20 | --dynamic True \ 21 | --seed $SEED 22 | done -------------------------------------------------------------------------------- /SingleDivide/task.sh: -------------------------------------------------------------------------------- 1 | for TASK in cola 2 | do 3 | python test_mrpc.py \ 4 | --model_name_or_path bert-base-uncased \ 5 | --choice quantize \ 6 | --training-bit all4bit \ 7 | --hadamard True \ 8 | --learnable_hadamard False \ 9 | --ifq lsq \ 10 | --wfq lsq \ 11 | --2gw True \ 12 | --2gi True \ 13 | --task $TASK \ 14 | --bmm False \ 15 | --dynamic True \ 16 | --fp16 True 17 | done 18 | -------------------------------------------------------------------------------- /SingleDivide/task_conll.sh: -------------------------------------------------------------------------------- 1 | for SEED in 27 28 29 2 | do 3 | accelerate launch test_token.py \ 4 | --model_name_or_path bert-base-uncased \ 5 | --dataset_name conll2003 \ 6 | --task_name ner \ 7 | --max_length 128 \ 8 | --per_device_train_batch_size 32 \ 9 | --learning_rate 2e-5 \ 10 | --num_train_epochs 3 \ 11 | --output_dir test_token_result_quantize/0 \ 12 | --arch BertForTokenClassification \ 13 | --with_tracking \ 14 | --choice quantize -c quantize \ 15 | --abits 4 --wbits 4 --bwbits 4 --bbits 4 \ 16 | --hadamard True \ 17 | --learnable_hadamard False \ 18 | --ifq lsq \ 19 | --wfq lsq \ 20 | --2gw True \ 21 | --2gi True \ 22 | --bmm False \ 23 | --dynamic True \ 24 | --seed $SEED 25 | done -------------------------------------------------------------------------------- /SingleDivide/task_swag.sh: -------------------------------------------------------------------------------- 1 | for SEED in 27 28 29 2 | do 3 | accelerate launch test_swag.py \ 4 | --model_name_or_path bert-base-cased \ 5 | --dataset_name swag \ 6 | --max_seq_length 128 \ 7 | --per_device_train_batch_size 32 \ 8 | --learning_rate 2e-5 \ 9 | --num_train_epochs 3 \ 10 | --output_dir /tmp/swag/ \ 11 | --arch BertForMultipleChoice \ 12 | --choice quantize -c quantize \ 13 | --abits 4 --wbits 4 --bwbits 4 --bbits 4 \ 14 | --hadamard True \ 15 | --learnable_hadamard False \ 16 | --ifq lsq \ 17 | --wfq lsq \ 18 | --2gw True \ 19 | --2gi True \ 20 | --bmm False \ 21 | --dynamic True \ 22 | --seed $SEED 23 | done -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/benchmark/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/haochengxi/Train_Transformers_with_INT4/a3244e235c0ffdba2bd18f25491316096217ed2f/SingleDivide/transformersLocal/benchmark/__init__.py -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/commands/__init__.py: -------------------------------------------------------------------------------- 1 | # Copyright 2020 The HuggingFace Team. All rights reserved. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | from abc import ABC, abstractmethod 16 | from argparse import ArgumentParser 17 | 18 | 19 | class BaseTransformersCLICommand(ABC): 20 | @staticmethod 21 | @abstractmethod 22 | def register_subcommand(parser: ArgumentParser): 23 | raise NotImplementedError() 24 | 25 | @abstractmethod 26 | def run(self): 27 | raise NotImplementedError() 28 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/commands/download.py: -------------------------------------------------------------------------------- 1 | # Copyright 2020 The HuggingFace Team. All rights reserved. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | from argparse import ArgumentParser 16 | 17 | from . import BaseTransformersCLICommand 18 | 19 | 20 | def download_command_factory(args): 21 | return DownloadCommand(args.model, args.cache_dir, args.force) 22 | 23 | 24 | class DownloadCommand(BaseTransformersCLICommand): 25 | @staticmethod 26 | def register_subcommand(parser: ArgumentParser): 27 | download_parser = parser.add_parser("download") 28 | download_parser.add_argument( 29 | "--cache-dir", type=str, default=None, help="Path to location to store the models" 30 | ) 31 | download_parser.add_argument( 32 | "--force", action="store_true", help="Force the model to be download even if already in cache-dir" 33 | ) 34 | download_parser.add_argument("model", type=str, help="Name of the model to download") 35 | download_parser.set_defaults(func=download_command_factory) 36 | 37 | def __init__(self, model: str, cache: str, force: bool): 38 | self._model = model 39 | self._cache = cache 40 | self._force = force 41 | 42 | def run(self): 43 | from ..models.auto import AutoModel, AutoTokenizer 44 | 45 | AutoModel.from_pretrained(self._model, cache_dir=self._cache, force_download=self._force) 46 | AutoTokenizer.from_pretrained(self._model, cache_dir=self._cache, force_download=self._force) 47 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/commands/transformers_cli.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # Copyright 2020 The HuggingFace Team. All rights reserved. 3 | # 4 | # Licensed under the Apache License, Version 2.0 (the "License"); 5 | # you may not use this file except in compliance with the License. 6 | # You may obtain a copy of the License at 7 | # 8 | # http://www.apache.org/licenses/LICENSE-2.0 9 | # 10 | # Unless required by applicable law or agreed to in writing, software 11 | # distributed under the License is distributed on an "AS IS" BASIS, 12 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | # See the License for the specific language governing permissions and 14 | # limitations under the License. 15 | 16 | from argparse import ArgumentParser 17 | 18 | from .add_new_model import AddNewModelCommand 19 | from .add_new_model_like import AddNewModelLikeCommand 20 | from .convert import ConvertCommand 21 | from .download import DownloadCommand 22 | from .env import EnvironmentCommand 23 | from .lfs import LfsCommands 24 | from .pt_to_tf import PTtoTFCommand 25 | from .run import RunCommand 26 | from .serving import ServeCommand 27 | from .user import UserCommands 28 | 29 | 30 | def main(): 31 | parser = ArgumentParser("Transformers CLI tool", usage="transformers-cli []") 32 | commands_parser = parser.add_subparsers(help="transformers-cli command helpers") 33 | 34 | # Register commands 35 | ConvertCommand.register_subcommand(commands_parser) 36 | DownloadCommand.register_subcommand(commands_parser) 37 | EnvironmentCommand.register_subcommand(commands_parser) 38 | RunCommand.register_subcommand(commands_parser) 39 | ServeCommand.register_subcommand(commands_parser) 40 | UserCommands.register_subcommand(commands_parser) 41 | AddNewModelCommand.register_subcommand(commands_parser) 42 | AddNewModelLikeCommand.register_subcommand(commands_parser) 43 | LfsCommands.register_subcommand(commands_parser) 44 | PTtoTFCommand.register_subcommand(commands_parser) 45 | 46 | # Let's go 47 | args = parser.parse_args() 48 | 49 | if not hasattr(args, "func"): 50 | parser.print_help() 51 | exit(1) 52 | 53 | # Run 54 | service = args.func(args) 55 | service.run() 56 | 57 | 58 | if __name__ == "__main__": 59 | main() 60 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/data/__init__.py: -------------------------------------------------------------------------------- 1 | # flake8: noqa 2 | # There's no way to ignore "F401 '...' imported but unused" warnings in this 3 | # module, but to preserve other warnings. So, don't check this module at all. 4 | 5 | # Copyright 2020 The HuggingFace Team. All rights reserved. 6 | # 7 | # Licensed under the Apache License, Version 2.0 (the "License"); 8 | # you may not use this file except in compliance with the License. 9 | # You may obtain a copy of the License at 10 | # 11 | # http://www.apache.org/licenses/LICENSE-2.0 12 | # 13 | # Unless required by applicable law or agreed to in writing, software 14 | # distributed under the License is distributed on an "AS IS" BASIS, 15 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | # See the License for the specific language governing permissions and 17 | # limitations under the License. 18 | 19 | from .data_collator import ( 20 | DataCollatorForLanguageModeling, 21 | DataCollatorForPermutationLanguageModeling, 22 | DataCollatorForSeq2Seq, 23 | DataCollatorForSOP, 24 | DataCollatorForTokenClassification, 25 | DataCollatorForWholeWordMask, 26 | DataCollatorWithPadding, 27 | DefaultDataCollator, 28 | default_data_collator, 29 | ) 30 | from .metrics import glue_compute_metrics, xnli_compute_metrics 31 | from .processors import ( 32 | DataProcessor, 33 | InputExample, 34 | InputFeatures, 35 | SingleSentenceClassificationProcessor, 36 | SquadExample, 37 | SquadFeatures, 38 | SquadV1Processor, 39 | SquadV2Processor, 40 | glue_convert_examples_to_features, 41 | glue_output_modes, 42 | glue_processors, 43 | glue_tasks_num_labels, 44 | squad_convert_examples_to_features, 45 | xnli_output_modes, 46 | xnli_processors, 47 | xnli_tasks_num_labels, 48 | ) 49 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/data/datasets/__init__.py: -------------------------------------------------------------------------------- 1 | # flake8: noqa 2 | # There's no way to ignore "F401 '...' imported but unused" warnings in this 3 | # module, but to preserve other warnings. So, don't check this module at all. 4 | 5 | # Copyright 2020 The HuggingFace Team. All rights reserved. 6 | # 7 | # Licensed under the Apache License, Version 2.0 (the "License"); 8 | # you may not use this file except in compliance with the License. 9 | # You may obtain a copy of the License at 10 | # 11 | # http://www.apache.org/licenses/LICENSE-2.0 12 | # 13 | # Unless required by applicable law or agreed to in writing, software 14 | # distributed under the License is distributed on an "AS IS" BASIS, 15 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | # See the License for the specific language governing permissions and 17 | # limitations under the License. 18 | 19 | from .glue import GlueDataset, GlueDataTrainingArguments 20 | from .language_modeling import ( 21 | LineByLineTextDataset, 22 | LineByLineWithRefDataset, 23 | LineByLineWithSOPTextDataset, 24 | TextDataset, 25 | TextDatasetForNextSentencePrediction, 26 | ) 27 | from .squad import SquadDataset, SquadDataTrainingArguments 28 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/data/processors/__init__.py: -------------------------------------------------------------------------------- 1 | # flake8: noqa 2 | # There's no way to ignore "F401 '...' imported but unused" warnings in this 3 | # module, but to preserve other warnings. So, don't check this module at all. 4 | 5 | # Copyright 2020 The HuggingFace Team. All rights reserved. 6 | # 7 | # Licensed under the Apache License, Version 2.0 (the "License"); 8 | # you may not use this file except in compliance with the License. 9 | # You may obtain a copy of the License at 10 | # 11 | # http://www.apache.org/licenses/LICENSE-2.0 12 | # 13 | # Unless required by applicable law or agreed to in writing, software 14 | # distributed under the License is distributed on an "AS IS" BASIS, 15 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | # See the License for the specific language governing permissions and 17 | # limitations under the License. 18 | 19 | from .glue import glue_convert_examples_to_features, glue_output_modes, glue_processors, glue_tasks_num_labels 20 | from .squad import SquadExample, SquadFeatures, SquadV1Processor, SquadV2Processor, squad_convert_examples_to_features 21 | from .utils import DataProcessor, InputExample, InputFeatures, SingleSentenceClassificationProcessor 22 | from .xnli import xnli_output_modes, xnli_processors, xnli_tasks_num_labels 23 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/dependency_versions_check.py: -------------------------------------------------------------------------------- 1 | # Copyright 2020 The HuggingFace Team. All rights reserved. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | import sys 15 | 16 | from .dependency_versions_table import deps 17 | from .utils.versions import require_version, require_version_core 18 | 19 | 20 | # define which module versions we always want to check at run time 21 | # (usually the ones defined in `install_requires` in setup.py) 22 | # 23 | # order specific notes: 24 | # - tqdm must be checked before tokenizers 25 | 26 | pkgs_to_check_at_runtime = "python tqdm regex requests packaging filelock numpy tokenizers".split() 27 | if sys.version_info < (3, 7): 28 | pkgs_to_check_at_runtime.append("dataclasses") 29 | if sys.version_info < (3, 8): 30 | pkgs_to_check_at_runtime.append("importlib_metadata") 31 | 32 | for pkg in pkgs_to_check_at_runtime: 33 | if pkg in deps: 34 | if pkg == "tokenizers": 35 | # must be loaded here, or else tqdm check may fail 36 | from .utils import is_tokenizers_available 37 | 38 | if not is_tokenizers_available(): 39 | continue # not required, check version only if installed 40 | 41 | require_version_core(deps[pkg]) 42 | else: 43 | raise ValueError(f"can't find {pkg} in {deps.keys()}, check dependency_versions_table.py") 44 | 45 | 46 | def dep_version_check(pkg, hint=None): 47 | require_version(deps[pkg], hint) 48 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/albert/convert_albert_original_tf_checkpoint_to_pytorch.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | # Copyright 2018 The HuggingFace Inc. team. 3 | # 4 | # Licensed under the Apache License, Version 2.0 (the "License"); 5 | # you may not use this file except in compliance with the License. 6 | # You may obtain a copy of the License at 7 | # 8 | # http://www.apache.org/licenses/LICENSE-2.0 9 | # 10 | # Unless required by applicable law or agreed to in writing, software 11 | # distributed under the License is distributed on an "AS IS" BASIS, 12 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | # See the License for the specific language governing permissions and 14 | # limitations under the License. 15 | """Convert ALBERT checkpoint.""" 16 | 17 | 18 | import argparse 19 | 20 | import torch 21 | 22 | from transformers import AlbertConfig, AlbertForPreTraining, load_tf_weights_in_albert 23 | from transformers.utils import logging 24 | 25 | 26 | logging.set_verbosity_info() 27 | 28 | 29 | def convert_tf_checkpoint_to_pytorch(tf_checkpoint_path, albert_config_file, pytorch_dump_path): 30 | # Initialise PyTorch model 31 | config = AlbertConfig.from_json_file(albert_config_file) 32 | print(f"Building PyTorch model from configuration: {config}") 33 | model = AlbertForPreTraining(config) 34 | 35 | # Load weights from tf checkpoint 36 | load_tf_weights_in_albert(model, config, tf_checkpoint_path) 37 | 38 | # Save pytorch-model 39 | print(f"Save PyTorch model to {pytorch_dump_path}") 40 | torch.save(model.state_dict(), pytorch_dump_path) 41 | 42 | 43 | if __name__ == "__main__": 44 | parser = argparse.ArgumentParser() 45 | # Required parameters 46 | parser.add_argument( 47 | "--tf_checkpoint_path", default=None, type=str, required=True, help="Path to the TensorFlow checkpoint path." 48 | ) 49 | parser.add_argument( 50 | "--albert_config_file", 51 | default=None, 52 | type=str, 53 | required=True, 54 | help=( 55 | "The config json file corresponding to the pre-trained ALBERT model. \n" 56 | "This specifies the model architecture." 57 | ), 58 | ) 59 | parser.add_argument( 60 | "--pytorch_dump_path", default=None, type=str, required=True, help="Path to the output PyTorch model." 61 | ) 62 | args = parser.parse_args() 63 | convert_tf_checkpoint_to_pytorch(args.tf_checkpoint_path, args.albert_config_file, args.pytorch_dump_path) 64 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/barthez/__init__.py: -------------------------------------------------------------------------------- 1 | # flake8: noqa 2 | # There's no way to ignore "F401 '...' imported but unused" warnings in this 3 | # module, but to preserve other warnings. So, don't check this module at all. 4 | 5 | # Copyright 2020 The HuggingFace Team. All rights reserved. 6 | # 7 | # Licensed under the Apache License, Version 2.0 (the "License"); 8 | # you may not use this file except in compliance with the License. 9 | # You may obtain a copy of the License at 10 | # 11 | # http://www.apache.org/licenses/LICENSE-2.0 12 | # 13 | # Unless required by applicable law or agreed to in writing, software 14 | # distributed under the License is distributed on an "AS IS" BASIS, 15 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | # See the License for the specific language governing permissions and 17 | # limitations under the License. 18 | 19 | from typing import TYPE_CHECKING 20 | 21 | from ...utils import OptionalDependencyNotAvailable, _LazyModule, is_sentencepiece_available, is_tokenizers_available 22 | 23 | 24 | _import_structure = {} 25 | 26 | try: 27 | if not is_sentencepiece_available(): 28 | raise OptionalDependencyNotAvailable() 29 | except OptionalDependencyNotAvailable: 30 | pass 31 | else: 32 | _import_structure["tokenization_barthez"] = ["BarthezTokenizer"] 33 | 34 | try: 35 | if not is_tokenizers_available(): 36 | raise OptionalDependencyNotAvailable() 37 | except OptionalDependencyNotAvailable: 38 | pass 39 | else: 40 | _import_structure["tokenization_barthez_fast"] = ["BarthezTokenizerFast"] 41 | 42 | 43 | if TYPE_CHECKING: 44 | 45 | try: 46 | if not is_sentencepiece_available(): 47 | raise OptionalDependencyNotAvailable() 48 | except OptionalDependencyNotAvailable: 49 | pass 50 | else: 51 | from .tokenization_barthez import BarthezTokenizer 52 | 53 | try: 54 | if not is_tokenizers_available(): 55 | raise OptionalDependencyNotAvailable() 56 | except OptionalDependencyNotAvailable: 57 | pass 58 | else: 59 | from .tokenization_barthez_fast import BarthezTokenizerFast 60 | 61 | else: 62 | import sys 63 | 64 | sys.modules[__name__] = _LazyModule(__name__, globals()["__file__"], _import_structure, module_spec=__spec__) 65 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/bartpho/__init__.py: -------------------------------------------------------------------------------- 1 | # flake8: noqa 2 | # There's no way to ignore "F401 '...' imported but unused" warnings in this 3 | # module, but to preserve other warnings. So, don't check this module at all. 4 | 5 | # Copyright 2021 The HuggingFace Team. All rights reserved. 6 | # 7 | # Licensed under the Apache License, Version 2.0 (the "License"); 8 | # you may not use this file except in compliance with the License. 9 | # You may obtain a copy of the License at 10 | # 11 | # http://www.apache.org/licenses/LICENSE-2.0 12 | # 13 | # Unless required by applicable law or agreed to in writing, software 14 | # distributed under the License is distributed on an "AS IS" BASIS, 15 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | # See the License for the specific language governing permissions and 17 | # limitations under the License. 18 | 19 | from typing import TYPE_CHECKING 20 | 21 | from ...utils import OptionalDependencyNotAvailable, _LazyModule, is_sentencepiece_available 22 | 23 | 24 | _import_structure = {} 25 | 26 | try: 27 | if not is_sentencepiece_available(): 28 | raise OptionalDependencyNotAvailable() 29 | except OptionalDependencyNotAvailable: 30 | pass 31 | else: 32 | _import_structure["tokenization_bartpho"] = ["BartphoTokenizer"] 33 | 34 | if TYPE_CHECKING: 35 | try: 36 | if not is_sentencepiece_available(): 37 | raise OptionalDependencyNotAvailable() 38 | except OptionalDependencyNotAvailable: 39 | pass 40 | else: 41 | from .tokenization_bartpho import BartphoTokenizer 42 | 43 | else: 44 | import sys 45 | 46 | sys.modules[__name__] = _LazyModule(__name__, globals()["__file__"], _import_structure, module_spec=__spec__) 47 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/bert/convert_bert_original_tf_checkpoint_to_pytorch.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | # Copyright 2018 The HuggingFace Inc. team. 3 | # 4 | # Licensed under the Apache License, Version 2.0 (the "License"); 5 | # you may not use this file except in compliance with the License. 6 | # You may obtain a copy of the License at 7 | # 8 | # http://www.apache.org/licenses/LICENSE-2.0 9 | # 10 | # Unless required by applicable law or agreed to in writing, software 11 | # distributed under the License is distributed on an "AS IS" BASIS, 12 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | # See the License for the specific language governing permissions and 14 | # limitations under the License. 15 | """Convert BERT checkpoint.""" 16 | 17 | 18 | import argparse 19 | 20 | import torch 21 | 22 | from transformers import BertConfig, BertForPreTraining, load_tf_weights_in_bert 23 | from transformers.utils import logging 24 | 25 | 26 | logging.set_verbosity_info() 27 | 28 | 29 | def convert_tf_checkpoint_to_pytorch(tf_checkpoint_path, bert_config_file, pytorch_dump_path): 30 | # Initialise PyTorch model 31 | config = BertConfig.from_json_file(bert_config_file) 32 | print(f"Building PyTorch model from configuration: {config}") 33 | model = BertForPreTraining(config) 34 | 35 | # Load weights from tf checkpoint 36 | load_tf_weights_in_bert(model, config, tf_checkpoint_path) 37 | 38 | # Save pytorch-model 39 | print(f"Save PyTorch model to {pytorch_dump_path}") 40 | torch.save(model.state_dict(), pytorch_dump_path) 41 | 42 | 43 | if __name__ == "__main__": 44 | parser = argparse.ArgumentParser() 45 | # Required parameters 46 | parser.add_argument( 47 | "--tf_checkpoint_path", default=None, type=str, required=True, help="Path to the TensorFlow checkpoint path." 48 | ) 49 | parser.add_argument( 50 | "--bert_config_file", 51 | default=None, 52 | type=str, 53 | required=True, 54 | help=( 55 | "The config json file corresponding to the pre-trained BERT model. \n" 56 | "This specifies the model architecture." 57 | ), 58 | ) 59 | parser.add_argument( 60 | "--pytorch_dump_path", default=None, type=str, required=True, help="Path to the output PyTorch model." 61 | ) 62 | args = parser.parse_args() 63 | convert_tf_checkpoint_to_pytorch(args.tf_checkpoint_path, args.bert_config_file, args.pytorch_dump_path) 64 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/bert/image_classification/__init__.py: -------------------------------------------------------------------------------- 1 | from . import logger 2 | from . import utils 3 | from . import mixup 4 | from . import smoothing 5 | from . import quantize 6 | from . import preconditioner 7 | from . import activation_quantizer_1 8 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/bert/image_classification/mixup.py: -------------------------------------------------------------------------------- 1 | import torch 2 | import torch.nn as nn 3 | import numpy as np 4 | 5 | 6 | def mixup(alpha, num_classes, data, target): 7 | with torch.no_grad(): 8 | bs = data.size(0) 9 | c = np.random.beta(alpha, alpha) 10 | 11 | perm = torch.randperm(bs).cuda() 12 | 13 | md = c * data + (1-c) * data[perm, :] 14 | mt = c * target + (1-c) * target[perm, :] 15 | return md, mt 16 | 17 | 18 | class MixUpWrapper(object): 19 | def __init__(self, alpha, num_classes, dataloader): 20 | self.alpha = alpha 21 | self.dataloader = dataloader 22 | self.num_classes = num_classes 23 | 24 | def mixup_loader(self, loader): 25 | for input, target in loader: 26 | i, t = mixup(self.alpha, self.num_classes, input, target) 27 | yield i, t 28 | 29 | def __iter__(self): 30 | return self.mixup_loader(self.dataloader) 31 | 32 | 33 | class NLLMultiLabelSmooth(nn.Module): 34 | def __init__(self, smoothing = 0.0): 35 | super(NLLMultiLabelSmooth, self).__init__() 36 | self.confidence = 1.0 - smoothing 37 | self.smoothing = smoothing 38 | 39 | def forward(self, x, target): 40 | if self.training: 41 | x = x.float() 42 | target = target.float() 43 | logprobs = torch.nn.functional.log_softmax(x, dim = -1) 44 | 45 | nll_loss = -logprobs * target 46 | nll_loss = nll_loss.sum(-1) 47 | 48 | smooth_loss = -logprobs.mean(dim=-1) 49 | 50 | loss = self.confidence * nll_loss + self.smoothing * smooth_loss 51 | 52 | return loss.mean() 53 | else: 54 | return torch.nn.functional.cross_entropy(x, target) 55 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/bert/image_classification/plot_variance.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import matplotlib 3 | matplotlib.use('Agg') 4 | from matplotlib import pyplot as plt 5 | from matplotlib.colors import LogNorm 6 | import pickle 7 | import seaborn as sns 8 | 9 | weight_names = pickle.load(open('layer_names.pkl', 'rb')) 10 | 11 | grads = np.load('error_profile_5.npy') 12 | grads = np.maximum(grads, 0) 13 | quantizers = grads.sum(1) 14 | variances = grads.sum(0) 15 | 16 | # grads = np.minimum(grads, 1) 17 | # grads *= 1000 18 | for i in range(grads.shape[0]): 19 | for j in range(grads.shape[1]): 20 | if j > i: 21 | grads[i, j] = 0 22 | 23 | fig, ax = plt.subplots(figsize=(20, 20)) 24 | im = ax.imshow(grads, cmap='Blues', norm=LogNorm(vmin=0.01, vmax=10.0)) 25 | ax.set_xticks(np.arange(len(weight_names))) 26 | ax.set_yticks(np.arange(len(weight_names)+1)) 27 | ax.set_xticklabels(weight_names) 28 | weight_names.append('sample') 29 | ax.set_yticklabels(weight_names) 30 | 31 | ax.tick_params(top=True, bottom=False, 32 | labeltop=True, labelbottom=False) 33 | plt.setp(ax.get_xticklabels(), rotation=-30, ha="right", 34 | rotation_mode="anchor") 35 | cbar = ax.figure.colorbar(im, ax=ax) 36 | 37 | for i in range(grads.shape[0]): 38 | for j in range(grads.shape[1]): 39 | text = ax.text(j, i, int(grads[i, j]*10), 40 | ha="center", va="center") 41 | 42 | 43 | fig.savefig('variance_profile.pdf') 44 | 45 | fig, ax = plt.subplots(figsize=(20, 20)) 46 | sns.barplot(x=np.arange(quantizers.shape[0]), y=quantizers, ax=ax) 47 | ax.set_xticks(np.arange(len(weight_names)) + 0.5) 48 | ax.set_xticklabels(weight_names) 49 | plt.setp(ax.get_xticklabels(), rotation=-30, ha="right", 50 | rotation_mode="anchor") 51 | ax.set_xlabel('quantizer') 52 | ax.set_ylabel('variance') 53 | fig.savefig('quantizers.pdf') 54 | 55 | fig, ax = plt.subplots(figsize=(20, 20)) 56 | sns.barplot(x=np.arange(variances.shape[0]), y=variances, ax=ax) 57 | weight_names.pop(-1) 58 | ax.set_xticks(np.arange(len(weight_names)) + 0.5) 59 | ax.set_xticklabels(weight_names) 60 | plt.setp(ax.get_xticklabels(), rotation=-30, ha="right", 61 | rotation_mode="anchor") 62 | ax.set_xlabel('parameter') 63 | ax.set_ylabel('variance') 64 | fig.savefig('parameter.pdf') 65 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/bert/image_classification/smoothing.py: -------------------------------------------------------------------------------- 1 | import torch 2 | import torch.nn as nn 3 | 4 | class LabelSmoothing(nn.Module): 5 | """ 6 | NLL loss with label smoothing. 7 | """ 8 | def __init__(self, smoothing=0.0): 9 | """ 10 | Constructor for the LabelSmoothing module. 11 | 12 | :param smoothing: label smoothing factor 13 | """ 14 | super(LabelSmoothing, self).__init__() 15 | self.confidence = 1.0 - smoothing 16 | self.smoothing = smoothing 17 | 18 | def forward(self, x, target): 19 | logprobs = torch.nn.functional.log_softmax(x, dim=-1) 20 | 21 | nll_loss = -logprobs.gather(dim=-1, index=target.unsqueeze(1)) 22 | nll_loss = nll_loss.squeeze(1) 23 | smooth_loss = -logprobs.mean(dim=-1) 24 | loss = self.confidence * nll_loss + self.smoothing * smooth_loss 25 | return loss.mean() 26 | 27 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/bert/image_classification/test.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "import torch" 10 | ] 11 | }, 12 | { 13 | "cell_type": "code", 14 | "execution_count": 1, 15 | "metadata": {}, 16 | "outputs": [ 17 | { 18 | "name": "stdout", 19 | "output_type": "stream", 20 | "text": [ 21 | "Original Tensor:\n", 22 | "tensor([ 1.7234, -0.6239, 0.3089, -1.1292, 1.0927])\n", 23 | "\n", 24 | "Indices where norm_x < 1:\n", 25 | "tensor([1, 2, 3])\n" 26 | ] 27 | } 28 | ], 29 | "source": [ 30 | "import torch\n", 31 | "\n", 32 | "# create a 1D tensor\n", 33 | "norm_x = torch.randn(5)\n", 34 | "\n", 35 | "# find indices where norm_x < 1\n", 36 | "indices = torch.where(norm_x < 1)[0]\n", 37 | "\n", 38 | "print(\"Original Tensor:\")\n", 39 | "print(norm_x)\n", 40 | "\n", 41 | "print(\"\\nIndices where norm_x < 1:\")\n", 42 | "print(indices)" 43 | ] 44 | } 45 | ], 46 | "metadata": { 47 | "kernelspec": { 48 | "display_name": "base", 49 | "language": "python", 50 | "name": "python3" 51 | }, 52 | "language_info": { 53 | "codemirror_mode": { 54 | "name": "ipython", 55 | "version": 3 56 | }, 57 | "file_extension": ".py", 58 | "mimetype": "text/x-python", 59 | "name": "python", 60 | "nbconvert_exporter": "python", 61 | "pygments_lexer": "ipython3", 62 | "version": "3.9.12" 63 | }, 64 | "orig_nbformat": 4, 65 | "vscode": { 66 | "interpreter": { 67 | "hash": "88279d2366fe020547cde40dd65aa0e3aa662a6ec1f3ca12d88834876c85e1a6" 68 | } 69 | } 70 | }, 71 | "nbformat": 4, 72 | "nbformat_minor": 2 73 | } 74 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/bert_japanese/__init__.py: -------------------------------------------------------------------------------- 1 | # flake8: noqa 2 | # There's no way to ignore "F401 '...' imported but unused" warnings in this 3 | # module, but to preserve other warnings. So, don't check this module at all. 4 | 5 | # Copyright 2020 The HuggingFace Team. All rights reserved. 6 | # 7 | # Licensed under the Apache License, Version 2.0 (the "License"); 8 | # you may not use this file except in compliance with the License. 9 | # You may obtain a copy of the License at 10 | # 11 | # http://www.apache.org/licenses/LICENSE-2.0 12 | # 13 | # Unless required by applicable law or agreed to in writing, software 14 | # distributed under the License is distributed on an "AS IS" BASIS, 15 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | # See the License for the specific language governing permissions and 17 | # limitations under the License. 18 | 19 | from typing import TYPE_CHECKING 20 | 21 | from ...utils import _LazyModule 22 | 23 | 24 | _import_structure = {"tokenization_bert_japanese": ["BertJapaneseTokenizer", "CharacterTokenizer", "MecabTokenizer"]} 25 | 26 | 27 | if TYPE_CHECKING: 28 | from .tokenization_bert_japanese import BertJapaneseTokenizer, CharacterTokenizer, MecabTokenizer 29 | 30 | else: 31 | import sys 32 | 33 | sys.modules[__name__] = _LazyModule(__name__, globals()["__file__"], _import_structure, module_spec=__spec__) 34 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/bertweet/__init__.py: -------------------------------------------------------------------------------- 1 | # flake8: noqa 2 | # There's no way to ignore "F401 '...' imported but unused" warnings in this 3 | # module, but to preserve other warnings. So, don't check this module at all. 4 | 5 | # Copyright 2020 The HuggingFace Team. All rights reserved. 6 | # 7 | # Licensed under the Apache License, Version 2.0 (the "License"); 8 | # you may not use this file except in compliance with the License. 9 | # You may obtain a copy of the License at 10 | # 11 | # http://www.apache.org/licenses/LICENSE-2.0 12 | # 13 | # Unless required by applicable law or agreed to in writing, software 14 | # distributed under the License is distributed on an "AS IS" BASIS, 15 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | # See the License for the specific language governing permissions and 17 | # limitations under the License. 18 | 19 | from typing import TYPE_CHECKING 20 | 21 | from ...utils import _LazyModule 22 | 23 | 24 | _import_structure = {"tokenization_bertweet": ["BertweetTokenizer"]} 25 | 26 | 27 | if TYPE_CHECKING: 28 | from .tokenization_bertweet import BertweetTokenizer 29 | 30 | else: 31 | import sys 32 | 33 | sys.modules[__name__] = _LazyModule(__name__, globals()["__file__"], _import_structure, module_spec=__spec__) 34 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/bort/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/haochengxi/Train_Transformers_with_INT4/a3244e235c0ffdba2bd18f25491316096217ed2f/SingleDivide/transformersLocal/models/bort/__init__.py -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/byt5/__init__.py: -------------------------------------------------------------------------------- 1 | # flake8: noqa 2 | # There's no way to ignore "F401 '...' imported but unused" warnings in this 3 | # module, but to preserve other warnings. So, don't check this module at all. 4 | 5 | # Copyright 2021 The HuggingFace Team. All rights reserved. 6 | # 7 | # Licensed under the Apache License, Version 2.0 (the "License"); 8 | # you may not use this file except in compliance with the License. 9 | # You may obtain a copy of the License at 10 | # 11 | # http://www.apache.org/licenses/LICENSE-2.0 12 | # 13 | # Unless required by applicable law or agreed to in writing, software 14 | # distributed under the License is distributed on an "AS IS" BASIS, 15 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | # See the License for the specific language governing permissions and 17 | # limitations under the License. 18 | 19 | from typing import TYPE_CHECKING 20 | 21 | from ...utils import _LazyModule 22 | 23 | 24 | _import_structure = {"tokenization_byt5": ["ByT5Tokenizer"]} 25 | 26 | 27 | if TYPE_CHECKING: 28 | from .tokenization_byt5 import ByT5Tokenizer 29 | else: 30 | import sys 31 | 32 | sys.modules[__name__] = _LazyModule(__name__, globals()["__file__"], _import_structure, module_spec=__spec__) 33 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/byt5/convert_byt5_original_tf_checkpoint_to_pytorch.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | # Copyright 2018 The T5 authors and HuggingFace Inc. team. 3 | # 4 | # Licensed under the Apache License, Version 2.0 (the "License"); 5 | # you may not use this file except in compliance with the License. 6 | # You may obtain a copy of the License at 7 | # 8 | # http://www.apache.org/licenses/LICENSE-2.0 9 | # 10 | # Unless required by applicable law or agreed to in writing, software 11 | # distributed under the License is distributed on an "AS IS" BASIS, 12 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | # See the License for the specific language governing permissions and 14 | # limitations under the License. 15 | """Convert T5 checkpoint.""" 16 | 17 | 18 | import argparse 19 | 20 | from transformers import T5Config, T5ForConditionalGeneration, load_tf_weights_in_t5 21 | from transformers.utils import logging 22 | 23 | 24 | logging.set_verbosity_info() 25 | 26 | 27 | def convert_tf_checkpoint_to_pytorch(tf_checkpoint_path, config_file, pytorch_dump_path): 28 | # Initialise PyTorch model 29 | config = T5Config.from_json_file(config_file) 30 | print(f"Building PyTorch model from configuration: {config}") 31 | model = T5ForConditionalGeneration(config) 32 | 33 | # Load weights from tf checkpoint 34 | load_tf_weights_in_t5(model, config, tf_checkpoint_path) 35 | 36 | # Save pytorch-model 37 | print(f"Save PyTorch model to {pytorch_dump_path}") 38 | model.save_pretrained(pytorch_dump_path) 39 | 40 | 41 | if __name__ == "__main__": 42 | parser = argparse.ArgumentParser() 43 | # Required parameters 44 | parser.add_argument( 45 | "--tf_checkpoint_path", default=None, type=str, required=True, help="Path to the TensorFlow checkpoint path." 46 | ) 47 | parser.add_argument( 48 | "--config_file", 49 | default=None, 50 | type=str, 51 | required=True, 52 | help=( 53 | "The config json file corresponding to the pre-trained T5 model. \nThis specifies the model architecture." 54 | ), 55 | ) 56 | parser.add_argument( 57 | "--pytorch_dump_path", default=None, type=str, required=True, help="Path to the output PyTorch model." 58 | ) 59 | args = parser.parse_args() 60 | convert_tf_checkpoint_to_pytorch(args.tf_checkpoint_path, args.config_file, args.pytorch_dump_path) 61 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/camembert/configuration_camembert.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | # Copyright 2018 The Google AI Language Team Authors and The HuggingFace Inc. team. 3 | # Copyright (c) 2018, NVIDIA CORPORATION. All rights reserved. 4 | # 5 | # Licensed under the Apache License, Version 2.0 (the "License"); 6 | # you may not use this file except in compliance with the License. 7 | # You may obtain a copy of the License at 8 | # 9 | # http://www.apache.org/licenses/LICENSE-2.0 10 | # 11 | # Unless required by applicable law or agreed to in writing, software 12 | # distributed under the License is distributed on an "AS IS" BASIS, 13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | # See the License for the specific language governing permissions and 15 | # limitations under the License. 16 | """ CamemBERT configuration""" 17 | 18 | from collections import OrderedDict 19 | from typing import Mapping 20 | 21 | from ...onnx import OnnxConfig 22 | from ...utils import logging 23 | from ..roberta.configuration_roberta import RobertaConfig 24 | 25 | 26 | logger = logging.get_logger(__name__) 27 | 28 | CAMEMBERT_PRETRAINED_CONFIG_ARCHIVE_MAP = { 29 | "camembert-base": "https://huggingface.co/camembert-base/resolve/main/config.json", 30 | "umberto-commoncrawl-cased-v1": ( 31 | "https://huggingface.co/Musixmatch/umberto-commoncrawl-cased-v1/resolve/main/config.json" 32 | ), 33 | "umberto-wikipedia-uncased-v1": ( 34 | "https://huggingface.co/Musixmatch/umberto-wikipedia-uncased-v1/resolve/main/config.json" 35 | ), 36 | } 37 | 38 | 39 | class CamembertConfig(RobertaConfig): 40 | """ 41 | This class overrides [`RobertaConfig`]. Please check the superclass for the appropriate documentation alongside 42 | usage examples. Instantiating a configuration with the defaults will yield a similar configuration to that of the 43 | Camembert [camembert-base](https://huggingface.co/camembert-base) architecture. 44 | """ 45 | 46 | model_type = "camembert" 47 | 48 | 49 | class CamembertOnnxConfig(OnnxConfig): 50 | @property 51 | def inputs(self) -> Mapping[str, Mapping[int, str]]: 52 | if self.task == "multiple-choice": 53 | dynamic_axis = {0: "batch", 1: "choice", 2: "sequence"} 54 | else: 55 | dynamic_axis = {0: "batch", 1: "sequence"} 56 | return OrderedDict( 57 | [ 58 | ("input_ids", dynamic_axis), 59 | ("attention_mask", dynamic_axis), 60 | ] 61 | ) 62 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/canine/convert_canine_original_tf_checkpoint_to_pytorch.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | # Copyright 2021 The HuggingFace Inc. team. 3 | # 4 | # Licensed under the Apache License, Version 2.0 (the "License"); 5 | # you may not use this file except in compliance with the License. 6 | # You may obtain a copy of the License at 7 | # 8 | # http://www.apache.org/licenses/LICENSE-2.0 9 | # 10 | # Unless required by applicable law or agreed to in writing, software 11 | # distributed under the License is distributed on an "AS IS" BASIS, 12 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | # See the License for the specific language governing permissions and 14 | # limitations under the License. 15 | """Convert CANINE checkpoint.""" 16 | 17 | 18 | import argparse 19 | 20 | from transformers import CanineConfig, CanineModel, CanineTokenizer, load_tf_weights_in_canine 21 | from transformers.utils import logging 22 | 23 | 24 | logging.set_verbosity_info() 25 | 26 | 27 | def convert_tf_checkpoint_to_pytorch(tf_checkpoint_path, pytorch_dump_path): 28 | 29 | # Initialize PyTorch model 30 | config = CanineConfig() 31 | model = CanineModel(config) 32 | model.eval() 33 | 34 | print(f"Building PyTorch model from configuration: {config}") 35 | 36 | # Load weights from tf checkpoint 37 | load_tf_weights_in_canine(model, config, tf_checkpoint_path) 38 | 39 | # Save pytorch-model (weights and configuration) 40 | print(f"Save PyTorch model to {pytorch_dump_path}") 41 | model.save_pretrained(pytorch_dump_path) 42 | 43 | # Save tokenizer files 44 | tokenizer = CanineTokenizer() 45 | print(f"Save tokenizer files to {pytorch_dump_path}") 46 | tokenizer.save_pretrained(pytorch_dump_path) 47 | 48 | 49 | if __name__ == "__main__": 50 | parser = argparse.ArgumentParser() 51 | # Required parameters 52 | parser.add_argument( 53 | "--tf_checkpoint_path", 54 | default=None, 55 | type=str, 56 | required=True, 57 | help="Path to the TensorFlow checkpoint. Should end with model.ckpt", 58 | ) 59 | parser.add_argument( 60 | "--pytorch_dump_path", 61 | default=None, 62 | type=str, 63 | required=True, 64 | help="Path to a folder where the PyTorch model will be placed.", 65 | ) 66 | args = parser.parse_args() 67 | convert_tf_checkpoint_to_pytorch(args.tf_checkpoint_path, args.pytorch_dump_path) 68 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/convbert/convert_convbert_original_tf1_checkpoint_to_pytorch_and_tf2.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | # Copyright 2020 The HuggingFace Inc. team. 3 | # 4 | # Licensed under the Apache License, Version 2.0 (the "License"); 5 | # you may not use this file except in compliance with the License. 6 | # You may obtain a copy of the License at 7 | # 8 | # http://www.apache.org/licenses/LICENSE-2.0 9 | # 10 | # Unless required by applicable law or agreed to in writing, software 11 | # distributed under the License is distributed on an "AS IS" BASIS, 12 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | # See the License for the specific language governing permissions and 14 | # limitations under the License. 15 | """Convert ConvBERT checkpoint.""" 16 | 17 | import argparse 18 | 19 | from transformers import ConvBertConfig, ConvBertModel, TFConvBertModel, load_tf_weights_in_convbert 20 | from transformers.utils import logging 21 | 22 | 23 | logging.set_verbosity_info() 24 | 25 | 26 | def convert_orig_tf1_checkpoint_to_pytorch(tf_checkpoint_path, convbert_config_file, pytorch_dump_path): 27 | conf = ConvBertConfig.from_json_file(convbert_config_file) 28 | model = ConvBertModel(conf) 29 | 30 | model = load_tf_weights_in_convbert(model, conf, tf_checkpoint_path) 31 | model.save_pretrained(pytorch_dump_path) 32 | 33 | tf_model = TFConvBertModel.from_pretrained(pytorch_dump_path, from_pt=True) 34 | tf_model.save_pretrained(pytorch_dump_path) 35 | 36 | 37 | if __name__ == "__main__": 38 | parser = argparse.ArgumentParser() 39 | # Required parameters 40 | parser.add_argument( 41 | "--tf_checkpoint_path", default=None, type=str, required=True, help="Path to the TensorFlow checkpoint path." 42 | ) 43 | parser.add_argument( 44 | "--convbert_config_file", 45 | default=None, 46 | type=str, 47 | required=True, 48 | help=( 49 | "The config json file corresponding to the pre-trained ConvBERT model. \n" 50 | "This specifies the model architecture." 51 | ), 52 | ) 53 | parser.add_argument( 54 | "--pytorch_dump_path", default=None, type=str, required=True, help="Path to the output PyTorch model." 55 | ) 56 | args = parser.parse_args() 57 | convert_orig_tf1_checkpoint_to_pytorch(args.tf_checkpoint_path, args.convbert_config_file, args.pytorch_dump_path) 58 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/convbert/tokenization_convbert.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | # Copyright 2018 The HuggingFace Inc. team. 3 | # 4 | # Licensed under the Apache License, Version 2.0 (the "License"); 5 | # you may not use this file except in compliance with the License. 6 | # You may obtain a copy of the License at 7 | # 8 | # http://www.apache.org/licenses/LICENSE-2.0 9 | # 10 | # Unless required by applicable law or agreed to in writing, software 11 | # distributed under the License is distributed on an "AS IS" BASIS, 12 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | # See the License for the specific language governing permissions and 14 | # limitations under the License. 15 | """Tokenization classes for ConvBERT.""" 16 | from ...utils import logging 17 | from ..bert.tokenization_bert import BertTokenizer 18 | 19 | 20 | logger = logging.get_logger(__name__) 21 | 22 | VOCAB_FILES_NAMES = {"vocab_file": "vocab.txt"} 23 | 24 | PRETRAINED_VOCAB_FILES_MAP = { 25 | "vocab_file": { 26 | "YituTech/conv-bert-base": "https://huggingface.co/YituTech/conv-bert-base/resolve/main/vocab.txt", 27 | "YituTech/conv-bert-medium-small": ( 28 | "https://huggingface.co/YituTech/conv-bert-medium-small/resolve/main/vocab.txt" 29 | ), 30 | "YituTech/conv-bert-small": "https://huggingface.co/YituTech/conv-bert-small/resolve/main/vocab.txt", 31 | } 32 | } 33 | 34 | PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES = { 35 | "YituTech/conv-bert-base": 512, 36 | "YituTech/conv-bert-medium-small": 512, 37 | "YituTech/conv-bert-small": 512, 38 | } 39 | 40 | 41 | PRETRAINED_INIT_CONFIGURATION = { 42 | "YituTech/conv-bert-base": {"do_lower_case": True}, 43 | "YituTech/conv-bert-medium-small": {"do_lower_case": True}, 44 | "YituTech/conv-bert-small": {"do_lower_case": True}, 45 | } 46 | 47 | 48 | class ConvBertTokenizer(BertTokenizer): 49 | r""" 50 | Construct a ConvBERT tokenizer. [`ConvBertTokenizer`] is identical to [`BertTokenizer`] and runs end-to-end 51 | tokenization: punctuation splitting and wordpiece. Refer to superclass [`BertTokenizer`] for usage examples and 52 | documentation concerning parameters. 53 | """ 54 | 55 | vocab_files_names = VOCAB_FILES_NAMES 56 | pretrained_vocab_files_map = PRETRAINED_VOCAB_FILES_MAP 57 | max_model_input_sizes = PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES 58 | pretrained_init_configuration = PRETRAINED_INIT_CONFIGURATION 59 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/convbert/tokenization_convbert_fast.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | # Copyright The HuggingFace Inc. team. All rights reserved. 3 | # 4 | # Licensed under the Apache License, Version 2.0 (the "License"); 5 | # you may not use this file except in compliance with the License. 6 | # You may obtain a copy of the License at 7 | # 8 | # http://www.apache.org/licenses/LICENSE-2.0 9 | # 10 | # Unless required by applicable law or agreed to in writing, software 11 | # distributed under the License is distributed on an "AS IS" BASIS, 12 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | # See the License for the specific language governing permissions and 14 | # limitations under the License. 15 | """Tokenization classes for ConvBERT.""" 16 | from ...utils import logging 17 | from ..bert.tokenization_bert_fast import BertTokenizerFast 18 | from .tokenization_convbert import ConvBertTokenizer 19 | 20 | 21 | logger = logging.get_logger(__name__) 22 | 23 | VOCAB_FILES_NAMES = {"vocab_file": "vocab.txt"} 24 | 25 | PRETRAINED_VOCAB_FILES_MAP = { 26 | "vocab_file": { 27 | "YituTech/conv-bert-base": "https://huggingface.co/YituTech/conv-bert-base/resolve/main/vocab.txt", 28 | "YituTech/conv-bert-medium-small": ( 29 | "https://huggingface.co/YituTech/conv-bert-medium-small/resolve/main/vocab.txt" 30 | ), 31 | "YituTech/conv-bert-small": "https://huggingface.co/YituTech/conv-bert-small/resolve/main/vocab.txt", 32 | } 33 | } 34 | 35 | PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES = { 36 | "YituTech/conv-bert-base": 512, 37 | "YituTech/conv-bert-medium-small": 512, 38 | "YituTech/conv-bert-small": 512, 39 | } 40 | 41 | 42 | PRETRAINED_INIT_CONFIGURATION = { 43 | "YituTech/conv-bert-base": {"do_lower_case": True}, 44 | "YituTech/conv-bert-medium-small": {"do_lower_case": True}, 45 | "YituTech/conv-bert-small": {"do_lower_case": True}, 46 | } 47 | 48 | 49 | class ConvBertTokenizerFast(BertTokenizerFast): 50 | r""" 51 | Construct a "fast" ConvBERT tokenizer (backed by HuggingFace's *tokenizers* library). 52 | 53 | [`ConvBertTokenizerFast`] is identical to [`BertTokenizerFast`] and runs end-to-end tokenization: punctuation 54 | splitting and wordpiece. 55 | 56 | Refer to superclass [`BertTokenizerFast`] for usage examples and documentation concerning parameters. 57 | """ 58 | vocab_files_names = VOCAB_FILES_NAMES 59 | pretrained_vocab_files_map = PRETRAINED_VOCAB_FILES_MAP 60 | max_model_input_sizes = PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES 61 | pretrained_init_configuration = PRETRAINED_INIT_CONFIGURATION 62 | slow_tokenizer_class = ConvBertTokenizer 63 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/cpm/__init__.py: -------------------------------------------------------------------------------- 1 | # flake8: noqa 2 | # There's no way to ignore "F401 '...' imported but unused" warnings in this 3 | # module, but to preserve other warnings. So, don't check this module at all. 4 | 5 | # Copyright 2020 The HuggingFace Team. All rights reserved. 6 | # 7 | # Licensed under the Apache License, Version 2.0 (the "License"); 8 | # you may not use this file except in compliance with the License. 9 | # You may obtain a copy of the License at 10 | # 11 | # http://www.apache.org/licenses/LICENSE-2.0 12 | # 13 | # Unless required by applicable law or agreed to in writing, software 14 | # distributed under the License is distributed on an "AS IS" BASIS, 15 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | # See the License for the specific language governing permissions and 17 | # limitations under the License. 18 | 19 | from typing import TYPE_CHECKING 20 | 21 | from ...utils import OptionalDependencyNotAvailable, _LazyModule, is_sentencepiece_available, is_tokenizers_available 22 | 23 | 24 | _import_structure = {} 25 | 26 | try: 27 | if not is_sentencepiece_available(): 28 | raise OptionalDependencyNotAvailable() 29 | except OptionalDependencyNotAvailable: 30 | pass 31 | else: 32 | _import_structure["tokenization_cpm"] = ["CpmTokenizer"] 33 | 34 | try: 35 | if not is_tokenizers_available(): 36 | raise OptionalDependencyNotAvailable() 37 | except OptionalDependencyNotAvailable: 38 | pass 39 | else: 40 | _import_structure["tokenization_cpm_fast"] = ["CpmTokenizerFast"] 41 | 42 | 43 | if TYPE_CHECKING: 44 | try: 45 | if not is_sentencepiece_available(): 46 | raise OptionalDependencyNotAvailable() 47 | except OptionalDependencyNotAvailable: 48 | pass 49 | else: 50 | from .tokenization_cpm import CpmTokenizer 51 | 52 | try: 53 | if not is_tokenizers_available(): 54 | raise OptionalDependencyNotAvailable() 55 | except OptionalDependencyNotAvailable: 56 | pass 57 | else: 58 | from .tokenization_cpm_fast import CpmTokenizerFast 59 | 60 | else: 61 | import sys 62 | 63 | sys.modules[__name__] = _LazyModule(__name__, globals()["__file__"], _import_structure, module_spec=__spec__) 64 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/cvt/__init__.py: -------------------------------------------------------------------------------- 1 | # flake8: noqa 2 | # There's no way to ignore "F401 '...' imported but unused" warnings in this 3 | # module, but to preserve other warnings. So, don't check this module at all. 4 | 5 | # Copyright 2022 The HuggingFace Team. All rights reserved. 6 | # 7 | # Licensed under the Apache License, Version 2.0 (the "License"); 8 | # you may not use this file except in compliance with the License. 9 | # You may obtain a copy of the License at 10 | # 11 | # http://www.apache.org/licenses/LICENSE-2.0 12 | # 13 | # Unless required by applicable law or agreed to in writing, software 14 | # distributed under the License is distributed on an "AS IS" BASIS, 15 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | # See the License for the specific language governing permissions and 17 | # limitations under the License. 18 | from typing import TYPE_CHECKING 19 | 20 | from ...utils import OptionalDependencyNotAvailable, _LazyModule, is_torch_available 21 | 22 | 23 | _import_structure = {"configuration_cvt": ["CVT_PRETRAINED_CONFIG_ARCHIVE_MAP", "CvtConfig"]} 24 | 25 | 26 | try: 27 | if not is_torch_available(): 28 | raise OptionalDependencyNotAvailable() 29 | except OptionalDependencyNotAvailable: 30 | pass 31 | else: 32 | _import_structure["modeling_cvt"] = [ 33 | "CVT_PRETRAINED_MODEL_ARCHIVE_LIST", 34 | "CvtForImageClassification", 35 | "CvtModel", 36 | "CvtPreTrainedModel", 37 | ] 38 | 39 | 40 | if TYPE_CHECKING: 41 | from .configuration_cvt import CVT_PRETRAINED_CONFIG_ARCHIVE_MAP, CvtConfig 42 | 43 | try: 44 | if not is_torch_available(): 45 | raise OptionalDependencyNotAvailable() 46 | except OptionalDependencyNotAvailable: 47 | pass 48 | else: 49 | from .modeling_cvt import ( 50 | CVT_PRETRAINED_MODEL_ARCHIVE_LIST, 51 | CvtForImageClassification, 52 | CvtModel, 53 | CvtPreTrainedModel, 54 | ) 55 | 56 | else: 57 | import sys 58 | 59 | sys.modules[__name__] = _LazyModule(__name__, globals()["__file__"], _import_structure, module_spec=__spec__) 60 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/decision_transformer/__init__.py: -------------------------------------------------------------------------------- 1 | # flake8: noqa 2 | # There's no way to ignore "F401 '...' imported but unused" warnings in this 3 | # module, but to preserve other warnings. So, don't check this module at all. 4 | 5 | # Copyright 2020 The HuggingFace Team. All rights reserved. 6 | # 7 | # Licensed under the Apache License, Version 2.0 (the "License"); 8 | # you may not use this file except in compliance with the License. 9 | # You may obtain a copy of the License at 10 | # 11 | # http://www.apache.org/licenses/LICENSE-2.0 12 | # 13 | # Unless required by applicable law or agreed to in writing, software 14 | # distributed under the License is distributed on an "AS IS" BASIS, 15 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | # See the License for the specific language governing permissions and 17 | # limitations under the License. 18 | from typing import TYPE_CHECKING 19 | 20 | # rely on isort to merge the imports 21 | from ...utils import OptionalDependencyNotAvailable, _LazyModule, is_torch_available 22 | 23 | 24 | _import_structure = { 25 | "configuration_decision_transformer": [ 26 | "DECISION_TRANSFORMER_PRETRAINED_CONFIG_ARCHIVE_MAP", 27 | "DecisionTransformerConfig", 28 | ], 29 | } 30 | 31 | try: 32 | if not is_torch_available(): 33 | raise OptionalDependencyNotAvailable() 34 | except OptionalDependencyNotAvailable: 35 | pass 36 | else: 37 | _import_structure["modeling_decision_transformer"] = [ 38 | "DECISION_TRANSFORMER_PRETRAINED_MODEL_ARCHIVE_LIST", 39 | "DecisionTransformerGPT2Model", 40 | "DecisionTransformerGPT2PreTrainedModel", 41 | "DecisionTransformerModel", 42 | "DecisionTransformerPreTrainedModel", 43 | ] 44 | 45 | 46 | if TYPE_CHECKING: 47 | from .configuration_decision_transformer import ( 48 | DECISION_TRANSFORMER_PRETRAINED_CONFIG_ARCHIVE_MAP, 49 | DecisionTransformerConfig, 50 | ) 51 | 52 | try: 53 | if not is_torch_available(): 54 | raise OptionalDependencyNotAvailable() 55 | except OptionalDependencyNotAvailable: 56 | pass 57 | else: 58 | from .modeling_decision_transformer import ( 59 | DECISION_TRANSFORMER_PRETRAINED_MODEL_ARCHIVE_LIST, 60 | DecisionTransformerGPT2Model, 61 | DecisionTransformerGPT2PreTrainedModel, 62 | DecisionTransformerModel, 63 | DecisionTransformerPreTrainedModel, 64 | ) 65 | 66 | 67 | else: 68 | import sys 69 | 70 | sys.modules[__name__] = _LazyModule(__name__, globals()["__file__"], _import_structure, module_spec=__spec__) 71 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/dialogpt/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/haochengxi/Train_Transformers_with_INT4/a3244e235c0ffdba2bd18f25491316096217ed2f/SingleDivide/transformersLocal/models/dialogpt/__init__.py -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/dialogpt/convert_dialogpt_original_pytorch_checkpoint_to_pytorch.py: -------------------------------------------------------------------------------- 1 | # Copyright 2020 The HuggingFace Team. All rights reserved. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | import argparse 16 | import os 17 | 18 | import torch 19 | 20 | from transformers.utils import WEIGHTS_NAME 21 | 22 | 23 | DIALOGPT_MODELS = ["small", "medium", "large"] 24 | 25 | OLD_KEY = "lm_head.decoder.weight" 26 | NEW_KEY = "lm_head.weight" 27 | 28 | 29 | def convert_dialogpt_checkpoint(checkpoint_path: str, pytorch_dump_folder_path: str): 30 | d = torch.load(checkpoint_path) 31 | d[NEW_KEY] = d.pop(OLD_KEY) 32 | os.makedirs(pytorch_dump_folder_path, exist_ok=True) 33 | torch.save(d, os.path.join(pytorch_dump_folder_path, WEIGHTS_NAME)) 34 | 35 | 36 | if __name__ == "__main__": 37 | parser = argparse.ArgumentParser() 38 | parser.add_argument("--dialogpt_path", default=".", type=str) 39 | args = parser.parse_args() 40 | for MODEL in DIALOGPT_MODELS: 41 | checkpoint_path = os.path.join(args.dialogpt_path, f"{MODEL}_ft.pkl") 42 | pytorch_dump_folder_path = f"./DialoGPT-{MODEL}" 43 | convert_dialogpt_checkpoint( 44 | checkpoint_path, 45 | pytorch_dump_folder_path, 46 | ) 47 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/dit/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/haochengxi/Train_Transformers_with_INT4/a3244e235c0ffdba2bd18f25491316096217ed2f/SingleDivide/transformersLocal/models/dit/__init__.py -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/fsmt/__init__.py: -------------------------------------------------------------------------------- 1 | # flake8: noqa 2 | # There's no way to ignore "F401 '...' imported but unused" warnings in this 3 | # module, but to preserve other warnings. So, don't check this module at all. 4 | 5 | # Copyright 2020 The HuggingFace Team. All rights reserved. 6 | # 7 | # Licensed under the Apache License, Version 2.0 (the "License"); 8 | # you may not use this file except in compliance with the License. 9 | # You may obtain a copy of the License at 10 | # 11 | # http://www.apache.org/licenses/LICENSE-2.0 12 | # 13 | # Unless required by applicable law or agreed to in writing, software 14 | # distributed under the License is distributed on an "AS IS" BASIS, 15 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | # See the License for the specific language governing permissions and 17 | # limitations under the License. 18 | 19 | from typing import TYPE_CHECKING 20 | 21 | from ...utils import OptionalDependencyNotAvailable, _LazyModule, is_torch_available 22 | 23 | 24 | _import_structure = { 25 | "configuration_fsmt": ["FSMT_PRETRAINED_CONFIG_ARCHIVE_MAP", "FSMTConfig"], 26 | "tokenization_fsmt": ["FSMTTokenizer"], 27 | } 28 | 29 | try: 30 | if not is_torch_available(): 31 | raise OptionalDependencyNotAvailable() 32 | except OptionalDependencyNotAvailable: 33 | pass 34 | else: 35 | _import_structure["modeling_fsmt"] = ["FSMTForConditionalGeneration", "FSMTModel", "PretrainedFSMTModel"] 36 | 37 | 38 | if TYPE_CHECKING: 39 | from .configuration_fsmt import FSMT_PRETRAINED_CONFIG_ARCHIVE_MAP, FSMTConfig 40 | from .tokenization_fsmt import FSMTTokenizer 41 | 42 | try: 43 | if not is_torch_available(): 44 | raise OptionalDependencyNotAvailable() 45 | except OptionalDependencyNotAvailable: 46 | pass 47 | else: 48 | from .modeling_fsmt import FSMTForConditionalGeneration, FSMTModel, PretrainedFSMTModel 49 | 50 | else: 51 | import sys 52 | 53 | sys.modules[__name__] = _LazyModule(__name__, globals()["__file__"], _import_structure, module_spec=__spec__) 54 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/funnel/convert_funnel_original_tf_checkpoint_to_pytorch.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | # Copyright 2020 The HuggingFace Inc. team. 3 | # 4 | # Licensed under the Apache License, Version 2.0 (the "License"); 5 | # you may not use this file except in compliance with the License. 6 | # You may obtain a copy of the License at 7 | # 8 | # http://www.apache.org/licenses/LICENSE-2.0 9 | # 10 | # Unless required by applicable law or agreed to in writing, software 11 | # distributed under the License is distributed on an "AS IS" BASIS, 12 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | # See the License for the specific language governing permissions and 14 | # limitations under the License. 15 | """Convert Funnel checkpoint.""" 16 | 17 | 18 | import argparse 19 | 20 | import torch 21 | 22 | from transformers import FunnelBaseModel, FunnelConfig, FunnelModel, load_tf_weights_in_funnel 23 | from transformers.utils import logging 24 | 25 | 26 | logging.set_verbosity_info() 27 | 28 | 29 | def convert_tf_checkpoint_to_pytorch(tf_checkpoint_path, config_file, pytorch_dump_path, base_model): 30 | # Initialise PyTorch model 31 | config = FunnelConfig.from_json_file(config_file) 32 | print(f"Building PyTorch model from configuration: {config}") 33 | model = FunnelBaseModel(config) if base_model else FunnelModel(config) 34 | 35 | # Load weights from tf checkpoint 36 | load_tf_weights_in_funnel(model, config, tf_checkpoint_path) 37 | 38 | # Save pytorch-model 39 | print(f"Save PyTorch model to {pytorch_dump_path}") 40 | torch.save(model.state_dict(), pytorch_dump_path) 41 | 42 | 43 | if __name__ == "__main__": 44 | parser = argparse.ArgumentParser() 45 | # Required parameters 46 | parser.add_argument( 47 | "--tf_checkpoint_path", default=None, type=str, required=True, help="Path to the TensorFlow checkpoint path." 48 | ) 49 | parser.add_argument( 50 | "--config_file", 51 | default=None, 52 | type=str, 53 | required=True, 54 | help="The config json file corresponding to the pre-trained model. \nThis specifies the model architecture.", 55 | ) 56 | parser.add_argument( 57 | "--pytorch_dump_path", default=None, type=str, required=True, help="Path to the output PyTorch model." 58 | ) 59 | parser.add_argument( 60 | "--base_model", action="store_true", help="Whether you want just the base model (no decoder) or not." 61 | ) 62 | args = parser.parse_args() 63 | convert_tf_checkpoint_to_pytorch( 64 | args.tf_checkpoint_path, args.config_file, args.pytorch_dump_path, args.base_model 65 | ) 66 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/groupvit/__init__.py: -------------------------------------------------------------------------------- 1 | # flake8: noqa 2 | # There's no way to ignore "F401 '...' imported but unused" warnings in this 3 | # module, but to preserve other warnings. So, don't check this module at all. 4 | 5 | # Copyright 2022 The HuggingFace Team. All rights reserved. 6 | # 7 | # Licensed under the Apache License, Version 2.0 (the "License"); 8 | # you may not use this file except in compliance with the License. 9 | # You may obtain a copy of the License at 10 | # 11 | # http://www.apache.org/licenses/LICENSE-2.0 12 | # 13 | # Unless required by applicable law or agreed to in writing, software 14 | # distributed under the License is distributed on an "AS IS" BASIS, 15 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | # See the License for the specific language governing permissions and 17 | # limitations under the License. 18 | from typing import TYPE_CHECKING 19 | 20 | from ...utils import OptionalDependencyNotAvailable, _LazyModule, is_torch_available 21 | 22 | 23 | _import_structure = { 24 | "configuration_groupvit": [ 25 | "GROUPVIT_PRETRAINED_CONFIG_ARCHIVE_MAP", 26 | "GroupViTConfig", 27 | "GroupViTTextConfig", 28 | "GroupViTVisionConfig", 29 | ], 30 | } 31 | 32 | try: 33 | if not is_torch_available(): 34 | raise OptionalDependencyNotAvailable() 35 | except OptionalDependencyNotAvailable: 36 | pass 37 | else: 38 | _import_structure["modeling_groupvit"] = [ 39 | "GROUPVIT_PRETRAINED_MODEL_ARCHIVE_LIST", 40 | "GroupViTModel", 41 | "GroupViTPreTrainedModel", 42 | "GroupViTTextModel", 43 | "GroupViTVisionModel", 44 | ] 45 | 46 | if TYPE_CHECKING: 47 | from .configuration_groupvit import ( 48 | GROUPVIT_PRETRAINED_CONFIG_ARCHIVE_MAP, 49 | GroupViTConfig, 50 | GroupViTTextConfig, 51 | GroupViTVisionConfig, 52 | ) 53 | 54 | try: 55 | if not is_torch_available(): 56 | raise OptionalDependencyNotAvailable() 57 | except OptionalDependencyNotAvailable: 58 | pass 59 | else: 60 | from .modeling_groupvit import ( 61 | GROUPVIT_PRETRAINED_MODEL_ARCHIVE_LIST, 62 | GroupViTModel, 63 | GroupViTPreTrainedModel, 64 | GroupViTTextModel, 65 | GroupViTVisionModel, 66 | ) 67 | 68 | else: 69 | import sys 70 | 71 | sys.modules[__name__] = _LazyModule(__name__, globals()["__file__"], _import_structure, module_spec=__spec__) 72 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/herbert/__init__.py: -------------------------------------------------------------------------------- 1 | # flake8: noqa 2 | # There's no way to ignore "F401 '...' imported but unused" warnings in this 3 | # module, but to preserve other warnings. So, don't check this module at all. 4 | 5 | # Copyright 2020 The HuggingFace Team. All rights reserved. 6 | # 7 | # Licensed under the Apache License, Version 2.0 (the "License"); 8 | # you may not use this file except in compliance with the License. 9 | # You may obtain a copy of the License at 10 | # 11 | # http://www.apache.org/licenses/LICENSE-2.0 12 | # 13 | # Unless required by applicable law or agreed to in writing, software 14 | # distributed under the License is distributed on an "AS IS" BASIS, 15 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | # See the License for the specific language governing permissions and 17 | # limitations under the License. 18 | 19 | from typing import TYPE_CHECKING 20 | 21 | from ...utils import OptionalDependencyNotAvailable, _LazyModule, is_tokenizers_available 22 | 23 | 24 | _import_structure = {"tokenization_herbert": ["HerbertTokenizer"]} 25 | 26 | try: 27 | if not is_tokenizers_available(): 28 | raise OptionalDependencyNotAvailable() 29 | except OptionalDependencyNotAvailable: 30 | pass 31 | else: 32 | _import_structure["tokenization_herbert_fast"] = ["HerbertTokenizerFast"] 33 | 34 | 35 | if TYPE_CHECKING: 36 | from .tokenization_herbert import HerbertTokenizer 37 | 38 | try: 39 | if not is_tokenizers_available(): 40 | raise OptionalDependencyNotAvailable() 41 | except OptionalDependencyNotAvailable: 42 | pass 43 | else: 44 | from .tokenization_herbert_fast import HerbertTokenizerFast 45 | 46 | else: 47 | import sys 48 | 49 | sys.modules[__name__] = _LazyModule(__name__, globals()["__file__"], _import_structure, module_spec=__spec__) 50 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/ibert/__init__.py: -------------------------------------------------------------------------------- 1 | # flake8: noqa 2 | # There's no way to ignore "F401 '...' imported but unused" warnings in this 3 | # module, but to preserve other warnings. So, don't check this module at all. 4 | 5 | # Copyright 2020 The HuggingFace Team. All rights reserved. 6 | # 7 | # Licensed under the Apache License, Version 2.0 (the "License"); 8 | # you may not use this file except in compliance with the License. 9 | # You may obtain a copy of the License at 10 | # 11 | # http://www.apache.org/licenses/LICENSE-2.0 12 | # 13 | # Unless required by applicable law or agreed to in writing, software 14 | # distributed under the License is distributed on an "AS IS" BASIS, 15 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | # See the License for the specific language governing permissions and 17 | # limitations under the License. 18 | 19 | from typing import TYPE_CHECKING 20 | 21 | from ...utils import OptionalDependencyNotAvailable, _LazyModule, is_torch_available 22 | 23 | 24 | _import_structure = {"configuration_ibert": ["IBERT_PRETRAINED_CONFIG_ARCHIVE_MAP", "IBertConfig", "IBertOnnxConfig"]} 25 | 26 | try: 27 | if not is_torch_available(): 28 | raise OptionalDependencyNotAvailable() 29 | except OptionalDependencyNotAvailable: 30 | pass 31 | else: 32 | _import_structure["modeling_ibert"] = [ 33 | "IBERT_PRETRAINED_MODEL_ARCHIVE_LIST", 34 | "IBertForMaskedLM", 35 | "IBertForMultipleChoice", 36 | "IBertForQuestionAnswering", 37 | "IBertForSequenceClassification", 38 | "IBertForTokenClassification", 39 | "IBertModel", 40 | "IBertPreTrainedModel", 41 | ] 42 | 43 | if TYPE_CHECKING: 44 | from .configuration_ibert import IBERT_PRETRAINED_CONFIG_ARCHIVE_MAP, IBertConfig, IBertOnnxConfig 45 | 46 | try: 47 | if not is_torch_available(): 48 | raise OptionalDependencyNotAvailable() 49 | except OptionalDependencyNotAvailable: 50 | pass 51 | else: 52 | from .modeling_ibert import ( 53 | IBERT_PRETRAINED_MODEL_ARCHIVE_LIST, 54 | IBertForMaskedLM, 55 | IBertForMultipleChoice, 56 | IBertForQuestionAnswering, 57 | IBertForSequenceClassification, 58 | IBertForTokenClassification, 59 | IBertModel, 60 | IBertPreTrainedModel, 61 | ) 62 | 63 | else: 64 | import sys 65 | 66 | sys.modules[__name__] = _LazyModule(__name__, globals()["__file__"], _import_structure, module_spec=__spec__) 67 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/layoutlm/tokenization_layoutlm.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | # Copyright 2018 The Microsoft Research Asia LayoutLM Team Authors. 3 | # 4 | # Licensed under the Apache License, Version 2.0 (the "License"); 5 | # you may not use this file except in compliance with the License. 6 | # You may obtain a copy of the License at 7 | # 8 | # http://www.apache.org/licenses/LICENSE-2.0 9 | # 10 | # Unless required by applicable law or agreed to in writing, software 11 | # distributed under the License is distributed on an "AS IS" BASIS, 12 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | # See the License for the specific language governing permissions and 14 | # limitations under the License. 15 | """ Tokenization class for model LayoutLM.""" 16 | 17 | 18 | from ...utils import logging 19 | from ..bert.tokenization_bert import BertTokenizer 20 | 21 | 22 | logger = logging.get_logger(__name__) 23 | 24 | VOCAB_FILES_NAMES = {"vocab_file": "vocab.txt"} 25 | 26 | PRETRAINED_VOCAB_FILES_MAP = { 27 | "vocab_file": { 28 | "microsoft/layoutlm-base-uncased": ( 29 | "https://huggingface.co/microsoft/layoutlm-base-uncased/resolve/main/vocab.txt" 30 | ), 31 | "microsoft/layoutlm-large-uncased": ( 32 | "https://huggingface.co/microsoft/layoutlm-large-uncased/resolve/main/vocab.txt" 33 | ), 34 | } 35 | } 36 | 37 | 38 | PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES = { 39 | "microsoft/layoutlm-base-uncased": 512, 40 | "microsoft/layoutlm-large-uncased": 512, 41 | } 42 | 43 | 44 | PRETRAINED_INIT_CONFIGURATION = { 45 | "microsoft/layoutlm-base-uncased": {"do_lower_case": True}, 46 | "microsoft/layoutlm-large-uncased": {"do_lower_case": True}, 47 | } 48 | 49 | 50 | class LayoutLMTokenizer(BertTokenizer): 51 | r""" 52 | Constructs a LayoutLM tokenizer. 53 | 54 | [`LayoutLMTokenizer`] is identical to [`BertTokenizer`] and runs end-to-end tokenization: punctuation splitting + 55 | wordpiece. 56 | 57 | Refer to superclass [`BertTokenizer`] for usage examples and documentation concerning parameters. 58 | """ 59 | 60 | vocab_files_names = VOCAB_FILES_NAMES 61 | pretrained_vocab_files_map = PRETRAINED_VOCAB_FILES_MAP 62 | pretrained_init_configuration = PRETRAINED_INIT_CONFIGURATION 63 | max_model_input_sizes = PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES 64 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/layoutxlm/__init__.py: -------------------------------------------------------------------------------- 1 | # flake8: noqa 2 | # There's no way to ignore "F401 '...' imported but unused" warnings in this 3 | # module, but to preserve other warnings. So, don't check this module at all. 4 | 5 | # Copyright 2021 The HuggingFace Team. All rights reserved. 6 | # 7 | # Licensed under the Apache License, Version 2.0 (the "License"); 8 | # you may not use this file except in compliance with the License. 9 | # You may obtain a copy of the License at 10 | # 11 | # http://www.apache.org/licenses/LICENSE-2.0 12 | # 13 | # Unless required by applicable law or agreed to in writing, software 14 | # distributed under the License is distributed on an "AS IS" BASIS, 15 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | # See the License for the specific language governing permissions and 17 | # limitations under the License. 18 | 19 | from typing import TYPE_CHECKING 20 | 21 | from ...utils import ( 22 | OptionalDependencyNotAvailable, 23 | _LazyModule, 24 | is_sentencepiece_available, 25 | is_tokenizers_available, 26 | is_torch_available, 27 | is_vision_available, 28 | ) 29 | 30 | 31 | _import_structure = {"processing_layoutxlm": ["LayoutXLMProcessor"]} 32 | 33 | try: 34 | if not is_sentencepiece_available(): 35 | raise OptionalDependencyNotAvailable() 36 | except OptionalDependencyNotAvailable: 37 | pass 38 | else: 39 | _import_structure["tokenization_layoutxlm"] = ["LayoutXLMTokenizer"] 40 | 41 | try: 42 | if not is_tokenizers_available(): 43 | raise OptionalDependencyNotAvailable() 44 | except OptionalDependencyNotAvailable: 45 | pass 46 | else: 47 | _import_structure["tokenization_layoutxlm_fast"] = ["LayoutXLMTokenizerFast"] 48 | 49 | if TYPE_CHECKING: 50 | from .processing_layoutxlm import LayoutXLMProcessor 51 | 52 | try: 53 | if not is_sentencepiece_available(): 54 | raise OptionalDependencyNotAvailable() 55 | except OptionalDependencyNotAvailable: 56 | pass 57 | else: 58 | from .tokenization_layoutxlm import LayoutXLMTokenizer 59 | 60 | try: 61 | if not is_tokenizers_available(): 62 | raise OptionalDependencyNotAvailable() 63 | except OptionalDependencyNotAvailable: 64 | pass 65 | else: 66 | from .tokenization_layoutxlm_fast import LayoutXLMTokenizerFast 67 | 68 | else: 69 | import sys 70 | 71 | sys.modules[__name__] = _LazyModule(__name__, globals()["__file__"], _import_structure, module_spec=__spec__) 72 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/lxmert/convert_lxmert_original_tf_checkpoint_to_pytorch.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | # Copyright 2018 The HuggingFace Inc. team. 3 | # 4 | # Licensed under the Apache License, Version 2.0 (the "License"); 5 | # you may not use this file except in compliance with the License. 6 | # You may obtain a copy of the License at 7 | # 8 | # http://www.apache.org/licenses/LICENSE-2.0 9 | # 10 | # Unless required by applicable law or agreed to in writing, software 11 | # distributed under the License is distributed on an "AS IS" BASIS, 12 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | # See the License for the specific language governing permissions and 14 | # limitations under the License. 15 | """Convert LXMERT checkpoint.""" 16 | 17 | 18 | import argparse 19 | 20 | import torch 21 | 22 | from transformers import LxmertConfig, LxmertForPreTraining, load_tf_weights_in_lxmert 23 | from transformers.utils import logging 24 | 25 | 26 | logging.set_verbosity_info() 27 | 28 | 29 | def convert_tf_checkpoint_to_pytorch(tf_checkpoint_path, config_file, pytorch_dump_path): 30 | # Initialise PyTorch model 31 | config = LxmertConfig.from_json_file(config_file) 32 | print(f"Building PyTorch model from configuration: {config}") 33 | model = LxmertForPreTraining(config) 34 | 35 | # Load weights from tf checkpoint 36 | load_tf_weights_in_lxmert(model, config, tf_checkpoint_path) 37 | 38 | # Save pytorch-model 39 | print(f"Save PyTorch model to {pytorch_dump_path}") 40 | torch.save(model.state_dict(), pytorch_dump_path) 41 | 42 | 43 | if __name__ == "__main__": 44 | parser = argparse.ArgumentParser() 45 | # Required parameters 46 | parser.add_argument( 47 | "--tf_checkpoint_path", default=None, type=str, required=True, help="Path to the TensorFlow checkpoint path." 48 | ) 49 | parser.add_argument( 50 | "--config_file", 51 | default=None, 52 | type=str, 53 | required=True, 54 | help="The config json file corresponding to the pre-trained model. \nThis specifies the model architecture.", 55 | ) 56 | parser.add_argument( 57 | "--pytorch_dump_path", default=None, type=str, required=True, help="Path to the output PyTorch model." 58 | ) 59 | args = parser.parse_args() 60 | convert_tf_checkpoint_to_pytorch(args.tf_checkpoint_path, args.config_file, args.pytorch_dump_path) 61 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/lxmert/tokenization_lxmert.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | # Copyright 2020 The Google AI Team, Stanford University and The HuggingFace Inc. team. 3 | # 4 | # Licensed under the Apache License, Version 2.0 (the "License"); 5 | # you may not use this file except in compliance with the License. 6 | # You may obtain a copy of the License at 7 | # 8 | # http://www.apache.org/licenses/LICENSE-2.0 9 | # 10 | # Unless required by applicable law or agreed to in writing, software 11 | # distributed under the License is distributed on an "AS IS" BASIS, 12 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | # See the License for the specific language governing permissions and 14 | # limitations under the License. 15 | 16 | from ..bert.tokenization_bert import BertTokenizer 17 | 18 | 19 | VOCAB_FILES_NAMES = {"vocab_file": "vocab.txt"} 20 | 21 | PRETRAINED_VOCAB_FILES_MAP = { 22 | "vocab_file": { 23 | "unc-nlp/lxmert-base-uncased": "https://huggingface.co/unc-nlp/lxmert-base-uncased/resolve/main/vocab.txt", 24 | } 25 | } 26 | 27 | PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES = { 28 | "unc-nlp/lxmert-base-uncased": 512, 29 | } 30 | 31 | PRETRAINED_INIT_CONFIGURATION = { 32 | "unc-nlp/lxmert-base-uncased": {"do_lower_case": True}, 33 | } 34 | 35 | 36 | class LxmertTokenizer(BertTokenizer): 37 | r""" 38 | Construct an LXMERT tokenizer. 39 | 40 | [`LxmertTokenizer`] is identical to [`BertTokenizer`] and runs end-to-end tokenization: punctuation splitting and 41 | wordpiece. 42 | 43 | Refer to superclass [`BertTokenizer`] for usage examples and documentation concerning parameters. 44 | """ 45 | 46 | vocab_files_names = VOCAB_FILES_NAMES 47 | pretrained_vocab_files_map = PRETRAINED_VOCAB_FILES_MAP 48 | max_model_input_sizes = PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES 49 | pretrained_init_configuration = PRETRAINED_INIT_CONFIGURATION 50 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/lxmert/tokenization_lxmert_fast.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | # Copyright 2020 The Google AI Team, Stanford University and The HuggingFace Inc. team. 3 | # 4 | # Licensed under the Apache License, Version 2.0 (the "License"); 5 | # you may not use this file except in compliance with the License. 6 | # You may obtain a copy of the License at 7 | # 8 | # http://www.apache.org/licenses/LICENSE-2.0 9 | # 10 | # Unless required by applicable law or agreed to in writing, software 11 | # distributed under the License is distributed on an "AS IS" BASIS, 12 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | # See the License for the specific language governing permissions and 14 | # limitations under the License. 15 | 16 | from ..bert.tokenization_bert_fast import BertTokenizerFast 17 | from .tokenization_lxmert import LxmertTokenizer 18 | 19 | 20 | VOCAB_FILES_NAMES = {"vocab_file": "vocab.txt", "tokenizer_file": "tokenizer.json"} 21 | 22 | PRETRAINED_VOCAB_FILES_MAP = { 23 | "vocab_file": { 24 | "unc-nlp/lxmert-base-uncased": "https://huggingface.co/unc-nlp/lxmert-base-uncased/resolve/main/vocab.txt", 25 | }, 26 | "tokenizer_file": { 27 | "unc-nlp/lxmert-base-uncased": ( 28 | "https://huggingface.co/unc-nlp/lxmert-base-uncased/resolve/main/tokenizer.json" 29 | ), 30 | }, 31 | } 32 | 33 | PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES = { 34 | "unc-nlp/lxmert-base-uncased": 512, 35 | } 36 | 37 | PRETRAINED_INIT_CONFIGURATION = { 38 | "unc-nlp/lxmert-base-uncased": {"do_lower_case": True}, 39 | } 40 | 41 | 42 | class LxmertTokenizerFast(BertTokenizerFast): 43 | r""" 44 | Construct a "fast" LXMERT tokenizer (backed by HuggingFace's *tokenizers* library). 45 | 46 | [`LxmertTokenizerFast`] is identical to [`BertTokenizerFast`] and runs end-to-end tokenization: punctuation 47 | splitting and wordpiece. 48 | 49 | Refer to superclass [`BertTokenizerFast`] for usage examples and documentation concerning parameters. 50 | """ 51 | vocab_files_names = VOCAB_FILES_NAMES 52 | pretrained_vocab_files_map = PRETRAINED_VOCAB_FILES_MAP 53 | max_model_input_sizes = PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES 54 | pretrained_init_configuration = PRETRAINED_INIT_CONFIGURATION 55 | slow_tokenizer_class = LxmertTokenizer 56 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/m2m_100/__init__.py: -------------------------------------------------------------------------------- 1 | # flake8: noqa 2 | # There's no way to ignore "F401 '...' imported but unused" warnings in this 3 | # module, but to preserve other warnings. So, don't check this module at all. 4 | 5 | # Copyright 2021 The Fairseq Authors and The HuggingFace Inc. team. All rights reserved. 6 | # 7 | # Licensed under the Apache License, Version 2.0 (the "License"); 8 | # you may not use this file except in compliance with the License. 9 | # You may obtain a copy of the License at 10 | # 11 | # http://www.apache.org/licenses/LICENSE-2.0 12 | # 13 | # Unless required by applicable law or agreed to in writing, software 14 | # distributed under the License is distributed on an "AS IS" BASIS, 15 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | # See the License for the specific language governing permissions and 17 | # limitations under the License. 18 | from typing import TYPE_CHECKING 19 | 20 | from ...utils import OptionalDependencyNotAvailable, _LazyModule, is_tokenizers_available, is_torch_available 21 | 22 | 23 | _import_structure = { 24 | "configuration_m2m_100": ["M2M_100_PRETRAINED_CONFIG_ARCHIVE_MAP", "M2M100Config", "M2M100OnnxConfig"], 25 | "tokenization_m2m_100": ["M2M100Tokenizer"], 26 | } 27 | 28 | 29 | try: 30 | if not is_torch_available(): 31 | raise OptionalDependencyNotAvailable() 32 | except OptionalDependencyNotAvailable: 33 | pass 34 | else: 35 | _import_structure["modeling_m2m_100"] = [ 36 | "M2M_100_PRETRAINED_MODEL_ARCHIVE_LIST", 37 | "M2M100ForConditionalGeneration", 38 | "M2M100Model", 39 | "M2M100PreTrainedModel", 40 | ] 41 | 42 | 43 | if TYPE_CHECKING: 44 | from .configuration_m2m_100 import M2M_100_PRETRAINED_CONFIG_ARCHIVE_MAP, M2M100Config, M2M100OnnxConfig 45 | from .tokenization_m2m_100 import M2M100Tokenizer 46 | 47 | try: 48 | if not is_torch_available(): 49 | raise OptionalDependencyNotAvailable() 50 | except OptionalDependencyNotAvailable: 51 | pass 52 | else: 53 | from .modeling_m2m_100 import ( 54 | M2M_100_PRETRAINED_MODEL_ARCHIVE_LIST, 55 | M2M100ForConditionalGeneration, 56 | M2M100Model, 57 | M2M100PreTrainedModel, 58 | ) 59 | 60 | 61 | else: 62 | import sys 63 | 64 | sys.modules[__name__] = _LazyModule(__name__, globals()["__file__"], _import_structure, module_spec=__spec__) 65 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/mbart50/__init__.py: -------------------------------------------------------------------------------- 1 | # flake8: noqa 2 | # There's no way to ignore "F401 '...' imported but unused" warnings in this 3 | # module, but to preserve other warnings. So, don't check this module at all. 4 | 5 | # Copyright 2020 The HuggingFace Team. All rights reserved. 6 | # 7 | # Licensed under the Apache License, Version 2.0 (the "License"); 8 | # you may not use this file except in compliance with the License. 9 | # You may obtain a copy of the License at 10 | # 11 | # http://www.apache.org/licenses/LICENSE-2.0 12 | # 13 | # Unless required by applicable law or agreed to in writing, software 14 | # distributed under the License is distributed on an "AS IS" BASIS, 15 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | # See the License for the specific language governing permissions and 17 | # limitations under the License. 18 | from typing import TYPE_CHECKING 19 | 20 | from ...utils import OptionalDependencyNotAvailable, _LazyModule, is_sentencepiece_available, is_tokenizers_available 21 | 22 | 23 | _import_structure = {} 24 | 25 | try: 26 | if not is_sentencepiece_available(): 27 | raise OptionalDependencyNotAvailable() 28 | except OptionalDependencyNotAvailable: 29 | pass 30 | else: 31 | _import_structure["tokenization_mbart50"] = ["MBart50Tokenizer"] 32 | 33 | try: 34 | if not is_tokenizers_available(): 35 | raise OptionalDependencyNotAvailable() 36 | except OptionalDependencyNotAvailable: 37 | pass 38 | else: 39 | _import_structure["tokenization_mbart50_fast"] = ["MBart50TokenizerFast"] 40 | 41 | 42 | if TYPE_CHECKING: 43 | try: 44 | if not is_sentencepiece_available(): 45 | raise OptionalDependencyNotAvailable() 46 | except OptionalDependencyNotAvailable: 47 | pass 48 | else: 49 | from .tokenization_mbart50 import MBart50Tokenizer 50 | 51 | try: 52 | if not is_tokenizers_available(): 53 | raise OptionalDependencyNotAvailable() 54 | except OptionalDependencyNotAvailable: 55 | pass 56 | else: 57 | from .tokenization_mbart50_fast import MBart50TokenizerFast 58 | 59 | else: 60 | import sys 61 | 62 | sys.modules[__name__] = _LazyModule(__name__, globals()["__file__"], _import_structure, module_spec=__spec__) 63 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/mctct/__init__.py: -------------------------------------------------------------------------------- 1 | # flake8: noqa 2 | # There's no way to ignore "F401 '...' imported but unused" warnings in this 3 | # module, but to preserve other warnings. So, don't check this module at all. 4 | 5 | # Copyright 2022 The HuggingFace Team. All rights reserved. 6 | # 7 | # Licensed under the Apache License, Version 2.0 (the "License"); 8 | # you may not use this file except in compliance with the License. 9 | # You may obtain a copy of the License at 10 | # 11 | # http://www.apache.org/licenses/LICENSE-2.0 12 | # 13 | # Unless required by applicable law or agreed to in writing, software 14 | # distributed under the License is distributed on an "AS IS" BASIS, 15 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | # See the License for the specific language governing permissions and 17 | # limitations under the License. 18 | from typing import TYPE_CHECKING 19 | 20 | from ...utils import OptionalDependencyNotAvailable, _LazyModule, is_speech_available, is_torch_available 21 | 22 | 23 | _import_structure = { 24 | "configuration_mctct": ["MCTCT_PRETRAINED_CONFIG_ARCHIVE_MAP", "MCTCTConfig"], 25 | "processing_mctct": ["MCTCTProcessor"], 26 | } 27 | 28 | 29 | try: 30 | if not is_speech_available(): 31 | raise OptionalDependencyNotAvailable() 32 | except OptionalDependencyNotAvailable: 33 | pass 34 | else: 35 | _import_structure["feature_extraction_mctct"] = ["MCTCTFeatureExtractor"] 36 | 37 | 38 | try: 39 | if not is_torch_available(): 40 | raise OptionalDependencyNotAvailable() 41 | except OptionalDependencyNotAvailable: 42 | pass 43 | else: 44 | _import_structure["modeling_mctct"] = [ 45 | "MCTCT_PRETRAINED_MODEL_ARCHIVE_LIST", 46 | "MCTCTForCTC", 47 | "MCTCTModel", 48 | "MCTCTPreTrainedModel", 49 | ] 50 | 51 | 52 | if TYPE_CHECKING: 53 | from .configuration_mctct import MCTCT_PRETRAINED_CONFIG_ARCHIVE_MAP, MCTCTConfig 54 | from .processing_mctct import MCTCTProcessor 55 | 56 | try: 57 | if not is_speech_available(): 58 | raise OptionalDependencyNotAvailable() 59 | except OptionalDependencyNotAvailable: 60 | pass 61 | else: 62 | from .feature_extraction_mctct import MCTCTFeatureExtractor 63 | 64 | try: 65 | if not is_torch_available(): 66 | raise OptionalDependencyNotAvailable() 67 | except OptionalDependencyNotAvailable: 68 | pass 69 | else: 70 | from .modeling_mctct import MCTCT_PRETRAINED_MODEL_ARCHIVE_LIST, MCTCTForCTC, MCTCTModel, MCTCTPreTrainedModel 71 | 72 | else: 73 | import sys 74 | 75 | sys.modules[__name__] = _LazyModule(__name__, globals()["__file__"], _import_structure, module_spec=__spec__) 76 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/megatron_gpt2/__init__.py: -------------------------------------------------------------------------------- 1 | # flake8: noqa 2 | # There's no way to ignore "F401 '...' imported but unused" warnings in this 3 | # module, but to preserve other warnings. So, don't check this module at all. 4 | 5 | # Copyright 2021 NVIDIA Corporation and The HuggingFace Team. All rights reserved. 6 | # 7 | # Licensed under the Apache License, Version 2.0 (the "License"); 8 | # you may not use this file except in compliance with the License. 9 | # You may obtain a copy of the License at 10 | # 11 | # http://www.apache.org/licenses/LICENSE-2.0 12 | # 13 | # Unless required by applicable law or agreed to in writing, software 14 | # distributed under the License is distributed on an "AS IS" BASIS, 15 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | # See the License for the specific language governing permissions and 17 | # limitations under the License. 18 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/mluke/__init__.py: -------------------------------------------------------------------------------- 1 | # flake8: noqa 2 | # There's no way to ignore "F401 '...' imported but unused" warnings in this 3 | # module, but to preserve other warnings. So, don't check this module at all. 4 | 5 | # Copyright 2021 The HuggingFace Team. All rights reserved. 6 | # 7 | # Licensed under the Apache License, Version 2.0 (the "License"); 8 | # you may not use this file except in compliance with the License. 9 | # You may obtain a copy of the License at 10 | # 11 | # http://www.apache.org/licenses/LICENSE-2.0 12 | # 13 | # Unless required by applicable law or agreed to in writing, software 14 | # distributed under the License is distributed on an "AS IS" BASIS, 15 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | # See the License for the specific language governing permissions and 17 | # limitations under the License. 18 | 19 | from typing import TYPE_CHECKING 20 | 21 | from ...utils import OptionalDependencyNotAvailable, _LazyModule, is_sentencepiece_available 22 | 23 | 24 | _import_structure = {} 25 | 26 | 27 | try: 28 | if not is_sentencepiece_available(): 29 | raise OptionalDependencyNotAvailable() 30 | except OptionalDependencyNotAvailable: 31 | pass 32 | else: 33 | _import_structure["tokenization_mluke"] = ["MLukeTokenizer"] 34 | 35 | if TYPE_CHECKING: 36 | try: 37 | if not is_sentencepiece_available(): 38 | raise OptionalDependencyNotAvailable() 39 | except OptionalDependencyNotAvailable: 40 | pass 41 | else: 42 | from .tokenization_mluke import MLukeTokenizer 43 | 44 | 45 | else: 46 | import sys 47 | 48 | sys.modules[__name__] = _LazyModule(__name__, globals()["__file__"], _import_structure, module_spec=__spec__) 49 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/mmbt/__init__.py: -------------------------------------------------------------------------------- 1 | # flake8: noqa 2 | # There's no way to ignore "F401 '...' imported but unused" warnings in this 3 | # module, but to preserve other warnings. So, don't check this module at all. 4 | 5 | # Copyright 2020 The HuggingFace Team. All rights reserved. 6 | # 7 | # Licensed under the Apache License, Version 2.0 (the "License"); 8 | # you may not use this file except in compliance with the License. 9 | # You may obtain a copy of the License at 10 | # 11 | # http://www.apache.org/licenses/LICENSE-2.0 12 | # 13 | # Unless required by applicable law or agreed to in writing, software 14 | # distributed under the License is distributed on an "AS IS" BASIS, 15 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | # See the License for the specific language governing permissions and 17 | # limitations under the License. 18 | 19 | from typing import TYPE_CHECKING 20 | 21 | from ...utils import OptionalDependencyNotAvailable, _LazyModule, is_torch_available 22 | 23 | 24 | _import_structure = {"configuration_mmbt": ["MMBTConfig"]} 25 | 26 | try: 27 | if not is_torch_available(): 28 | raise OptionalDependencyNotAvailable() 29 | except OptionalDependencyNotAvailable: 30 | pass 31 | else: 32 | _import_structure["modeling_mmbt"] = ["MMBTForClassification", "MMBTModel", "ModalEmbeddings"] 33 | 34 | 35 | if TYPE_CHECKING: 36 | from .configuration_mmbt import MMBTConfig 37 | 38 | try: 39 | if not is_torch_available(): 40 | raise OptionalDependencyNotAvailable() 41 | except OptionalDependencyNotAvailable: 42 | pass 43 | else: 44 | from .modeling_mmbt import MMBTForClassification, MMBTModel, ModalEmbeddings 45 | 46 | else: 47 | import sys 48 | 49 | sys.modules[__name__] = _LazyModule(__name__, globals()["__file__"], _import_structure, module_spec=__spec__) 50 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/mmbt/configuration_mmbt.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | # Copyright (c) Facebook, Inc. and its affiliates. 3 | # Copyright (c) HuggingFace Inc. team. 4 | # 5 | # Licensed under the Apache License, Version 2.0 (the "License"); 6 | # you may not use this file except in compliance with the License. 7 | # You may obtain a copy of the License at 8 | # 9 | # http://www.apache.org/licenses/LICENSE-2.0 10 | # 11 | # Unless required by applicable law or agreed to in writing, software 12 | # distributed under the License is distributed on an "AS IS" BASIS, 13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | # See the License for the specific language governing permissions and 15 | # limitations under the License. 16 | """ MMBT configuration""" 17 | 18 | from ...utils import logging 19 | 20 | 21 | logger = logging.get_logger(__name__) 22 | 23 | 24 | class MMBTConfig(object): 25 | """ 26 | This is the configuration class to store the configuration of a [`MMBTModel`]. It is used to instantiate a MMBT 27 | model according to the specified arguments, defining the model architecture. 28 | 29 | Args: 30 | config ([`PreTrainedConfig`]): 31 | Config of the underlying Transformer models. Its values are copied over to use a single config. 32 | num_labels (`int`, *optional*): 33 | Size of final Linear layer for classification. 34 | modal_hidden_size (`int`, *optional*, defaults to 2048): 35 | Embedding dimension of the non-text modality encoder. 36 | """ 37 | 38 | def __init__(self, config, num_labels=None, modal_hidden_size=2048): 39 | self.__dict__ = config.__dict__ 40 | self.modal_hidden_size = modal_hidden_size 41 | if num_labels: 42 | self.num_labels = num_labels 43 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/mobilebert/convert_mobilebert_original_tf_checkpoint_to_pytorch.py: -------------------------------------------------------------------------------- 1 | # Copyright 2020 The HuggingFace Team. All rights reserved. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | import argparse 16 | 17 | import torch 18 | 19 | from transformers import MobileBertConfig, MobileBertForPreTraining, load_tf_weights_in_mobilebert 20 | from transformers.utils import logging 21 | 22 | 23 | logging.set_verbosity_info() 24 | 25 | 26 | def convert_tf_checkpoint_to_pytorch(tf_checkpoint_path, mobilebert_config_file, pytorch_dump_path): 27 | # Initialise PyTorch model 28 | config = MobileBertConfig.from_json_file(mobilebert_config_file) 29 | print(f"Building PyTorch model from configuration: {config}") 30 | model = MobileBertForPreTraining(config) 31 | # Load weights from tf checkpoint 32 | model = load_tf_weights_in_mobilebert(model, config, tf_checkpoint_path) 33 | # Save pytorch-model 34 | print(f"Save PyTorch model to {pytorch_dump_path}") 35 | torch.save(model.state_dict(), pytorch_dump_path) 36 | 37 | 38 | if __name__ == "__main__": 39 | parser = argparse.ArgumentParser() 40 | # Required parameters 41 | parser.add_argument( 42 | "--tf_checkpoint_path", default=None, type=str, required=True, help="Path to the TensorFlow checkpoint path." 43 | ) 44 | parser.add_argument( 45 | "--mobilebert_config_file", 46 | default=None, 47 | type=str, 48 | required=True, 49 | help=( 50 | "The config json file corresponding to the pre-trained MobileBERT model. \n" 51 | "This specifies the model architecture." 52 | ), 53 | ) 54 | parser.add_argument( 55 | "--pytorch_dump_path", default=None, type=str, required=True, help="Path to the output PyTorch model." 56 | ) 57 | args = parser.parse_args() 58 | convert_tf_checkpoint_to_pytorch(args.tf_checkpoint_path, args.mobilebert_config_file, args.pytorch_dump_path) 59 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/mobilebert/tokenization_mobilebert.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | # 3 | # Copyright 2020 The HuggingFace Team. All rights reserved. 4 | # 5 | # Licensed under the Apache License, Version 2.0 (the "License"); 6 | # you may not use this file except in compliance with the License. 7 | # You may obtain a copy of the License at 8 | # 9 | # http://www.apache.org/licenses/LICENSE-2.0 10 | # 11 | # Unless required by applicable law or agreed to in writing, software 12 | # distributed under the License is distributed on an "AS IS" BASIS, 13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | # See the License for the specific language governing permissions and 15 | # limitations under the License. 16 | """Tokenization classes for MobileBERT.""" 17 | 18 | from ...utils import logging 19 | from ..bert.tokenization_bert import BertTokenizer 20 | 21 | 22 | logger = logging.get_logger(__name__) 23 | 24 | VOCAB_FILES_NAMES = {"vocab_file": "vocab.txt"} 25 | 26 | PRETRAINED_VOCAB_FILES_MAP = { 27 | "vocab_file": {"mobilebert-uncased": "https://huggingface.co/google/mobilebert-uncased/resolve/main/vocab.txt"} 28 | } 29 | 30 | PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES = {"mobilebert-uncased": 512} 31 | 32 | 33 | PRETRAINED_INIT_CONFIGURATION = {} 34 | 35 | 36 | class MobileBertTokenizer(BertTokenizer): 37 | r""" 38 | Construct a MobileBERT tokenizer. 39 | 40 | [`MobileBertTokenizer`] is identical to [`BertTokenizer`] and runs end-to-end tokenization: punctuation splitting 41 | and wordpiece. 42 | 43 | Refer to superclass [`BertTokenizer`] for usage examples and documentation concerning parameters. 44 | """ 45 | 46 | vocab_files_names = VOCAB_FILES_NAMES 47 | pretrained_vocab_files_map = PRETRAINED_VOCAB_FILES_MAP 48 | max_model_input_sizes = PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES 49 | pretrained_init_configuration = PRETRAINED_INIT_CONFIGURATION 50 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/mobilebert/tokenization_mobilebert_fast.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | # 3 | # Copyright 2020 The HuggingFace Team. All rights reserved. 4 | # 5 | # Licensed under the Apache License, Version 2.0 (the "License"); 6 | # you may not use this file except in compliance with the License. 7 | # You may obtain a copy of the License at 8 | # 9 | # http://www.apache.org/licenses/LICENSE-2.0 10 | # 11 | # Unless required by applicable law or agreed to in writing, software 12 | # distributed under the License is distributed on an "AS IS" BASIS, 13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | # See the License for the specific language governing permissions and 15 | # limitations under the License. 16 | """Tokenization classes for MobileBERT.""" 17 | 18 | from ...utils import logging 19 | from ..bert.tokenization_bert_fast import BertTokenizerFast 20 | from .tokenization_mobilebert import MobileBertTokenizer 21 | 22 | 23 | logger = logging.get_logger(__name__) 24 | 25 | VOCAB_FILES_NAMES = {"vocab_file": "vocab.txt", "tokenizer_file": "tokenizer.json"} 26 | 27 | PRETRAINED_VOCAB_FILES_MAP = { 28 | "vocab_file": {"mobilebert-uncased": "https://huggingface.co/google/mobilebert-uncased/resolve/main/vocab.txt"}, 29 | "tokenizer_file": { 30 | "mobilebert-uncased": "https://huggingface.co/google/mobilebert-uncased/resolve/main/tokenizer.json" 31 | }, 32 | } 33 | 34 | PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES = {"mobilebert-uncased": 512} 35 | 36 | 37 | PRETRAINED_INIT_CONFIGURATION = {} 38 | 39 | 40 | class MobileBertTokenizerFast(BertTokenizerFast): 41 | r""" 42 | Construct a "fast" MobileBERT tokenizer (backed by HuggingFace's *tokenizers* library). 43 | 44 | [`MobileBertTokenizerFast`] is identical to [`BertTokenizerFast`] and runs end-to-end tokenization: punctuation 45 | splitting and wordpiece. 46 | 47 | Refer to superclass [`BertTokenizerFast`] for usage examples and documentation concerning parameters. 48 | """ 49 | 50 | vocab_files_names = VOCAB_FILES_NAMES 51 | pretrained_vocab_files_map = PRETRAINED_VOCAB_FILES_MAP 52 | max_model_input_sizes = PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES 53 | pretrained_init_configuration = PRETRAINED_INIT_CONFIGURATION 54 | slow_tokenizer_class = MobileBertTokenizer 55 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/nllb/__init__.py: -------------------------------------------------------------------------------- 1 | # flake8: noqa 2 | # There's no way to ignore "F401 '...' imported but unused" warnings in this 3 | # module, but to preserve other warnings. So, don't check this module at all. 4 | 5 | # Copyright 2022 The HuggingFace Team. All rights reserved. 6 | # 7 | # Licensed under the Apache License, Version 2.0 (the "License"); 8 | # you may not use this file except in compliance with the License. 9 | # You may obtain a copy of the License at 10 | # 11 | # http://www.apache.org/licenses/LICENSE-2.0 12 | # 13 | # Unless required by applicable law or agreed to in writing, software 14 | # distributed under the License is distributed on an "AS IS" BASIS, 15 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | # See the License for the specific language governing permissions and 17 | # limitations under the License. 18 | from typing import TYPE_CHECKING 19 | 20 | from ...utils import ( 21 | OptionalDependencyNotAvailable, 22 | _LazyModule, 23 | is_sentencepiece_available, 24 | is_tokenizers_available, 25 | is_torch_available, 26 | ) 27 | 28 | 29 | _import_structure = {} 30 | 31 | try: 32 | if not is_sentencepiece_available(): 33 | raise OptionalDependencyNotAvailable() 34 | except OptionalDependencyNotAvailable: 35 | pass 36 | else: 37 | _import_structure["tokenization_nllb"] = ["NllbTokenizer"] 38 | 39 | try: 40 | if not is_tokenizers_available(): 41 | raise OptionalDependencyNotAvailable() 42 | except OptionalDependencyNotAvailable: 43 | pass 44 | else: 45 | _import_structure["tokenization_nllb_fast"] = ["NllbTokenizerFast"] 46 | 47 | 48 | if TYPE_CHECKING: 49 | try: 50 | if not is_sentencepiece_available(): 51 | raise OptionalDependencyNotAvailable() 52 | except OptionalDependencyNotAvailable: 53 | pass 54 | else: 55 | from .tokenization_nllb import NllbTokenizer 56 | 57 | try: 58 | if not is_tokenizers_available(): 59 | raise OptionalDependencyNotAvailable() 60 | except OptionalDependencyNotAvailable: 61 | pass 62 | else: 63 | from .tokenization_nllb_fast import NllbTokenizerFast 64 | 65 | else: 66 | import sys 67 | 68 | sys.modules[__name__] = _LazyModule(__name__, globals()["__file__"], _import_structure, module_spec=__spec__) 69 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/opt/image_classification/__init__.py: -------------------------------------------------------------------------------- 1 | from . import logger 2 | from . import utils 3 | from . import mixup 4 | from . import smoothing 5 | from . import quantize 6 | from . import preconditioner 7 | from . import activation_quantizer_1 8 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/opt/image_classification/mixup.py: -------------------------------------------------------------------------------- 1 | import torch 2 | import torch.nn as nn 3 | import numpy as np 4 | 5 | 6 | def mixup(alpha, num_classes, data, target): 7 | with torch.no_grad(): 8 | bs = data.size(0) 9 | c = np.random.beta(alpha, alpha) 10 | 11 | perm = torch.randperm(bs).cuda() 12 | 13 | md = c * data + (1-c) * data[perm, :] 14 | mt = c * target + (1-c) * target[perm, :] 15 | return md, mt 16 | 17 | 18 | class MixUpWrapper(object): 19 | def __init__(self, alpha, num_classes, dataloader): 20 | self.alpha = alpha 21 | self.dataloader = dataloader 22 | self.num_classes = num_classes 23 | 24 | def mixup_loader(self, loader): 25 | for input, target in loader: 26 | i, t = mixup(self.alpha, self.num_classes, input, target) 27 | yield i, t 28 | 29 | def __iter__(self): 30 | return self.mixup_loader(self.dataloader) 31 | 32 | 33 | class NLLMultiLabelSmooth(nn.Module): 34 | def __init__(self, smoothing = 0.0): 35 | super(NLLMultiLabelSmooth, self).__init__() 36 | self.confidence = 1.0 - smoothing 37 | self.smoothing = smoothing 38 | 39 | def forward(self, x, target): 40 | if self.training: 41 | x = x.float() 42 | target = target.float() 43 | logprobs = torch.nn.functional.log_softmax(x, dim = -1) 44 | 45 | nll_loss = -logprobs * target 46 | nll_loss = nll_loss.sum(-1) 47 | 48 | smooth_loss = -logprobs.mean(dim=-1) 49 | 50 | loss = self.confidence * nll_loss + self.smoothing * smooth_loss 51 | 52 | return loss.mean() 53 | else: 54 | return torch.nn.functional.cross_entropy(x, target) 55 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/opt/image_classification/plot_variance.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import matplotlib 3 | matplotlib.use('Agg') 4 | from matplotlib import pyplot as plt 5 | from matplotlib.colors import LogNorm 6 | import pickle 7 | import seaborn as sns 8 | 9 | weight_names = pickle.load(open('layer_names.pkl', 'rb')) 10 | 11 | grads = np.load('error_profile_5.npy') 12 | grads = np.maximum(grads, 0) 13 | quantizers = grads.sum(1) 14 | variances = grads.sum(0) 15 | 16 | # grads = np.minimum(grads, 1) 17 | # grads *= 1000 18 | for i in range(grads.shape[0]): 19 | for j in range(grads.shape[1]): 20 | if j > i: 21 | grads[i, j] = 0 22 | 23 | fig, ax = plt.subplots(figsize=(20, 20)) 24 | im = ax.imshow(grads, cmap='Blues', norm=LogNorm(vmin=0.01, vmax=10.0)) 25 | ax.set_xticks(np.arange(len(weight_names))) 26 | ax.set_yticks(np.arange(len(weight_names)+1)) 27 | ax.set_xticklabels(weight_names) 28 | weight_names.append('sample') 29 | ax.set_yticklabels(weight_names) 30 | 31 | ax.tick_params(top=True, bottom=False, 32 | labeltop=True, labelbottom=False) 33 | plt.setp(ax.get_xticklabels(), rotation=-30, ha="right", 34 | rotation_mode="anchor") 35 | cbar = ax.figure.colorbar(im, ax=ax) 36 | 37 | for i in range(grads.shape[0]): 38 | for j in range(grads.shape[1]): 39 | text = ax.text(j, i, int(grads[i, j]*10), 40 | ha="center", va="center") 41 | 42 | 43 | fig.savefig('variance_profile.pdf') 44 | 45 | fig, ax = plt.subplots(figsize=(20, 20)) 46 | sns.barplot(x=np.arange(quantizers.shape[0]), y=quantizers, ax=ax) 47 | ax.set_xticks(np.arange(len(weight_names)) + 0.5) 48 | ax.set_xticklabels(weight_names) 49 | plt.setp(ax.get_xticklabels(), rotation=-30, ha="right", 50 | rotation_mode="anchor") 51 | ax.set_xlabel('quantizer') 52 | ax.set_ylabel('variance') 53 | fig.savefig('quantizers.pdf') 54 | 55 | fig, ax = plt.subplots(figsize=(20, 20)) 56 | sns.barplot(x=np.arange(variances.shape[0]), y=variances, ax=ax) 57 | weight_names.pop(-1) 58 | ax.set_xticks(np.arange(len(weight_names)) + 0.5) 59 | ax.set_xticklabels(weight_names) 60 | plt.setp(ax.get_xticklabels(), rotation=-30, ha="right", 61 | rotation_mode="anchor") 62 | ax.set_xlabel('parameter') 63 | ax.set_ylabel('variance') 64 | fig.savefig('parameter.pdf') 65 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/opt/image_classification/smoothing.py: -------------------------------------------------------------------------------- 1 | import torch 2 | import torch.nn as nn 3 | 4 | class LabelSmoothing(nn.Module): 5 | """ 6 | NLL loss with label smoothing. 7 | """ 8 | def __init__(self, smoothing=0.0): 9 | """ 10 | Constructor for the LabelSmoothing module. 11 | 12 | :param smoothing: label smoothing factor 13 | """ 14 | super(LabelSmoothing, self).__init__() 15 | self.confidence = 1.0 - smoothing 16 | self.smoothing = smoothing 17 | 18 | def forward(self, x, target): 19 | logprobs = torch.nn.functional.log_softmax(x, dim=-1) 20 | 21 | nll_loss = -logprobs.gather(dim=-1, index=target.unsqueeze(1)) 22 | nll_loss = nll_loss.squeeze(1) 23 | smooth_loss = -logprobs.mean(dim=-1) 24 | loss = self.confidence * nll_loss + self.smoothing * smooth_loss 25 | return loss.mean() 26 | 27 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/opt/image_classification/test.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "import torch" 10 | ] 11 | }, 12 | { 13 | "cell_type": "code", 14 | "execution_count": 1, 15 | "metadata": {}, 16 | "outputs": [ 17 | { 18 | "name": "stdout", 19 | "output_type": "stream", 20 | "text": [ 21 | "Original Tensor:\n", 22 | "tensor([ 1.7234, -0.6239, 0.3089, -1.1292, 1.0927])\n", 23 | "\n", 24 | "Indices where norm_x < 1:\n", 25 | "tensor([1, 2, 3])\n" 26 | ] 27 | } 28 | ], 29 | "source": [ 30 | "import torch\n", 31 | "\n", 32 | "# create a 1D tensor\n", 33 | "norm_x = torch.randn(5)\n", 34 | "\n", 35 | "# find indices where norm_x < 1\n", 36 | "indices = torch.where(norm_x < 1)[0]\n", 37 | "\n", 38 | "print(\"Original Tensor:\")\n", 39 | "print(norm_x)\n", 40 | "\n", 41 | "print(\"\\nIndices where norm_x < 1:\")\n", 42 | "print(indices)" 43 | ] 44 | } 45 | ], 46 | "metadata": { 47 | "kernelspec": { 48 | "display_name": "base", 49 | "language": "python", 50 | "name": "python3" 51 | }, 52 | "language_info": { 53 | "codemirror_mode": { 54 | "name": "ipython", 55 | "version": 3 56 | }, 57 | "file_extension": ".py", 58 | "mimetype": "text/x-python", 59 | "name": "python", 60 | "nbconvert_exporter": "python", 61 | "pygments_lexer": "ipython3", 62 | "version": "3.9.12" 63 | }, 64 | "orig_nbformat": 4, 65 | "vscode": { 66 | "interpreter": { 67 | "hash": "88279d2366fe020547cde40dd65aa0e3aa662a6ec1f3ca12d88834876c85e1a6" 68 | } 69 | } 70 | }, 71 | "nbformat": 4, 72 | "nbformat_minor": 2 73 | } 74 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/phobert/__init__.py: -------------------------------------------------------------------------------- 1 | # flake8: noqa 2 | # There's no way to ignore "F401 '...' imported but unused" warnings in this 3 | # module, but to preserve other warnings. So, don't check this module at all. 4 | 5 | # Copyright 2020 The HuggingFace Team. All rights reserved. 6 | # 7 | # Licensed under the Apache License, Version 2.0 (the "License"); 8 | # you may not use this file except in compliance with the License. 9 | # You may obtain a copy of the License at 10 | # 11 | # http://www.apache.org/licenses/LICENSE-2.0 12 | # 13 | # Unless required by applicable law or agreed to in writing, software 14 | # distributed under the License is distributed on an "AS IS" BASIS, 15 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | # See the License for the specific language governing permissions and 17 | # limitations under the License. 18 | 19 | from typing import TYPE_CHECKING 20 | 21 | from ...utils import _LazyModule 22 | 23 | 24 | _import_structure = {"tokenization_phobert": ["PhobertTokenizer"]} 25 | 26 | 27 | if TYPE_CHECKING: 28 | from .tokenization_phobert import PhobertTokenizer 29 | 30 | else: 31 | import sys 32 | 33 | sys.modules[__name__] = _LazyModule(__name__, globals()["__file__"], _import_structure, module_spec=__spec__) 34 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/prophetnet/__init__.py: -------------------------------------------------------------------------------- 1 | # flake8: noqa 2 | # There's no way to ignore "F401 '...' imported but unused" warnings in this 3 | # module, but to preserve other warnings. So, don't check this module at all. 4 | 5 | # Copyright 2020 The HuggingFace Team. All rights reserved. 6 | # 7 | # Licensed under the Apache License, Version 2.0 (the "License"); 8 | # you may not use this file except in compliance with the License. 9 | # You may obtain a copy of the License at 10 | # 11 | # http://www.apache.org/licenses/LICENSE-2.0 12 | # 13 | # Unless required by applicable law or agreed to in writing, software 14 | # distributed under the License is distributed on an "AS IS" BASIS, 15 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | # See the License for the specific language governing permissions and 17 | # limitations under the License. 18 | 19 | from typing import TYPE_CHECKING 20 | 21 | from ...utils import OptionalDependencyNotAvailable, _LazyModule, is_torch_available 22 | 23 | 24 | _import_structure = { 25 | "configuration_prophetnet": ["PROPHETNET_PRETRAINED_CONFIG_ARCHIVE_MAP", "ProphetNetConfig"], 26 | "tokenization_prophetnet": ["ProphetNetTokenizer"], 27 | } 28 | 29 | try: 30 | if not is_torch_available(): 31 | raise OptionalDependencyNotAvailable() 32 | except OptionalDependencyNotAvailable: 33 | pass 34 | else: 35 | _import_structure["modeling_prophetnet"] = [ 36 | "PROPHETNET_PRETRAINED_MODEL_ARCHIVE_LIST", 37 | "ProphetNetDecoder", 38 | "ProphetNetEncoder", 39 | "ProphetNetForCausalLM", 40 | "ProphetNetForConditionalGeneration", 41 | "ProphetNetModel", 42 | "ProphetNetPreTrainedModel", 43 | ] 44 | 45 | 46 | if TYPE_CHECKING: 47 | from .configuration_prophetnet import PROPHETNET_PRETRAINED_CONFIG_ARCHIVE_MAP, ProphetNetConfig 48 | from .tokenization_prophetnet import ProphetNetTokenizer 49 | 50 | try: 51 | if not is_torch_available(): 52 | raise OptionalDependencyNotAvailable() 53 | except OptionalDependencyNotAvailable: 54 | pass 55 | else: 56 | from .modeling_prophetnet import ( 57 | PROPHETNET_PRETRAINED_MODEL_ARCHIVE_LIST, 58 | ProphetNetDecoder, 59 | ProphetNetEncoder, 60 | ProphetNetForCausalLM, 61 | ProphetNetForConditionalGeneration, 62 | ProphetNetModel, 63 | ProphetNetPreTrainedModel, 64 | ) 65 | 66 | else: 67 | import sys 68 | 69 | sys.modules[__name__] = _LazyModule(__name__, globals()["__file__"], _import_structure, module_spec=__spec__) 70 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/rembert/convert_rembert_tf_checkpoint_to_pytorch.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | # Copyright 2018 The HuggingFace Inc. team. 3 | # 4 | # Licensed under the Apache License, Version 2.0 (the "License"); 5 | # you may not use this file except in compliance with the License. 6 | # You may obtain a copy of the License at 7 | # 8 | # http://www.apache.org/licenses/LICENSE-2.0 9 | # 10 | # Unless required by applicable law or agreed to in writing, software 11 | # distributed under the License is distributed on an "AS IS" BASIS, 12 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | # See the License for the specific language governing permissions and 14 | # limitations under the License. 15 | """Convert RemBERT checkpoint.""" 16 | 17 | 18 | import argparse 19 | 20 | import torch 21 | 22 | from transformers import RemBertConfig, RemBertModel, load_tf_weights_in_rembert 23 | from transformers.utils import logging 24 | 25 | 26 | logging.set_verbosity_info() 27 | 28 | 29 | def convert_rembert_tf_checkpoint_to_pytorch(tf_checkpoint_path, bert_config_file, pytorch_dump_path): 30 | # Initialise PyTorch model 31 | config = RemBertConfig.from_json_file(bert_config_file) 32 | print("Building PyTorch model from configuration: {}".format(str(config))) 33 | model = RemBertModel(config) 34 | 35 | # Load weights from tf checkpoint 36 | load_tf_weights_in_rembert(model, config, tf_checkpoint_path) 37 | 38 | # Save pytorch-model 39 | print("Save PyTorch model to {}".format(pytorch_dump_path)) 40 | torch.save(model.state_dict(), pytorch_dump_path) 41 | 42 | 43 | if __name__ == "__main__": 44 | parser = argparse.ArgumentParser() 45 | # Required parameters 46 | parser.add_argument( 47 | "--tf_checkpoint_path", default=None, type=str, required=True, help="Path to the TensorFlow checkpoint path." 48 | ) 49 | parser.add_argument( 50 | "--rembert_config_file", 51 | default=None, 52 | type=str, 53 | required=True, 54 | help=( 55 | "The config json file corresponding to the pre-trained RemBERT model. \n" 56 | "This specifies the model architecture." 57 | ), 58 | ) 59 | parser.add_argument( 60 | "--pytorch_dump_path", default=None, type=str, required=True, help="Path to the output PyTorch model." 61 | ) 62 | args = parser.parse_args() 63 | convert_rembert_tf_checkpoint_to_pytorch(args.tf_checkpoint_path, args.rembert_config_file, args.pytorch_dump_path) 64 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/retribert/tokenization_retribert.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | # Copyright 2018 The HuggingFace Inc. team. 3 | # 4 | # Licensed under the Apache License, Version 2.0 (the "License"); 5 | # you may not use this file except in compliance with the License. 6 | # You may obtain a copy of the License at 7 | # 8 | # http://www.apache.org/licenses/LICENSE-2.0 9 | # 10 | # Unless required by applicable law or agreed to in writing, software 11 | # distributed under the License is distributed on an "AS IS" BASIS, 12 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | # See the License for the specific language governing permissions and 14 | # limitations under the License. 15 | """Tokenization classes for RetriBERT.""" 16 | 17 | from ...utils import logging 18 | from ..bert.tokenization_bert import BertTokenizer 19 | 20 | 21 | logger = logging.get_logger(__name__) 22 | 23 | VOCAB_FILES_NAMES = {"vocab_file": "vocab.txt"} 24 | 25 | PRETRAINED_VOCAB_FILES_MAP = { 26 | "vocab_file": { 27 | "yjernite/retribert-base-uncased": ( 28 | "https://huggingface.co/yjernite/retribert-base-uncased/resolve/main/vocab.txt" 29 | ), 30 | } 31 | } 32 | 33 | PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES = { 34 | "yjernite/retribert-base-uncased": 512, 35 | } 36 | 37 | 38 | PRETRAINED_INIT_CONFIGURATION = { 39 | "yjernite/retribert-base-uncased": {"do_lower_case": True}, 40 | } 41 | 42 | 43 | class RetriBertTokenizer(BertTokenizer): 44 | r""" 45 | Constructs a RetriBERT tokenizer. 46 | 47 | [`RetriBertTokenizer`] is identical to [`BertTokenizer`] and runs end-to-end tokenization: punctuation splitting 48 | and wordpiece. 49 | 50 | Refer to superclass [`BertTokenizer`] for usage examples and documentation concerning parameters. 51 | """ 52 | 53 | vocab_files_names = VOCAB_FILES_NAMES 54 | pretrained_vocab_files_map = PRETRAINED_VOCAB_FILES_MAP 55 | max_model_input_sizes = PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES 56 | pretrained_init_configuration = PRETRAINED_INIT_CONFIGURATION 57 | model_input_names = ["input_ids", "attention_mask"] 58 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/retribert/tokenization_retribert_fast.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | # Copyright 2018 The HuggingFace Inc. team. 3 | # 4 | # Licensed under the Apache License, Version 2.0 (the "License"); 5 | # you may not use this file except in compliance with the License. 6 | # You may obtain a copy of the License at 7 | # 8 | # http://www.apache.org/licenses/LICENSE-2.0 9 | # 10 | # Unless required by applicable law or agreed to in writing, software 11 | # distributed under the License is distributed on an "AS IS" BASIS, 12 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | # See the License for the specific language governing permissions and 14 | # limitations under the License. 15 | """Tokenization classes for RetriBERT.""" 16 | 17 | from ...utils import logging 18 | from ..bert.tokenization_bert_fast import BertTokenizerFast 19 | from .tokenization_retribert import RetriBertTokenizer 20 | 21 | 22 | logger = logging.get_logger(__name__) 23 | 24 | VOCAB_FILES_NAMES = {"vocab_file": "vocab.txt", "tokenizer_file": "tokenizer.json"} 25 | 26 | PRETRAINED_VOCAB_FILES_MAP = { 27 | "vocab_file": { 28 | "yjernite/retribert-base-uncased": ( 29 | "https://huggingface.co/yjernite/retribert-base-uncased/resolve/main/vocab.txt" 30 | ), 31 | }, 32 | "tokenizer_file": { 33 | "yjernite/retribert-base-uncased": ( 34 | "https://huggingface.co/yjernite/retribert-base-uncased/resolve/main/tokenizer.json" 35 | ), 36 | }, 37 | } 38 | 39 | PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES = { 40 | "yjernite/retribert-base-uncased": 512, 41 | } 42 | 43 | 44 | PRETRAINED_INIT_CONFIGURATION = { 45 | "yjernite/retribert-base-uncased": {"do_lower_case": True}, 46 | } 47 | 48 | 49 | class RetriBertTokenizerFast(BertTokenizerFast): 50 | r""" 51 | Construct a "fast" RetriBERT tokenizer (backed by HuggingFace's *tokenizers* library). 52 | 53 | [`RetriBertTokenizerFast`] is identical to [`BertTokenizerFast`] and runs end-to-end tokenization: punctuation 54 | splitting and wordpiece. 55 | 56 | Refer to superclass [`BertTokenizerFast`] for usage examples and documentation concerning parameters. 57 | """ 58 | 59 | vocab_files_names = VOCAB_FILES_NAMES 60 | pretrained_vocab_files_map = PRETRAINED_VOCAB_FILES_MAP 61 | max_model_input_sizes = PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES 62 | pretrained_init_configuration = PRETRAINED_INIT_CONFIGURATION 63 | slow_tokenizer_class = RetriBertTokenizer 64 | model_input_names = ["input_ids", "attention_mask"] 65 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/roberta/image_classification/__init__.py: -------------------------------------------------------------------------------- 1 | from . import logger 2 | from . import utils 3 | from . import mixup 4 | from . import smoothing 5 | from . import quantize 6 | from . import preconditioner 7 | from . import activation_quantizer_1 8 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/roberta/image_classification/mixup.py: -------------------------------------------------------------------------------- 1 | import torch 2 | import torch.nn as nn 3 | import numpy as np 4 | 5 | 6 | def mixup(alpha, num_classes, data, target): 7 | with torch.no_grad(): 8 | bs = data.size(0) 9 | c = np.random.beta(alpha, alpha) 10 | 11 | perm = torch.randperm(bs).cuda() 12 | 13 | md = c * data + (1-c) * data[perm, :] 14 | mt = c * target + (1-c) * target[perm, :] 15 | return md, mt 16 | 17 | 18 | class MixUpWrapper(object): 19 | def __init__(self, alpha, num_classes, dataloader): 20 | self.alpha = alpha 21 | self.dataloader = dataloader 22 | self.num_classes = num_classes 23 | 24 | def mixup_loader(self, loader): 25 | for input, target in loader: 26 | i, t = mixup(self.alpha, self.num_classes, input, target) 27 | yield i, t 28 | 29 | def __iter__(self): 30 | return self.mixup_loader(self.dataloader) 31 | 32 | 33 | class NLLMultiLabelSmooth(nn.Module): 34 | def __init__(self, smoothing = 0.0): 35 | super(NLLMultiLabelSmooth, self).__init__() 36 | self.confidence = 1.0 - smoothing 37 | self.smoothing = smoothing 38 | 39 | def forward(self, x, target): 40 | if self.training: 41 | x = x.float() 42 | target = target.float() 43 | logprobs = torch.nn.functional.log_softmax(x, dim = -1) 44 | 45 | nll_loss = -logprobs * target 46 | nll_loss = nll_loss.sum(-1) 47 | 48 | smooth_loss = -logprobs.mean(dim=-1) 49 | 50 | loss = self.confidence * nll_loss + self.smoothing * smooth_loss 51 | 52 | return loss.mean() 53 | else: 54 | return torch.nn.functional.cross_entropy(x, target) 55 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/roberta/image_classification/plot_variance.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import matplotlib 3 | matplotlib.use('Agg') 4 | from matplotlib import pyplot as plt 5 | from matplotlib.colors import LogNorm 6 | import pickle 7 | import seaborn as sns 8 | 9 | weight_names = pickle.load(open('layer_names.pkl', 'rb')) 10 | 11 | grads = np.load('error_profile_5.npy') 12 | grads = np.maximum(grads, 0) 13 | quantizers = grads.sum(1) 14 | variances = grads.sum(0) 15 | 16 | # grads = np.minimum(grads, 1) 17 | # grads *= 1000 18 | for i in range(grads.shape[0]): 19 | for j in range(grads.shape[1]): 20 | if j > i: 21 | grads[i, j] = 0 22 | 23 | fig, ax = plt.subplots(figsize=(20, 20)) 24 | im = ax.imshow(grads, cmap='Blues', norm=LogNorm(vmin=0.01, vmax=10.0)) 25 | ax.set_xticks(np.arange(len(weight_names))) 26 | ax.set_yticks(np.arange(len(weight_names)+1)) 27 | ax.set_xticklabels(weight_names) 28 | weight_names.append('sample') 29 | ax.set_yticklabels(weight_names) 30 | 31 | ax.tick_params(top=True, bottom=False, 32 | labeltop=True, labelbottom=False) 33 | plt.setp(ax.get_xticklabels(), rotation=-30, ha="right", 34 | rotation_mode="anchor") 35 | cbar = ax.figure.colorbar(im, ax=ax) 36 | 37 | for i in range(grads.shape[0]): 38 | for j in range(grads.shape[1]): 39 | text = ax.text(j, i, int(grads[i, j]*10), 40 | ha="center", va="center") 41 | 42 | 43 | fig.savefig('variance_profile.pdf') 44 | 45 | fig, ax = plt.subplots(figsize=(20, 20)) 46 | sns.barplot(x=np.arange(quantizers.shape[0]), y=quantizers, ax=ax) 47 | ax.set_xticks(np.arange(len(weight_names)) + 0.5) 48 | ax.set_xticklabels(weight_names) 49 | plt.setp(ax.get_xticklabels(), rotation=-30, ha="right", 50 | rotation_mode="anchor") 51 | ax.set_xlabel('quantizer') 52 | ax.set_ylabel('variance') 53 | fig.savefig('quantizers.pdf') 54 | 55 | fig, ax = plt.subplots(figsize=(20, 20)) 56 | sns.barplot(x=np.arange(variances.shape[0]), y=variances, ax=ax) 57 | weight_names.pop(-1) 58 | ax.set_xticks(np.arange(len(weight_names)) + 0.5) 59 | ax.set_xticklabels(weight_names) 60 | plt.setp(ax.get_xticklabels(), rotation=-30, ha="right", 61 | rotation_mode="anchor") 62 | ax.set_xlabel('parameter') 63 | ax.set_ylabel('variance') 64 | fig.savefig('parameter.pdf') 65 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/roberta/image_classification/smoothing.py: -------------------------------------------------------------------------------- 1 | import torch 2 | import torch.nn as nn 3 | 4 | class LabelSmoothing(nn.Module): 5 | """ 6 | NLL loss with label smoothing. 7 | """ 8 | def __init__(self, smoothing=0.0): 9 | """ 10 | Constructor for the LabelSmoothing module. 11 | 12 | :param smoothing: label smoothing factor 13 | """ 14 | super(LabelSmoothing, self).__init__() 15 | self.confidence = 1.0 - smoothing 16 | self.smoothing = smoothing 17 | 18 | def forward(self, x, target): 19 | logprobs = torch.nn.functional.log_softmax(x, dim=-1) 20 | 21 | nll_loss = -logprobs.gather(dim=-1, index=target.unsqueeze(1)) 22 | nll_loss = nll_loss.squeeze(1) 23 | smooth_loss = -logprobs.mean(dim=-1) 24 | loss = self.confidence * nll_loss + self.smoothing * smooth_loss 25 | return loss.mean() 26 | 27 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/roberta/image_classification/test.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "import torch" 10 | ] 11 | }, 12 | { 13 | "cell_type": "code", 14 | "execution_count": 1, 15 | "metadata": {}, 16 | "outputs": [ 17 | { 18 | "name": "stdout", 19 | "output_type": "stream", 20 | "text": [ 21 | "Original Tensor:\n", 22 | "tensor([ 1.7234, -0.6239, 0.3089, -1.1292, 1.0927])\n", 23 | "\n", 24 | "Indices where norm_x < 1:\n", 25 | "tensor([1, 2, 3])\n" 26 | ] 27 | } 28 | ], 29 | "source": [ 30 | "import torch\n", 31 | "\n", 32 | "# create a 1D tensor\n", 33 | "norm_x = torch.randn(5)\n", 34 | "\n", 35 | "# find indices where norm_x < 1\n", 36 | "indices = torch.where(norm_x < 1)[0]\n", 37 | "\n", 38 | "print(\"Original Tensor:\")\n", 39 | "print(norm_x)\n", 40 | "\n", 41 | "print(\"\\nIndices where norm_x < 1:\")\n", 42 | "print(indices)" 43 | ] 44 | } 45 | ], 46 | "metadata": { 47 | "kernelspec": { 48 | "display_name": "base", 49 | "language": "python", 50 | "name": "python3" 51 | }, 52 | "language_info": { 53 | "codemirror_mode": { 54 | "name": "ipython", 55 | "version": 3 56 | }, 57 | "file_extension": ".py", 58 | "mimetype": "text/x-python", 59 | "name": "python", 60 | "nbconvert_exporter": "python", 61 | "pygments_lexer": "ipython3", 62 | "version": "3.9.12" 63 | }, 64 | "orig_nbformat": 4, 65 | "vscode": { 66 | "interpreter": { 67 | "hash": "88279d2366fe020547cde40dd65aa0e3aa662a6ec1f3ca12d88834876c85e1a6" 68 | } 69 | } 70 | }, 71 | "nbformat": 4, 72 | "nbformat_minor": 2 73 | } 74 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/roformer/convert_roformer_original_tf_checkpoint_to_pytorch.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | # Copyright 2021 The HuggingFace Inc. team. All rights reserved. 3 | # 4 | # Licensed under the Apache License, Version 2.0 (the "License"); 5 | # you may not use this file except in compliance with the License. 6 | # You may obtain a copy of the License at 7 | # 8 | # http://www.apache.org/licenses/LICENSE-2.0 9 | # 10 | # Unless required by applicable law or agreed to in writing, software 11 | # distributed under the License is distributed on an "AS IS" BASIS, 12 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | # See the License for the specific language governing permissions and 14 | # limitations under the License. 15 | """Convert RoFormer checkpoint.""" 16 | 17 | 18 | import argparse 19 | 20 | import torch 21 | 22 | from transformers import RoFormerConfig, RoFormerForMaskedLM, load_tf_weights_in_roformer 23 | from transformers.utils import logging 24 | 25 | 26 | logging.set_verbosity_info() 27 | 28 | 29 | def convert_tf_checkpoint_to_pytorch(tf_checkpoint_path, bert_config_file, pytorch_dump_path): 30 | # Initialise PyTorch model 31 | config = RoFormerConfig.from_json_file(bert_config_file) 32 | print(f"Building PyTorch model from configuration: {config}") 33 | model = RoFormerForMaskedLM(config) 34 | 35 | # Load weights from tf checkpoint 36 | load_tf_weights_in_roformer(model, config, tf_checkpoint_path) 37 | 38 | # Save pytorch-model 39 | print(f"Save PyTorch model to {pytorch_dump_path}") 40 | torch.save(model.state_dict(), pytorch_dump_path, _use_new_zipfile_serialization=False) 41 | 42 | 43 | if __name__ == "__main__": 44 | parser = argparse.ArgumentParser() 45 | # Required parameters 46 | parser.add_argument( 47 | "--tf_checkpoint_path", default=None, type=str, required=True, help="Path to the TensorFlow checkpoint path." 48 | ) 49 | parser.add_argument( 50 | "--bert_config_file", 51 | default=None, 52 | type=str, 53 | required=True, 54 | help=( 55 | "The config json file corresponding to the pre-trained BERT model. \n" 56 | "This specifies the model architecture." 57 | ), 58 | ) 59 | parser.add_argument( 60 | "--pytorch_dump_path", default=None, type=str, required=True, help="Path to the output PyTorch model." 61 | ) 62 | args = parser.parse_args() 63 | convert_tf_checkpoint_to_pytorch(args.tf_checkpoint_path, args.bert_config_file, args.pytorch_dump_path) 64 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/sew/__init__.py: -------------------------------------------------------------------------------- 1 | # flake8: noqa 2 | # There's no way to ignore "F401 '...' imported but unused" warnings in this 3 | # module, but to preserve other warnings. So, don't check this module at all. 4 | 5 | # Copyright 2021 The HuggingFace Team. All rights reserved. 6 | # 7 | # Licensed under the Apache License, Version 2.0 (the "License"); 8 | # you may not use this file except in compliance with the License. 9 | # You may obtain a copy of the License at 10 | # 11 | # http://www.apache.org/licenses/LICENSE-2.0 12 | # 13 | # Unless required by applicable law or agreed to in writing, software 14 | # distributed under the License is distributed on an "AS IS" BASIS, 15 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | # See the License for the specific language governing permissions and 17 | # limitations under the License. 18 | from typing import TYPE_CHECKING 19 | 20 | from ...utils import OptionalDependencyNotAvailable, _LazyModule, is_torch_available 21 | 22 | 23 | _import_structure = {"configuration_sew": ["SEW_PRETRAINED_CONFIG_ARCHIVE_MAP", "SEWConfig"]} 24 | 25 | try: 26 | if not is_torch_available(): 27 | raise OptionalDependencyNotAvailable() 28 | except OptionalDependencyNotAvailable: 29 | pass 30 | else: 31 | _import_structure["modeling_sew"] = [ 32 | "SEW_PRETRAINED_MODEL_ARCHIVE_LIST", 33 | "SEWForCTC", 34 | "SEWForSequenceClassification", 35 | "SEWModel", 36 | "SEWPreTrainedModel", 37 | ] 38 | 39 | if TYPE_CHECKING: 40 | from .configuration_sew import SEW_PRETRAINED_CONFIG_ARCHIVE_MAP, SEWConfig 41 | 42 | try: 43 | if not is_torch_available(): 44 | raise OptionalDependencyNotAvailable() 45 | except OptionalDependencyNotAvailable: 46 | pass 47 | else: 48 | from .modeling_sew import ( 49 | SEW_PRETRAINED_MODEL_ARCHIVE_LIST, 50 | SEWForCTC, 51 | SEWForSequenceClassification, 52 | SEWModel, 53 | SEWPreTrainedModel, 54 | ) 55 | 56 | 57 | else: 58 | import sys 59 | 60 | sys.modules[__name__] = _LazyModule(__name__, globals()["__file__"], _import_structure, module_spec=__spec__) 61 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/sew_d/__init__.py: -------------------------------------------------------------------------------- 1 | # flake8: noqa 2 | # There's no way to ignore "F401 '...' imported but unused" warnings in this 3 | # module, but to preserve other warnings. So, don't check this module at all. 4 | 5 | # Copyright 2021 The HuggingFace Team. All rights reserved. 6 | # 7 | # Licensed under the Apache License, Version 2.0 (the "License"); 8 | # you may not use this file except in compliance with the License. 9 | # You may obtain a copy of the License at 10 | # 11 | # http://www.apache.org/licenses/LICENSE-2.0 12 | # 13 | # Unless required by applicable law or agreed to in writing, software 14 | # distributed under the License is distributed on an "AS IS" BASIS, 15 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | # See the License for the specific language governing permissions and 17 | # limitations under the License. 18 | from typing import TYPE_CHECKING 19 | 20 | from ...utils import OptionalDependencyNotAvailable, _LazyModule, is_torch_available 21 | 22 | 23 | _import_structure = {"configuration_sew_d": ["SEW_D_PRETRAINED_CONFIG_ARCHIVE_MAP", "SEWDConfig"]} 24 | 25 | try: 26 | if not is_torch_available(): 27 | raise OptionalDependencyNotAvailable() 28 | except OptionalDependencyNotAvailable: 29 | pass 30 | else: 31 | _import_structure["modeling_sew_d"] = [ 32 | "SEW_D_PRETRAINED_MODEL_ARCHIVE_LIST", 33 | "SEWDForCTC", 34 | "SEWDForSequenceClassification", 35 | "SEWDModel", 36 | "SEWDPreTrainedModel", 37 | ] 38 | 39 | if TYPE_CHECKING: 40 | from .configuration_sew_d import SEW_D_PRETRAINED_CONFIG_ARCHIVE_MAP, SEWDConfig 41 | 42 | try: 43 | if not is_torch_available(): 44 | raise OptionalDependencyNotAvailable() 45 | except OptionalDependencyNotAvailable: 46 | pass 47 | else: 48 | from .modeling_sew_d import ( 49 | SEW_D_PRETRAINED_MODEL_ARCHIVE_LIST, 50 | SEWDForCTC, 51 | SEWDForSequenceClassification, 52 | SEWDModel, 53 | SEWDPreTrainedModel, 54 | ) 55 | 56 | 57 | else: 58 | import sys 59 | 60 | sys.modules[__name__] = _LazyModule(__name__, globals()["__file__"], _import_structure, module_spec=__spec__) 61 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/speech_encoder_decoder/__init__.py: -------------------------------------------------------------------------------- 1 | # flake8: noqa 2 | # There's no way to ignore "F401 '...' imported but unused" warnings in this 3 | # module, but to preserve other warnings. So, don't check this module at all. 4 | 5 | # Copyright 2021 The HuggingFace Team. All rights reserved. 6 | # 7 | # Licensed under the Apache License, Version 2.0 (the "License"); 8 | # you may not use this file except in compliance with the License. 9 | # You may obtain a copy of the License at 10 | # 11 | # http://www.apache.org/licenses/LICENSE-2.0 12 | # 13 | # Unless required by applicable law or agreed to in writing, software 14 | # distributed under the License is distributed on an "AS IS" BASIS, 15 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | # See the License for the specific language governing permissions and 17 | # limitations under the License. 18 | 19 | from typing import TYPE_CHECKING 20 | 21 | from ...utils import OptionalDependencyNotAvailable, _LazyModule, is_flax_available, is_torch_available 22 | 23 | 24 | _import_structure = {"configuration_speech_encoder_decoder": ["SpeechEncoderDecoderConfig"]} 25 | 26 | try: 27 | if not is_torch_available(): 28 | raise OptionalDependencyNotAvailable() 29 | except OptionalDependencyNotAvailable: 30 | pass 31 | else: 32 | _import_structure["modeling_speech_encoder_decoder"] = ["SpeechEncoderDecoderModel"] 33 | 34 | try: 35 | if not is_flax_available(): 36 | raise OptionalDependencyNotAvailable() 37 | except OptionalDependencyNotAvailable: 38 | pass 39 | else: 40 | _import_structure["modeling_flax_speech_encoder_decoder"] = ["FlaxSpeechEncoderDecoderModel"] 41 | 42 | if TYPE_CHECKING: 43 | from .configuration_speech_encoder_decoder import SpeechEncoderDecoderConfig 44 | 45 | try: 46 | if not is_torch_available(): 47 | raise OptionalDependencyNotAvailable() 48 | except OptionalDependencyNotAvailable: 49 | pass 50 | else: 51 | from .modeling_speech_encoder_decoder import SpeechEncoderDecoderModel 52 | 53 | try: 54 | if not is_flax_available(): 55 | raise OptionalDependencyNotAvailable() 56 | except OptionalDependencyNotAvailable: 57 | pass 58 | else: 59 | from .modeling_flax_speech_encoder_decoder import FlaxSpeechEncoderDecoderModel 60 | 61 | else: 62 | import sys 63 | 64 | sys.modules[__name__] = _LazyModule(__name__, globals()["__file__"], _import_structure, module_spec=__spec__) 65 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/speech_to_text_2/__init__.py: -------------------------------------------------------------------------------- 1 | # flake8: noqa 2 | # There's no way to ignore "F401 '...' imported but unused" warnings in this 3 | # module, but to preserve other warnings. So, don't check this module at all. 4 | 5 | # Copyright 2021 The HuggingFace Team. All rights reserved. 6 | # 7 | # Licensed under the Apache License, Version 2.0 (the "License"); 8 | # you may not use this file except in compliance with the License. 9 | # You may obtain a copy of the License at 10 | # 11 | # http://www.apache.org/licenses/LICENSE-2.0 12 | # 13 | # Unless required by applicable law or agreed to in writing, software 14 | # distributed under the License is distributed on an "AS IS" BASIS, 15 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | # See the License for the specific language governing permissions and 17 | # limitations under the License. 18 | from typing import TYPE_CHECKING 19 | 20 | from ...utils import ( 21 | OptionalDependencyNotAvailable, 22 | _LazyModule, 23 | is_sentencepiece_available, 24 | is_speech_available, 25 | is_torch_available, 26 | ) 27 | 28 | 29 | _import_structure = { 30 | "configuration_speech_to_text_2": ["SPEECH_TO_TEXT_2_PRETRAINED_CONFIG_ARCHIVE_MAP", "Speech2Text2Config"], 31 | "processing_speech_to_text_2": ["Speech2Text2Processor"], 32 | "tokenization_speech_to_text_2": ["Speech2Text2Tokenizer"], 33 | } 34 | 35 | 36 | try: 37 | if not is_torch_available(): 38 | raise OptionalDependencyNotAvailable() 39 | except OptionalDependencyNotAvailable: 40 | pass 41 | else: 42 | _import_structure["modeling_speech_to_text_2"] = [ 43 | "SPEECH_TO_TEXT_2_PRETRAINED_MODEL_ARCHIVE_LIST", 44 | "Speech2Text2ForCausalLM", 45 | "Speech2Text2PreTrainedModel", 46 | ] 47 | 48 | 49 | if TYPE_CHECKING: 50 | from .configuration_speech_to_text_2 import SPEECH_TO_TEXT_2_PRETRAINED_CONFIG_ARCHIVE_MAP, Speech2Text2Config 51 | from .processing_speech_to_text_2 import Speech2Text2Processor 52 | from .tokenization_speech_to_text_2 import Speech2Text2Tokenizer 53 | 54 | try: 55 | if not is_torch_available(): 56 | raise OptionalDependencyNotAvailable() 57 | except OptionalDependencyNotAvailable: 58 | pass 59 | else: 60 | from .modeling_speech_to_text_2 import ( 61 | SPEECH_TO_TEXT_2_PRETRAINED_MODEL_ARCHIVE_LIST, 62 | Speech2Text2ForCausalLM, 63 | Speech2Text2PreTrainedModel, 64 | ) 65 | 66 | else: 67 | import sys 68 | 69 | sys.modules[__name__] = _LazyModule(__name__, globals()["__file__"], _import_structure, module_spec=__spec__) 70 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/squeezebert/tokenization_squeezebert.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | # Copyright 2020 The SqueezeBert authors and The HuggingFace Inc. team. 3 | # 4 | # Licensed under the Apache License, Version 2.0 (the "License"); 5 | # you may not use this file except in compliance with the License. 6 | # You may obtain a copy of the License at 7 | # 8 | # http://www.apache.org/licenses/LICENSE-2.0 9 | # 10 | # Unless required by applicable law or agreed to in writing, software 11 | # distributed under the License is distributed on an "AS IS" BASIS, 12 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | # See the License for the specific language governing permissions and 14 | # limitations under the License. 15 | """Tokenization classes for SqueezeBERT.""" 16 | 17 | from ...utils import logging 18 | from ..bert.tokenization_bert import BertTokenizer 19 | 20 | 21 | logger = logging.get_logger(__name__) 22 | 23 | VOCAB_FILES_NAMES = {"vocab_file": "vocab.txt"} 24 | 25 | PRETRAINED_VOCAB_FILES_MAP = { 26 | "vocab_file": { 27 | "squeezebert/squeezebert-uncased": ( 28 | "https://huggingface.co/squeezebert/squeezebert-uncased/resolve/main/vocab.txt" 29 | ), 30 | "squeezebert/squeezebert-mnli": "https://huggingface.co/squeezebert/squeezebert-mnli/resolve/main/vocab.txt", 31 | "squeezebert/squeezebert-mnli-headless": ( 32 | "https://huggingface.co/squeezebert/squeezebert-mnli-headless/resolve/main/vocab.txt" 33 | ), 34 | } 35 | } 36 | 37 | PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES = { 38 | "squeezebert/squeezebert-uncased": 512, 39 | "squeezebert/squeezebert-mnli": 512, 40 | "squeezebert/squeezebert-mnli-headless": 512, 41 | } 42 | 43 | 44 | PRETRAINED_INIT_CONFIGURATION = { 45 | "squeezebert/squeezebert-uncased": {"do_lower_case": True}, 46 | "squeezebert/squeezebert-mnli": {"do_lower_case": True}, 47 | "squeezebert/squeezebert-mnli-headless": {"do_lower_case": True}, 48 | } 49 | 50 | 51 | class SqueezeBertTokenizer(BertTokenizer): 52 | r""" 53 | Constructs a SqueezeBert tokenizer. 54 | 55 | [`SqueezeBertTokenizer`] is identical to [`BertTokenizer`] and runs end-to-end tokenization: punctuation splitting 56 | + wordpiece. 57 | 58 | Refer to superclass [`BertTokenizer`] for usage examples and documentation concerning parameters. 59 | """ 60 | 61 | vocab_files_names = VOCAB_FILES_NAMES 62 | pretrained_vocab_files_map = PRETRAINED_VOCAB_FILES_MAP 63 | max_model_input_sizes = PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES 64 | pretrained_init_configuration = PRETRAINED_INIT_CONFIGURATION 65 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/swinv2/__init__.py: -------------------------------------------------------------------------------- 1 | # flake8: noqa 2 | # There's no way to ignore "F401 '...' imported but unused" warnings in this 3 | # module, but to preserve other warnings. So, don't check this module at all. 4 | 5 | # Copyright 2022 The HuggingFace Team. All rights reserved. 6 | # 7 | # Licensed under the Apache License, Version 2.0 (the "License"); 8 | # you may not use this file except in compliance with the License. 9 | # You may obtain a copy of the License at 10 | # 11 | # http://www.apache.org/licenses/LICENSE-2.0 12 | # 13 | # Unless required by applicable law or agreed to in writing, software 14 | # distributed under the License is distributed on an "AS IS" BASIS, 15 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | # See the License for the specific language governing permissions and 17 | # limitations under the License. 18 | from typing import TYPE_CHECKING 19 | 20 | # rely on isort to merge the imports 21 | from ...utils import OptionalDependencyNotAvailable, _LazyModule, is_torch_available 22 | 23 | 24 | _import_structure = { 25 | "configuration_swinv2": ["SWINV2_PRETRAINED_CONFIG_ARCHIVE_MAP", "Swinv2Config"], 26 | } 27 | 28 | 29 | try: 30 | if not is_torch_available(): 31 | raise OptionalDependencyNotAvailable() 32 | except OptionalDependencyNotAvailable: 33 | pass 34 | else: 35 | _import_structure["modeling_swinv2"] = [ 36 | "SWINV2_PRETRAINED_MODEL_ARCHIVE_LIST", 37 | "Swinv2ForImageClassification", 38 | "Swinv2ForMaskedImageModeling", 39 | "Swinv2Model", 40 | "Swinv2PreTrainedModel", 41 | ] 42 | 43 | 44 | if TYPE_CHECKING: 45 | from .configuration_swinv2 import SWINV2_PRETRAINED_CONFIG_ARCHIVE_MAP, Swinv2Config 46 | 47 | try: 48 | if not is_torch_available(): 49 | raise OptionalDependencyNotAvailable() 50 | except OptionalDependencyNotAvailable: 51 | pass 52 | else: 53 | from .modeling_swinv2 import ( 54 | SWINV2_PRETRAINED_MODEL_ARCHIVE_LIST, 55 | Swinv2ForImageClassification, 56 | Swinv2ForMaskedImageModeling, 57 | Swinv2Model, 58 | Swinv2PreTrainedModel, 59 | ) 60 | 61 | 62 | else: 63 | import sys 64 | 65 | sys.modules[__name__] = _LazyModule(__name__, globals()["__file__"], _import_structure, module_spec=__spec__) 66 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/t5/convert_t5_original_tf_checkpoint_to_pytorch.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | # Copyright 2018 The T5 authors and HuggingFace Inc. team. 3 | # 4 | # Licensed under the Apache License, Version 2.0 (the "License"); 5 | # you may not use this file except in compliance with the License. 6 | # You may obtain a copy of the License at 7 | # 8 | # http://www.apache.org/licenses/LICENSE-2.0 9 | # 10 | # Unless required by applicable law or agreed to in writing, software 11 | # distributed under the License is distributed on an "AS IS" BASIS, 12 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | # See the License for the specific language governing permissions and 14 | # limitations under the License. 15 | """Convert T5 checkpoint.""" 16 | 17 | 18 | import argparse 19 | 20 | from transformers import T5Config, T5ForConditionalGeneration, load_tf_weights_in_t5 21 | from transformers.utils import logging 22 | 23 | 24 | logging.set_verbosity_info() 25 | 26 | 27 | def convert_tf_checkpoint_to_pytorch(tf_checkpoint_path, config_file, pytorch_dump_path): 28 | # Initialise PyTorch model 29 | config = T5Config.from_json_file(config_file) 30 | print(f"Building PyTorch model from configuration: {config}") 31 | model = T5ForConditionalGeneration(config) 32 | 33 | # Load weights from tf checkpoint 34 | load_tf_weights_in_t5(model, config, tf_checkpoint_path) 35 | 36 | # Save pytorch-model 37 | print(f"Save PyTorch model to {pytorch_dump_path}") 38 | model.save_pretrained(pytorch_dump_path) 39 | 40 | 41 | if __name__ == "__main__": 42 | parser = argparse.ArgumentParser() 43 | # Required parameters 44 | parser.add_argument( 45 | "--tf_checkpoint_path", default=None, type=str, required=True, help="Path to the TensorFlow checkpoint path." 46 | ) 47 | parser.add_argument( 48 | "--config_file", 49 | default=None, 50 | type=str, 51 | required=True, 52 | help=( 53 | "The config json file corresponding to the pre-trained T5 model. \nThis specifies the model architecture." 54 | ), 55 | ) 56 | parser.add_argument( 57 | "--pytorch_dump_path", default=None, type=str, required=True, help="Path to the output PyTorch model." 58 | ) 59 | args = parser.parse_args() 60 | convert_tf_checkpoint_to_pytorch(args.tf_checkpoint_path, args.config_file, args.pytorch_dump_path) 61 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/t5/download_from_gcp.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | # Use this script as follows ./download_from_gcp.sh /path/to/folder/to/store/downloads 3 | folder_to_store_downloads=${1} 4 | 5 | # Replace by gcp_path to T5 cloud bucket folder here 6 | # To download the official `t5-small` model of https://github.com/google-research/text-to-text-transfer-transformer#released-model-checkpoints: 7 | gcp_path="gs://t5-data/pretrained_models/small" 8 | 9 | # Number of files the checkpoint is split into 10 | num_of_checks=16 11 | 12 | # Create dir if not exist 13 | mkdir -p ${folder_to_store_downloads} 14 | 15 | # Copy all meta information files 16 | gsutil cp "${gcp_path}/operative_config.gin" ${folder_to_store_downloads} 17 | gsutil cp "${gcp_path}/checkpoint" ${folder_to_store_downloads} 18 | gsutil cp "${gcp_path}/model.ckpt-1000000.index" ${folder_to_store_downloads} 19 | gsutil cp "${gcp_path}/model.ckpt-1000000.meta" ${folder_to_store_downloads} 20 | 21 | # Copy all model weights 22 | # single digit num checkpoitns 23 | for ((i = 0 ; i < ${num_of_checks} ; i++)); do 24 | gsutil cp "${gcp_path}/model.ckpt-1000000.data-0000${i}-of-000${num_of_checks}" ${folder_to_store_downloads} 25 | done 26 | 27 | # double digit num checkpoints 28 | for ((i = 0 ; i < ${num_of_checks} ; i++)); do 29 | gsutil cp "${gcp_path}/model.ckpt-1000000.data-000${i}-of-000${num_of_checks}" ${folder_to_store_downloads} 30 | done 31 | 32 | 33 | # Having run this script, you should create a suitable config.json, *e.g.* by 34 | # looking at `https://huggingface.co/t5-small`. 35 | # Then you can run `python convert_t5_original_tf_checkpoint_to_pytorch.py --tf_checkpoint_path "${folder_to_store_downloads}" --config_file "config.json" --pytorch_dump_path "/path/to/store/pytorch/weights" 36 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/tapex/__init__.py: -------------------------------------------------------------------------------- 1 | # flake8: noqa 2 | # There's no way to ignore "F401 '...' imported but unused" warnings in this 3 | # module, but to preserve other warnings. So, don't check this module at all. 4 | 5 | # Copyright 2022 The HuggingFace Team. All rights reserved. 6 | # 7 | # Licensed under the Apache License, Version 2.0 (the "License"); 8 | # you may not use this file except in compliance with the License. 9 | # You may obtain a copy of the License at 10 | # 11 | # http://www.apache.org/licenses/LICENSE-2.0 12 | # 13 | # Unless required by applicable law or agreed to in writing, software 14 | # distributed under the License is distributed on an "AS IS" BASIS, 15 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | # See the License for the specific language governing permissions and 17 | # limitations under the License. 18 | from typing import TYPE_CHECKING 19 | 20 | # rely on isort to merge the imports 21 | from ...file_utils import _LazyModule 22 | 23 | 24 | _import_structure = {"tokenization_tapex": ["TapexTokenizer"]} 25 | 26 | 27 | if TYPE_CHECKING: 28 | from .tokenization_tapex import TapexTokenizer 29 | 30 | 31 | else: 32 | import sys 33 | 34 | sys.modules[__name__] = _LazyModule(__name__, globals()["__file__"], _import_structure) 35 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/trajectory_transformer/__init__.py: -------------------------------------------------------------------------------- 1 | # flake8: noqa 2 | # There's no way to ignore "F401 '...' imported but unused" warnings in this 3 | # module, but to preserve other warnings. So, don't check this module at all. 4 | 5 | # Copyright 2022 The HuggingFace Team. All rights reserved. 6 | # 7 | # Licensed under the Apache License, Version 2.0 (the "License"); 8 | # you may not use this file except in compliance with the License. 9 | # You may obtain a copy of the License at 10 | # 11 | # http://www.apache.org/licenses/LICENSE-2.0 12 | # 13 | # Unless required by applicable law or agreed to in writing, software 14 | # distributed under the License is distributed on an "AS IS" BASIS, 15 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | # See the License for the specific language governing permissions and 17 | # limitations under the License. 18 | from typing import TYPE_CHECKING 19 | 20 | # rely on isort to merge the imports 21 | from ...utils import OptionalDependencyNotAvailable, _LazyModule, is_torch_available 22 | 23 | 24 | _import_structure = { 25 | "configuration_trajectory_transformer": [ 26 | "TRAJECTORY_TRANSFORMER_PRETRAINED_CONFIG_ARCHIVE_MAP", 27 | "TrajectoryTransformerConfig", 28 | ], 29 | } 30 | 31 | try: 32 | if not is_torch_available(): 33 | raise OptionalDependencyNotAvailable() 34 | except OptionalDependencyNotAvailable: 35 | pass 36 | else: 37 | _import_structure["modeling_trajectory_transformer"] = [ 38 | "TRAJECTORY_TRANSFORMER_PRETRAINED_MODEL_ARCHIVE_LIST", 39 | "TrajectoryTransformerModel", 40 | "TrajectoryTransformerPreTrainedModel", 41 | "load_tf_weights_in_trajectory_transformer", 42 | ] 43 | 44 | 45 | if TYPE_CHECKING: 46 | from .configuration_trajectory_transformer import ( 47 | TRAJECTORY_TRANSFORMER_PRETRAINED_CONFIG_ARCHIVE_MAP, 48 | TrajectoryTransformerConfig, 49 | ) 50 | 51 | try: 52 | if not is_torch_available(): 53 | raise OptionalDependencyNotAvailable() 54 | except OptionalDependencyNotAvailable: 55 | pass 56 | else: 57 | from .modeling_trajectory_transformer import ( 58 | TRAJECTORY_TRANSFORMER_PRETRAINED_MODEL_ARCHIVE_LIST, 59 | TrajectoryTransformerModel, 60 | TrajectoryTransformerPreTrainedModel, 61 | load_tf_weights_in_trajectory_transformer, 62 | ) 63 | 64 | 65 | else: 66 | import sys 67 | 68 | sys.modules[__name__] = _LazyModule(__name__, globals()["__file__"], _import_structure, module_spec=__spec__) 69 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/trocr/__init__.py: -------------------------------------------------------------------------------- 1 | # flake8: noqa 2 | # There's no way to ignore "F401 '...' imported but unused" warnings in this 3 | # module, but to preserve other warnings. So, don't check this module at all. 4 | 5 | # Copyright 2021 The HuggingFace Team. All rights reserved. 6 | # 7 | # Licensed under the Apache License, Version 2.0 (the "License"); 8 | # you may not use this file except in compliance with the License. 9 | # You may obtain a copy of the License at 10 | # 11 | # http://www.apache.org/licenses/LICENSE-2.0 12 | # 13 | # Unless required by applicable law or agreed to in writing, software 14 | # distributed under the License is distributed on an "AS IS" BASIS, 15 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | # See the License for the specific language governing permissions and 17 | # limitations under the License. 18 | from typing import TYPE_CHECKING 19 | 20 | from ...utils import ( 21 | OptionalDependencyNotAvailable, 22 | _LazyModule, 23 | is_sentencepiece_available, 24 | is_speech_available, 25 | is_torch_available, 26 | ) 27 | 28 | 29 | _import_structure = { 30 | "configuration_trocr": ["TROCR_PRETRAINED_CONFIG_ARCHIVE_MAP", "TrOCRConfig"], 31 | "processing_trocr": ["TrOCRProcessor"], 32 | } 33 | 34 | 35 | try: 36 | if not is_torch_available(): 37 | raise OptionalDependencyNotAvailable() 38 | except OptionalDependencyNotAvailable: 39 | pass 40 | else: 41 | _import_structure["modeling_trocr"] = [ 42 | "TROCR_PRETRAINED_MODEL_ARCHIVE_LIST", 43 | "TrOCRForCausalLM", 44 | "TrOCRPreTrainedModel", 45 | ] 46 | 47 | 48 | if TYPE_CHECKING: 49 | from .configuration_trocr import TROCR_PRETRAINED_CONFIG_ARCHIVE_MAP, TrOCRConfig 50 | from .processing_trocr import TrOCRProcessor 51 | 52 | try: 53 | if not is_torch_available(): 54 | raise OptionalDependencyNotAvailable() 55 | except OptionalDependencyNotAvailable: 56 | pass 57 | else: 58 | from .modeling_trocr import TROCR_PRETRAINED_MODEL_ARCHIVE_LIST, TrOCRForCausalLM, TrOCRPreTrainedModel 59 | 60 | else: 61 | import sys 62 | 63 | sys.modules[__name__] = _LazyModule(__name__, globals()["__file__"], _import_structure, module_spec=__spec__) 64 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/unispeech/__init__.py: -------------------------------------------------------------------------------- 1 | # flake8: noqa 2 | # There's no way to ignore "F401 '...' imported but unused" warnings in this 3 | # module, but to preserve other warnings. So, don't check this module at all. 4 | 5 | # Copyright 2021 The HuggingFace Team. All rights reserved. 6 | # 7 | # Licensed under the Apache License, Version 2.0 (the "License"); 8 | # you may not use this file except in compliance with the License. 9 | # You may obtain a copy of the License at 10 | # 11 | # http://www.apache.org/licenses/LICENSE-2.0 12 | # 13 | # Unless required by applicable law or agreed to in writing, software 14 | # distributed under the License is distributed on an "AS IS" BASIS, 15 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | # See the License for the specific language governing permissions and 17 | # limitations under the License. 18 | from typing import TYPE_CHECKING 19 | 20 | from ...utils import ( 21 | OptionalDependencyNotAvailable, 22 | _LazyModule, 23 | is_flax_available, 24 | is_tf_available, 25 | is_torch_available, 26 | ) 27 | 28 | 29 | _import_structure = {"configuration_unispeech": ["UNISPEECH_PRETRAINED_CONFIG_ARCHIVE_MAP", "UniSpeechConfig"]} 30 | 31 | try: 32 | if not is_torch_available(): 33 | raise OptionalDependencyNotAvailable() 34 | except OptionalDependencyNotAvailable: 35 | pass 36 | else: 37 | _import_structure["modeling_unispeech"] = [ 38 | "UNISPEECH_PRETRAINED_MODEL_ARCHIVE_LIST", 39 | "UniSpeechForCTC", 40 | "UniSpeechForPreTraining", 41 | "UniSpeechForSequenceClassification", 42 | "UniSpeechModel", 43 | "UniSpeechPreTrainedModel", 44 | ] 45 | 46 | if TYPE_CHECKING: 47 | from .configuration_unispeech import UNISPEECH_PRETRAINED_CONFIG_ARCHIVE_MAP, UniSpeechConfig 48 | 49 | try: 50 | if not is_torch_available(): 51 | raise OptionalDependencyNotAvailable() 52 | except OptionalDependencyNotAvailable: 53 | pass 54 | else: 55 | from .modeling_unispeech import ( 56 | UNISPEECH_PRETRAINED_MODEL_ARCHIVE_LIST, 57 | UniSpeechForCTC, 58 | UniSpeechForPreTraining, 59 | UniSpeechForSequenceClassification, 60 | UniSpeechModel, 61 | UniSpeechPreTrainedModel, 62 | ) 63 | 64 | else: 65 | import sys 66 | 67 | sys.modules[__name__] = _LazyModule(__name__, globals()["__file__"], _import_structure, module_spec=__spec__) 68 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/van/__init__.py: -------------------------------------------------------------------------------- 1 | # flake8: noqa 2 | # There's no way to ignore "F401 '...' imported but unused" warnings in this 3 | # module, but to preserve other warnings. So, don't check this module at all. 4 | 5 | # Copyright 2022 The HuggingFace Team. All rights reserved. 6 | # 7 | # Licensed under the Apache License, Version 2.0 (the "License"); 8 | # you may not use this file except in compliance with the License. 9 | # You may obtain a copy of the License at 10 | # 11 | # http://www.apache.org/licenses/LICENSE-2.0 12 | # 13 | # Unless required by applicable law or agreed to in writing, software 14 | # distributed under the License is distributed on an "AS IS" BASIS, 15 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | # See the License for the specific language governing permissions and 17 | # limitations under the License. 18 | from typing import TYPE_CHECKING 19 | 20 | # rely on isort to merge the imports 21 | from ...utils import OptionalDependencyNotAvailable, _LazyModule, is_torch_available, is_vision_available 22 | 23 | 24 | _import_structure = {"configuration_van": ["VAN_PRETRAINED_CONFIG_ARCHIVE_MAP", "VanConfig"]} 25 | 26 | 27 | try: 28 | if not is_torch_available(): 29 | raise OptionalDependencyNotAvailable() 30 | except OptionalDependencyNotAvailable: 31 | pass 32 | else: 33 | _import_structure["modeling_van"] = [ 34 | "VAN_PRETRAINED_MODEL_ARCHIVE_LIST", 35 | "VanForImageClassification", 36 | "VanModel", 37 | "VanPreTrainedModel", 38 | ] 39 | 40 | if TYPE_CHECKING: 41 | from .configuration_van import VAN_PRETRAINED_CONFIG_ARCHIVE_MAP, VanConfig 42 | 43 | try: 44 | if not is_torch_available(): 45 | raise OptionalDependencyNotAvailable() 46 | except OptionalDependencyNotAvailable: 47 | pass 48 | else: 49 | from .modeling_van import ( 50 | VAN_PRETRAINED_MODEL_ARCHIVE_LIST, 51 | VanForImageClassification, 52 | VanModel, 53 | VanPreTrainedModel, 54 | ) 55 | 56 | else: 57 | import sys 58 | 59 | sys.modules[__name__] = _LazyModule(__name__, globals()["__file__"], _import_structure) 60 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/vision_text_dual_encoder/__init__.py: -------------------------------------------------------------------------------- 1 | # flake8: noqa 2 | # There's no way to ignore "F401 '...' imported but unused" warnings in this 3 | # module, but to preserve other warnings. So, don't check this module at all. 4 | 5 | # Copyright 2021 The HuggingFace Team. All rights reserved. 6 | # 7 | # Licensed under the Apache License, Version 2.0 (the "License"); 8 | # you may not use this file except in compliance with the License. 9 | # You may obtain a copy of the License at 10 | # 11 | # http://www.apache.org/licenses/LICENSE-2.0 12 | # 13 | # Unless required by applicable law or agreed to in writing, software 14 | # distributed under the License is distributed on an "AS IS" BASIS, 15 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | # See the License for the specific language governing permissions and 17 | # limitations under the License. 18 | from typing import TYPE_CHECKING 19 | 20 | # rely on isort to merge the imports 21 | from ...utils import OptionalDependencyNotAvailable, _LazyModule, is_flax_available, is_torch_available 22 | 23 | 24 | _import_structure = { 25 | "configuration_vision_text_dual_encoder": ["VisionTextDualEncoderConfig"], 26 | "processing_vision_text_dual_encoder": ["VisionTextDualEncoderProcessor"], 27 | } 28 | 29 | 30 | try: 31 | if not is_torch_available(): 32 | raise OptionalDependencyNotAvailable() 33 | except OptionalDependencyNotAvailable: 34 | pass 35 | else: 36 | _import_structure["modeling_vision_text_dual_encoder"] = ["VisionTextDualEncoderModel"] 37 | 38 | 39 | try: 40 | if not is_flax_available(): 41 | raise OptionalDependencyNotAvailable() 42 | except OptionalDependencyNotAvailable: 43 | pass 44 | else: 45 | _import_structure["modeling_flax_vision_text_dual_encoder"] = ["FlaxVisionTextDualEncoderModel"] 46 | 47 | 48 | if TYPE_CHECKING: 49 | from .configuration_vision_text_dual_encoder import VisionTextDualEncoderConfig 50 | from .processing_visiotn_text_dual_encoder import VisionTextDualEncoderProcessor 51 | 52 | try: 53 | if not is_torch_available(): 54 | raise OptionalDependencyNotAvailable() 55 | except OptionalDependencyNotAvailable: 56 | pass 57 | else: 58 | from .modeling_vision_text_dual_encoder import VisionTextDualEncoderModel 59 | 60 | try: 61 | if not is_flax_available(): 62 | raise OptionalDependencyNotAvailable() 63 | except OptionalDependencyNotAvailable: 64 | pass 65 | else: 66 | from .modeling_vision_text_dual_encoder import FlaxVisionTextDualEncoderModel 67 | 68 | 69 | else: 70 | import sys 71 | 72 | sys.modules[__name__] = _LazyModule(__name__, globals()["__file__"], _import_structure) 73 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/visual_bert/__init__.py: -------------------------------------------------------------------------------- 1 | # flake8: noqa 2 | # There's no way to ignore "F401 '...' imported but unused" warnings in this 3 | # module, but to preserve other warnings. So, don't check this module at all. 4 | 5 | # Copyright 2021 The HuggingFace Team. All rights reserved. 6 | # 7 | # Licensed under the Apache License, Version 2.0 (the "License"); 8 | # you may not use this file except in compliance with the License. 9 | # You may obtain a copy of the License at 10 | # 11 | # http://www.apache.org/licenses/LICENSE-2.0 12 | # 13 | # Unless required by applicable law or agreed to in writing, software 14 | # distributed under the License is distributed on an "AS IS" BASIS, 15 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | # See the License for the specific language governing permissions and 17 | # limitations under the License. 18 | from typing import TYPE_CHECKING 19 | 20 | from ...utils import OptionalDependencyNotAvailable, _LazyModule, is_torch_available 21 | 22 | 23 | _import_structure = {"configuration_visual_bert": ["VISUAL_BERT_PRETRAINED_CONFIG_ARCHIVE_MAP", "VisualBertConfig"]} 24 | 25 | try: 26 | if not is_torch_available(): 27 | raise OptionalDependencyNotAvailable() 28 | except OptionalDependencyNotAvailable: 29 | pass 30 | else: 31 | _import_structure["modeling_visual_bert"] = [ 32 | "VISUAL_BERT_PRETRAINED_MODEL_ARCHIVE_LIST", 33 | "VisualBertForMultipleChoice", 34 | "VisualBertForPreTraining", 35 | "VisualBertForQuestionAnswering", 36 | "VisualBertForRegionToPhraseAlignment", 37 | "VisualBertForVisualReasoning", 38 | "VisualBertLayer", 39 | "VisualBertModel", 40 | "VisualBertPreTrainedModel", 41 | ] 42 | 43 | 44 | if TYPE_CHECKING: 45 | from .configuration_visual_bert import VISUAL_BERT_PRETRAINED_CONFIG_ARCHIVE_MAP, VisualBertConfig 46 | 47 | try: 48 | if not is_torch_available(): 49 | raise OptionalDependencyNotAvailable() 50 | except OptionalDependencyNotAvailable: 51 | pass 52 | else: 53 | from .modeling_visual_bert import ( 54 | VISUAL_BERT_PRETRAINED_MODEL_ARCHIVE_LIST, 55 | VisualBertForMultipleChoice, 56 | VisualBertForPreTraining, 57 | VisualBertForQuestionAnswering, 58 | VisualBertForRegionToPhraseAlignment, 59 | VisualBertForVisualReasoning, 60 | VisualBertLayer, 61 | VisualBertModel, 62 | VisualBertPreTrainedModel, 63 | ) 64 | 65 | 66 | else: 67 | import sys 68 | 69 | sys.modules[__name__] = _LazyModule(__name__, globals()["__file__"], _import_structure, module_spec=__spec__) 70 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/wav2vec2_phoneme/__init__.py: -------------------------------------------------------------------------------- 1 | # flake8: noqa 2 | # There's no way to ignore "F401 '...' imported but unused" warnings in this 3 | # module, but to preserve other warnings. So, don't check this module at all. 4 | 5 | # Copyright 2021 The HuggingFace Team. All rights reserved. 6 | # 7 | # Licensed under the Apache License, Version 2.0 (the "License"); 8 | # you may not use this file except in compliance with the License. 9 | # You may obtain a copy of the License at 10 | # 11 | # http://www.apache.org/licenses/LICENSE-2.0 12 | # 13 | # Unless required by applicable law or agreed to in writing, software 14 | # distributed under the License is distributed on an "AS IS" BASIS, 15 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | # See the License for the specific language governing permissions and 17 | # limitations under the License. 18 | from typing import TYPE_CHECKING 19 | 20 | from ...utils import _LazyModule 21 | 22 | 23 | _import_structure = {"tokenization_wav2vec2_phoneme": ["Wav2Vec2PhonemeCTCTokenizer"]} 24 | 25 | 26 | if TYPE_CHECKING: 27 | from .tokenization_wav2vec2_phoneme import Wav2Vec2PhonemeCTCTokenizer 28 | else: 29 | import sys 30 | 31 | sys.modules[__name__] = _LazyModule(__name__, globals()["__file__"], _import_structure, module_spec=__spec__) 32 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/wav2vec2_with_lm/__init__.py: -------------------------------------------------------------------------------- 1 | # flake8: noqa 2 | # There's no way to ignore "F401 '...' imported but unused" warnings in this 3 | # module, but to preserve other warnings. So, don't check this module at all. 4 | 5 | # Copyright 2021 The HuggingFace Team. All rights reserved. 6 | # 7 | # Licensed under the Apache License, Version 2.0 (the "License"); 8 | # you may not use this file except in compliance with the License. 9 | # You may obtain a copy of the License at 10 | # 11 | # http://www.apache.org/licenses/LICENSE-2.0 12 | # 13 | # Unless required by applicable law or agreed to in writing, software 14 | # distributed under the License is distributed on an "AS IS" BASIS, 15 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | # See the License for the specific language governing permissions and 17 | # limitations under the License. 18 | from typing import TYPE_CHECKING 19 | 20 | from ...utils import _LazyModule 21 | 22 | 23 | _import_structure = {"processing_wav2vec2_with_lm": ["Wav2Vec2ProcessorWithLM"]} 24 | 25 | 26 | if TYPE_CHECKING: 27 | from .processing_wav2vec2_with_lm import Wav2Vec2ProcessorWithLM 28 | else: 29 | import sys 30 | 31 | sys.modules[__name__] = _LazyModule(__name__, globals()["__file__"], _import_structure, module_spec=__spec__) 32 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/wavlm/__init__.py: -------------------------------------------------------------------------------- 1 | # flake8: noqa 2 | # There's no way to ignore "F401 '...' imported but unused" warnings in this 3 | # module, but to preserve other warnings. So, don't check this module at all. 4 | 5 | # Copyright 2021 The HuggingFace Team. All rights reserved. 6 | # 7 | # Licensed under the Apache License, Version 2.0 (the "License"); 8 | # you may not use this file except in compliance with the License. 9 | # You may obtain a copy of the License at 10 | # 11 | # http://www.apache.org/licenses/LICENSE-2.0 12 | # 13 | # Unless required by applicable law or agreed to in writing, software 14 | # distributed under the License is distributed on an "AS IS" BASIS, 15 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | # See the License for the specific language governing permissions and 17 | # limitations under the License. 18 | from typing import TYPE_CHECKING 19 | 20 | from ...utils import OptionalDependencyNotAvailable, _LazyModule, is_torch_available 21 | 22 | 23 | _import_structure = {"configuration_wavlm": ["WAVLM_PRETRAINED_CONFIG_ARCHIVE_MAP", "WavLMConfig"]} 24 | 25 | try: 26 | if not is_torch_available(): 27 | raise OptionalDependencyNotAvailable() 28 | except OptionalDependencyNotAvailable: 29 | pass 30 | else: 31 | _import_structure["modeling_wavlm"] = [ 32 | "WAVLM_PRETRAINED_MODEL_ARCHIVE_LIST", 33 | "WavLMForAudioFrameClassification", 34 | "WavLMForCTC", 35 | "WavLMForSequenceClassification", 36 | "WavLMForXVector", 37 | "WavLMModel", 38 | "WavLMPreTrainedModel", 39 | ] 40 | 41 | if TYPE_CHECKING: 42 | from .configuration_wavlm import WAVLM_PRETRAINED_CONFIG_ARCHIVE_MAP, WavLMConfig 43 | 44 | try: 45 | if not is_torch_available(): 46 | raise OptionalDependencyNotAvailable() 47 | except OptionalDependencyNotAvailable: 48 | pass 49 | else: 50 | from .modeling_wavlm import ( 51 | WAVLM_PRETRAINED_MODEL_ARCHIVE_LIST, 52 | WavLMForAudioFrameClassification, 53 | WavLMForCTC, 54 | WavLMForSequenceClassification, 55 | WavLMForXVector, 56 | WavLMModel, 57 | WavLMPreTrainedModel, 58 | ) 59 | 60 | else: 61 | import sys 62 | 63 | sys.modules[__name__] = _LazyModule(__name__, globals()["__file__"], _import_structure, module_spec=__spec__) 64 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/xglm/convert_xglm_original_ckpt_to_trfms.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | from argparse import Namespace 3 | 4 | import torch 5 | from torch import nn 6 | 7 | from transformers import XGLMConfig, XGLMForCausalLM 8 | 9 | 10 | def remove_ignore_keys_(state_dict): 11 | ignore_keys = [ 12 | "decoder.version", 13 | "decoder.output_projection.weight", 14 | "_float_tensor", 15 | "decoder.embed_positions._float_tensor", 16 | ] 17 | for k in ignore_keys: 18 | state_dict.pop(k, None) 19 | 20 | 21 | def make_linear_from_emb(emb): 22 | vocab_size, emb_size = emb.weight.shape 23 | lin_layer = nn.Linear(vocab_size, emb_size, bias=False) 24 | lin_layer.weight.data = emb.weight.data 25 | return lin_layer 26 | 27 | 28 | def convert_fairseq_xglm_checkpoint_from_disk(checkpoint_path): 29 | checkpoint = torch.load(checkpoint_path, map_location="cpu") 30 | args = Namespace(**checkpoint["cfg"]["model"]) 31 | state_dict = checkpoint["model"] 32 | remove_ignore_keys_(state_dict) 33 | vocab_size = state_dict["decoder.embed_tokens.weight"].shape[0] 34 | 35 | state_dict = {key.replace("decoder", "model"): val for key, val in state_dict.items()} 36 | 37 | config = XGLMConfig( 38 | vocab_size=vocab_size, 39 | max_position_embeddings=args.max_target_positions, 40 | num_layers=args.decoder_layers, 41 | attention_heads=args.decoder_attention_heads, 42 | ffn_dim=args.decoder_ffn_embed_dim, 43 | d_model=args.decoder_embed_dim, 44 | layerdrop=args.decoder_layerdrop, 45 | dropout=args.dropout, 46 | attention_dropout=args.attention_dropout, 47 | activation_dropout=args.activation_dropout, 48 | activation_function="gelu", 49 | scale_embedding=not args.no_scale_embedding, 50 | tie_word_embeddings=args.share_decoder_input_output_embed, 51 | ) 52 | 53 | model = XGLMForCausalLM(config) 54 | missing = model.load_state_dict(state_dict, strict=False) 55 | print(missing) 56 | model.lm_head = make_linear_from_emb(model.model.embed_tokens) 57 | 58 | return model 59 | 60 | 61 | if __name__ == "__main__": 62 | parser = argparse.ArgumentParser() 63 | # Required parameters 64 | parser.add_argument("fairseq_path", type=str, help="path to a model.pt on local filesystem.") 65 | parser.add_argument("pytorch_dump_folder_path", default=None, type=str, help="Path to the output PyTorch model.") 66 | args = parser.parse_args() 67 | model = convert_fairseq_xglm_checkpoint_from_disk(args.fairseq_path) 68 | model.save_pretrained(args.pytorch_dump_folder_path) 69 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/xlm_prophetnet/configuration_xlm_prophetnet.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | # Copyright 2020 The Microsoft Authors and The HuggingFace Inc. team. 3 | # 4 | # Licensed under the Apache License, Version 2.0 (the "License"); 5 | # you may not use this file except in compliance with the License. 6 | # You may obtain a copy of the License at 7 | # 8 | # http://www.apache.org/licenses/LICENSE-2.0 9 | # 10 | # Unless required by applicable law or agreed to in writing, software 11 | # distributed under the License is distributed on an "AS IS" BASIS, 12 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | # See the License for the specific language governing permissions and 14 | # limitations under the License. 15 | """ XLM-ProphetNet model configuration""" 16 | 17 | 18 | from ...utils import logging 19 | from ..prophetnet.configuration_prophetnet import ProphetNetConfig 20 | 21 | 22 | logger = logging.get_logger(__name__) 23 | 24 | XLM_PROPHETNET_PRETRAINED_CONFIG_ARCHIVE_MAP = { 25 | "microsoft/xprophetnet-large-wiki100-cased": ( 26 | "https://huggingface.co/microsoft/xprophetnet-large-wiki100-cased/resolve/main/config.json" 27 | ), 28 | } 29 | 30 | 31 | class XLMProphetNetConfig(ProphetNetConfig): 32 | """ 33 | This class overrides [`ProphetNetConfig`]. Please check the superclass for the appropriate documentation alongside 34 | usage examples. Instantiating a configuration with the defaults will yield a similar configuration to that of the 35 | XLMProphetNet 36 | [microsoft/xprophetnet-large-wiki100-cased](https://huggingface.co/microsoft/xprophetnet-large-wiki100-cased) 37 | architecture. 38 | """ 39 | 40 | model_type = "xlm-prophetnet" 41 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/xlm_roberta/image_classification/__init__.py: -------------------------------------------------------------------------------- 1 | from . import logger 2 | from . import utils 3 | from . import mixup 4 | from . import smoothing 5 | from . import quantize 6 | from . import preconditioner 7 | from . import activation_quantizer_1 8 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/xlm_roberta/image_classification/mixup.py: -------------------------------------------------------------------------------- 1 | import torch 2 | import torch.nn as nn 3 | import numpy as np 4 | 5 | 6 | def mixup(alpha, num_classes, data, target): 7 | with torch.no_grad(): 8 | bs = data.size(0) 9 | c = np.random.beta(alpha, alpha) 10 | 11 | perm = torch.randperm(bs).cuda() 12 | 13 | md = c * data + (1-c) * data[perm, :] 14 | mt = c * target + (1-c) * target[perm, :] 15 | return md, mt 16 | 17 | 18 | class MixUpWrapper(object): 19 | def __init__(self, alpha, num_classes, dataloader): 20 | self.alpha = alpha 21 | self.dataloader = dataloader 22 | self.num_classes = num_classes 23 | 24 | def mixup_loader(self, loader): 25 | for input, target in loader: 26 | i, t = mixup(self.alpha, self.num_classes, input, target) 27 | yield i, t 28 | 29 | def __iter__(self): 30 | return self.mixup_loader(self.dataloader) 31 | 32 | 33 | class NLLMultiLabelSmooth(nn.Module): 34 | def __init__(self, smoothing = 0.0): 35 | super(NLLMultiLabelSmooth, self).__init__() 36 | self.confidence = 1.0 - smoothing 37 | self.smoothing = smoothing 38 | 39 | def forward(self, x, target): 40 | if self.training: 41 | x = x.float() 42 | target = target.float() 43 | logprobs = torch.nn.functional.log_softmax(x, dim = -1) 44 | 45 | nll_loss = -logprobs * target 46 | nll_loss = nll_loss.sum(-1) 47 | 48 | smooth_loss = -logprobs.mean(dim=-1) 49 | 50 | loss = self.confidence * nll_loss + self.smoothing * smooth_loss 51 | 52 | return loss.mean() 53 | else: 54 | return torch.nn.functional.cross_entropy(x, target) 55 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/xlm_roberta/image_classification/plot_variance.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import matplotlib 3 | matplotlib.use('Agg') 4 | from matplotlib import pyplot as plt 5 | from matplotlib.colors import LogNorm 6 | import pickle 7 | import seaborn as sns 8 | 9 | weight_names = pickle.load(open('layer_names.pkl', 'rb')) 10 | 11 | grads = np.load('error_profile_5.npy') 12 | grads = np.maximum(grads, 0) 13 | quantizers = grads.sum(1) 14 | variances = grads.sum(0) 15 | 16 | # grads = np.minimum(grads, 1) 17 | # grads *= 1000 18 | for i in range(grads.shape[0]): 19 | for j in range(grads.shape[1]): 20 | if j > i: 21 | grads[i, j] = 0 22 | 23 | fig, ax = plt.subplots(figsize=(20, 20)) 24 | im = ax.imshow(grads, cmap='Blues', norm=LogNorm(vmin=0.01, vmax=10.0)) 25 | ax.set_xticks(np.arange(len(weight_names))) 26 | ax.set_yticks(np.arange(len(weight_names)+1)) 27 | ax.set_xticklabels(weight_names) 28 | weight_names.append('sample') 29 | ax.set_yticklabels(weight_names) 30 | 31 | ax.tick_params(top=True, bottom=False, 32 | labeltop=True, labelbottom=False) 33 | plt.setp(ax.get_xticklabels(), rotation=-30, ha="right", 34 | rotation_mode="anchor") 35 | cbar = ax.figure.colorbar(im, ax=ax) 36 | 37 | for i in range(grads.shape[0]): 38 | for j in range(grads.shape[1]): 39 | text = ax.text(j, i, int(grads[i, j]*10), 40 | ha="center", va="center") 41 | 42 | 43 | fig.savefig('variance_profile.pdf') 44 | 45 | fig, ax = plt.subplots(figsize=(20, 20)) 46 | sns.barplot(x=np.arange(quantizers.shape[0]), y=quantizers, ax=ax) 47 | ax.set_xticks(np.arange(len(weight_names)) + 0.5) 48 | ax.set_xticklabels(weight_names) 49 | plt.setp(ax.get_xticklabels(), rotation=-30, ha="right", 50 | rotation_mode="anchor") 51 | ax.set_xlabel('quantizer') 52 | ax.set_ylabel('variance') 53 | fig.savefig('quantizers.pdf') 54 | 55 | fig, ax = plt.subplots(figsize=(20, 20)) 56 | sns.barplot(x=np.arange(variances.shape[0]), y=variances, ax=ax) 57 | weight_names.pop(-1) 58 | ax.set_xticks(np.arange(len(weight_names)) + 0.5) 59 | ax.set_xticklabels(weight_names) 60 | plt.setp(ax.get_xticklabels(), rotation=-30, ha="right", 61 | rotation_mode="anchor") 62 | ax.set_xlabel('parameter') 63 | ax.set_ylabel('variance') 64 | fig.savefig('parameter.pdf') 65 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/xlm_roberta/image_classification/smoothing.py: -------------------------------------------------------------------------------- 1 | import torch 2 | import torch.nn as nn 3 | 4 | class LabelSmoothing(nn.Module): 5 | """ 6 | NLL loss with label smoothing. 7 | """ 8 | def __init__(self, smoothing=0.0): 9 | """ 10 | Constructor for the LabelSmoothing module. 11 | 12 | :param smoothing: label smoothing factor 13 | """ 14 | super(LabelSmoothing, self).__init__() 15 | self.confidence = 1.0 - smoothing 16 | self.smoothing = smoothing 17 | 18 | def forward(self, x, target): 19 | logprobs = torch.nn.functional.log_softmax(x, dim=-1) 20 | 21 | nll_loss = -logprobs.gather(dim=-1, index=target.unsqueeze(1)) 22 | nll_loss = nll_loss.squeeze(1) 23 | smooth_loss = -logprobs.mean(dim=-1) 24 | loss = self.confidence * nll_loss + self.smoothing * smooth_loss 25 | return loss.mean() 26 | 27 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/xlm_roberta/image_classification/test.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "import torch" 10 | ] 11 | }, 12 | { 13 | "cell_type": "code", 14 | "execution_count": 1, 15 | "metadata": {}, 16 | "outputs": [ 17 | { 18 | "name": "stdout", 19 | "output_type": "stream", 20 | "text": [ 21 | "Original Tensor:\n", 22 | "tensor([ 1.7234, -0.6239, 0.3089, -1.1292, 1.0927])\n", 23 | "\n", 24 | "Indices where norm_x < 1:\n", 25 | "tensor([1, 2, 3])\n" 26 | ] 27 | } 28 | ], 29 | "source": [ 30 | "import torch\n", 31 | "\n", 32 | "# create a 1D tensor\n", 33 | "norm_x = torch.randn(5)\n", 34 | "\n", 35 | "# find indices where norm_x < 1\n", 36 | "indices = torch.where(norm_x < 1)[0]\n", 37 | "\n", 38 | "print(\"Original Tensor:\")\n", 39 | "print(norm_x)\n", 40 | "\n", 41 | "print(\"\\nIndices where norm_x < 1:\")\n", 42 | "print(indices)" 43 | ] 44 | } 45 | ], 46 | "metadata": { 47 | "kernelspec": { 48 | "display_name": "base", 49 | "language": "python", 50 | "name": "python3" 51 | }, 52 | "language_info": { 53 | "codemirror_mode": { 54 | "name": "ipython", 55 | "version": 3 56 | }, 57 | "file_extension": ".py", 58 | "mimetype": "text/x-python", 59 | "name": "python", 60 | "nbconvert_exporter": "python", 61 | "pygments_lexer": "ipython3", 62 | "version": "3.9.12" 63 | }, 64 | "orig_nbformat": 4, 65 | "vscode": { 66 | "interpreter": { 67 | "hash": "88279d2366fe020547cde40dd65aa0e3aa662a6ec1f3ca12d88834876c85e1a6" 68 | } 69 | } 70 | }, 71 | "nbformat": 4, 72 | "nbformat_minor": 2 73 | } 74 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/yoso/__init__.py: -------------------------------------------------------------------------------- 1 | # flake8: noqa 2 | # There's no way to ignore "F401 '...' imported but unused" warnings in this 3 | # module, but to preserve other warnings. So, don't check this module at all. 4 | 5 | # Copyright 2022 The HuggingFace Team. All rights reserved. 6 | # 7 | # Licensed under the Apache License, Version 2.0 (the "License"); 8 | # you may not use this file except in compliance with the License. 9 | # You may obtain a copy of the License at 10 | # 11 | # http://www.apache.org/licenses/LICENSE-2.0 12 | # 13 | # Unless required by applicable law or agreed to in writing, software 14 | # distributed under the License is distributed on an "AS IS" BASIS, 15 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | # See the License for the specific language governing permissions and 17 | # limitations under the License. 18 | from typing import TYPE_CHECKING 19 | 20 | # rely on isort to merge the imports 21 | from ...utils import OptionalDependencyNotAvailable, _LazyModule, is_tokenizers_available, is_torch_available 22 | 23 | 24 | _import_structure = {"configuration_yoso": ["YOSO_PRETRAINED_CONFIG_ARCHIVE_MAP", "YosoConfig"]} 25 | 26 | try: 27 | if not is_torch_available(): 28 | raise OptionalDependencyNotAvailable() 29 | except OptionalDependencyNotAvailable: 30 | pass 31 | else: 32 | _import_structure["modeling_yoso"] = [ 33 | "YOSO_PRETRAINED_MODEL_ARCHIVE_LIST", 34 | "YosoForMaskedLM", 35 | "YosoForMultipleChoice", 36 | "YosoForQuestionAnswering", 37 | "YosoForSequenceClassification", 38 | "YosoForTokenClassification", 39 | "YosoLayer", 40 | "YosoModel", 41 | "YosoPreTrainedModel", 42 | ] 43 | 44 | 45 | if TYPE_CHECKING: 46 | from .configuration_yoso import YOSO_PRETRAINED_CONFIG_ARCHIVE_MAP, YosoConfig 47 | 48 | try: 49 | if not is_torch_available(): 50 | raise OptionalDependencyNotAvailable() 51 | except OptionalDependencyNotAvailable: 52 | pass 53 | else: 54 | from .modeling_yoso import ( 55 | YOSO_PRETRAINED_MODEL_ARCHIVE_LIST, 56 | YosoForMaskedLM, 57 | YosoForMultipleChoice, 58 | YosoForQuestionAnswering, 59 | YosoForSequenceClassification, 60 | YosoForTokenClassification, 61 | YosoLayer, 62 | YosoModel, 63 | YosoPreTrainedModel, 64 | ) 65 | 66 | 67 | else: 68 | import sys 69 | 70 | sys.modules[__name__] = _LazyModule(__name__, globals()["__file__"], _import_structure) 71 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/yoso/common.h: -------------------------------------------------------------------------------- 1 | 2 | #define min(a, b) ((a)<(b)?(a):(b)) 3 | #define max(a, b) ((a)>(b)?(a):(b)) 4 | #define ceil_divide(a, b) ((a)/(b)+((a)%(b)!=0)) 5 | #define select(cond, a, b) ((cond)?(a):(b)) 6 | #define PI 3.141592 7 | #define EPSILON 1e-8 8 | #define MAX_VAL 1e12 9 | #define MIN_VAL -1e12 10 | #define EMPTY_VALUE -1 11 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/yoso/common_cuda.h: -------------------------------------------------------------------------------- 1 | 2 | #define MAX_THREADS_PER_BLOCK 1024 3 | #define OPTIMAL_THREADS_PER_BLOCK 256 4 | #define WARP_SIZE 32 5 | #define MAX_NUM_BLOCK_X 2147483647 6 | #define MAX_NUM_BLOCK_Y 65535 7 | #define MAX_NUM_BLOCK_Z 65535 8 | #define MAX_SHARED_MEM_PER_BLOCK 48000 9 | #define FULL_MASK 0xffffffff 10 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/yoso/common_cuda_device.h: -------------------------------------------------------------------------------- 1 | 2 | #include "common.h" 3 | 4 | template 5 | __device__ int set_insert(T *set, int set_size, T value) { 6 | int slot = value % set_size; 7 | int start_slot = slot; 8 | while (true) { 9 | T prev = atomicCAS(&set[slot], EMPTY_VALUE, value); 10 | if (prev == EMPTY_VALUE || prev == value) { 11 | return slot; 12 | } 13 | slot = (slot + 1) % set_size; 14 | if (slot == start_slot) { 15 | return -1; 16 | } 17 | } 18 | return -1; 19 | } 20 | 21 | template 22 | __device__ int set_lookup(T *set, int set_size, T value) { 23 | int slot = value % set_size; 24 | int start_slot = slot; 25 | while (true) { 26 | if (set[slot] == value) { 27 | return slot; 28 | } 29 | slot = (slot + 1) % set_size; 30 | if (slot == start_slot) { 31 | return -1; 32 | } 33 | } 34 | return -1; 35 | } 36 | 37 | template 38 | __device__ void init_buffer(T init_value, T *buffer, int buffer_size, int num_threads, int thread_id) { 39 | __syncthreads(); 40 | for (int i = 0; i < buffer_size; i = i + num_threads) { 41 | int offset_idx = i + thread_id; 42 | if (offset_idx < buffer_size) { 43 | buffer[offset_idx] = init_value; 44 | } 45 | } 46 | __syncthreads(); 47 | } 48 | 49 | template 50 | __device__ void copy_data(T *src_pt, T *dist_pt, int data_length, int num_threads, int thread_id) { 51 | __syncthreads(); 52 | for (int i = 0; i < data_length; i = i + num_threads) { 53 | int offset_idx = i + thread_id; 54 | if (offset_idx < data_length) { 55 | dist_pt[offset_idx] = src_pt[offset_idx]; 56 | } 57 | } 58 | __syncthreads(); 59 | } 60 | 61 | template 62 | __device__ void init_buffer_nonblocking(T init_value, T *buffer, int buffer_size, int num_threads, int thread_id) { 63 | for (int i = 0; i < buffer_size; i = i + num_threads) { 64 | int offset_idx = i + thread_id; 65 | if (offset_idx < buffer_size) { 66 | buffer[offset_idx] = init_value; 67 | } 68 | } 69 | } 70 | 71 | template 72 | __device__ void copy_data_nonblocking(T *src_pt, T *dist_pt, int data_length, int num_threads, int thread_id) { 73 | for (int i = 0; i < data_length; i = i + num_threads) { 74 | int offset_idx = i + thread_id; 75 | if (offset_idx < data_length) { 76 | dist_pt[offset_idx] = src_pt[offset_idx]; 77 | } 78 | } 79 | } 80 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/models/yoso/fast_lsh_cumulation.h: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | #include 4 | 5 | std::vector fast_hash_ver1_kernel( 6 | at::Tensor query_mask, 7 | at::Tensor query_vector, 8 | at::Tensor key_mask, 9 | at::Tensor key_vector, 10 | int num_hash_f, 11 | int hash_code_len, 12 | bool use_cuda 13 | ); 14 | 15 | at::Tensor lsh_cumulation_ver1_kernel( 16 | at::Tensor query_mask, 17 | at::Tensor query_hash_code, 18 | at::Tensor key_mask, 19 | at::Tensor key_hash_code, 20 | at::Tensor value, 21 | int hashtable_capacity, 22 | bool use_cuda 23 | ); 24 | 25 | at::Tensor lsh_weighted_cumulation_ver1_kernel( 26 | at::Tensor query_mask, 27 | at::Tensor query_hash_code, 28 | at::Tensor query_weight, 29 | at::Tensor key_mask, 30 | at::Tensor key_hash_code, 31 | at::Tensor key_weight, 32 | at::Tensor value, 33 | int hashtable_capacity, 34 | bool use_cuda 35 | ); 36 | 37 | at::Tensor lsh_weighted_cumulation_ver2_kernel( 38 | at::Tensor query_mask, 39 | at::Tensor query_hash_code, 40 | at::Tensor query_weight, 41 | at::Tensor key_mask, 42 | at::Tensor key_hash_code, 43 | at::Tensor key_weight, 44 | at::Tensor value, 45 | int hashtable_capacity, 46 | bool use_cuda 47 | ); 48 | 49 | at::Tensor lsh_weighted_cumulation_ver3_kernel( 50 | at::Tensor query_mask, 51 | at::Tensor query_hash_code, 52 | at::Tensor query_weight, 53 | at::Tensor key_mask, 54 | at::Tensor key_hash_code, 55 | at::Tensor key_weight, 56 | at::Tensor value, 57 | int hashtable_capacity, 58 | bool use_cuda 59 | ); 60 | 61 | at::Tensor lsh_weighted_cumulation_ver4_kernel( 62 | at::Tensor query_mask, 63 | at::Tensor query_hash_code, 64 | at::Tensor query_weight, 65 | at::Tensor key_mask, 66 | at::Tensor key_hash_code, 67 | at::Tensor key_weight, 68 | at::Tensor value, 69 | int hashtable_capacity, 70 | bool use_cuda 71 | ); 72 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/onnx/__init__.py: -------------------------------------------------------------------------------- 1 | # flake8: noqa 2 | # Copyright 2020 The HuggingFace Team. All rights reserved. 3 | # 4 | # Licensed under the Apache License, Version 2.0 (the "License"); 5 | # you may not use this file except in compliance with the License. 6 | # You may obtain a copy of the License at 7 | # 8 | # http://www.apache.org/licenses/LICENSE-2.0 9 | # 10 | # Unless required by applicable law or agreed to in writing, software 11 | # distributed under the License is distributed on an "AS IS" BASIS, 12 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | # See the License for the specific language governing permissions and 14 | # limitations under the License. 15 | 16 | from typing import TYPE_CHECKING 17 | 18 | from ..utils import _LazyModule 19 | 20 | 21 | _import_structure = { 22 | "config": [ 23 | "EXTERNAL_DATA_FORMAT_SIZE_LIMIT", 24 | "OnnxConfig", 25 | "OnnxConfigWithPast", 26 | "OnnxSeq2SeqConfigWithPast", 27 | "PatchingSpec", 28 | ], 29 | "convert": ["export", "validate_model_outputs"], 30 | "features": ["FeaturesManager"], 31 | "utils": ["ParameterFormat", "compute_serialized_parameters_size"], 32 | } 33 | 34 | 35 | if TYPE_CHECKING: 36 | from .config import ( 37 | EXTERNAL_DATA_FORMAT_SIZE_LIMIT, 38 | OnnxConfig, 39 | OnnxConfigWithPast, 40 | OnnxSeq2SeqConfigWithPast, 41 | PatchingSpec, 42 | ) 43 | from .convert import export, validate_model_outputs 44 | from .features import FeaturesManager 45 | from .utils import ParameterFormat, compute_serialized_parameters_size 46 | 47 | else: 48 | import sys 49 | 50 | sys.modules[__name__] = _LazyModule(__name__, globals()["__file__"], _import_structure, module_spec=__spec__) 51 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/sagemaker/__init__.py: -------------------------------------------------------------------------------- 1 | # flake8: noqa 2 | # There's no way to ignore "F401 '...' imported but unused" warnings in this 3 | # module, but to preserve other warnings. So, don't check this module at all. 4 | 5 | # Copyright 2021 The HuggingFace Team. All rights reserved. 6 | # 7 | # Licensed under the Apache License, Version 2.0 (the "License"); 8 | # you may not use this file except in compliance with the License. 9 | # You may obtain a copy of the License at 10 | # 11 | # http://www.apache.org/licenses/LICENSE-2.0 12 | # 13 | # Unless required by applicable law or agreed to in writing, software 14 | # distributed under the License is distributed on an "AS IS" BASIS, 15 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | # See the License for the specific language governing permissions and 17 | # limitations under the License. 18 | 19 | from .trainer_sm import SageMakerTrainer 20 | from .training_args_sm import SageMakerTrainingArguments, is_sagemaker_dp_enabled 21 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/sagemaker/trainer_sm.py: -------------------------------------------------------------------------------- 1 | # Copyright 2021 The HuggingFace Team. All rights reserved. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | import warnings 15 | 16 | from ..trainer import Trainer 17 | from ..utils import logging 18 | 19 | 20 | logger = logging.get_logger(__name__) 21 | 22 | 23 | class SageMakerTrainer(Trainer): 24 | def __init__(self, args=None, **kwargs): 25 | warnings.warn( 26 | "`SageMakerTrainer` is deprecated and will be removed in v5 of Transformers. You can use `Trainer` " 27 | "instead.", 28 | FutureWarning, 29 | ) 30 | super().__init__(args=args, **kwargs) 31 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/utils/constants.py: -------------------------------------------------------------------------------- 1 | IMAGENET_DEFAULT_MEAN = [0.485, 0.456, 0.406] 2 | IMAGENET_DEFAULT_STD = [0.229, 0.224, 0.225] 3 | IMAGENET_STANDARD_MEAN = [0.5, 0.5, 0.5] 4 | IMAGENET_STANDARD_STD = [0.5, 0.5, 0.5] 5 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/utils/dummy_detectron2_objects.py: -------------------------------------------------------------------------------- 1 | # This file is autogenerated by the command `make fix-copies`, do not edit. 2 | from ..utils import requires_backends 3 | 4 | 5 | LAYOUTLM_V2_PRETRAINED_MODEL_ARCHIVE_LIST = None 6 | 7 | 8 | class LayoutLMv2Model: 9 | def __init__(self, *args, **kwargs): 10 | requires_backends(self, ["detectron2"]) 11 | 12 | @classmethod 13 | def from_pretrained(cls, *args, **kwargs): 14 | requires_backends(cls, ["detectron2"]) 15 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/utils/dummy_scatter_objects.py: -------------------------------------------------------------------------------- 1 | # This file is autogenerated by the command `make fix-copies`, do not edit. 2 | # flake8: noqa 3 | from ..utils import DummyObject, requires_backends 4 | 5 | 6 | TAPAS_PRETRAINED_MODEL_ARCHIVE_LIST = None 7 | 8 | 9 | class TapasForMaskedLM(metaclass=DummyObject): 10 | _backends = ["scatter"] 11 | 12 | def __init__(self, *args, **kwargs): 13 | requires_backends(self, ["scatter"]) 14 | 15 | 16 | class TapasForQuestionAnswering(metaclass=DummyObject): 17 | _backends = ["scatter"] 18 | 19 | def __init__(self, *args, **kwargs): 20 | requires_backends(self, ["scatter"]) 21 | 22 | 23 | class TapasForSequenceClassification(metaclass=DummyObject): 24 | _backends = ["scatter"] 25 | 26 | def __init__(self, *args, **kwargs): 27 | requires_backends(self, ["scatter"]) 28 | 29 | 30 | class TapasModel(metaclass=DummyObject): 31 | _backends = ["scatter"] 32 | 33 | def __init__(self, *args, **kwargs): 34 | requires_backends(self, ["scatter"]) 35 | 36 | 37 | class TapasPreTrainedModel(metaclass=DummyObject): 38 | _backends = ["scatter"] 39 | 40 | def __init__(self, *args, **kwargs): 41 | requires_backends(self, ["scatter"]) 42 | 43 | 44 | def load_tf_weights_in_tapas(*args, **kwargs): 45 | requires_backends(load_tf_weights_in_tapas, ["scatter"]) 46 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/utils/dummy_sentencepiece_and_speech_objects.py: -------------------------------------------------------------------------------- 1 | # This file is autogenerated by the command `make fix-copies`, do not edit. 2 | # flake8: noqa 3 | from ..utils import DummyObject, requires_backends 4 | 5 | 6 | class Speech2TextProcessor(metaclass=DummyObject): 7 | _backends = ["sentencepiece", "speech"] 8 | 9 | def __init__(self, *args, **kwargs): 10 | requires_backends(self, ["sentencepiece", "speech"]) 11 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/utils/dummy_sentencepiece_and_tokenizers_objects.py: -------------------------------------------------------------------------------- 1 | # This file is autogenerated by the command `make fix-copies`, do not edit. 2 | # flake8: noqa 3 | from ..utils import DummyObject, requires_backends 4 | 5 | 6 | SLOW_TO_FAST_CONVERTERS = None 7 | 8 | 9 | def convert_slow_tokenizer(*args, **kwargs): 10 | requires_backends(convert_slow_tokenizer, ["sentencepiece", "tokenizers"]) 11 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/utils/dummy_speech_objects.py: -------------------------------------------------------------------------------- 1 | # This file is autogenerated by the command `make fix-copies`, do not edit. 2 | # flake8: noqa 3 | from ..utils import DummyObject, requires_backends 4 | 5 | 6 | class MCTCTFeatureExtractor(metaclass=DummyObject): 7 | _backends = ["speech"] 8 | 9 | def __init__(self, *args, **kwargs): 10 | requires_backends(self, ["speech"]) 11 | 12 | 13 | class Speech2TextFeatureExtractor(metaclass=DummyObject): 14 | _backends = ["speech"] 15 | 16 | def __init__(self, *args, **kwargs): 17 | requires_backends(self, ["speech"]) 18 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/utils/dummy_tensorflow_text_objects.py: -------------------------------------------------------------------------------- 1 | # This file is autogenerated by the command `make fix-copies`, do not edit. 2 | # flake8: noqa 3 | from ..utils import DummyObject, requires_backends 4 | 5 | 6 | class TFBertTokenizer(metaclass=DummyObject): 7 | _backends = ["tensorflow_text"] 8 | 9 | def __init__(self, *args, **kwargs): 10 | requires_backends(self, ["tensorflow_text"]) 11 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/utils/dummy_timm_and_vision_objects.py: -------------------------------------------------------------------------------- 1 | # This file is autogenerated by the command `make fix-copies`, do not edit. 2 | # flake8: noqa 3 | from ..utils import DummyObject, requires_backends 4 | 5 | 6 | DETR_PRETRAINED_MODEL_ARCHIVE_LIST = None 7 | 8 | 9 | class DetrForObjectDetection(metaclass=DummyObject): 10 | _backends = ["timm", "vision"] 11 | 12 | def __init__(self, *args, **kwargs): 13 | requires_backends(self, ["timm", "vision"]) 14 | 15 | 16 | class DetrForSegmentation(metaclass=DummyObject): 17 | _backends = ["timm", "vision"] 18 | 19 | def __init__(self, *args, **kwargs): 20 | requires_backends(self, ["timm", "vision"]) 21 | 22 | 23 | class DetrModel(metaclass=DummyObject): 24 | _backends = ["timm", "vision"] 25 | 26 | def __init__(self, *args, **kwargs): 27 | requires_backends(self, ["timm", "vision"]) 28 | 29 | 30 | class DetrPreTrainedModel(metaclass=DummyObject): 31 | _backends = ["timm", "vision"] 32 | 33 | def __init__(self, *args, **kwargs): 34 | requires_backends(self, ["timm", "vision"]) 35 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/utils/dummy_timm_objects.py: -------------------------------------------------------------------------------- 1 | # This file is autogenerated by the command `make fix-copies`, do not edit. 2 | from ..utils import requires_backends 3 | 4 | 5 | DETR_PRETRAINED_MODEL_ARCHIVE_LIST = None 6 | 7 | 8 | class DetrForObjectDetection: 9 | def __init__(self, *args, **kwargs): 10 | requires_backends(self, ["timm"]) 11 | 12 | @classmethod 13 | def from_pretrained(cls, *args, **kwargs): 14 | requires_backends(cls, ["timm"]) 15 | 16 | 17 | class DetrForSegmentation: 18 | def __init__(self, *args, **kwargs): 19 | requires_backends(self, ["timm"]) 20 | 21 | @classmethod 22 | def from_pretrained(cls, *args, **kwargs): 23 | requires_backends(cls, ["timm"]) 24 | 25 | 26 | class DetrModel: 27 | def __init__(self, *args, **kwargs): 28 | requires_backends(self, ["timm"]) 29 | 30 | @classmethod 31 | def from_pretrained(cls, *args, **kwargs): 32 | requires_backends(cls, ["timm"]) 33 | -------------------------------------------------------------------------------- /SingleDivide/transformersLocal/utils/model_parallel_utils.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | # Copyright 2020 The HuggingFace Team. All rights reserved. 3 | # 4 | # Licensed under the Apache License, Version 2.0 (the "License"); 5 | # you may not use this file except in compliance with the License. 6 | # You may obtain a copy of the License at 7 | # 8 | # http://www.apache.org/licenses/LICENSE-2.0 9 | # 10 | # Unless required by applicable law or agreed to in writing, software 11 | # distributed under the License is distributed on an "AS IS" BASIS, 12 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | # See the License for the specific language governing permissions and 14 | # limitations under the License. 15 | 16 | from math import ceil 17 | 18 | 19 | def assert_device_map(device_map, num_blocks): 20 | blocks = list(range(0, num_blocks)) 21 | 22 | device_map_blocks = [item for sublist in list(device_map.values()) for item in sublist] 23 | 24 | # Duplicate check 25 | duplicate_blocks = [] 26 | for i in device_map_blocks: 27 | if device_map_blocks.count(i) > 1 and i not in duplicate_blocks: 28 | duplicate_blocks.append(i) 29 | # Missing blocks 30 | missing_blocks = [i for i in blocks if i not in device_map_blocks] 31 | extra_blocks = [i for i in device_map_blocks if i not in blocks] 32 | 33 | if len(duplicate_blocks) != 0: 34 | raise ValueError( 35 | "Duplicate attention blocks specified in device_map. Attention blocks must be specified to one device." 36 | " These attention blocks were specified more than once: " 37 | + str(duplicate_blocks) 38 | ) 39 | if len(missing_blocks) != 0: 40 | raise ValueError( 41 | "There are attention blocks for this model that are not specified in the device_map. Add these attention " 42 | "blocks to a device on the device_map: " 43 | + str(missing_blocks) 44 | ) 45 | if len(extra_blocks) != 0: 46 | raise ValueError( 47 | "The device_map contains more attention blocks than this model has. Remove these from the device_map:" 48 | + str(extra_blocks) 49 | ) 50 | 51 | 52 | def get_device_map(n_layers, devices): 53 | """Returns a dictionary of layers distributed evenly across all devices.""" 54 | layers = list(range(n_layers)) 55 | n_blocks = int(ceil(n_layers / len(devices))) 56 | layers_list = list(layers[i : i + n_blocks] for i in range(0, n_layers, n_blocks)) 57 | 58 | return dict(zip(devices, layers_list)) 59 | -------------------------------------------------------------------------------- /readme.md: -------------------------------------------------------------------------------- 1 | ### Training Transformers with 4-bit Integers 2 | 3 | This is the official repo for https://arxiv.org/abs//2306.11987. 4 | 5 | >**Abstract**:
6 | >Quantizing the activation, weight, and gradient to 4-bit is promising to accelerate neural network training. However, existing 4-bit training methods require custom numerical formats which are not supported by contemporary hardware. In this work, we propose a training method for transformers with all matrix multiplications implemented with the INT4 arithmetic. 7 | >Training with an ultra-low INT4 precision is challenging. To achieve this, we carefully analyze the specific structures of activation and gradients in transformers to propose dedicated quantizers for them. For forward propagation, we identify the challenge of outliers and propose a Hadamard quantizer to suppress the outliers. For backpropagation, we leverage the structural sparsity of gradients by proposing bit splitting and leverage score sampling techniques to quantize gradients accurately. Our algorithm achieves competitive accuracy on a wide range of tasks including natural language understanding, machine translation, and image classification. Unlike previous 4-bit training methods, our algorithm can be implemented on the current generation of GPUs. Our prototypical linear operator implementation is up to 2.2 times faster than the FP16 counterparts and speeds up the training by up to 35.1\%. 8 | 9 | Cuda_LSS_HQ provides the CUDA implementation of our algorithm. SingleDivide provides the training recipe to reproduce the result. 10 | 11 | To reproduce the result, please first change the cutlass_path in each of the setup.py in Cuda_LSS_HQ to the absolute path of cutlass/include. 12 | 13 | Then, please run 14 | ``` 15 | bash task.sh 16 | ``` 17 | in SingleDivide reproduce the result of GLUE dataset. --------------------------------------------------------------------------------