├── .gitignore ├── accurate_timing_of_strava_segments ├── requirements.txt ├── marienfeld_a.png └── accurate_timing_of_strava_segments.py ├── programming_language_learning_curves ├── cpp.png ├── java.png ├── lisp.png ├── php.png ├── haskell.png ├── python.png ├── Humor-Sans.txt ├── Humor-Sans.ttf ├── javascript.png └── generate.py ├── basic_measures_of_descriptive_statistics ├── cdf.png ├── iqr.png ├── pdf.png ├── cdf_with_median.png ├── few_values_histogram.png └── many_values_histogram.png ├── how_i_got_rid_of_the_crosstalk_in_my_headset ├── box1.jpg ├── box2.jpg ├── box3.jpg ├── trrs.jpg ├── cut_jack.jpg ├── jabra_usb.jpg ├── soundcard.jpg ├── strands.jpg ├── two_plugs.jpg ├── melted_off.jpg ├── multimeter.jpg ├── plug_labels.jpg ├── adapter_merge.jpg ├── adapter_split.jpg ├── opened_headset.jpg ├── plug_standards.jpg ├── strain_relief.jpg ├── terminal_blocks.jpg ├── recording_experiment.jpg └── share_ground_adapters.jpg ├── threads_can_infect_each_other_with_their_low_priority ├── 1.png ├── 2.png ├── 3.png └── 4.png ├── fixing_underengineered_code_vs_fixing_overengineered_code ├── puzzle.jpg ├── puzzle_overengineering.jpg └── puzzle_underengineering.jpg ├── mechanical_analogies_for_basic_measures_of_descriptive_statistics ├── pdf.png ├── pdf_wooden.png ├── pdf_rotation.png ├── pdf_with_mean.png └── pdf_with_median.png ├── covariance_and_contravariance_explained_without_code ├── jane_bike_crusher.jpg ├── john_bike_factory.jpg ├── jane_vehicle_crusher.jpg └── john_vehicle_factory.jpg ├── from_spaghetti_to_ravioli_-_a_refactoring_example_using_kotlin ├── diff_a_b.png ├── diff_b_c.png ├── diff_c_d.png ├── diff_d_e.png ├── diff_e_f.png ├── diff_f_g.png ├── diff_g_h.png └── from_spaghetti_to_ravioli.png ├── a_too_naive_approach_to_video_compression_using_artificial_neural_networks_files ├── nn.jpg ├── tanh.png ├── nn_snapshot.jpg └── original_snapshot.jpg ├── on_feeling_deceived_when_receiving_non_labeled_llm_generated_messages └── four_sides_model.png ├── when_does_a_when_expression_in_kotlin_need_to_be_exhaustive_and_when_does_it_not ├── clau.png ├── clav.png ├── clsu.png ├── clsv.png ├── cnau.png ├── cnav.png ├── cnsu.png ├── cnsv.png ├── elau.png ├── elav.png ├── elsu.png ├── elsv.png ├── enau.png ├── enav.png ├── ensu.png ├── ensv.png ├── error.png └── recommendation.png ├── CITATION.cff ├── functional_programming_in_cpp_with_the_functionalplus_library_today_hackerrank_challange_gemstones └── api_search_fold_left_1_001.png ├── programming_language_learning_curves.md ├── mechanical_analogies_for_basic_measures_of_descriptive_statistics.md ├── fixing_underengineered_code_vs_fixing_overengineered_code.md ├── refactoring_suggestions_are_a_compliment.md ├── on_feeling_deceived_when_receiving_non_labeled_llm_generated_messages.md ├── using_fold_to_mimic_mutation.md ├── implementation_inheritance_is_bad_-_the_fragile_base_class_problem.md ├── how_i_got_rid_of_the_crosstalk_in_my_headset.md ├── internals_of_the_async_await_pattern_from_first_principles └── internals_of_the_async_await_pattern_from_first_principles.py ├── README.md ├── the_expressive_power_of_constraints.md ├── avoid_methods_to_reduce_coupling.md ├── why_a_generic_implementation_can_be_the_easier_to_understand_solution.md ├── covariance_and_contravariance_explained_without_code.md ├── what_kotlin_could_learn_from_cpps_keyword_const.md ├── basic_measures_of_descriptive_statistics.md ├── when_does_a_when_expression_in_kotlin_need_to_be_exhaustive_and_when_does_it_not.md ├── accurate_timing_of_strava_segments.md ├── a_too_naive_approach_to_video_compression_using_artificial_neural_networks.md ├── threads_can_infect_each_other_with_their_low_priority.md ├── do_a_b_tests_because_correlation_does_not_imply_causation.md ├── how_touch_typing_and_keyboard_shortcuts_can_improve_the_quality_of_the_software_you_create.md ├── from_goto_to_std-transform.md ├── llm_agents_demystified.md ├── creating_a_replacement_for_the_switch_statement_in_cpp_that_also_works_for_strings.md ├── functional_programming_in_cpp_with_the_functionalplus_library_today_hackerrank_challange_gemstones.md ├── simple_collaborative_filtering_in_pure_postgresql.md ├── a_personal_generic_things_i_learned_as_a_software_developer_list.md ├── internals_of_the_async_await_pattern_from_first_principles.md └── a_monad_is_just_a_monoid_in_the_category_of_endofunctors_explained.md /.gitignore: -------------------------------------------------------------------------------- 1 | dev 2 | .vscode 3 | -------------------------------------------------------------------------------- /accurate_timing_of_strava_segments/requirements.txt: -------------------------------------------------------------------------------- 1 | geopy 2 | requests 3 | sympy 4 | tcxreader 5 | -------------------------------------------------------------------------------- /programming_language_learning_curves/cpp.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/programming_language_learning_curves/cpp.png -------------------------------------------------------------------------------- /programming_language_learning_curves/java.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/programming_language_learning_curves/java.png -------------------------------------------------------------------------------- /programming_language_learning_curves/lisp.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/programming_language_learning_curves/lisp.png -------------------------------------------------------------------------------- /programming_language_learning_curves/php.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/programming_language_learning_curves/php.png -------------------------------------------------------------------------------- /basic_measures_of_descriptive_statistics/cdf.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/basic_measures_of_descriptive_statistics/cdf.png -------------------------------------------------------------------------------- /basic_measures_of_descriptive_statistics/iqr.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/basic_measures_of_descriptive_statistics/iqr.png -------------------------------------------------------------------------------- /basic_measures_of_descriptive_statistics/pdf.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/basic_measures_of_descriptive_statistics/pdf.png -------------------------------------------------------------------------------- /programming_language_learning_curves/haskell.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/programming_language_learning_curves/haskell.png -------------------------------------------------------------------------------- /programming_language_learning_curves/python.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/programming_language_learning_curves/python.png -------------------------------------------------------------------------------- /programming_language_learning_curves/Humor-Sans.txt: -------------------------------------------------------------------------------- 1 | The font is from here: http://xkcdsucks.blogspot.de/2009/03/xkcdsucks-is-proud-to-present-humor.html -------------------------------------------------------------------------------- /accurate_timing_of_strava_segments/marienfeld_a.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/accurate_timing_of_strava_segments/marienfeld_a.png -------------------------------------------------------------------------------- /how_i_got_rid_of_the_crosstalk_in_my_headset/box1.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/how_i_got_rid_of_the_crosstalk_in_my_headset/box1.jpg -------------------------------------------------------------------------------- /how_i_got_rid_of_the_crosstalk_in_my_headset/box2.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/how_i_got_rid_of_the_crosstalk_in_my_headset/box2.jpg -------------------------------------------------------------------------------- /how_i_got_rid_of_the_crosstalk_in_my_headset/box3.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/how_i_got_rid_of_the_crosstalk_in_my_headset/box3.jpg -------------------------------------------------------------------------------- /how_i_got_rid_of_the_crosstalk_in_my_headset/trrs.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/how_i_got_rid_of_the_crosstalk_in_my_headset/trrs.jpg -------------------------------------------------------------------------------- /programming_language_learning_curves/Humor-Sans.ttf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/programming_language_learning_curves/Humor-Sans.ttf -------------------------------------------------------------------------------- /programming_language_learning_curves/javascript.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/programming_language_learning_curves/javascript.png -------------------------------------------------------------------------------- /how_i_got_rid_of_the_crosstalk_in_my_headset/cut_jack.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/how_i_got_rid_of_the_crosstalk_in_my_headset/cut_jack.jpg -------------------------------------------------------------------------------- /how_i_got_rid_of_the_crosstalk_in_my_headset/jabra_usb.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/how_i_got_rid_of_the_crosstalk_in_my_headset/jabra_usb.jpg -------------------------------------------------------------------------------- /how_i_got_rid_of_the_crosstalk_in_my_headset/soundcard.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/how_i_got_rid_of_the_crosstalk_in_my_headset/soundcard.jpg -------------------------------------------------------------------------------- /how_i_got_rid_of_the_crosstalk_in_my_headset/strands.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/how_i_got_rid_of_the_crosstalk_in_my_headset/strands.jpg -------------------------------------------------------------------------------- /how_i_got_rid_of_the_crosstalk_in_my_headset/two_plugs.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/how_i_got_rid_of_the_crosstalk_in_my_headset/two_plugs.jpg -------------------------------------------------------------------------------- /basic_measures_of_descriptive_statistics/cdf_with_median.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/basic_measures_of_descriptive_statistics/cdf_with_median.png -------------------------------------------------------------------------------- /how_i_got_rid_of_the_crosstalk_in_my_headset/melted_off.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/how_i_got_rid_of_the_crosstalk_in_my_headset/melted_off.jpg -------------------------------------------------------------------------------- /how_i_got_rid_of_the_crosstalk_in_my_headset/multimeter.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/how_i_got_rid_of_the_crosstalk_in_my_headset/multimeter.jpg -------------------------------------------------------------------------------- /how_i_got_rid_of_the_crosstalk_in_my_headset/plug_labels.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/how_i_got_rid_of_the_crosstalk_in_my_headset/plug_labels.jpg -------------------------------------------------------------------------------- /threads_can_infect_each_other_with_their_low_priority/1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/threads_can_infect_each_other_with_their_low_priority/1.png -------------------------------------------------------------------------------- /threads_can_infect_each_other_with_their_low_priority/2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/threads_can_infect_each_other_with_their_low_priority/2.png -------------------------------------------------------------------------------- /threads_can_infect_each_other_with_their_low_priority/3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/threads_can_infect_each_other_with_their_low_priority/3.png -------------------------------------------------------------------------------- /threads_can_infect_each_other_with_their_low_priority/4.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/threads_can_infect_each_other_with_their_low_priority/4.png -------------------------------------------------------------------------------- /how_i_got_rid_of_the_crosstalk_in_my_headset/adapter_merge.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/how_i_got_rid_of_the_crosstalk_in_my_headset/adapter_merge.jpg -------------------------------------------------------------------------------- /how_i_got_rid_of_the_crosstalk_in_my_headset/adapter_split.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/how_i_got_rid_of_the_crosstalk_in_my_headset/adapter_split.jpg -------------------------------------------------------------------------------- /how_i_got_rid_of_the_crosstalk_in_my_headset/opened_headset.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/how_i_got_rid_of_the_crosstalk_in_my_headset/opened_headset.jpg -------------------------------------------------------------------------------- /how_i_got_rid_of_the_crosstalk_in_my_headset/plug_standards.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/how_i_got_rid_of_the_crosstalk_in_my_headset/plug_standards.jpg -------------------------------------------------------------------------------- /how_i_got_rid_of_the_crosstalk_in_my_headset/strain_relief.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/how_i_got_rid_of_the_crosstalk_in_my_headset/strain_relief.jpg -------------------------------------------------------------------------------- /basic_measures_of_descriptive_statistics/few_values_histogram.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/basic_measures_of_descriptive_statistics/few_values_histogram.png -------------------------------------------------------------------------------- /how_i_got_rid_of_the_crosstalk_in_my_headset/terminal_blocks.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/how_i_got_rid_of_the_crosstalk_in_my_headset/terminal_blocks.jpg -------------------------------------------------------------------------------- /basic_measures_of_descriptive_statistics/many_values_histogram.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/basic_measures_of_descriptive_statistics/many_values_histogram.png -------------------------------------------------------------------------------- /fixing_underengineered_code_vs_fixing_overengineered_code/puzzle.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/fixing_underengineered_code_vs_fixing_overengineered_code/puzzle.jpg -------------------------------------------------------------------------------- /how_i_got_rid_of_the_crosstalk_in_my_headset/recording_experiment.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/how_i_got_rid_of_the_crosstalk_in_my_headset/recording_experiment.jpg -------------------------------------------------------------------------------- /how_i_got_rid_of_the_crosstalk_in_my_headset/share_ground_adapters.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/how_i_got_rid_of_the_crosstalk_in_my_headset/share_ground_adapters.jpg -------------------------------------------------------------------------------- /mechanical_analogies_for_basic_measures_of_descriptive_statistics/pdf.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/mechanical_analogies_for_basic_measures_of_descriptive_statistics/pdf.png -------------------------------------------------------------------------------- /covariance_and_contravariance_explained_without_code/jane_bike_crusher.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/covariance_and_contravariance_explained_without_code/jane_bike_crusher.jpg -------------------------------------------------------------------------------- /covariance_and_contravariance_explained_without_code/john_bike_factory.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/covariance_and_contravariance_explained_without_code/john_bike_factory.jpg -------------------------------------------------------------------------------- /from_spaghetti_to_ravioli_-_a_refactoring_example_using_kotlin/diff_a_b.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/from_spaghetti_to_ravioli_-_a_refactoring_example_using_kotlin/diff_a_b.png -------------------------------------------------------------------------------- /from_spaghetti_to_ravioli_-_a_refactoring_example_using_kotlin/diff_b_c.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/from_spaghetti_to_ravioli_-_a_refactoring_example_using_kotlin/diff_b_c.png -------------------------------------------------------------------------------- /from_spaghetti_to_ravioli_-_a_refactoring_example_using_kotlin/diff_c_d.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/from_spaghetti_to_ravioli_-_a_refactoring_example_using_kotlin/diff_c_d.png -------------------------------------------------------------------------------- /from_spaghetti_to_ravioli_-_a_refactoring_example_using_kotlin/diff_d_e.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/from_spaghetti_to_ravioli_-_a_refactoring_example_using_kotlin/diff_d_e.png -------------------------------------------------------------------------------- /from_spaghetti_to_ravioli_-_a_refactoring_example_using_kotlin/diff_e_f.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/from_spaghetti_to_ravioli_-_a_refactoring_example_using_kotlin/diff_e_f.png -------------------------------------------------------------------------------- /from_spaghetti_to_ravioli_-_a_refactoring_example_using_kotlin/diff_f_g.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/from_spaghetti_to_ravioli_-_a_refactoring_example_using_kotlin/diff_f_g.png -------------------------------------------------------------------------------- /from_spaghetti_to_ravioli_-_a_refactoring_example_using_kotlin/diff_g_h.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/from_spaghetti_to_ravioli_-_a_refactoring_example_using_kotlin/diff_g_h.png -------------------------------------------------------------------------------- /covariance_and_contravariance_explained_without_code/jane_vehicle_crusher.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/covariance_and_contravariance_explained_without_code/jane_vehicle_crusher.jpg -------------------------------------------------------------------------------- /covariance_and_contravariance_explained_without_code/john_vehicle_factory.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/covariance_and_contravariance_explained_without_code/john_vehicle_factory.jpg -------------------------------------------------------------------------------- /mechanical_analogies_for_basic_measures_of_descriptive_statistics/pdf_wooden.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/mechanical_analogies_for_basic_measures_of_descriptive_statistics/pdf_wooden.png -------------------------------------------------------------------------------- /mechanical_analogies_for_basic_measures_of_descriptive_statistics/pdf_rotation.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/mechanical_analogies_for_basic_measures_of_descriptive_statistics/pdf_rotation.png -------------------------------------------------------------------------------- /mechanical_analogies_for_basic_measures_of_descriptive_statistics/pdf_with_mean.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/mechanical_analogies_for_basic_measures_of_descriptive_statistics/pdf_with_mean.png -------------------------------------------------------------------------------- /fixing_underengineered_code_vs_fixing_overengineered_code/puzzle_overengineering.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/fixing_underengineered_code_vs_fixing_overengineered_code/puzzle_overengineering.jpg -------------------------------------------------------------------------------- /fixing_underengineered_code_vs_fixing_overengineered_code/puzzle_underengineering.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/fixing_underengineered_code_vs_fixing_overengineered_code/puzzle_underengineering.jpg -------------------------------------------------------------------------------- /mechanical_analogies_for_basic_measures_of_descriptive_statistics/pdf_with_median.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/mechanical_analogies_for_basic_measures_of_descriptive_statistics/pdf_with_median.png -------------------------------------------------------------------------------- /a_too_naive_approach_to_video_compression_using_artificial_neural_networks_files/nn.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/a_too_naive_approach_to_video_compression_using_artificial_neural_networks_files/nn.jpg -------------------------------------------------------------------------------- /a_too_naive_approach_to_video_compression_using_artificial_neural_networks_files/tanh.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/a_too_naive_approach_to_video_compression_using_artificial_neural_networks_files/tanh.png -------------------------------------------------------------------------------- /on_feeling_deceived_when_receiving_non_labeled_llm_generated_messages/four_sides_model.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/on_feeling_deceived_when_receiving_non_labeled_llm_generated_messages/four_sides_model.png -------------------------------------------------------------------------------- /when_does_a_when_expression_in_kotlin_need_to_be_exhaustive_and_when_does_it_not/clau.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/when_does_a_when_expression_in_kotlin_need_to_be_exhaustive_and_when_does_it_not/clau.png -------------------------------------------------------------------------------- /when_does_a_when_expression_in_kotlin_need_to_be_exhaustive_and_when_does_it_not/clav.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/when_does_a_when_expression_in_kotlin_need_to_be_exhaustive_and_when_does_it_not/clav.png -------------------------------------------------------------------------------- /when_does_a_when_expression_in_kotlin_need_to_be_exhaustive_and_when_does_it_not/clsu.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/when_does_a_when_expression_in_kotlin_need_to_be_exhaustive_and_when_does_it_not/clsu.png -------------------------------------------------------------------------------- /when_does_a_when_expression_in_kotlin_need_to_be_exhaustive_and_when_does_it_not/clsv.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/when_does_a_when_expression_in_kotlin_need_to_be_exhaustive_and_when_does_it_not/clsv.png -------------------------------------------------------------------------------- /when_does_a_when_expression_in_kotlin_need_to_be_exhaustive_and_when_does_it_not/cnau.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/when_does_a_when_expression_in_kotlin_need_to_be_exhaustive_and_when_does_it_not/cnau.png -------------------------------------------------------------------------------- /when_does_a_when_expression_in_kotlin_need_to_be_exhaustive_and_when_does_it_not/cnav.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/when_does_a_when_expression_in_kotlin_need_to_be_exhaustive_and_when_does_it_not/cnav.png -------------------------------------------------------------------------------- /when_does_a_when_expression_in_kotlin_need_to_be_exhaustive_and_when_does_it_not/cnsu.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/when_does_a_when_expression_in_kotlin_need_to_be_exhaustive_and_when_does_it_not/cnsu.png -------------------------------------------------------------------------------- /when_does_a_when_expression_in_kotlin_need_to_be_exhaustive_and_when_does_it_not/cnsv.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/when_does_a_when_expression_in_kotlin_need_to_be_exhaustive_and_when_does_it_not/cnsv.png -------------------------------------------------------------------------------- /when_does_a_when_expression_in_kotlin_need_to_be_exhaustive_and_when_does_it_not/elau.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/when_does_a_when_expression_in_kotlin_need_to_be_exhaustive_and_when_does_it_not/elau.png -------------------------------------------------------------------------------- /when_does_a_when_expression_in_kotlin_need_to_be_exhaustive_and_when_does_it_not/elav.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/when_does_a_when_expression_in_kotlin_need_to_be_exhaustive_and_when_does_it_not/elav.png -------------------------------------------------------------------------------- /when_does_a_when_expression_in_kotlin_need_to_be_exhaustive_and_when_does_it_not/elsu.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/when_does_a_when_expression_in_kotlin_need_to_be_exhaustive_and_when_does_it_not/elsu.png -------------------------------------------------------------------------------- /when_does_a_when_expression_in_kotlin_need_to_be_exhaustive_and_when_does_it_not/elsv.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/when_does_a_when_expression_in_kotlin_need_to_be_exhaustive_and_when_does_it_not/elsv.png -------------------------------------------------------------------------------- /when_does_a_when_expression_in_kotlin_need_to_be_exhaustive_and_when_does_it_not/enau.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/when_does_a_when_expression_in_kotlin_need_to_be_exhaustive_and_when_does_it_not/enau.png -------------------------------------------------------------------------------- /when_does_a_when_expression_in_kotlin_need_to_be_exhaustive_and_when_does_it_not/enav.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/when_does_a_when_expression_in_kotlin_need_to_be_exhaustive_and_when_does_it_not/enav.png -------------------------------------------------------------------------------- /when_does_a_when_expression_in_kotlin_need_to_be_exhaustive_and_when_does_it_not/ensu.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/when_does_a_when_expression_in_kotlin_need_to_be_exhaustive_and_when_does_it_not/ensu.png -------------------------------------------------------------------------------- /when_does_a_when_expression_in_kotlin_need_to_be_exhaustive_and_when_does_it_not/ensv.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/when_does_a_when_expression_in_kotlin_need_to_be_exhaustive_and_when_does_it_not/ensv.png -------------------------------------------------------------------------------- /when_does_a_when_expression_in_kotlin_need_to_be_exhaustive_and_when_does_it_not/error.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/when_does_a_when_expression_in_kotlin_need_to_be_exhaustive_and_when_does_it_not/error.png -------------------------------------------------------------------------------- /CITATION.cff: -------------------------------------------------------------------------------- 1 | cff-version: 1.2.0 2 | title: "articles" 3 | url: "https://github.com/Dobiasd/articles" 4 | authors: 5 | - family-names: "Hermann" 6 | given-names: "Tobias" 7 | orcid: "https://orcid.org/0009-0007-4792-4904" 8 | -------------------------------------------------------------------------------- /from_spaghetti_to_ravioli_-_a_refactoring_example_using_kotlin/from_spaghetti_to_ravioli.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/from_spaghetti_to_ravioli_-_a_refactoring_example_using_kotlin/from_spaghetti_to_ravioli.png -------------------------------------------------------------------------------- /a_too_naive_approach_to_video_compression_using_artificial_neural_networks_files/nn_snapshot.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/a_too_naive_approach_to_video_compression_using_artificial_neural_networks_files/nn_snapshot.jpg -------------------------------------------------------------------------------- /when_does_a_when_expression_in_kotlin_need_to_be_exhaustive_and_when_does_it_not/recommendation.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/when_does_a_when_expression_in_kotlin_need_to_be_exhaustive_and_when_does_it_not/recommendation.png -------------------------------------------------------------------------------- /a_too_naive_approach_to_video_compression_using_artificial_neural_networks_files/original_snapshot.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/a_too_naive_approach_to_video_compression_using_artificial_neural_networks_files/original_snapshot.jpg -------------------------------------------------------------------------------- /functional_programming_in_cpp_with_the_functionalplus_library_today_hackerrank_challange_gemstones/api_search_fold_left_1_001.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dobiasd/articles/HEAD/functional_programming_in_cpp_with_the_functionalplus_library_today_hackerrank_challange_gemstones/api_search_fold_left_1_001.png -------------------------------------------------------------------------------- /programming_language_learning_curves.md: -------------------------------------------------------------------------------- 1 | # Learning Curves (for different programming languages) 2 | 3 | ![Javascript](programming_language_learning_curves/javascript.png) 4 | ![Java](programming_language_learning_curves/java.png) 5 | ![C++](programming_language_learning_curves/cpp.png) 6 | ![Python](programming_language_learning_curves/python.png) 7 | ![Lisp](programming_language_learning_curves/lisp.png) 8 | ![Haskell](programming_language_learning_curves/haskell.png) 9 | ![PHP](programming_language_learning_curves/php.png) 10 | 11 | Disclaimer: The only purpose of this is to entertain. It has no empirical base whatsoever. ;-) 12 | 13 | [Discuss on reddit](http://www.reddit.com/r/ProgrammerHumor/comments/2qeami/learning_curves_of_programming_languages/) -------------------------------------------------------------------------------- /mechanical_analogies_for_basic_measures_of_descriptive_statistics.md: -------------------------------------------------------------------------------- 1 | # Mechanical analogies for basic measures of descriptive statistics 2 | 3 | (This article assumes you to know what the "mean", median and variance of a distribution are. In case you could need a quick refresher on this topic, please have a look at [*Basic measures of descriptive statistics*](basic_measures_of_descriptive_statistics.md) first.) 4 | 5 | ![pdf](mechanical_analogies_for_basic_measures_of_descriptive_statistics/pdf.png) 6 | 7 | Let's assume we would cut a thick, but flat, wooden model from a distribution like this. The usual basic measures of descriptive statistics would represent real physical properties of it. 8 | 9 | ![pdf_wooden](mechanical_analogies_for_basic_measures_of_descriptive_statistics/pdf_wooden.png) 10 | 11 | (I know, this looks amazingly realistic. Sadly I'm currently not available to accept design projects. ^_-) 12 | 13 | ## Mean 14 | 15 | ![pdf_with_mean](mechanical_analogies_for_basic_measures_of_descriptive_statistics/pdf_with_mean.png) 16 | 17 | The mean is the center of gravity in regards to the x-axis. So we could balance it on a rod aligned with the mean line. 18 | 19 | Also, we could hang it on a thread at one point on the mean line, and it would not tilt. I think this would make a very nice crib mobile. :) 20 | 21 | 22 | ## Median 23 | 24 | ![pdf_with_median](mechanical_analogies_for_basic_measures_of_descriptive_statistics/pdf_with_median.png) 25 | 26 | We cut our wooden distribution model in two parts along the median, both halves have the exact same weight. 27 | 28 | ## Variance 29 | 30 | ![rotation](mechanical_analogies_for_basic_measures_of_descriptive_statistics/pdf_rotation.png) 31 | 32 | We spin our model around its vertical axis. It spins around its mean if we don't attach it to something. And the moment of inertia for this rotation is equivalent to the variance of our distribution. So the variance is proportional to the torque needed for some angular acceleration. -------------------------------------------------------------------------------- /fixing_underengineered_code_vs_fixing_overengineered_code.md: -------------------------------------------------------------------------------- 1 | # Fixing under-engineered code vs. fixing over-engineered code 2 | 3 | ![(puzzle)](fixing_underengineered_code_vs_fixing_overengineered_code/puzzle.jpg) 4 | 5 | Recently I was playing around with the following question: 6 | 7 | > **Is it easier to fix underengineering in a codebase or overengineering?** 8 | 9 | This question naively assumes that there is a "right" level of engineering, which makes maintenance and enhancements most efficient. In reality, of course, this level depends on the requirements, expected future of the project, company culture, etc. But let's just pretend there is such an ideal, non-subjective, level. 10 | 11 | Let's also pretend we all have some similar understanding of what both terms mean. 12 | 13 | **Underengineering might have the following ingredients:** 14 | - too few abstractions 15 | - entangled concerns (non-SOLID code) 16 | - spaghetti 17 | 18 | **Overengineering might consist of the following:** 19 | - too many abstractions 20 | - consequently, very deep call stacks 21 | - violating the YAGNI principle 22 | 23 | While talking to people about this, I heard different opinions, many of them, naturally, including an "it depends", but I found the analogy my colleague Benjamin (CPO, former CTO, former ingenious software developer) spontaneously came up with especially striking, so I'd like to present it here. :slightly_smiling_face: 24 | 25 | **Fixing underengineering is like solving a jigsaw puzzle.** It's easy in the small, but the complexity grows exponentially with the number of pieces. 26 | 27 | ![(puzzle_underengineering)](fixing_underengineered_code_vs_fixing_overengineered_code/puzzle_underengineering.jpg) 28 | 29 | **Fixing overengineering is removing superfluous abstractions.** It scales linearly with the size. 30 | 31 | ![(puzzle_overengineering)](fixing_underengineered_code_vs_fixing_overengineered_code/puzzle_overengineering.jpg) 32 | 33 | Thus, for small projects, underengineering might be easier to fix, but once the project grows in size, overengineering is less troublesome to fix later. (Of course, the initial cost to develop the overengineered solution might be somewhat higher, but that's not the concern of this article. :wink:) -------------------------------------------------------------------------------- /refactoring_suggestions_are_a_compliment.md: -------------------------------------------------------------------------------- 1 | # Refactoring suggestions are a compliment 2 | 3 | Bob is a software developer. He implements a lot of nice new features. Alice, his colleague, occasionally gives suggestions on how to refactor the code - sometimes in a formal code review, sometimes casually, sometimes as actual commits in a merge request she opens. 4 | 5 | Bob interprets her ideas as a compliment because deep in his heart he knows that the intentions of Alice are noble and that she is making these suggestions because 6 | 7 | - she thinks the stuff Bob is working on is important enough to receive such care. She believes the code will be used and extended for quite some time in the future, and that investing her time in it is worth it. 8 | - she wants to reduce the amount of work on Bob and others in the long-term. They will be more efficient (and happier) maintaining a clean code base compared to something less SOLID. 9 | - she is positive that Bob is interested in learning and improving his skills. She cares about his personal growth as a developer. 10 | 11 | Bob constantly reminds himself of these conditions. Bob is awesome. Think as Bob thinks. 12 | 13 | Alice facilitates Bob's mindset by 14 | 15 | - conveying that these refactorings are not urgent but still important and by keeping them at a reasonable size. 16 | - being aware that it is usually easier to recognize smells (opportunities for improvement) if one looks at code from a distance compared to being knee-deep in the details. 17 | - striving towards an agreement of two equals by explaining her points and asking questions to understand Bob's way of thinking. She does not try to use authority. 18 | - being open to learning why Bob's original way might be better than her suggestion. 19 | 20 | Alice talks more in terms of "we" instead of "you" and "I". Alice is awesome. Do as Alice does. 21 | 22 | Carlos, their CTO, is happy about this friendly culture, which 23 | 24 | - helps to reduce the bus factor thanks to shared knowledge. 25 | - allows everyone to grow and results in low employee churn. 26 | - makes the team more productive and lets them create quality software. 27 | 28 | Carlos fosters this culture by being a good role model. Carlos is awesome. Lead as Carlos leads. 29 | -------------------------------------------------------------------------------- /on_feeling_deceived_when_receiving_non_labeled_llm_generated_messages.md: -------------------------------------------------------------------------------- 1 | # On feeling deceived when receiving (non-labeled) LLM-generated messages 2 | 3 | Seeing LLM-generated articles or social media posts is nothing new. We're used to double-checking important things, so we don't blindly believe factually wrong hallucinations of some GPT. 4 | 5 | However, the more personal the communication becomes, the stronger I feel against receiving generated messages without them being labeled as such. As an extreme example, imagine you write to your significant other (about some topic that is important to you), and they copy-paste your text to the "AI" and prompt it to "please respond on my behalf". When you find out you might feel angry, sad, and deceived. 6 | 7 | This extreme example shows that it's not just about the annoyance of having invested the effort in reading and understanding something that the sender did not invest the effort in thinking through and writing. To understand where the feeling of deception originates from, we need to appreciate that there is [more](https://en.wikipedia.org/wiki/Four-sides_model) to communication than just the factual information in a message. And this is not only true for love relationships. Even when you exchange texts with a friend or work colleague, and even if it's about a sober topic (like in a technical discussion in the comment section of a company-internal document), usually, 8 | - the sender reveals something (emotions, etc.) about themselves. 9 | - one can find hints about the relationship between sender and recipient "between the lines". 10 | - the nuances in the phrasing imply some appeal the writer intends to invoke in the reader. 11 | 12 | ![four_sides_model](on_feeling_deceived_when_receiving_non_labeled_llm_generated_messages/four_sides_model.png) 13 | 14 | When the alleged sender did not actually write the text, these additional sides of the message are no longer real. For an unaware recipient, who (maybe subconsciously) also processes these three other sides, this can lead to incorrect assumptions and expectations in the future, and to a false feeling of connectedness. 15 | 16 | Sure, there is a continuous spectrum from 17 | - "outsourcing a conversation completely to AI" 18 | - over "generating only a part of a message" 19 | - to "just using a fact or idea that you got from an AI at some point" 20 | 21 | and it's subjective where to set the "I should label this as LLM-powered" threshold. 22 | 23 | But please keep the four-sides model in mind when sending "AI-augmented" messages, even if it's not a very intimate relationship. Your recipient will very likely not only read the factual side but interpret the three other sides as well. And in case of doubt, default to being transparent. It's more empathic and more honest. And the mutual trust is worth it in the long run. 24 | -------------------------------------------------------------------------------- /using_fold_to_mimic_mutation.md: -------------------------------------------------------------------------------- 1 | # Using `fold` to mimic mutation 2 | 3 | (This article uses Kotlin to depict the idea with code. However, the actual idea is language-agnostic.) 4 | 5 | Usually, when thinking of [`fold`](https://en.wikipedia.org/wiki/Fold_(higher-order_function)), something like the following comes to one's mind: 6 | 7 | ```kotlin 8 | fun multiply(a: Int, b: Int) = a * b 9 | 10 | fun product(xs: List) = xs.fold(1, ::multiply) 11 | ``` 12 | 13 | I.e., using it to simply combine all elements of some iterable thing with an operator. (`listOf(5, 6, 7).fold(1, ::multiply)` expands to `((1 * 5) * 6) * 7`.) 14 | 15 | But it can also be used to simulate mutation in a purely functional style. But before going into this, let's set up a trivial mutation example: 16 | 17 | ```kotin 18 | data class Foo( 19 | var bar: Int, 20 | val baz: String 21 | ) 22 | 23 | fun mutateTimes2(foo: Foo) { 24 | foo.bar = foo.bar * 2 25 | } 26 | 27 | fun mutatePlus3(foo: Foo) { 28 | foo.bar = foo.bar + 3 29 | } 30 | 31 | fun main() { 32 | val x = Foo(1, "hi") 33 | mutateTimes2(x) 34 | mutatePlus3(x) 35 | println(x) // Foo(bar=5, baz=hi) 36 | } 37 | ``` 38 | 39 | In a functional setting, we would not mutate our `foo`, but instead, create new instances with a different value for `bar` (because of thread safety and stuff): 40 | 41 | ```kotlin 42 | data class Foo( 43 | val bar: Int, 44 | val baz: String 45 | ) 46 | 47 | fun withBarTimes2(foo: Foo) = 48 | foo.copy(bar = foo.bar * 2) 49 | 50 | fun withBarPlus3(foo: Foo) = 51 | foo.copy(bar = foo.bar + 3) 52 | 53 | fun main() { 54 | val x = Foo(1, "hi") 55 | val temp = withBarTimes2(x) 56 | val y = withBarPlus3(temp) 57 | println(y) // Foo(bar=5, baz=hi) 58 | } 59 | ``` 60 | 61 | In our mutation example, let's first put the mutating functions into a list to iterate over: 62 | 63 | ```kotlin 64 | data class Foo( 65 | var bar: Int, 66 | val baz: String 67 | ) 68 | 69 | fun mutateTimes2(foo: Foo) { 70 | foo.bar = foo.bar * 2 71 | } 72 | 73 | fun mutatePlus3(foo: Foo) { 74 | foo.bar = foo.bar + 3 75 | } 76 | 77 | fun main() { 78 | var x = Foo(1, "hi") 79 | listOf( 80 | ::mutateTimes2, 81 | ::mutatePlus3 82 | ).forEach { 83 | it(x) 84 | } 85 | println(x) // Foo(bar=5, baz=hi) 86 | } 87 | ``` 88 | 89 | Now, when squinting hard enough, we can see how to express this analogously in our functional code by using a (left) [`fold`](https://kotlinlang.org/api/latest/jvm/stdlib/kotlin.collections/fold.html): 90 | 91 | ```kotlin 92 | data class Foo( 93 | val bar: Int, 94 | val baz: String 95 | ) 96 | 97 | fun withBarTimes2(foo: Foo) = 98 | foo.copy(bar = foo.bar * 2) 99 | 100 | fun withBarPlus3(foo: Foo) = 101 | foo.copy(bar = foo.bar + 3) 102 | 103 | fun main() { 104 | val x = Foo(1, "hi") 105 | val y = listOf( 106 | ::withBarTimes2, 107 | ::withBarPlus3 108 | ).fold(x, { x, f -> f(x) }) 109 | println(y) // Foo(bar=5, baz=hi) 110 | } 111 | ``` 112 | 113 | This pattern can become quite useful if the collection of functions to apply is not yet known at compile-time, and is built dynamically. 114 | -------------------------------------------------------------------------------- /implementation_inheritance_is_bad_-_the_fragile_base_class_problem.md: -------------------------------------------------------------------------------- 1 | # Implementation inheritance is bad - the fragile base class problem 2 | 3 | One could argue, that implementation inheritance should not be supported at all 4 | by any sane object-oriented programming language. 5 | It should only be allowed to inherit (in the sense of "implement") interfaces. 6 | 7 | A common example in favor of this argument is the so-called "fragile base class problem". 8 | 9 | Imagine you have written the following class (using Python as an example here): 10 | 11 | ```python 12 | class Person: 13 | def __init__(self, name): 14 | self.name = name 15 | 16 | def say(self, text): 17 | print(f'{self.name}: {text}') 18 | 19 | def greet_casual(self): 20 | self.say("Hi.") 21 | 22 | def greet_formal(self): 23 | self.say("Hi.") 24 | self.say("How are you?") 25 | ``` 26 | 27 | And at some point you decide to change it like so: 28 | 29 | ```python 30 | class Person: 31 | def __init__(self, name): 32 | self.name = name 33 | 34 | def say(self, text): 35 | print(f'{self.name}: {text}') 36 | 37 | def greet_casual(self): 38 | self.say("Hi.") 39 | 40 | def greet_formal(self): 41 | self.greet_casual() # <--- Here is the change. 42 | self.say("How are you?") 43 | ``` 44 | 45 | This should be totally OK. It's fully compatible. 46 | Neither the interface nor the functionality changed. 47 | All unit tests are green. 48 | 49 | But with this change, you accidentally broke some other part of the project. 50 | A class is inheriting from your class: 51 | 52 | ```python 53 | class VeryPolitePerson(Person): 54 | def greet_casual(self): 55 | self.greet_formal() 56 | 57 | 58 | p = VeryPolitePerson("John") 59 | p.greet_casual() 60 | ``` 61 | 62 | Before your change, this worked fine. But now it results in the following: 63 | 64 | ```text 65 | RecursionError: maximum recursion depth exceeded 66 | ``` 67 | 68 | The occurrence of such a problem should not even be possible. 69 | However, support for inheriting implementation and method overriding 70 | allows for it to happen. 71 | 72 | OK, "It's only method overriding, which is evil." you might say. 73 | "Programming languages should make methods `final` or non-`virtual` by default, 74 | so users have to consciously opt-in to this feature when writing a class, 75 | which somebody might inherit from." 76 | 77 | Yes, but even without method overriding, 78 | just with implementation inheritance only, 79 | innocent behavior can lead to mean bugs. 80 | 81 | Let's say we write the following (this time with type annotations for clarity): 82 | 83 | ```python 84 | # library code 85 | from typing import List 86 | 87 | class ContainerBase: 88 | def __init__(self, initial_values: List[int]) -> None: 89 | self._values = initial_values 90 | ``` 91 | 92 | and then we add something: 93 | 94 | ```python 95 | # library code 96 | from typing import List 97 | 98 | class ContainerBase: 99 | def __init__(self, initial_values: List[int]) -> None: 100 | self._values = initial_values 101 | 102 | def clear(self) -> None: 103 | self._values = [] 104 | 105 | 106 | def do_something_with_container_and_clear(container: ContainerBase) -> None: 107 | # do something 108 | container.clear() 109 | ``` 110 | 111 | However, in between those two steps, somebody else wrote that: 112 | 113 | ```python 114 | # client code 115 | 116 | class SummingContainer(ContainerBase): 117 | def __init__(self) -> None: 118 | super(SummingContainer, self).__init__([]) 119 | self.sum = 0 120 | 121 | def add(self, value: int) -> None: 122 | self.sum += value 123 | self._values.append(value) 124 | ``` 125 | 126 | Our new function (`ContainerBase.clear`) will break an invariant 127 | of `SummingContainer`, i.e., `SummingContainer.sum` will no longer 128 | be guaranteed to reflect the sum of all numbers in the container. 129 | 130 | Thus `SummingContainer` can not be used with 131 | `do_something_with_container_and_clear`, 132 | This violates the [Liskov substitution principle](https://en.wikipedia.org/wiki/Liskov_substitution_principle), 133 | which is bad. 134 | 135 | A language not allowing implementation inheritance 136 | (or at least making every class `final` by default) 137 | could prevent this from happening. 138 | 139 | Code reuse can be achieved by other means. 140 | Composition (with [delegation](https://en.wikipedia.org/wiki/Delegation_pattern)) can provide one viable way to do so. 141 | -------------------------------------------------------------------------------- /programming_language_learning_curves/generate.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*- coding: ascii -*- 3 | """programming language learning curves 4 | """ 5 | __author__ = 'Tobias Hermann' 6 | __version__ = '1.0.0' 7 | 8 | from matplotlib import pyplot as plt 9 | import numpy as np 10 | import matplotlib.font_manager as fm 11 | import matplotlib 12 | 13 | prop = fm.FontProperties(fname='Humor-Sans.ttf', size=18) 14 | 15 | def annotate(text, pos, to): 16 | plt.annotate(text, xy=to, arrowprops=dict(arrowstyle='->'), xytext=pos, 17 | fontproperties=prop) 18 | 19 | def defaults1(): 20 | plt.xkcd() 21 | 22 | fig = plt.figure() 23 | ax = fig.add_subplot(1, 1, 1) 24 | ax.spines['right'].set_color('none') 25 | ax.spines['top'].set_color('none') 26 | plt.xticks([]) 27 | plt.yticks([]) 28 | 29 | ax.set_ylim([0,100]) 30 | ax.set_xlim([0,100]) 31 | 32 | def defaults2(title, loc='upper right', filename=None): 33 | plt.legend(prop=prop, loc=loc) 34 | plt.title(title, fontproperties=prop) 35 | plt.xlabel('experience', fontproperties=prop) 36 | if not filename: 37 | filename = title.lower() 38 | plt.savefig(filename.lower() + '.png', dpi=75) 39 | plt.clf() 40 | 41 | def javascript(): 42 | defaults1() 43 | 44 | annotate('callbacks', (5, 38), (30, 60)) 45 | annotate('callbacks', (5, 38), (30, 20)) 46 | 47 | x = [0, 30, 40, 100] 48 | y = [ [60, 60, 20, 20], [20, 20, 60, 60] ] 49 | labels = ['productivity', 'self-assessment'] 50 | 51 | for y_arr, label in zip(y, labels): 52 | plt.plot(x, y_arr, label=label) 53 | 54 | defaults2('Javascript') 55 | 56 | def python(): 57 | defaults1() 58 | 59 | annotate('unit tests', (40, 20), (30, 42)) 60 | annotate('decorators', (40, 70), (60, 55)) 61 | 62 | x1 = [ 0, 30, 35, 100] 63 | y1 = [40, 42, 50, 55] 64 | x2 = [ 0, 60, 65, 100] 65 | y2 = [50, 55, 60, 62] 66 | 67 | plt.plot(x1, y1, label='productivity') 68 | plt.plot(x2, y2, label='self-assessment') 69 | 70 | defaults2('Python') 71 | 72 | def lisp(): 73 | defaults1() 74 | 75 | annotate('macros', (30, 50), (50, 35)) 76 | annotate('macros', (30, 50), (50, 25)) 77 | 78 | x1 = [ 0, 50, 100] 79 | y1 = [10, 35, 80] 80 | x2 = [ 0, 50, 100] 81 | y2 = [ 5, 25, 70] 82 | 83 | plt.plot(x1, y1, label='productivity') 84 | plt.plot(x2, y2, label='self-assessment') 85 | 86 | defaults2('Lisp', 'upper left') 87 | 88 | def php(): 89 | defaults1() 90 | 91 | x1 = [ 0, 100] 92 | y1 = [10, 10] 93 | x2 = [ 0, 100] 94 | y2 = [90, 90] 95 | 96 | plt.plot(x1, y1, label='productivity') 97 | plt.plot(x2, y2, label='self-assessment') 98 | 99 | defaults2('php', 'center') 100 | 101 | def java(): 102 | defaults1() 103 | 104 | annotate('design patterns', (25, 20), (30, 30)) 105 | annotate('design patterns', (25, 20), (30, 35)) 106 | 107 | x1 = [ 0, 30, 100] 108 | y1 = [20, 30, 30] 109 | x2 = [ 0, 30, 100] 110 | y2 = [30, 35, 65] 111 | 112 | plt.plot(x1, y1, label='productivity') 113 | plt.plot(x2, y2, label='self-assessment') 114 | 115 | defaults2('Java') 116 | 117 | def cpp(): 118 | defaults1() 119 | 120 | annotate('templates', (25, 35), (50, 60)) 121 | 122 | x1 = [ 0, 100] 123 | y1 = [20, 30] 124 | x2 = [ 0, 50, 52, 100] 125 | y2 = [30, 60, 30, 90] 126 | 127 | plt.plot(x1, y1, label='productivity') 128 | plt.plot(x2, y2, label='self-assessment') 129 | 130 | defaults2('C++', 'upper left', 'cpp') 131 | 132 | def haskell(): 133 | defaults1() 134 | 135 | annotate('My brain hurts!', (1, 20), (2, 8)) 136 | annotate('My brain hurts!', (1, 20), (6, 8)) 137 | annotate('My brain hurts!', (1, 20), (10, 8)) 138 | annotate('My brain hurts!', (1, 20), (14, 8)) 139 | annotate('My brain hurts!', (1, 20), (18, 8)) 140 | annotate('My brain hurts!', (1, 20), (22, 8)) 141 | annotate('My brain hurts!', (1, 20), (26, 8)) 142 | 143 | annotate('Monads', (25, 40), (40, 20)) 144 | annotate('Monads', (25, 40), (40, 30)) 145 | 146 | annotate('Category theory', (65, 43), (70, 30)) 147 | 148 | x1 = [ 0, 5, 28, 40, 50, 100] 149 | y1 = [ 1, 1, 1, 30, 25, 100] 150 | x2 = [0, 2, 4, 6, 8,10,12,14,16,18,20,22,24,26,28, 40,42,44,46,48,50, 70, 72, 100] 151 | y2 = [3, 8, 3, 8, 3, 8, 3, 8, 3, 8, 3, 8, 3, 8, 3, 20, 3, 8, 3, 8, 3, 30, 14, 40] 152 | 153 | plt.plot(x1, y1, label='productivity') 154 | plt.plot(x2, y2, label='self-assessment') 155 | 156 | defaults2('Haskell', 'upper left') 157 | 158 | def main(): 159 | python() 160 | javascript() 161 | lisp() 162 | php() 163 | java() 164 | cpp() 165 | haskell() 166 | 167 | if __name__ == "__main__": 168 | main() -------------------------------------------------------------------------------- /how_i_got_rid_of_the_crosstalk_in_my_headset.md: -------------------------------------------------------------------------------- 1 | # How I got rid of the crosstalk in my headset 2 | 3 | Many modern headsets have a TRRS audio jack, combining the headphones and the microphone into one plug. 4 | 5 | ![trrs](how_i_got_rid_of_the_crosstalk_in_my_headset/trrs.jpg) 6 | 7 | While this is convenient for usage with smartphones, etc., it would be much better if headsets would provide two separate plugs, one for the headphones part and one for the microphone, 8 | 9 | ![two_plugs](how_i_got_rid_of_the_crosstalk_in_my_headset/two_plugs.jpg) 10 | 11 | and maybe add an adapter, 12 | 13 | ![adapter_merge](how_i_got_rid_of_the_crosstalk_in_my_headset/adapter_merge.jpg) 14 | 15 | because in the single-plug version, the headphone and the microphone (electrically) share the "ground", which makes the stuff you listen to (headphones) leak (crosstalk) into the stuff the others hear coming from you (microphone). 16 | - When gaming online while in a Discord channel, and you do something in the game, only you are supposed to hear, the others will hear it too. 17 | - When on a Zoom call, and you have some system sounds from notifications popping up (or prefer listening to music instead of your colleagues :-P), they hear it too. 18 | - ... 19 | 20 | Wait, you think, does this guy really state that there is a fundamental design flaw in the general concept of many headphone plugs? 21 | 22 | Yes, I do! :grin: 23 | 24 | And it's not a minor effect, which you only recognize when really paying attention. Instead, it's very obvious. I've tested this with 3 different headsets on two different PCs (front and back audio ports, with the needed [adapter](how_i_got_rid_of_the_crosstalk_in_my_headset/adapter_split.jpg) to match the [ports](how_i_got_rid_of_the_crosstalk_in_my_headset/soundcard.jpg)), and the effect was strong in all cases. I invite you to experiment yourself if you have such a headset: 25 | - Record your voice (use Audacity, a Zoom call, whatever). 26 | - Play some music on your headphones. 27 | - Listen to the recording and be surprised. 28 | 29 | ![recording_experiment](how_i_got_rid_of_the_crosstalk_in_my_headset/recording_experiment.jpg) 30 | 31 | If you are lucky and have a headset with two separate plugs, you can re-create this effect by coupling the two ground signals with adapters like this. 32 | 33 | ![share_ground_adapters](how_i_got_rid_of_the_crosstalk_in_my_headset/share_ground_adapters.jpg) 34 | 35 | **Conclusion: Don't ever buy a headset with a combined (TRRS) audio jack!** (Instead, buy something with two separate audio jacks, or USB, or wireless.) 36 | 37 | So here the general story already ends. But in case you're interested in how I got rid of this annoying effect in my day-to-day headset (Jabra Evolve 40 UC), please read on. :slightly_smiling_face: 38 | 39 | First, I opened it and was happily surprised, that the ground of the headphone and the microphone is not yet shared directly. Instead, they go through the cable as separate strands. 40 | 41 | ![opened_headset](how_i_got_rid_of_the_crosstalk_in_my_headset/opened_headset.jpg) 42 | 43 | ![strands](how_i_got_rid_of_the_crosstalk_in_my_headset/strands.jpg) 44 | 45 | Then, I cut off the jack 46 | 47 | ![cut_jack](how_i_got_rid_of_the_crosstalk_in_my_headset/cut_jack.jpg) 48 | 49 | melted off the insulation 50 | 51 | ![melted_off](how_i_got_rid_of_the_crosstalk_in_my_headset/melted_off.jpg) 52 | 53 | and then, using a multimeter 54 | 55 | ![multimeter](how_i_got_rid_of_the_crosstalk_in_my_headset/multimeter.jpg) 56 | 57 | I checked which colors are connected to what part of the jack. 58 | 59 | ![plug_labels](how_i_got_rid_of_the_crosstalk_in_my_headset/plug_labels.jpg) 60 | 61 | From that one can see the function of each colored strand. (The pink one is some proprietary Jabra thing, probably for their [USB-link thingy](how_i_got_rid_of_the_crosstalk_in_my_headset/jabra_usb.jpg). We can ignore it here.) 62 | 63 | ![plug_standards](how_i_got_rid_of_the_crosstalk_in_my_headset/plug_standards.jpg) 64 | 65 | So after un-insulating the strands in the cable going to the headset, I connected them to separate plugs with terminal blocks. (You can, of course, do this much more nicely and compactly by soldering, maybe even directly in the headset, replacing the cable completely.) 66 | 67 | ![terminal_blocks](how_i_got_rid_of_the_crosstalk_in_my_headset/terminal_blocks.jpg) 68 | 69 | Then, putting this all into some custom-made case with strain relief, 70 | 71 | ![strain_relief](how_i_got_rid_of_the_crosstalk_in_my_headset/strain_relief.jpg) 72 | 73 | And we have a stupidly huge box. (Luckily I only use the headset at my desk, so it does not matter.) 74 | 75 | ![box1](how_i_got_rid_of_the_crosstalk_in_my_headset/box1.jpg) ![box2](how_i_got_rid_of_the_crosstalk_in_my_headset/box2.jpg) ![box3](how_i_got_rid_of_the_crosstalk_in_my_headset/box3.jpg) 76 | 77 | It works perfectly. :tada: 78 | 79 | I have no idea how long it will last, but at least I'm crosstalk-free now. :sunglasses: 80 | 81 | (Next time, I'll definitely not buy a headset with such a TRRS plug. -_-) 82 | -------------------------------------------------------------------------------- /internals_of_the_async_await_pattern_from_first_principles/internals_of_the_async_await_pattern_from_first_principles.py: -------------------------------------------------------------------------------- 1 | import time 2 | from collections import deque 3 | from operator import itemgetter 4 | from select import select 5 | from socket import socket, AF_INET, SOCK_STREAM 6 | from typing import Deque, Any, Dict, Tuple, Awaitable, Generator 7 | 8 | 9 | class Executor: 10 | current = None 11 | _ready: Deque[Any] = deque() 12 | _scheduled: Deque[Any] = deque() 13 | _read_pending: Dict[Any, Any] = {} 14 | _write_pending: Dict[Any, Any] = {} 15 | 16 | def submit(self, coroutine) -> None: 17 | self._ready.append(coroutine) 18 | 19 | def schedule(self, timestamp, coroutine) -> None: 20 | self._scheduled.append((timestamp, coroutine)) 21 | # A real priority queue would do this automatically and more efficiently. 22 | self._scheduled = deque(sorted(self._scheduled, key=itemgetter(0))) 23 | 24 | def run(self) -> None: 25 | while any([self._ready, self._scheduled, self._read_pending, self._write_pending]): 26 | if not self._ready: 27 | timeout = None 28 | if self._scheduled: 29 | wakeup_time = self._scheduled[0][0] 30 | timeout = max(0, wakeup_time - time.time()) 31 | 32 | # Sleep till the timeout expires or a handle becomes ready. 33 | read_ready, write_ready, _ = select(self._read_pending.keys(), 34 | self._write_pending.keys(), 35 | [], timeout) 36 | for handle in read_ready: 37 | self.submit(self._read_pending.pop(handle)) 38 | for handle in write_ready: 39 | self.submit(self._write_pending.pop(handle)) 40 | 41 | now = time.time() 42 | while self._scheduled: 43 | if now >= self._scheduled[0][0]: 44 | self.submit(self._scheduled.pop()[1]) 45 | else: 46 | break 47 | 48 | self.current = self._ready.popleft() 49 | try: 50 | # Step the current coroutine for one state transition. 51 | self.current.send(None) 52 | # We only re-submit the coroutine when it was not removed/scheduled by async_sleep. 53 | if self.current: 54 | self.submit(self.current) 55 | self.current = None 56 | except StopIteration: 57 | pass 58 | 59 | async def accept(self, sock) -> Tuple[Any, Tuple[str, int]]: 60 | self._read_pending[sock] = executor.current 61 | self.current = None 62 | await YieldOnAwait() 63 | return sock.accept() 64 | 65 | async def recv(self, sock, max_bytes) -> bytes: 66 | self._read_pending[sock] = self.current 67 | self.current = None 68 | await YieldOnAwait() 69 | return sock.recv(max_bytes) 70 | 71 | async def send(self, sock, data) -> None: 72 | self._write_pending[sock] = self.current 73 | self.current = None 74 | await YieldOnAwait() 75 | return sock.send(data) 76 | 77 | 78 | executor = Executor() 79 | 80 | 81 | # A python technicality - we need an awaitable to switch tasks. 82 | class YieldOnAwait(Awaitable): 83 | def __await__(self) -> Generator: 84 | yield 85 | 86 | 87 | async def async_sleep(duration) -> None: 88 | executor.schedule(time.time() + duration, executor.current) 89 | executor.current = None 90 | await YieldOnAwait() 91 | 92 | 93 | def sync_sleep(duration) -> None: 94 | if duration > 0: 95 | time.sleep(duration) 96 | 97 | 98 | async def foo() -> None: 99 | print("a") 100 | await async_sleep(1) 101 | print("b") 102 | await async_sleep(1) 103 | print("c") 104 | 105 | 106 | async def bar() -> None: 107 | print("x") 108 | await async_sleep(1) 109 | print("y") 110 | await async_sleep(1) 111 | print("z") 112 | 113 | 114 | def open_socket(host: str, port: int) -> Any: 115 | sock = socket(AF_INET, SOCK_STREAM) 116 | sock.bind((host, port)) 117 | sock.listen(1) 118 | return sock 119 | 120 | 121 | async def tcp_server(): 122 | server_sock = open_socket("", 8080) 123 | while True: 124 | client_sock, address = await executor.accept(server_sock) 125 | print(f"Accepted connection from {address}") 126 | executor.submit(echo_handler(client_sock)) 127 | 128 | 129 | async def echo_handler(sock): 130 | while True: 131 | data = await executor.recv(sock, 10000) 132 | if not data: 133 | break 134 | await executor.send(sock, b'echo ' + data) 135 | print("Connection closed") 136 | sock.close() 137 | 138 | 139 | executor.submit(foo()) 140 | executor.submit(bar()) 141 | executor.submit(tcp_server()) 142 | executor.run() 143 | # Cou can try it out like that (linux): nc localhost 8080 144 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # [Dobiasd](https://github.com/dobiasd)'s articles 2 | 3 | This repository serves as a programming-related brain dump. 4 | 5 | If you enjoy my articles, feel free to [![buymeacoffee](https://www.buymeacoffee.com/assets/img/custom_images/orange_img.png)](https://buymeacoffee.com/tobiashermann). ;-) 6 | 7 | ### C++ 8 | 9 | [From goto to std::transform](https://github.com/Dobiasd/articles/blob/master/from_goto_to_std-transform.md) 10 | 11 | [Functional programming in C++ with the FunctionalPlus library; today: HackerRank challange "Gemstones"](https://github.com/Dobiasd/articles/blob/master/functional_programming_in_cpp_with_the_functionalplus_library_today_hackerrank_challange_gemstones.md) 12 | 13 | [Creating a replacement for the switch statement in C++ that also works for strings](https://github.com/Dobiasd/articles/blob/master/creating_a_replacement_for_the_switch_statement_in_cpp_that_also_works_for_strings.md) 14 | 15 | ### Elm 16 | 17 | [Switching from imperative to functional programming with games in Elm](https://github.com/Dobiasd/articles/blob/master/switching_from_imperative_to_functional_programming_with_games_in_Elm.md) 18 | 19 | ### Fun 20 | 21 | [Learning Curves (for different programming languages)](https://github.com/Dobiasd/articles/blob/master/programming_language_learning_curves.md) 22 | 23 | [Programming language subreddits and their choice of words](https://github.com/Dobiasd/programming-language-subreddits-and-their-choice-of-words) 24 | 25 | ### Haskell 26 | 27 | [From Object Oriented Programming to Functional Programming - Inheritance and the Expression Problem](https://github.com/Dobiasd/articles/blob/master/from_oop_to_fp_-_inheritance_and_the_expression_problem.md) 28 | 29 | ### Kotlin 30 | 31 | [What Kotlin could learn from C++'s keyword `const`](https://github.com/Dobiasd/articles/blob/master/what_kotlin_could_learn_from_cpps_keyword_const.md) 32 | 33 | [When does a when expression in Kotlin need to be exhaustive, and when does it not](https://github.com/Dobiasd/articles/blob/master/when_does_a_when_expression_in_kotlin_need_to_be_exhaustive_and_when_does_it_not.md) 34 | 35 | [From Spaghetti to Ravioli - a Refactoring Example (using Kotlin)](https://github.com/Dobiasd/articles/blob/master/from_spaghetti_to_ravioli_-_a_refactoring_example_using_kotlin.md) 36 | 37 | [Using `fold` to mimic mutation](https://github.com/Dobiasd/articles/blob/master/using_fold_to_mimic_mutation.md) 38 | 39 | ### Statistics 40 | 41 | [Basic measures of descriptive statistics](https://github.com/Dobiasd/articles/blob/master/basic_measures_of_descriptive_statistics.md) 42 | 43 | [Mechanical analogies for basic measures of descriptive statistics](https://github.com/Dobiasd/articles/blob/master/mechanical_analogies_for_basic_measures_of_descriptive_statistics.md) 44 | 45 | [Do A/B tests - because correlation does not imply causation](https://github.com/Dobiasd/articles/blob/master/do_a_b_tests_because_correlation_does_not_imply_causation.md) 46 | 47 | [Simple collaborative filtering in pure PostgreSQL](simple_collaborative_filtering_in_pure_postgresql.md) 48 | 49 | ### Misc 50 | 51 | [How touch typing and keyboard shortcuts can improve the quality of the software you create](https://github.com/Dobiasd/articles/blob/master/how_touch_typing_and_keyboard_shortcuts_can_improve_the_quality_of_the_software_you_create.md) 52 | 53 | [A too naive approach to video compression using artificial neural networks](https://github.com/Dobiasd/articles/blob/master/a_too_naive_approach_to_video_compression_using_artificial_neural_networks.md) 54 | 55 | [A personal, generic, things-I-learned-as-a-software-developer list](https://github.com/Dobiasd/articles/blob/master/a_personal_generic_things_i_learned_as_a_software_developer_list.md) 56 | 57 | [Implementation inheritance is bad - the fragile base class problem](https://github.com/Dobiasd/articles/blob/master/implementation_inheritance_is_bad_-_the_fragile_base_class_problem.md) 58 | 59 | [Refactoring suggestions are a compliment](https://github.com/Dobiasd/articles/blob/master/refactoring_suggestions_are_a_compliment.md) 60 | 61 | [Threads can infect each other with their low priority](https://github.com/Dobiasd/articles/blob/master/threads_can_infect_each_other_with_their_low_priority.md) 62 | 63 | [Why a generic implementation can be the easier-to-understand solution](https://github.com/Dobiasd/articles/blob/master/why_a_generic_implementation_can_be_the_easier_to_understand_solution.md) 64 | 65 | [Covariance and contravariance explained without code](https://github.com/Dobiasd/articles/blob/master/covariance_and_contravariance_explained_without_code.md) 66 | 67 | [How I got rid of the crosstalk in my headset](https://github.com/Dobiasd/articles/blob/master/how_i_got_rid_of_the_crosstalk_in_my_headset.md) 68 | 69 | [Avoid methods to reduce coupling](https://github.com/Dobiasd/articles/blob/master/avoid_methods_to_reduce_coupling.md) 70 | 71 | ["A monad is just a monoid in the category of endofunctors." - explained](https://github.com/Dobiasd/articles/blob/master/a_monad_is_just_a_monoid_in_the_category_of_endofunctors_explained.md) 72 | 73 | [Accurate timing of Strava segments](https://github.com/Dobiasd/articles/blob/master/accurate_timing_of_strava_segments.md) 74 | 75 | [Internals of the async/await pattern from first principles](https://github.com/Dobiasd/articles/blob/master/internals_of_the_async_await_pattern_from_first_principles.md) 76 | 77 | [On feeling deceived when receiving (non-labeled) LLM-generated messages](on_feeling_deceived_when_receiving_non_labeled_llm_generated_messages.md) 78 | 79 | [The expressive power of constraints](the_expressive_power_of_constraints.md) 80 | 81 | [LLM agents demystified](llm_agents_demystified.md) 82 | -------------------------------------------------------------------------------- /the_expressive_power_of_constraints.md: -------------------------------------------------------------------------------- 1 | # The expressive power of constraints 2 | 3 | Since we write code not primarily for the machine to execute, but mainly for the human reader to understand, we want our code to easily state what it does. One great way to achieve this is to let it state what it does *not*! When somebody looks into existing source code, initially, they don't know which of the endless things that could be going on, actually are. Reading the code allows them to narrow down their mental model to exactly one (hopefully the correct) possibility. Constraints can help a lot with that. 4 | 5 | 6 | ## Some examples 7 | 8 | 9 | ### Type annotations (trivial) 10 | 11 | Something like (fantasy language syntax) 12 | 13 | ``` 14 | function switch_something_on_or_off(x: Any) 15 | ``` 16 | 17 | allows `x` to be anything. It's better to restrict it 18 | 19 | ``` 20 | function switch_something_on_or_off(x: int) 21 | ``` 22 | 23 | and if we only need "on" and "off", using a type that only allows these two options is even better: 24 | 25 | ``` 26 | function switch_something_on_or_off(x: bool) 27 | ``` 28 | 29 | By reducing the cardinality of the set (the type), we reduce the possible states the program can have. 30 | 31 | 32 | ### Immutability (simple) 33 | 34 | When a variable (primitive or a complex object) is declared as constant, the reader immediately knows that it will never be mutated, because the type system guarantees it. 35 | 36 | 37 | ### Private fields/methods (basic) 38 | 39 | Every non-public thing in a class is guaranteed not to be accessed from the outside. 40 | 41 | 42 | ### Principle of Least Knowledge (classic) 43 | 44 | Given 45 | 46 | ``` 47 | class Foo 48 | name: Name 49 | age: int 50 | ... 51 | ``` 52 | 53 | this 54 | 55 | ``` 56 | function say_hello_to(name: Name) 57 | print("Hello, {name}.") 58 | ``` 59 | 60 | is better than 61 | 62 | ``` 63 | function say_hello_to(foo: Foo) 64 | print("Hello, {foo.name}.") 65 | ``` 66 | 67 | because passing only the data that the function actually needs to operate already tells the reader about all the things that the function does not do, e.g., doing something with `.age`, etc. 68 | 69 | 70 | ### Avoiding generic loops (functional) 71 | 72 | Given a list 73 | 74 | ``` 75 | xs = [1, 23, 42] 76 | ``` 77 | 78 | and we want to only keep the even elements. 79 | 80 | ``` 81 | ys = [] 82 | i = 0 83 | while i < length(xs) 84 | if is_even(xs[i]) 85 | ys.append(xs[i]) 86 | i += 1 87 | ``` 88 | 89 | is one way to do it, but obviously not the most obvious one. 90 | 91 | ``` 92 | ys = [] 93 | for x in xs: 94 | if is_even(x) 95 | ys.append(x) 96 | ``` 97 | 98 | already is better, because the `for` header already guarantees that no element of `xs` will be skipped or visited out of order by this loop. In the `while` version above, one needed to read the loop body to know this. 99 | 100 | A declarative 101 | 102 | ``` 103 | ys = filter(is_even, xs) 104 | ``` 105 | 106 | is even better, not just because it's shorter than the imperative options, but it guarantees that `ys` will never contain any element that was not present in `xs`. For this, one does not even need to know what `is_even` does. In the `for` example, one needed to look into the loop body to learn about that restriction. 107 | 108 | 109 | ### Abstraction with generics (fancier) 110 | 111 | If we implement a `filter` function like the one used above, we could have 112 | 113 | ``` 114 | function filter(predicate: (int -> bool), xs: List[int]) -> List[int] 115 | ... 116 | ``` 117 | 118 | however, the reader might wonder why it's specific to this list-element type, i.e., does the function utilize some special properties of `int`s? 119 | 120 | A generic version like this 121 | 122 | ``` 123 | function filter(predicate: (T -> bool), xs: List[T]) -> List[T] 124 | ... 125 | ``` 126 | 127 | on the other hand, states that the function does not do anything that is specific to some list-element type. Since the function knows nothing about `T` except what the type variable guarantees (in this case, just that it's some type), it cannot perform int-specific operations like arithmetic or comparison. In this case, it's somewhat the other way around than in the "type annotation" example: The less we know about the parameter type, the fewer things the function can possibly do. 128 | 129 | 130 | ### Library/tool selection (beyond code) 131 | 132 | When using some library (or any other tool), using one that is specialized for the task at hand can be a good idea. If some huge framework is pulled in to do something simple, the reader might ask themselves why this is the case, and that maybe some other things are happening in the code that they don't yet know about. By using a lib/tool that does just one thing, the possibilities are restricted, which is (according to this article) good for readability. 133 | 134 | For example, if you just need to parse dates, using a dedicated date-parsing library immediately tells the reader that only date parsing is happening. Suppose instead you import a full web framework that happens to include date utilities. In that case, the reader might wonder whether HTTP requests, database operations, or other framework features are also being used elsewhere in the code. 135 | 136 | 137 | ## Conclusion 138 | 139 | We've seen a common pattern: By restricting what our code *can* do, we make it clearer what our code *does* do. Whether through types, immutability, or specialized abstractions, constraints communicate intent. The reader spends less time wondering about possibilities and more time understanding reality. 140 | -------------------------------------------------------------------------------- /avoid_methods_to_reduce_coupling.md: -------------------------------------------------------------------------------- 1 | # Avoid methods to reduce coupling 2 | 3 | The following uses Python only as an example to show some code. The logic applies in many other programming languages too. 4 | (E.g., you can find the snippets translated to Kotlin [here](https://gist.github.com/Dobiasd/6f7324630f4e589ef8f6c5dccf9b31d1).) 5 | 6 | ## Global variables (make things complicated, yada yada yada) 7 | 8 | ```python3 9 | def foo(): 10 | x = 21 11 | bar() 12 | print(2 * x) 13 | 14 | def bar(): 15 | # imagine something very long nobody wants to read 16 | pass 17 | 18 | foo() 19 | ``` 20 | 21 | This is fine. No matter what `bar` does (ignoring exceptions, infinite loops, etc.), we know that `foo` will print `42`. `bar` can not influence that by changing the value of `x`. 22 | 23 | This would be different if `x` were a global variable: 24 | 25 | ```python3 26 | x = 21 27 | 28 | def foo(): 29 | bar() 30 | print(2 * x) 31 | 32 | def bar(): 33 | # imagine something very long nobody wants to read 34 | pass 35 | 36 | foo() 37 | ``` 38 | 39 | `bar` could change the value of x, so we're not sure what `foo` will print. 40 | 41 | So, of course, if not really needed, we try to avoid global variables (**rule 1**). 42 | 43 | 44 | ## Nested functions 45 | 46 | ```python3 47 | def foo(): 48 | x = 21 49 | 50 | def bar(): 51 | # imagine something very long nobody wants to read 52 | pass 53 | 54 | bar() 55 | print(2 * x) 56 | 57 | foo() 58 | ``` 59 | 60 | With `bar` being inside `foo`, here, again, `bar` could change the value of x (e.g., `nonlocal x; x = 1`), so we're not sure what `foo` will print. 61 | 62 | So, if not really needed, we should try to avoid nested functions (**rule 2**). 63 | 64 | 65 | ## Methods 66 | 67 | ```python3 68 | class Thing(): 69 | def __init__(self): 70 | self.x = 21 71 | 72 | def foo(self): 73 | self.bar() 74 | print(2 * self.x) 75 | 76 | def bar(self): 77 | # imagine something very long nobody wants to read 78 | pass 79 | 80 | my_thing = Thing() 81 | my_thing.foo() 82 | ``` 83 | 84 | With `bar` being inside `Thing`, here, again, `bar` could change the value of x (e.g., `self.x = 1`), so we're not sure what `foo` will print. 85 | 86 | So, in case `bar` does not use `self`, we should move it out of `Thing` and make it a free function. 87 | 88 | But even if it does use `self.something`, like this 89 | 90 | ```python3 91 | class Thing(): 92 | def __init__(self): 93 | self.x = 21 94 | self.y = 2 95 | 96 | def foo(self): 97 | self.bar() 98 | print(self.y * self.x) 99 | 100 | def bar(self): 101 | print(self.y) 102 | 103 | my_thing = Thing() 104 | my_thing.foo() 105 | ``` 106 | 107 | we should move it out of `Thing` 108 | 109 | ```python3 110 | class Thing(): 111 | def __init__(self): 112 | self.x = 21 113 | self.y = 2 114 | 115 | def foo(self): 116 | bar(self.y) 117 | print(self.y * self.x) 118 | 119 | def bar(y): 120 | print(y) 121 | 122 | my_thing = Thing() 123 | my_thing.foo() 124 | ``` 125 | 126 | or, make it static (which basically is the same thing in this context, despite the changed syntax at the call site) 127 | 128 | ```python3 129 | class Thing(): 130 | def __init__(self): 131 | self.x = 21 132 | self.y = 2 133 | 134 | def foo(self): 135 | Thing.bar(self.y) 136 | print(self.y * self.x) 137 | 138 | @staticmethod 139 | def bar(y): 140 | print(y) 141 | 142 | my_thing = Thing() 143 | my_thing.foo() 144 | ``` 145 | 146 | because this: 147 | - gives us the guarantee that `bar` will not mutate any of the member variables of our `Thing`. 148 | - makes the dependency (`bar` depends on a value for `y`) visible and explicit. 149 | - shows the reader that `bar` does not depend on something else from `Thing`. 150 | 151 | It simply makes it easier to reason about our code, because `bar` no longer is potentially coupled to all member variables (like a free function is couples to global variables). Also testing `bar` will be simpler, because we don't need to instantiate a `Thing` object anymore, just to invoke `bar`. 152 | 153 | So, if not really needed, we should try to avoid (non-static) methods (**rule 3**) too! 154 | 155 | When no fancy OOP stuff (implementation inheritance/polymorphism) is needed, the best idea might be to use the class only as a plain struct and keep the logic out. `Thing` can now even become a `NamedTuple`. 156 | 157 | ```python3 158 | from typing import NamedTuple 159 | 160 | class Thing(NamedTuple): 161 | x: int = 21 162 | y: int = 2 163 | z: int = 3 164 | 165 | # Or use a normal class and @staticmethod 166 | def foo(x, y): 167 | bar(y) 168 | print(y * x) 169 | 170 | def bar(y): 171 | print(y) 172 | 173 | my_thing = Thing() 174 | foo(my_thing.x, my_thing.y) 175 | ``` 176 | 177 | ## Conclusion 178 | 179 | When using methods, we suffer from a similar problem as we do when using nested functions or global variables, i.e., we lose guarantees about what a function will definitely **not** do. And the more of such guarantees we have, the lower the cognitive load is when maintaining (understanding/fixing/refactoring/extending) the code later. :-) 180 | 181 | This, of course, does not only apply to methods, but to any other constructs that take hidden inputs (member variables of an instance) or have hidden outputs (produce side effects). If possible, prefer pure functions, i.e. functions that only depend on the given input and have their return value as output, and isolate side effects. -------------------------------------------------------------------------------- /why_a_generic_implementation_can_be_the_easier_to_understand_solution.md: -------------------------------------------------------------------------------- 1 | # Why a generic implementation can be the easier-to-understand solution 2 | 3 | Following YAGNI, one might avoid implementing a generic version of some algorithm if it is only needed in one place and for one specific type. 4 | 5 | In this article, I'll argue that a generic implementation might be easier to understand when maintaining code, and thus should be preferred, if feasible. (I'll use Java in the code examples, but the concept applies to other languages, like C++, Swift, (type-annotated) Python, Kotlin, or C#, too.) 6 | 7 | Imagine you're working on the order system of a pizza delivery, which has a hierarchy like this: 8 | 9 | ```java 10 | interface Customer { 11 | long getId(); 12 | String getName(); 13 | // ... 14 | } 15 | 16 | interface SpecialCustomer extends Customer { 17 | boolean checkIfIsBirthday(LocalDate today); 18 | // Some other special things here. 19 | } 20 | ``` 21 | 22 | You're tasked with implementing a feature that prints all special customers, and you implement something like the following: 23 | 24 | ```java 25 | static void printSpecialCustomers(List specialCustomers) { 26 | specialCustomers.forEach((customer) -> { 27 | System.out.println(String.format("%s: %s", customer.getId(), customer.getName())); 28 | }); 29 | } 30 | ``` 31 | 32 | ```java 33 | printSpecialCustomers(ourSpecialCustomers); 34 | ``` 35 | 36 | Since only methods of `Customer` are used, a future maintainer might wonder, why the function takes `List` and not `List`. Since `SpecialCustomer` is a more complex type compared to `Customer`, using it exposes more (but unneeded) possibilities to the function. The maintainer might spend time trying to find out why this was done, just to conclude that is was not needed. The following implementation would have avoided this confusion. 37 | 38 | ```java 39 | static void printCustomers(List customers) { 40 | customers.forEach((customer) -> { 41 | System.out.println(String.format("%s: %s", customer.getId(), customer.getName())); 42 | }); 43 | } 44 | ``` 45 | 46 | So even though the function is never used for `Customer`s, which are not `SpecialCustomer`s, it is the better solution, because the parameter type is more restrictive in what possible things the function could do. But it is not just helpful when maintaining the function implementation, but also when just using is, because having `Customer` instead of `SpecialCustomer` already tell the client-code developer about all the things, this function is guaranteed to not be able to do, which is all the particular `SpecialCustomer` things. Thus using `Customer` instead of `SpecialCustomer` has made things simpler. 47 | 48 | And we don't need to stop here with this line of reasoning. Let's imagine the typical pizza-delivery-business logic of having a non-reliable order queue shall be implemented. Maybe it swaps order from time to time or even randomly drops an order now and then. 49 | 50 | ```java 51 | interface Order { 52 | Customer getCustomer(); 53 | List getOrderLines(); 54 | } 55 | ``` 56 | 57 | ```java 58 | class ShittyOrderQueue { 59 | private List orders; 60 | 61 | public void add(Order order) { 62 | // todo: Sometimes just don't add the order. 63 | } 64 | 65 | public Optional poll() { 66 | // todo: Sometimes return not from the front of the queue. 67 | } 68 | } 69 | ``` 70 | 71 | People using a class that implements this interface, or reading its implementation, might wonder why it is constrained to work with `Order`s. "Wouldn't it also work with arbitrary other Objects? And if so, why isn't it implemented that way? There must be a reason for it, and I should find out to avoid doing something unintended." 72 | 73 | So it is much better to implement the queue it in a generic way, even though it will actually be used for `Order`s exclusively. 74 | 75 | ```java 76 | class ShittyQueue { 77 | private List items; 78 | 79 | public void add(T item) { 80 | // todo: Sometimes just don't add the item. 81 | } 82 | 83 | public Optional poll() { 84 | // todo: Sometimes return not from the front of the queue. 85 | } 86 | } 87 | ``` 88 | 89 | (Using a generic type is just like using a maximally general interface.) 90 | 91 | Basically, with a bit of squinting, we can look at the version specific to `Order` (`ShittyOrderQueue`) as if it was a generic implementation, just with a very strong constraint. 92 | 93 | ```java 94 | class ShittyOrderQueue { 95 | // ... 96 | } 97 | ``` 98 | 99 | This way makes the issue obvious since you would not use other non-needed constraints, like 100 | 101 | ```java 102 | class ShittyQueue { 103 | // ... 104 | } 105 | ``` 106 | 107 | either, because your fellow developers (including future you) would unnecessarily try to understand why `T` needs to implement the `Comparable` interface. (In languages like Scala or Rust, one would have traits here instead, but the concept is the same.) 108 | 109 | - constrained type -> unconstrained possibilities -> more cognitive load 110 | - uncontrained type -> constrained possibilities -> less cognitive load 111 | 112 | So the rule "*Always let your functions take (and return) the widest possible types as parameters.*" not only makes sense for re-usability but also for clarity. 113 | 114 | In conclusion, generics, as abstraction in general, can reduce complexity. They can be helpful even if the implemented class/function is never going to be used with more than one type, because being less concrete with the types allows the code to concretely express what it does not do, resulting in decreased cognitive load. 115 | -------------------------------------------------------------------------------- /covariance_and_contravariance_explained_without_code.md: -------------------------------------------------------------------------------- 1 | # Covariance and contravariance explained without code 2 | 3 | Covariance and contravariance are concepts one can bump into (and initially be confused by) when working with object-oriented programming. This article explains the basic idea without requiring any programming knowledge. 4 | 5 | Imagine you are a merchant of large industrial things, and you sell the following two types of products: 6 | - `factory`: A thing that produces (spits out) stuff. 7 | - `crusher`: A thing that consumes stuff, i.e., you can throw stuff, you want to get rid of, in it. 8 | 9 | Your customers are mainly interested in vehicles, some especially in bikes. And it's common wisdom, that: 10 | 11 | - Every `bike` is a `vehicle`. (But not every `vehicle` is a `bike`.) 12 | 13 | ``` 14 | ----------- 15 | | vehicle | 16 | ----------- 17 | ^ 18 | | 19 | | is a 20 | | 21 | -------- 22 | | bike | 23 | -------- 24 | ``` 25 | 26 | ## Covariance 27 | 28 | Your customer John wants to purchase a `vehicle factory`. He does not care what kind of vehicles it produces. 29 | You supply him with a `bike factory`, and he is happy because a `bike factory` is a `vehicle factory`. It's just a special kind, which is ok. 30 | 31 | ![(john_bike_factory)](covariance_and_contravariance_explained_without_code/john_bike_factory.jpg) 32 | 33 | It would not have worked the other way around! If John would have wanted a `bike factory` and you would have delivered a `vehicle factory` instead, he would be angry, because one can not control which type of random vehicle it produces. 34 | 35 | ![(john_vehicle_factory)](covariance_and_contravariance_explained_without_code/john_vehicle_factory.jpg) 36 | 37 | This is covariance in action: 38 | - Every `bike` is a `vehicle`. (But not every `vehicle` is a `bike`.) 39 | - Every `bike factory` is a `vehicle factory`. (But not every `vehicle factory` is a `bike factory`.) 40 | 41 | ``` 42 | ----------- ------------------- 43 | | vehicle | | vehicle factory | 44 | ----------- ------------------- 45 | ^ ^ 46 | | | 47 | | is a | is a 48 | | | 49 | -------- ---------------- 50 | | bike | | bike factory | 51 | -------- ---------------- 52 | ``` 53 | 54 | (Both "is a" arrows point in the same direction.) 55 | 56 | ## Contravariance 57 | 58 | Now, your next customer, Jane, wants to buy a `bike crusher` from you. You don't have one on stock, so you provide a generic `vehicle crusher` instead, which is totally ok. 59 | 60 | ![(jane_vehicle_crusher)](covariance_and_contravariance_explained_without_code/jane_vehicle_crusher.jpg) 61 | 62 | It would not have worked the other way around! If Jane would have wanted a `vehicle crusher` and you would have delivered a `bike crusher` instead, she would be angry. 63 | 64 | ![(jane_bike_crusher)](covariance_and_contravariance_explained_without_code/jane_bike_crusher.jpg) 65 | 66 | This is contravariance in action: 67 | - Every `bike` is a `vehicle`. (But not every `vehicle` is a `bike`.) 68 | - Every `vehicle crusher` is a `bike crusher`. (But not every `bike crusher` is a `vehicle crusher`.) 69 | 70 | ``` 71 | ----------- ------------------- 72 | | vehicle | | vehicle crusher | 73 | ----------- ------------------- 74 | ^ | 75 | | | is a 76 | | is a | 77 | | v 78 | -------- ---------------- 79 | | bike | | bike crusher | 80 | -------- ---------------- 81 | ``` 82 | 83 | (The two "is a" arrows point in opposite directions.) 84 | 85 | ## The difference between a `factory` and a `crusher` 86 | 87 | The crucial point is already in the definitions. 88 | - A `factory` produces. 89 | - A `crusher` consumes. 90 | 91 | Producing things (in programming: functions returning the object in question) exhibit covariance with the class of the object (`vehicle`/`bike`). 92 | 93 | Consuming things (in programming: functions taking the object in question as an argument) exhibit contravariance with the class of the object (`vehicle`/`bike`). 94 | 95 | ## Additional terms 96 | 97 | - Invariance/Nonvariance: If there is no arrow in any of the two directions, the relationship is invariant. 98 | - Bivariance: There are arrows in both directions. 99 | - Variance: Variance is given if the relationship is either covariant, contravariant, or bivariant. 100 | 101 | ## Code (if you really want to see it) 102 | 103 | The below example is written in Kotlin, but the same logic applies to many other languages too. 104 | 105 | ```kotlin 106 | interface Vehicle 107 | 108 | interface Bike : Vehicle 109 | 110 | // Covariance 111 | 112 | interface VehicleFactory { 113 | fun produce(): Vehicle 114 | } 115 | 116 | interface BikeFactory : VehicleFactory { 117 | override fun produce(): Bike 118 | } 119 | 120 | // Contravariance 121 | 122 | interface BikeCrusher { 123 | fun consume(bike: Bike) 124 | } 125 | 126 | interface VehicleCrusher : BikeCrusher { 127 | fun consume(vehicle: Vehicle) 128 | } 129 | 130 | fun john(factory: VehicleFactory) { 131 | val someVehicle = factory.produce() 132 | } 133 | 134 | fun jane(crusher: BikeCrusher) { 135 | val someBike: Bike = ... 136 | crusher.consume(someBike) 137 | } 138 | 139 | 140 | val someVehicleFactory: VehicleFactory = ... 141 | val someBikeFactory: BikeFactory = ... 142 | val someVehicleCrusher: VehicleCrusher = ... 143 | val someBikeCrusher: BikeCrusher = ... 144 | 145 | john(someVehicleFactory) 146 | john(someBikeFactory) 147 | jane(someVehicleCrusher) 148 | jane(someBikeCrusher) 149 | ``` 150 | 151 | ([discussion on reddit](https://www.reddit.com/r/programming/comments/iqqtp0/covariance_and_contravariance_explained_without/)) 152 | -------------------------------------------------------------------------------- /what_kotlin_could_learn_from_cpps_keyword_const.md: -------------------------------------------------------------------------------- 1 | # What Kotlin could learn from C++'s keyword `const` 2 | 3 | Let's say, in Kotlin we have some simple 2D-vector class. 4 | It has two methods, one (`normalize`) which mutates the object, 5 | and one (`length`) which does not: 6 | 7 | ```kotlin 8 | import kotlin.math.sqrt 9 | 10 | class Vector(private var x: Double, private var y: Double) { 11 | fun length() = sqrt(x * x + y * y) 12 | fun normalize() { 13 | val l = length() 14 | x /= l 15 | y /= l 16 | } 17 | } 18 | ``` 19 | 20 | Imagine `Vector` being some kind of domain entity (with an ID, etc.), 21 | such that `normalize` needs to mutate it 22 | instead of just returning a new, immutable normalized `Vector` (value object). 23 | 24 | We could now use it like that: 25 | 26 | ```kotlin 27 | fun foo(v: Vector) { 28 | v.normalize() 29 | } 30 | 31 | fun bar(v: Vector) { 32 | println(v.length()) 33 | } 34 | 35 | fun main() { 36 | val myVector = Vector(3.0, 4.0) 37 | foo(myVector) 38 | bar(myVector) 39 | } 40 | ``` 41 | 42 | However, we might want to make sure `bar` can not mutate our object. 43 | It should only be allowed to call the non-mutating methods. 44 | To achieve this, we need to provide two different classes: 45 | 46 | ```kotlin 47 | open class Vector(protected var x: Double, protected var y: Double) { 48 | fun length() = sqrt(x * x + y * y) 49 | } 50 | 51 | class MutableVector(x: Double, y: Double) : Vector(x, y) { 52 | fun normalize() { 53 | val l = super.length() 54 | x /= l 55 | y /= l 56 | } 57 | } 58 | ``` 59 | 60 | Now we can adjust the rest of our code, such that the desired property of `bar` is satisfied: 61 | 62 | ```kotlin 63 | fun foo(v: MutableVector) { 64 | v.normalize() 65 | } 66 | 67 | fun bar(v: Vector) { 68 | println(v.length()) 69 | } 70 | 71 | fun main() { 72 | val myVector = MutableVector(3.0, 4.0) 73 | foo(myVector) 74 | bar(myVector) 75 | } 76 | ``` 77 | 78 | [`MutableList`](https://kotlinlang.org/api/latest/jvm/stdlib/kotlin.collections/-mutable-list/index.html) 79 | and [`List`](https://kotlinlang.org/api/latest/jvm/stdlib/kotlin.collections/-list/index.html) 80 | in Kotlin's standard library follow a similar approach. 81 | Actually, Kotlin is in good company here. 82 | Other languages (not just Java) utilize similar mechanisms. 83 | 84 | So, let's translate this straight into C++: 85 | 86 | ```cpp 87 | #include 88 | #include 89 | 90 | class vector { 91 | public: 92 | vector(double x, double y): x_(x), y_(y) {} 93 | double length() { 94 | return std::sqrt(x_ * x_ + y_ * y_); 95 | } 96 | protected: 97 | double x_; 98 | double y_; 99 | }; 100 | 101 | class mutable_vector: public vector { 102 | public: 103 | mutable_vector(double x, double y): vector(x, y) {} 104 | void normalize() { 105 | auto l = length(); 106 | x_ /= l; 107 | y_ /= l; 108 | } 109 | }; 110 | 111 | void foo(mutable_vector& v) { 112 | v.normalize(); 113 | } 114 | 115 | void bar(vector& v) { 116 | std::cout << v.length() << std::endl; 117 | } 118 | 119 | int main() { 120 | mutable_vector my_vector(3.0, 4.0); 121 | foo(my_vector); 122 | bar(my_vector); 123 | } 124 | ``` 125 | 126 | Looks fine. However, in a code review, 127 | this would raise a huge laugh because C++ provides a much better solution. 128 | Using the [`const`](https://en.cppreference.com/w/cpp/keyword/const) keyword, 129 | one can let the compiler do the tedious work of providing two different interfaces! 130 | 131 | It looks as follows: 132 | 133 | ```cpp 134 | #include 135 | #include 136 | 137 | class vector { 138 | public: 139 | vector(double x, double y): x_(x), y_(y) {} 140 | double length() const { 141 | return std::sqrt(x_ * x_ + y_ * y_); 142 | } 143 | void normalize() { 144 | auto l = length(); 145 | x_ /= l; 146 | y_ /= l; 147 | } 148 | private: 149 | double x_; 150 | double y_; 151 | }; 152 | 153 | void foo(vector& v) { 154 | v.normalize(); 155 | } 156 | 157 | void bar(const vector& v) { 158 | std::cout << v.length() << std::endl; 159 | } 160 | 161 | int main() { 162 | vector my_vector(3.0, 4.0); 163 | foo(my_vector); 164 | bar(my_vector); 165 | } 166 | ``` 167 | 168 | By marking `vector::length` as const, but not `vector::normalize`, 169 | the compiler knows which member functions can be called 170 | depending on the `const` qualification of an instance or a reference to one. 171 | 172 | `bar` now takes a reference-to-const parameter, 173 | which produces the exact effect we wanted, 174 | i.e., `v` is immutable and trying to call `v.normalize()` in its body would result in a compile-time error: 175 | 176 | ```cpp 177 | void bar(const vector& v) { 178 | v.normalize(); 179 | } 180 | ``` 181 | 182 | Error (generated by `clang++`): 183 | 184 | ```text 185 | error: 'this' argument to member function 'normalize' has type 'const vector', but function is not marked const 186 | v.normalize(); 187 | ^ 188 | ``` 189 | 190 | :tada: 191 | 192 | Good old C++, despite all its idiosyncrasies, seems to be doing something right here. :) 193 | 194 | ([Rust too.](https://gist.github.com/Dobiasd/4d44fc681c6a51dc2e516f682e8e9366)) 195 | 196 | I'd love to hear your [feedback](https://www.reddit.com/r/Kotlin/comments/cixncc/what_kotlin_could_learn_from_cs_keyword_const/). 197 | 198 | --- 199 | 200 | Edit: The Kotlin community seems to already discuss such a possible feature 201 | ([1](https://discuss.kotlinlang.org/t/transitive-const/576), 202 | [2](https://discuss.kotlinlang.org/t/what-is-meant-by-immutable-data/3294), 203 | [3](https://discuss.kotlinlang.org/t/object-immutability/6875)), 204 | however, it seems to be quite tricky to integrate it into the language. 205 | I hope somebody finds a clean way, 206 | because I'd really love to have this in my day-to-day work with Kotlin. 207 | -------------------------------------------------------------------------------- /basic_measures_of_descriptive_statistics.md: -------------------------------------------------------------------------------- 1 | # Basic measures of descriptive statistics 2 | 3 | Let's assume you want to analyze the distribution of one variable (univariate analysis), like the height of people, the sum of the value of purchases per user in your online shop, or the number of cats per house in your neighborhood. In any case, you always end up with one (sometimes quite large) a list of numbers. 4 | 5 | To go with the online-shop example, assume your users made 7 purchases and paid the following amounts of money (the actual unit, i.e., dollars, can be omitted here): 6 | `[4, 17, 2, 7, 18, 4, 11]` 7 | 8 | The histogram looks as follows: 9 | 10 | ![value_histogram](basic_measures_of_descriptive_statistics/few_values_histogram.png) 11 | 12 | What follows are some basic statistical measures and how they are calculated. 13 | 14 | ## Location parameters 15 | 16 | The most common measures of central tendency. 17 | 18 | ### Mean 19 | 20 | The (arithmetic) mean is just the sum of all values divided by the number of values. 21 | 22 | `mean = (4 + 17 + 2 + 7 + 18 + 4 + 11) / 7 = 9` 23 | 24 | So it represents what one might call the "average amount spent in the shop". 25 | 26 | ### Median 27 | 28 | The median is the value of the middle position after sorting the numbers. 29 | 30 | `median = middle_value([2, 4, 4, 7, 11, 17, 18]) = 7` 31 | 32 | While from the mean amount and the number of users we can, of course, get back to the overall amount easily: `overall_amount = mean * number_of_values`, from the median this is not possible. However, the median better represents the *typical* value. It is less vulnerable to outliers, i.e., the medians of `[2, 4, 4, 7, 11, 17, 18]` and `[2, 4, 4, 7, 11, 17, 918]` are the same (`7`), while the mean differs a lot. The median is also called the 50th percentile because 50% of the values are below it (and the other 50% are above). One can think of it as the value that most other values are clustered around. 33 | 34 | ### Mode 35 | 36 | The mode is the value which appears most often. In our example: 37 | `mode([2, 4, 4, 7, 11, 17, 18]) = 4` 38 | 39 | ## Scale parameters 40 | 41 | ### Variance 42 | 43 | The variance is the mean of the squared distances of the values to the distribution's mean. In our example the mean is `9` so we end up with: 44 | 45 | ``` 46 | variance = mean([(2-9)², (4-9)², (4-9)², (7-9)², (11-9)², (17-9)², (18-9)²]) 47 | = mean([49, 25, 25, 4, 4, 64, 81]) 48 | = 36 49 | ``` 50 | 51 | One of the reasons to have the distances squared instead of plain, is to weight outliers more. (Others include advanced properties like being continuously differentiable etc.) 52 | 53 | The intuition for it is: A small variance means, the values, in general, are quite close to each other, while a big variance indicates a more spread-out distribution. 54 | 55 | While, e.g., `[1, 4, 7]` and `[3, 4, 5]` have the same mean and median (both `3`), their variance differs significantly. 56 | 57 | ## Standard deviation 58 | 59 | The standard deviation is simply the square root of the variance. 60 | `standard_deviation = sqrt(variance) = sqrt(36) = 6` 61 | 62 | Qualitatively, the standard deviation tells the same fact as the variance does. However, often the standard deviation is preferred because it's unit is the same as the one of the original distribution. So in our online-shop example, the standard deviation is 6 dollars. 63 | 64 | ## Probability density function 65 | 66 | Your online shop grows, and you get more samples (purchases). The histogram now might look as follows: 67 | 68 | ![many_values_histogram](basic_measures_of_descriptive_statistics/many_values_histogram.png) 69 | 70 | Now we can fit a curve to it and normalize it in such a way, that the area under the curve (the integral) equals `1`. The result is called the probability density function (PDF). It represents the probability (y-axis) of purchase with a certain dollar value to occur. In that particular case, it has a positive skew, i.e., a longer tail at the right side. 71 | 72 | ![pdf](basic_measures_of_descriptive_statistics/pdf.png) 73 | 74 | ## Cumulative distribution function 75 | 76 | The integral of the PDF is the CDF (cumulative distribution function). It shows the probability that a random sample will have a value less than or equal to x. 77 | 78 | ![cdf](basic_measures_of_descriptive_statistics/cdf.png) 79 | 80 | What's nice about it, that we can simply read the median, i.e., the value with 1/2 of the samples on each side of it, from it. 81 | 82 | ![cdf_with_median](basic_measures_of_descriptive_statistics/cdf_with_median.png) 83 | 84 | Due to this property, the median is also called the "50th percentile". Other percentiles, e.g., the 90th one, are also often used. They can quickly be read from the CDF in the same way. 85 | 86 | ## Scale parameters (revisited) 87 | 88 | ### Interquartile range 89 | 90 | The Quartiles are the three cuts, dividing a distribution into four groups of the same size: 91 | - 25th percentile 92 | - 50th percentile (median) 93 | - 75th percentile 94 | 95 | We can use them to overcome a problem of the variance (or standard deviation), i.e., weighting outliers too much in certain situations. Distributions with long thin tails (high "kurtosis") in either direction can bloat those measures quite a lot, which may be unwanted. 96 | 97 | If we instead use the range from the 25th percentile to the 75th percentile as a measure of how spread out the distribution is, we are more resilient towards outliers. 98 | 99 | ![iqr](basic_measures_of_descriptive_statistics/iqr.png) 100 | 101 | This range is called the IQR (interquartile range). 102 | 103 | `IQR = 75th_percentile - 25th_percentile ≈ 11.6844 - 4.8349 = 6.8495` 104 | 105 | One way to eliminate outliers completely before calculating the other measures would be to only consider the values in the IQR. This process is called a 50% winsorization. A more typical value for winsorizing however would be 90%, i.e., dropping only the bottom and top 5% of the initial values. 106 | 107 | --- 108 | 109 | In case you are interested, here is a short follow-up article: [Mechanical analogies for basic measures of descriptive statistics](mechanical_analogies_for_basic_measures_of_descriptive_statistics.md) -------------------------------------------------------------------------------- /when_does_a_when_expression_in_kotlin_need_to_be_exhaustive_and_when_does_it_not.md: -------------------------------------------------------------------------------- 1 | # When does a when expression in Kotlin need to be exhaustive, and when does it not 2 | 3 | Recently, while working with Kotlin (version `1.3.50`), a colleague and I recognized 4 | that we were unable to predict, in what cases, 5 | and in what way (error or just a recommendation), 6 | the IDE (IntelliJ IDEA version `2019.2.1`) does complain about non-exhaustive when expressions. 7 | 8 | So I ran a short experiment, of which I'll present the results here. 9 | 10 | The general test pattern looks as follows: 11 | 12 | ```kotlin 13 | fun singleExperiment(x: EnumOrClass) { 14 | when (x) { 15 | Foo.A -> { 16 | someFunction() 17 | } 18 | } 19 | } 20 | ``` 21 | 22 | So, the first thing to compare is `x` being 23 | an instance of a derived class (passed as base-class reference) 24 | or an enum: 25 | 26 | ```kotlin 27 | enum class Foo { A, B } 28 | 29 | open class Base 30 | class Derived1 : Base() 31 | class Derived2 : Base() 32 | ``` 33 | 34 | Also `someFunction` can return a value `Unit` or some actual type: 35 | 36 | ```kotlin 37 | fun printSomething() = println(42) 38 | 39 | fun returnSomething(): Int = 42 40 | ``` 41 | 42 | The third thing to explore is 43 | whether it makes a difference to assign the expression to a variable: 44 | 45 | ```kotlin 46 | val y = when (x) { 47 | ... 48 | } 49 | ``` 50 | 51 | This may sound absurd, but please bear with me. It will become relevant soon. 52 | 53 | Last, we will check if using `let` in the following way changes things: 54 | 55 | ```kotlin 56 | x.let { 57 | when (it) { 58 | ... 59 | } 60 | } 61 | ``` 62 | 63 | So, we have 4 binary dimensions in our experiment space. 64 | 65 | - `E`/`C` = enum or class (type of `x`) 66 | - `N`/`L` = normal or let (way `x` is used) 67 | - `S`/`A` = statement or assignment (usage of `when` expression) 68 | - `U`/`V` = unit or value (`someFunction`'s return type) 69 | 70 | The result of each possible combination can be: 71 | 72 | - `O` = OK 73 | - `R` = Recommendation only 74 | - `F` = Failure (compilation error) 75 | 76 | Let's explore all 16 corners of this hypercube one by one. 77 | 78 | ![ensu](when_does_a_when_expression_in_kotlin_need_to_be_exhaustive_and_when_does_it_not/ensu.png) 79 | 80 | The given recommendation looks as follows: 81 | 82 | ![recommendation](when_does_a_when_expression_in_kotlin_need_to_be_exhaustive_and_when_does_it_not/recommendation.png) 83 | 84 | ![cnsu](when_does_a_when_expression_in_kotlin_need_to_be_exhaustive_and_when_does_it_not/cnsu.png) 85 | 86 | ![ensv](when_does_a_when_expression_in_kotlin_need_to_be_exhaustive_and_when_does_it_not/ensv.png) 87 | 88 | ![cnsv](when_does_a_when_expression_in_kotlin_need_to_be_exhaustive_and_when_does_it_not/cnsv.png) 89 | 90 | ![elsu](when_does_a_when_expression_in_kotlin_need_to_be_exhaustive_and_when_does_it_not/elsu.png) 91 | 92 | ![clsu](when_does_a_when_expression_in_kotlin_need_to_be_exhaustive_and_when_does_it_not/clsu.png) 93 | 94 | ![elsv](when_does_a_when_expression_in_kotlin_need_to_be_exhaustive_and_when_does_it_not/elsv.png) 95 | 96 | The error message looks as follows: 97 | 98 | ![error](when_does_a_when_expression_in_kotlin_need_to_be_exhaustive_and_when_does_it_not/error.png) 99 | 100 | ![clsv](when_does_a_when_expression_in_kotlin_need_to_be_exhaustive_and_when_does_it_not/clsv.png) 101 | 102 | ![enau](when_does_a_when_expression_in_kotlin_need_to_be_exhaustive_and_when_does_it_not/enau.png) 103 | 104 | ![cnau](when_does_a_when_expression_in_kotlin_need_to_be_exhaustive_and_when_does_it_not/cnau.png) 105 | 106 | ![enav](when_does_a_when_expression_in_kotlin_need_to_be_exhaustive_and_when_does_it_not/enav.png) 107 | 108 | ![cnav](when_does_a_when_expression_in_kotlin_need_to_be_exhaustive_and_when_does_it_not/cnav.png) 109 | 110 | ![elau](when_does_a_when_expression_in_kotlin_need_to_be_exhaustive_and_when_does_it_not/elau.png) 111 | 112 | ![clau](when_does_a_when_expression_in_kotlin_need_to_be_exhaustive_and_when_does_it_not/clau.png) 113 | 114 | ![elav](when_does_a_when_expression_in_kotlin_need_to_be_exhaustive_and_when_does_it_not/elav.png) 115 | 116 | ![clav](when_does_a_when_expression_in_kotlin_need_to_be_exhaustive_and_when_does_it_not/clav.png) 117 | 118 | That was interesting and somewhat surprising (at least to me). 119 | 120 | To finalize, here is a summary table to of the results: 121 | 122 | | e/c | n/l | s/a | u/v | o/r/f | 123 | |-------|--------|------------|-------|----------------| 124 | | enum | normal | statement | unit | recommendation | 125 | | class | normal | statement | unit | ok | 126 | | enum | normal | statement | value | recommendation | 127 | | class | normal | statement | value | ok | 128 | | enum | let | statement | unit | recommendation | 129 | | class | let | statement | unit | ok | 130 | | enum | let | statement | value | failure | 131 | | class | let | statement | value | failure | 132 | | enum | normal | assignment | unit | failure | 133 | | class | normal | assignment | unit | failure | 134 | | enum | normal | assignment | value | failure | 135 | | class | normal | assignment | value | failure | 136 | | enum | let | assignment | unit | recommendation | 137 | | class | let | assignment | unit | ok | 138 | | enum | let | assignment | value | failure | 139 | | class | let | assignment | value | failure | 140 | 141 | You can find the full source code to play around with [here](https://gist.github.com/Dobiasd/a7292aaf8f818e3303a2cfc55c69c6ab). 142 | 143 | Thanks to very helpful people on reddit ([1](https://www.reddit.com/r/Kotlin/comments/cura4x/when_does_a_when_expression_in_kotlin_need_to_be/), [2](https://www.reddit.com/r/programming/comments/cura13/when_does_a_when_expression_in_kotlin_need_to_be/)) we can read an explanation for these observations [there](https://www.reddit.com/r/Kotlin/comments/cura4x/when_does_a_when_expression_in_kotlin_need_to_be/exzm7li/). 144 | They show the pattern behind the behavior that seems chaotic when only observed superficially like here. 145 | -------------------------------------------------------------------------------- /accurate_timing_of_strava_segments.md: -------------------------------------------------------------------------------- 1 | # Accurate timing of Strava segments 2 | 3 | ## tl;dr 4 | 5 | [Strava](https://en.wikipedia.org/wiki/Strava "Strava is an app for tracking physical exercise, for example, bike rides. 🚴") calculates [segment](https://support.strava.com/hc/en-us/articles/216918167-Strava-Segments "Segments are portions of roads or trails where athletes can compare times and thus compete. 👑") times in an unnecessarily inaccurate way, resulting in unfair leaderboards. But there's an elegant way to improve the situation. 🌈 6 | 7 | ## Problem 8 | 9 | [Strava explains](https://support.strava.com/hc/en-us/articles/216918187-Segment-Matching-Issues): 10 | 11 | > When we match you to a segment, we choose the GPS points from your file that fall closest to the segment's start and end. Because of the nature of GPS and the differences in device recording intervals, it's unlikely that there will be a GPS point exactly on top of the segment's start/end. This means that you can be timed over portions of your activity that are slightly longer or shorter than the actual segment length. Everyone is susceptible to this, but you may get a short match that works in your favor or a long match that adds a few extra seconds. 12 | 13 | [and](https://support.strava.com/hc/en-us/articles/216918227-Optimizing-Segment-Creation-how-to-create-good-segments#short): 14 | > [...] the sampling rate of your GPS device; some devices only record a GPS point every 5 to 10 seconds 15 | 16 | [Garmin confirms that devices can have varying recording invervals](https://support.garmin.com/en-US/?faq=s4w6kZmbmK0P6l20SgpW28): 17 | > Smart Recording captures key data points as changes occur in direction, speed, heart rate, or elevation. This method is recommended [...] 18 | 19 | 20 | ## Visual depiction 21 | 22 | Following from the above, when two cyclists ride (also valid for, e.g., runners, btw.) along a segment with the same speed, depending on random chance, one might get a much better time than the other: 23 | 24 | ``` 25 | start finish 26 | | | 27 | v v 28 | 🟢-----------------[segment]---------------🏁 29 | 30 | 31 | --🔴-----🔴-----🔵-----🔵---[lucky effort]---🔵-----🔵-----🔴-----🔴-- 32 | ^ ^ 33 | | | 34 | start end 35 | 36 | 37 | --🔴-----🔵-----🔵-----🔵--[unlucky effort]--🔵-----🔵-----🔵-----🔴-- 38 | ^ ^ 39 | | | 40 | start end 41 | ``` 42 | 43 | ## Solution 44 | 45 | Instead of using the recorded GPS points closest to the segment start/end, one can calculate the closest point on the polygonal chain of the activity points, deduce the timestamp by interpolating the timestamps of the two neighbor points, and use that: 46 | 47 | ``` 48 | start finish 49 | | | 50 | v v 51 | 🟢-----------------[segment]---------------🏁 52 | 53 | 54 | --🔴-----🔴--🔵--🔵-----🔵--[actual effort]--🔵-----🔵--🔵--🔴-----🔴-- 55 | ^ ^ 56 | | | 57 | calculated start calculated end 58 | ``` 59 | 60 | In case a recorded point is closest, it will simply be used without interpolation. 61 | 62 | This nicely fixes the problem resulting from low sampling rates in a stream of GPS points, but it also further improves the accuracy of recordings with a high(-ish) sampling frequency (like 1 Hz). 63 | 64 | (Of course, actual inaccuracies in the recorded coordinates of single GPS points can not be addressed by this.) 65 | 66 | ## Implementation 67 | 68 | [`accurate_timing_of_strava_segments.py`](accurate_timing_of_strava_segments/accurate_timing_of_strava_segments.py) is a Python script containing an exemplaric implementation of the described solution. ⏱️ 69 | 70 | ## Real-life example 71 | 72 | Taking [this segment](https://www.strava.com/segments/4391619) as an example, the leaderboard contains the following two efforts: 73 | - `A`: Jun 25 2017: 33 s 74 | - `B`: Sep 1, 2021: 34 s 75 | 76 | When looking closer at the two activities, something seems fishy. `B` has fewer recorded GPS points than `A`. Also, the average speed of `A` seems higher when manually selecting the segment part of the activity on the analysis page. 77 | 78 | ![marienfeld_a](accurate_timing_of_strava_segments/marienfeld_a.png) 79 | 80 | (The blue points are `A`s GPS recordings closest to the segment start (green) and end (checkered).) 81 | 82 | Running the script linked to above, we get the following results: 83 | - `A`: **37.8 s** (instead of Strava's 33 s, i.e., +4.8 s difference) 84 | - `B`: **34.3 s** (instead of Strava's 34 s, i.e., +0.3 s difference) 85 | 86 | This shows, that `B` actually performed significantly better in that segment than `A`. 🚀 87 | 88 | ## Plea 89 | 90 | Dear Strava developers, 91 | 92 | your app is a great source of joy for KOM hunters, but the above introduces an unnecessary and avoidable random factor into the segment leaderboards. Please consider implementing the described solution to remedy the situation. 93 | 94 | And while you're at it, in 2022, you increased the minimum length for new segments to 500 m, presumably to reduce unfairness due to inaccurate measurements. The solution described here would fix this more cleanly, and you could reduce the minimum segment length again. (Especially for steep hills, 200 m often is more than enough.) Please remember, not all of us users are time trialists, some are sprinters. 😉 95 | 96 | ## Links 97 | - [discussion in /r/Strava](https://www.reddit.com/r/Strava/comments/10thlkc/how_strava_could_drastically_improve_the_timing/) 98 | - [suggestion in the Stava community hub](https://communityhub.strava.com/t5/ideas/improve-the-timing-precision-of-segment-efforts-by-interpolating/idi-p/6612/emcs_t/S2h8ZW1haWx8dG9waWNfc3Vic2NyaXB0aW9ufExPTjUwWU05SDNZTDVTfDE5NjQ3fFNVQlNDUklQVElPTlN8aEs) 99 | -------------------------------------------------------------------------------- /a_too_naive_approach_to_video_compression_using_artificial_neural_networks.md: -------------------------------------------------------------------------------- 1 | A too naive approach to video compression using artificial neural networks 2 | ========================================================================== 3 | 4 | Introduction 5 | ------------ 6 | In computer vision neural networks often are used to classify images by the objects they depict. So one could theoretically also use them to detect if an image is a frame of a specific movie and if so at what position it occurs. 7 | So why not try it the other way around? 8 | In this article I will present my naive attempt to use a very simple neural network to learn the frames of a video in order to regenerate them later. Spoiler alert: It failed. But it was fun to try nonetheless. ;) 9 | 10 | First, some background 11 | ---------------------- 12 | (Feel free to skip this section if you already have a basic idea of how ANNs work.) 13 | 14 | An artificial neural network consists of neurons and the connections between them. One of the most simple neural networks is a feed forward perceptron. It has no circles in its directed graph. Here is a very small one: 15 | 16 | ![nn](a_too_naive_approach_to_video_compression_using_artificial_neural_networks_files/nn.jpg) 17 | 18 | It is composed of three neurons in the input layer and two neurons in the output layer. But there could a an arbitrary number of additional layers, called hidden layers, between them. 19 | 20 | If you put values (usually 0 <= value <= 1) onto the input neurons, they propagate through the net to the right according to the weights of the connections. Every neuron has an [activation function](https://en.wikipedia.org/wiki/Sigmoid_function), which basically maps its summed input values back onto the interval [0,1]: 21 | 22 | ![tanh](a_too_naive_approach_to_video_compression_using_artificial_neural_networks_files/tanh.png) 23 | 24 | (image source: [Wikipedia](https://de.wikipedia.org/wiki/Tangens_Hyperbolicus_und_Kotangens_Hyperbolicus#/media/File:Hyperbolic_Tangent.svg)) 25 | 26 | The only modifiable thing of the net are the weights of the connections, so all information is stored there. To get a net to learn something you train it with data consisting of input vectors (values for the neurons in the input layer) and corresponding output values. During training the initially randomly preinitialized weights are optimized to make the output layer yield something close to the desired values of the training set. 27 | 28 | Test Video 29 | ---------- 30 | Our test video consists of 84 single frames with a resolution of 480 * 360 pixels at 25 frames per second. And it looks like this: 31 | 32 | ![(Picture missing, uh oh)](a_too_naive_approach_to_video_compression_using_artificial_neural_networks_files/original_snapshot.jpg) 33 | 34 | [youtube link](https://www.youtube.com/watch?v=GqusTv0wp4c) 35 | 36 | If you save it raw with 3 bytes per pixel (B, G and R) it is about 43 MB in size. 37 | 38 | Neural Network without compression 39 | ---------------------------------- 40 | If we now construct a neural network with 84 input neurons, no hidden layer and 518400 (480 * 360 * 3) output neorons, it can easily be trained to learn the video perfectly. [Overfitting](https://en.wikipedia.org/wiki/Overfitting) is not a problem here, but even wanted. Every entry in the training data represents one frame. In the input layer only the neuron representing the current frame number is set to 1 while all others are 0. E.g. frame number 3/8 would be coded in the input layer with `[0,0,1,0,0,0,0,0]`. The output layer gets every pixel value (converted from [0,255] to [0.0,1.0]) mapped onto it. 41 | 42 | After a very short training phase, the weights converge to the raw pixel values. But of course nothing is gained here, since there is no compression. We solely developed a fancy way to store raw pixel values in the form of 43545600 (84 * 480 * 360 * 3) connection weights. 43 | 44 | Neural Network with compression 45 | ------------------------------- 46 | The idea now is that we do not need to store all pixel values as weights. Since almost every video has recurring patters and constant or single-colored sectors it could be possible to store the information needed to reproduce frames sufficiently similar the the original ones with less than 43545600 weights. So we introduce two hidden layers. 47 | 48 | `layersizes = [84, 10, 10, 518400]` 49 | 50 | Now our net only weights 5184940 (84 * 10 + 10 * 10 + 10 * 518400) connections, i.e. roughly 12% of the prior version. So if we store our weights as 4-byte floats, we have 20 MB (20739760 bytes) in storage size. This is less than half the size of an uncompressed video format using 3 bytes per pixel, but still about 50 times the size of an H264 (1010 kBit/s) compressed version of the video with 0.36 MB (380007 bytes). 51 | 52 | After about one hour of training on all my four CPU cores with [resilient backpropagation](https://en.wikipedia.org/wiki/Rprop) the result looks as follows: 53 | 54 | ![(Picture missing, uh oh)](a_too_naive_approach_to_video_compression_using_artificial_neural_networks_files/nn_snapshot.jpg) 55 | 56 | [youtube link](https://www.youtube.com/watch?v=Qku14b_v-B8) 57 | 58 | I experimented with different layer configurations and training parameters but did not obtain significantly better results. 59 | 60 | Conclusion 61 | ---------- 62 | We can save a version of the video smaller than the raw format, and we see some blurry dream/ghost/drug-like effects, but have nothing suitable for real world video compression. Compression time, file size and video quality are terrible. 63 | 64 | If you would like to play around with the source code, e.g. use different settings, try out another input video etc., you can find it here: [https://gist.github.com/Dobiasd/9234a8fe7ba958f79227](https://gist.github.com/Dobiasd/9234a8fe7ba958f79227) 65 | 66 | Different possibilies to decode the frame number for the input neurons are already given by the instances of `NumToNNValuesFunc`. 67 | 68 | Further links 69 | ------------- 70 | In case you are wondering, if there are any successful attempts to use neural networks to generate images: Yes, there are plenty. Here are some very cool ones: 71 | 72 | * [Inceptionism: Going Deeper into Neural Networks](http://googleresearch.blogspot.com/2015/06/inceptionism-going-deeper-into-neural.html) 73 | * [New algorithm gives photos Picasso-style makeovers](http://mashable.com/2015/08/29/computer-photos/) 74 | 75 | [discussion on reddit](https://www.reddit.com/r/programming/comments/3mvnz6/a_too_naive_approach_to_video_compression_using/) -------------------------------------------------------------------------------- /threads_can_infect_each_other_with_their_low_priority.md: -------------------------------------------------------------------------------- 1 | # Threads can infect each other with their low priority 2 | 3 | Imagine having something like the following situation: 4 | 5 | ```c++ 6 | ThreadSafeQueue queue; 7 | 8 | void thread_a() { 9 | // Infinite loop just for the sake of this example. 10 | while(true) { 11 | ... 12 | queue.push(...); 13 | ... 14 | } 15 | } 16 | 17 | void thread_b() { 18 | while(true) { 19 | ... 20 | x = queue.pop(); 21 | ... 22 | } 23 | } 24 | ``` 25 | (Pseudo C++, but the concept applies to other languages too.) 26 | 27 | Both functions (`thread_a` and `thread_b`) are running concurrently on different OS threads. It's a typical producer-consumer situation. 28 | 29 | Maybe `thread_a` is doing something that should not have too much latency, e.g., communicating with a PLC, and `thread_b` does some work, that is allowed to have high latency, e.g., updating some UI elements. 30 | 31 | Of course, if you had hard real-time requirements, you would use a real-time OS. Nevertheless, since the job of `thread_a` is more urgent than the job of `thread_b`, you decide to change the priorities: 32 | 33 | - `thread_a`: `HIGH` priority (low latency) 34 | - `thread_b`: `LOW` priority (high latency) 35 | 36 | So with the help of the preemptive scheduler of the OS, you now should have low latency on `thread_a`, because with its priority it's not only preferred over `thread_b` but also over all other threads (not shown above) with priority `NORMAL` (the default). 37 | 38 | Seems fine, right? But in reality, it likely is not. Assuming our `ThreadSafeQueue` isn't a specially designed lock-free with all its quirks, but protected by a mutex instead, with a bit of manual inlining, the code reveals more of what it is doing: 39 | 40 | ```c++ 41 | Mutex queue_mutex; 42 | 43 | void thread_a() { 44 | while(true) { 45 | ... 46 | queueMutex.lock(); 47 | // push 48 | queueMutex.unlock(); 49 | ... 50 | } 51 | } 52 | 53 | void thread_b() { 54 | while(true) { 55 | ... 56 | queueMutex.lock(); 57 | // pop 58 | queueMutex.unlock(); 59 | ... 60 | } 61 | } 62 | ``` 63 | 64 | (Exception-safety using RAII purposely ignored for clarity.) 65 | 66 | This means the following might (read "will") happen at some point: 67 | 68 | - `thread_b` locks the mutex. 69 | - The scheduler interrupts it's execution and gives the CPU to some other thread. 70 | - It might take quite long until `thread_b` gets its next time slice. Some other threads with priority `NORMAL`, which can also live in other processes running on the same machine, might have quite some time-consuming work to do, so the scheduler prefers them over `thread_b`. 71 | - `thread_a` thus will be forced to wait on it's blocking `.lock()` call for quite long. 72 | - Consequently, `thread_a` has lost its low-latency property. 73 | 74 | So you thought you have the following: 75 | 76 | --- 77 | 78 | ![1](threads_can_infect_each_other_with_their_low_priority/1.png?raw=true) 79 | 80 | --- 81 | 82 | But in reality, you have: 83 | 84 | --- 85 | 86 | ![2](threads_can_infect_each_other_with_their_low_priority/2.png?raw=true) 87 | 88 | --- 89 | 90 | `thread_b` infected `thread_a` with its low priority. 91 | 92 | This brings us to a new theorem: 93 | 94 | **A thread's observed priority is not higher than the lowest priority of any thread it shares a mutex with.** 95 | 96 | The strength of the effect this ([priority inversion](https://en.wikipedia.org/wiki/Priority_inversion)) has on latency may differ depending on what OS/scheduler is used, but in general, it holds that, as soon as thread priorities are changed deliberately for any reason, one suddenly is forced to investigate the thread-mutex graph of one's application and check for lower-prioritized neighbors. OS schedulers might implement [different techniques](https://en.wikipedia.org/wiki/Priority_inversion#Solutions) to lessen the severity of this problem, each of them with different upsides and downsides. 97 | 98 | Naturally, the question arises, if this spreading can be transitive too, i.e., if, in the following graph example, the priority of `thread_e` can have an influence on the latency / observes priority of `thread_c`: 99 | 100 | --- 101 | 102 | ![3](threads_can_infect_each_other_with_their_low_priority/3.png?raw=true) 103 | 104 | --- 105 | 106 | The answer is: Yes, it can. It depends on how the acquisitions of the locks in `thread_d` are intertwined. 107 | 108 | ```cpp 109 | Mutex mutex_1; 110 | Mutex mutex_2; 111 | 112 | // prio HIGH 113 | void thread_c() { 114 | while(true) { 115 | ... 116 | mutex_1.lock(); 117 | ... 118 | mutex_1.unlock(); 119 | ... 120 | } 121 | } 122 | 123 | // prio NORMAL 124 | void thread_d() { 125 | while(true) { 126 | ... 127 | mutex_1.lock(); 128 | ... 129 | mutex_2.lock(); 130 | ... 131 | mutex_2.unlock(); 132 | ... 133 | mutex_1.unlock(); 134 | ... 135 | } 136 | } 137 | 138 | // prio LOW 139 | void thread_e() { 140 | while(true) { 141 | ... 142 | mutex_2.lock(); 143 | ... 144 | mutex_2.unlock(); 145 | ... 146 | } 147 | } 148 | ``` 149 | 150 | (The situation in `thread_d` might be hidden in calls so some other functions.) 151 | 152 | So, what can happen here? 153 | 154 | - `thread_e` does its `mutex_2.lock()`. 155 | - The scheduler interrupts `thread_e`. 156 | - `thread_d` does its `mutex_1.lock()`. 157 | - `thread_d` wants to do its `mutex_2.lock()`, but is blocked, because `thread_e` is already sleeping inside `mutex_2`. 158 | - `thread_c` tries do it its `mutex_1.lock()`, but is blocked, because `thread_d` already has acquired `mutex_1`. 159 | 160 | Thus, `thread_c` is blocked at least as long as `thread_e` does not reach its `mutex_2.unlock()`. And this, again, can take quite a while. 161 | 162 | So, the latency of `thread_c` was ruined, transitively, by `thread_e`, 163 | 164 | and you end up with the following: 165 | 166 | --- 167 | 168 | ![4](threads_can_infect_each_other_with_their_low_priority/4.png?raw=true) 169 | 170 | --- 171 | 172 | This can haunt you not only when using classical mutexes, but with other [types of mutual exclusion devices](https://en.wikipedia.org/wiki/Mutual_exclusion#Types_of_mutual_exclusion_devices) too. 173 | 174 | Conclusion: **If possible, avoid headaches by just not fiddling around with thread priorities.** 175 | 176 | ([discussion on reddit](https://www.reddit.com/r/programming/comments/e7sb5p/threads_can_infect_each_other_with_their_low/)) 177 | -------------------------------------------------------------------------------- /do_a_b_tests_because_correlation_does_not_imply_causation.md: -------------------------------------------------------------------------------- 1 | # Do A/B tests - because correlation does not imply causation 2 | 3 | Let's imagine we (hypothetical) collect habits and health data from people. We find that, on average, people who regularly go to the sauna had less sick days last year. 4 | 5 | ```text 6 | (totally hypothetical data) 7 | 8 | | number of people | avg sick days / person | 9 | |----------|------------------|------------------------| 10 | | sauna | 1000 | 5 | 11 | | no-sauna | 90000 | 10 | 12 | ``` 13 | 14 | We now might want to conclude that regularly having a sauna prevents one from getting sick, and we start to recommend doing this. But this would be a big mistake! 15 | 16 | We don't know at all if the sauna is the cause for being less sick. Some other factor could be making people use the sauna and be less sick. 17 | 18 | ```text 19 | |====> sauna 20 | other_factor ====| 21 | |====> being less sick 22 | ``` 23 | 24 | 25 | This other factor could be (for example) [socioeconomic status](https://en.wikipedia.org/wiki/Socioeconomic_status#Health), which simply means: Wealthy people are less sick in general, and poor people just don't go to the sauna that much. 26 | 27 | The only productive thing we can do with this sauna-less-sick correlation that we found, is to conduct an experiment, i.e., an A/B test. We need to randomly assign (a bunch of) people into two test groups. One group will be compelled to regularly go to the sauna and the other group is forbidden to do so. It's important, that people are not allowed to choose their group. We then let them do their assigned task for some time (quite long in that case), and then we count sick days again. Possible result: 28 | 29 | ```text 30 | from hypothetical A/B test 31 | 32 | | number of people | avg sick days / person | 33 | |----------|------------------|------------------------| 34 | | sauna | 50 | 8 | 35 | | no-sauna | 50 | 7 | 36 | ``` 37 | 38 | Oh, so sauna does not cause less sick days at all. On the contrary, the sauna-test group had one sick day more per year on average compared to the no-sauna group. Sauna might even cause a bit more sickness. (Remember, this is just a contrived example, and is not related to any real numbers evaluating the effect of sauna.) 39 | 40 | When doing such an experiment (A/B test) we are not allowed to separate the two groups in time or space, meaning: 41 | 42 | - Both groups have to be measured simultaneously. If we test the sauna-sick days in one year, and the no-sauna-test-sick days the next year, other non-related factors, will have changed. 43 | - The groups may not be separated in space, i.e., move one group to Finland, but not the other one. The individuals have to stay locally mixed as they were before the random assignment. 44 | If we don't adhere to one of these rules, we will measure effects that are caused by some external factors instead of the suggested cause we want to measure (sauna). 45 | 46 | --- 47 | 48 | So when we find a positive correlation between two things `X` and `Y`. 49 | 50 | ```text 51 | X <----> Y 52 | ``` 53 | 54 | It can mean one causes the other: 55 | 56 | ```text 57 | X ====> Y 58 | ``` 59 | 60 | or 61 | 62 | ```text 63 | X <==== Y 64 | ``` 65 | 66 | But it does not have to. Other factors might always be involved: 67 | 68 | ```text 69 | |====> X 70 | Z ====| 71 | |====> Y 72 | ``` 73 | 74 | And we can only find the true causation with randomized controlled trials (A/B tests). 75 | 76 | --- 77 | 78 | Examples showing the absurdity: 79 | 80 | 1) `X` might be "Kids watching more TV", and `Y` "Kids being more violent". Watching TV could cause violence or peacefulness. We don't know. Maybe violent kids just tend to watch more TV. 81 | 82 | 2) "stork population" by region strongly correlates with "human birthrate" by region. However, storks don't deliver babies. People in rural areas (storks don't live in big cities) just have more kids. 83 | 84 | 3) Countries with more ice-cream consumption (`X`) have more deaths by drowning (`Y`). But eating ice cream does not make you drown. It's just that in colder countries, people eat less ice cream and don't go swimming that much. Because it's cold, duh. 85 | 86 | In the media, one can find an enormous number of instances of this exact causal fallacy. 87 | 88 | Regarding the last example, i.e., the one with the ice-cream, one might think, that one just has to filter the data in better way, e.g., by considering only the people that actually went swimming, and then compare those that ate ice-cream during the 30 minutes before going into the water with those who did not. In case the ice-cream eaters would on average drown more often, would this mean eating ice-cream increases the risk or drowning? No, it would not! There can still be endlessly many confounding factors. For example, people who take care of their fitness level might be less likely to eat ice-cream and also be better swimmers. Or it could be that kids tend to eat ice-cream more often compared to adults and that sadly, they also are more likely to drown. One does not, and can not know about (and exclude) all these possibilities. The only realistic chance we have to really find out if ice-cream causes drowning is, again, to conduct a randomized trial, i.e., randomly assign a lot of people to an ice-cream group and a placebo-ice-cream group, throw both into the water, and measure the drowning rate. 89 | 90 | --- 91 | 92 | An example related to web/app development: 93 | 94 | Let's say we are developing a mobile (freemium) game, we might find a positive correlation between "actually playing the tutorial level instead of skipping it" and "buying the full version". 95 | 96 | Playing the tutorial level *might* cause more purchases, but we can not possibly know this from those statistics. It might be the case that people who are the "buying type of person" for some reason just also (on average) skip tutorials less often. 97 | 98 | ```text 99 | |====> using the tutorial 100 | other_factor ====| 101 | |====> purchasing 102 | ``` 103 | 104 | So to really find out if we can make more of our users buy the full version by making playing the tutorial level mandatory, we need to conduct an A/B test with users randomly assigned into two groups. One gets the usual version of the app (default group), the other one gets the force-tutorial version (feature group). After collecting enough data by letting the test run long enough, we can make an informed decision on if we want to fully roll out the feature. 105 | -------------------------------------------------------------------------------- /how_touch_typing_and_keyboard_shortcuts_can_improve_the_quality_of_the_software_you_create.md: -------------------------------------------------------------------------------- 1 | # How touch typing and keyboard shortcuts can improve the quality of the software you create 2 | 3 | Recently on [proggit](http://www.reddit.com/r/programming/comments/2nvt2w/the_case_for_slow_programming/) I found an [article](https://ventrellathing.wordpress.com/2013/06/18/the-case-for-slow-programming/) advocating a well thought out design process in software development. I mostly agreed. But one sentence made me cringe: 4 | 5 | > I'm Glad I'm not a Touch-Typist. 6 | 7 | First of all, I do not think that being able to touch type is an indispensable prerequisite for being a good programmer since we often write [astonishing few lines of code per day](http://skeptics.stackexchange.com/questions/17224/do-professional-software-developers-write-an-average-of-10-lines-of-code-per-day) and have many of our ideas while not sitting in front of our computer. Yet, I'm very glad I can use **touch typing**, because it [makes one type faster](http://en.wikipedia.org/wiki/Touch_typing#Speed), and I think even though this does not improve development speed significantly in the short run, it **improves the quality of the software one produces**. In this short article, I will explain why I believe this. Although it is just a gut feeling probably based on subjective observation, and I do not have empirical evidence, I still hope you will find some points here that make sense to you. 8 | 9 | ## Nano decisions and threshold 10 | 11 | In our everyday lives, we unconsciously make very small decisions all the time. Every few seconds we do something that can be done in many different ways, but we mostly stick to our habits, and the skill set we have and the environment we live in influences our nano decisions. 12 | 13 | Take cooking as an example: You are in the middle of preparing a delicious dinner and you want to cut your veggies in a special way. A [tourné knife](http://cooking.stackexchange.com/questions/37818/what-is-a-tourne-knife-used-for) would be ideal for this, but yours is somewhere in the conservatory from you having your last snack there. You probably will just use your paring knife, because it already is right in front of you on your chopping board, and using it instead will decrease the quality of your meal only very slightly. 14 | 15 | Being a slow typist is like having your kitchen tools badly organized. 16 | 17 | You are writing some code that is partly self-explanatory. It would be nicer with, e.g., a more descriptive name for that function/variable, but not really disastrous without it. The cost/effort of making that tiny change, measured in time of thought-interruption and thus cognitive load, is lower if you are able to type fast. So the benefit-cost ratio can end up below the "make the change" threshold if you are a slow typist and above it otherwise. Effort thresholds can be about different things relevant to the quality of your work: 18 | - Make this small refactoring 19 | - Ask a question on [stackoverflow](http://stackoverflow.com)" 20 | - Write the better version of the commit message, pull-request description, slack message, email, ... 21 | 22 | 23 | ## Keyboard shortcuts 24 | 25 | Perhaps even more important than fast typing is the skill to use keyboard shortcuts efficiently. Not having to look at your fingers while using them also helps, since you can react much better. If I see someone looking at his keyboard while editing text, I always think of a tennis player not watching the ball and his opponent, but staring at his racket the whole game. It gets even more [funny](http://steve-yegge.blogspot.de/2008/09/programmings-dirtiest-little-secret.html), if their IDE is constantly providing the correct autocomplete suggestions, but they just don't notice it. 26 | The keyboard should not feel like an external device you are trying to manipulate but like a natural extension of your body, that enhances your capabilities. 27 | In this condition, your typing is less likely to destructively interrupt your train of thought due to its duration and shift of focus. 28 | 29 | 30 | So if you do not have to carry the constant burden of having to watch your fingers, some things become much easier. If you know the shortcuts, you automatically start to use "ctrl+f, F3" for searching something instead of scrolling with your mouse, write stuff in text files instead of paper ([greppability](http://en.wiktionary.org/wiki/greppable) is a big advantage.), etc. 31 | 32 | This again influences your software development decisions. Say you are not 100% sure about a fact. You could just assume it or look it up, e.g. in a log file or online API documentation. If you are able to open the file/page and navigate to the right spot in a few seconds, you are more likely to really do it, which reduces the risk of making a mistake by relying on a wrong assumption. 33 | 34 | 35 | ## Pair programming and alike 36 | 37 | If you are at the computer with somebody else, it can even improve your relationship if you are skilled in the things mentioned above. If you are, and the other person is not, you are a good example, and perhaps even an inspiration if you are really impressive. 38 | On the other hand, if the person by your side would be much faster than you, they can get really annoyed by just having to watch you using the PC unnecessary slowly. 39 | 40 | 41 | ## Conclusion 42 | 43 | Fast typing (+blind, +shortcuts) can not only improve your speed but particularly also the quality you produce. 44 | 45 | Having your flow interrupted while programming is rarely a good thing. Causes for this include disruptive colleagues/managers, slow hardware (compile times, etc.) and tying disability. If you not yet have eliminated the third one (and I've seen it many times even in experienced developers), I suggest the following: 46 | 47 | * Learn to type without looking at your fingers. Just using a Sharpie to paint your keys black can work wonders. Perhaps use a game like [Z-Type](http://phoboslab.org/ztype/). 48 | * Find and Memorize the keyboard shortcuts for your operating system and software. 49 | * Put your mouse away and operate the PC without it for a few days to practice the shortcuts and to evaluate which of them are helpful and which are not. 50 | * Use the console/terminal more. 51 | * Additionally, become a master at rearranging text (select, copy, paste, etc.) with your keyboard only. To practice it you can go to the [EditGym](http://www.editgym.com/text-editing-training/). 52 | 53 | 54 | So, do you think CEOs should give their developers [black keyboards](http://www.daskeyboard.com/daskeyboard-4-ultimate/) and throw their mice out of the window to increase their productivity in the long run? ;-) ([comment on reddit](http://www.reddit.com/r/programming/comments/2q3uwe/how_touch_typing_and_keyboard_shortcuts_can/)) 55 | -------------------------------------------------------------------------------- /from_goto_to_std-transform.md: -------------------------------------------------------------------------------- 1 | # C++ - From goto to std::transform 2 | 3 | In nearly every programming languages there are often many different possibilities to write code that in the end does exactly the same thing. In most cases we want to use the version that is the easiest to understand to a human reader. 4 | 5 | Let's say we have a vector of integers and want another one with all elements being the square of the elements in the first one. 6 | `[4, 1, 7] -> [16, 1, 49]` 7 | 8 | 9 | ## Goto 10 | Sure, one could use gotos for this, but I guess nobody in their right mind would do this voluntarily, unless for trolling: 11 | 12 | ```c++ 13 | vector squareVec1(vector v) 14 | { 15 | auto it = begin(v); 16 | loopBegin: 17 | if (it == end(v)) 18 | goto loopEnd; 19 | *it = *it * *it; 20 | ++it; 21 | goto loopBegin; 22 | loopEnd: 23 | return v; 24 | } 25 | ``` 26 | On the first look you have no idea what this code is doing. You have to kind of interpret it in your head and follow the control flow manually to find out what's going on. 27 | 28 | 29 | ## While loop 30 | The next more readable step would be to use a loop as control structure. 31 | ```c++ 32 | vector squareVec2(vector v) 33 | { 34 | auto it = begin(v); 35 | while (it != end(v)) 36 | { 37 | *it = *it * *it; 38 | ++it; 39 | } 40 | return v; 41 | } 42 | ``` 43 | This is bit better, at least you can immediatly see that there is some kind of loop, but most people would probably use a 44 | 45 | 46 | ## For loop 47 | ```c++ 48 | vector squareVec3(vector v) 49 | { 50 | for (auto it = begin(v); it != end(v); ++it) 51 | { 52 | *it = *it * *it; 53 | } 54 | return v; 55 | } 56 | ``` 57 | Here you can immediately see that the algorithm iterates ofter the elements of `v` but you still have to read the whole line `for (auto it = begin(v); it != end(v); ++it)` until you know that every single element is used and not e.g. every second, since the increase could also be `it += 2` or something else instead of `++it`. Still a `++it` could be hidden somewhere in the loop body. 58 | 59 | 60 | ## Range-based for loop 61 | ```c++ 62 | vector squareVec4(vector v) 63 | { 64 | for (int& i : v) 65 | { 66 | i = i * i; 67 | } 68 | return v; 69 | } 70 | ``` 71 | 72 | This time the `for` line already tells the reader that probably every element of `v` is used, but still only probably. One still has to look into the body of the for loop and look for `if`, `continue` or even `break` statements to really know that every single elemnt in `v` will be iterated over the end. Also the header does not tell what will happen to the elements. 73 | 74 | Many people stop here, but we can continue do better in terms of readability ease. 75 | 76 | 77 | ## std::transform 78 | OK, how can we express more clearly without explicit comments what our code does, i.e. make it self explaining? 79 | 80 | ```c++ 81 | vector squareVec5(vector v) 82 | { 83 | transform(begin(v), end(v), begin(v), [](int i) 84 | { 85 | return i*i; 86 | }); 87 | return v; 88 | } 89 | ``` 90 | `std::transform` tells the reader at one glance that all `v.size()` elements of `v` will be transformed into something else. 91 | Now one just has to look at `return i*i` and he directly knows everything. 92 | This should be much easier than decyphering a for loop every single time. 93 | 94 | 95 | ## Range-based for vs. [``](http://en.cppreference.com/w/cpp/algorithm) 96 | A for loop also beginning with `for (int i : v)` could do something totally unrelated to `std::transform`. E.g. it could implement a filter: 97 | ```c++ 98 | vector result; 99 | for (int i : v) 100 | { 101 | if (i % 2 == 0) 102 | result.push_back(i); 103 | } 104 | ``` 105 | 106 | Here a more expressive version would be: 107 | ```c++ 108 | vector result; 109 | copy_if(begin(v), end(v), back_inserter(result), [](int i) 110 | { 111 | return i % 2 == 0; 112 | }); 113 | ``` 114 | 115 | `transform` and `copy_if` show the [map](http://en.wikipedia.org/wiki/Map_%28higher-order_function%29) [filter](http://en.wikipedia.org/wiki/Filter_%28higher-order_function%29) difference more clearly than the two range-based for loops with the same header and just a differing body. 116 | 117 | `copy_if` immediately tells the reader of our code that only elements from `v` will show up in `result` and that `result.size() <= v.size()`. Additionally bugs caused by inattentiveness like infinite loops or invalid iterators are ruled out automatically by this style compared to `for (auto it = ...)` versions. 118 | 119 | This higher level ob abstraction not only gives our code a self-documenting flavour but also makes it easier to perhaps later replace `copy_if` with a hypothetical `copy_if_multithread` or something like that. 120 | 121 | 122 | "But the range-based for loop is shorter and thus more readable." you say? In this very small example, this may be the case, but if the loop body would be much longer, the character count difference dissolves and you will be happy that you do not have to look at the body at all in the `transform`/`find_if` version to figure out what it is doing. 123 | 124 | Also passing along a strategy in form of a [`std::function`](http://en.cppreference.com/w/cpp/utility/functional/function) will become easier, since you can just plug it in. 125 | 126 | 127 | ## Convenience wrappers 128 | If you just can not stand the manual usage of `begin(v)` and `end(v)` you are free to write a wrapper in case you have to use `transform` often enough: 129 | ```c++ 130 | template 131 | Container transformCont(Container xs, Functor op) 132 | { 133 | transform(begin(xs), end(xs), begin(xs), op); 134 | return xs; 135 | } 136 | 137 | vector squareVec6(const vector& v) 138 | { 139 | return transformCont(v, [](int i) 140 | { 141 | return i*i; 142 | }); 143 | } 144 | ``` 145 | 146 | ## Performance 147 | "But I have to use the hand written for loop for better performance!" - Nope, you do not have to. 148 | Even if the `std::transform` version looks like much abstraction induced function call overhead, especially with the lambda function, there is none. It is all optimized away by the compiler. 149 | 150 | For 50000 runs over 10000 values the different implementations ([source code](https://gist.github.com/Dobiasd/839acc2bc7a1f48a5063)) took the following cpu times on my machine: 151 | ``` 152 | goto - elapsed time: 0.538397s 153 | while - elapsed time: 0.538062s 154 | for - elapsed time: 0.537738s 155 | range-based for - elapsed time: 0.538066s 156 | std::transform - elapsed time: 0.537909s 157 | wrapped std::transform - elapsed time: 0.537213s 158 | ``` 159 | 160 | 161 | ## Conclusion 162 | Sure, readability also has something to with taste or to be precise familiarity, but in my opinion you should avoid explicit loops and make use of the cool stuff in the [`` header](http://en.cppreference.com/w/cpp/algorithm) for better maintainability of your C++ software. Once you get used to the odd lambda syntax in C++ you will enjoy every for loop you do *not* have to read. ;-) 163 | 164 | 165 | ## Further reading 166 | With [effective stl](http://www.amazon.com/dp/0201749629) Scott Meyers has written a very nice book covering this and more in depths. 167 | Herb Sutter's [talk about lambdas](https://www.youtube.com/watch?v=rcgRY7sOA58) can also help to get more into this topic. 168 | Also you can discuss this article on [reddit](http://redd.it/22q18m) or [Hacker News](https://news.ycombinator.com/item?id=7571490). 169 | 170 | By the way, if you are interested in learning more about functional programming using C++ you might enjoy [my video course on Udemy](https://www.udemy.com/functional-programming-using-cpp). -------------------------------------------------------------------------------- /llm_agents_demystified.md: -------------------------------------------------------------------------------- 1 | # LLM agents demystified 2 | 3 | By now, we all know that LLMs (ChatGPT, etc.) are autoregressive and just spit out tokens one by one in a loop, eating their incomplete output in each iteration. We might also have read about how tokenizers are trained and what the implementation of Attention layers roughly looks like. 4 | 5 | But agents (and RAG) might still be a bit enigmatic. Let's demystify them (using oversimplified pseudo-Python code). 6 | 7 | First, a quick refresher of the basics: 8 | 9 | ```python 10 | class LLM: 11 | weights: ... 12 | def invoke(self, context: List[int]) -> List[float]: 13 | """Returns the logits for the next token. All the GPU things happen here.""" 14 | ... 15 | 16 | def softmax(logits: List[float]) -> List[float]: 17 | """Converts to probabilities.""" 18 | ... 19 | 20 | def infer_next_token(context: List[int]) -> int: 21 | token_probabilities = softmax(llm.invoke(context)) 22 | return argmax(token_probabilities) # In reality, temperature and top-k/top-p make it somewhat random. 23 | 24 | def tokenize(text: str) -> List[int]: 25 | """Splits the string and returns the tokens.""" 26 | ... 27 | 28 | def detokenize(tokens: List[int]) -> str: 29 | """Converts tokens back to text.""" 30 | ... 31 | 32 | def inference(input_text: str) -> str: 33 | """Autoregressively extracts a response from the LLM""" 34 | input_tokens = tokenize(input_text) 35 | output_tokens: List[int] = [] 36 | next_token = None 37 | while True: 38 | context: List[int] = input_tokens + output_tokens 39 | next_token = infer_next_token(context) 40 | if next_token == EOS or len(output_tokens) >= max_tokens: 41 | break 42 | output_tokens.append(next_token) 43 | return detokenize(output_tokens) 44 | ``` 45 | 46 | So, with `inference`, we have a function that takes input (user prompt, a full conversation, whatever), and returns the response. 47 | 48 | Letting the model respond with an object of a specific schema can be sketched like this: 49 | 50 | ```python 51 | def structured_inference(input_text: str, schema: Schema) -> Dict: 52 | """Generates output that conforms to the given schema (e.g., JSON schema).""" 53 | prompt = input_text + "\nRespond with valid JSON matching this schema: " + str(schema) 54 | # In reality, constrained decoding would ensure validity token-by-token. 55 | json_string = inference(prompt) 56 | return json.loads(json_string) 57 | ``` 58 | 59 | Retrieval-Augmented Generation (RAG) is, on an abstract level, also not too fancy. It just enriches the context (`prompt`) before the actual LLM inference with text from documents: 60 | 61 | ```python 62 | def rag(user_query: str, knowledge_base: List[str]) -> str: 63 | """Retrieval Augmented Generation - answers using relevant documents.""" 64 | # Uses embeddings to find semantically similar documents from a vector database 65 | relevant_docs = retrieve_most_relevant(user_query, knowledge_base) 66 | prompt = f"{'\n'.join(relevant_docs)}\n\n{user_query}" 67 | return inference(prompt) 68 | ``` 69 | 70 | Let's tackle agentic AI with tool calls next. It's basically just a loop around an LLM giving (structured) responses. When such a response requests a tool call, the selected tool is executed, its result is concatenated to the context (`conversation`), and thrown into the LLM again. Once the LLM no longer "wants" to call any tool, but gives the final answer instead, the loop is done. 71 | 72 | The key insight: the agent builds up a conversation history where tool results become part of the context for the next LLM call, allowing it to "see" what it learned and decide what to do next. 73 | 74 | ```python 75 | def agent(user_input: str) -> str: 76 | """Agent that can use tools to answer questions.""" 77 | 78 | # We accumulate tool calls and results here. 79 | conversation: List[Dict] = [{"role": "user", "content": user_input}] 80 | 81 | while True: # In practice, add max_iterations to prevent infinite loops 82 | # format_conversation turns the list of messages into a single prompt string. 83 | prompt = format_conversation(conversation) + "\nAvailable tools: " + str(tools) 84 | prompt += "\nRespond with either a tool call or a final answer." 85 | 86 | response_schema = { 87 | "type": "one_of", 88 | "options": [ 89 | {"action": "tool_call", "tool": "string", "arguments": "dict"}, 90 | {"action": "final_answer", "answer": "string"} 91 | ] 92 | } 93 | response = structured_inference(prompt, response_schema) 94 | 95 | if response["action"] == "final_answer": 96 | return response["answer"] 97 | 98 | # call_tool looks up the tool by name and calls it with the arguments. 99 | tool_result = call_tool(response["tool"], response["arguments"]) 100 | 101 | conversation.append({"role": "assistant", "tool_call": response}) 102 | conversation.append({"role": "tool", "content": tool_result}) 103 | ``` 104 | 105 | And, provided the right tools like these 106 | 107 | ```python 108 | tools = [ 109 | { 110 | "name": "get_weather", 111 | "description": "Gets the current weather for a location", 112 | "parameters": {"location": "string"} 113 | }, 114 | { 115 | "name": "search_web", 116 | "description": "Searches the web for information", 117 | "parameters": {"query": "string"} 118 | } 119 | ] 120 | # RAG could also be a tool. 121 | ``` 122 | 123 | we already have an agent that we can ask things like 124 | 125 | > How's the weather at El Portet beach? 126 | 127 | If we're lucky, the agent uses 128 | - `search_web` to find out that it's a beach in Moraira 129 | - and then `get_weather` to get the current weather at the Costa Blanca. 130 | 131 | Here's what the conversation might look like internally for this query: 132 | 133 | 1. **Initial state:** 134 | ```python 135 | [ 136 | {"role": "user", "content": "How's the weather at El Portet beach?"} 137 | ] 138 | ``` 139 | 140 | 2. **LLM responds with tool call:** The agent calls `search_web` to find the location. 141 | 142 | 3. **After first tool execution:** 143 | ```python 144 | [ 145 | # ... previous messages ... 146 | {"role": "assistant", "tool_call": {"action": "tool_call", "tool": "search_web", "arguments": {"query": "El Portet beach location"}}}, 147 | {"role": "tool", "content": "El Portet is a beach in Moraira, Spain."} 148 | ] 149 | ``` 150 | 151 | 4. **LLM responds with another tool call:** Seeing the location, it calls `get_weather`. 152 | 153 | 5. **After second tool execution:** 154 | ```python 155 | [ 156 | # ... previous messages ... 157 | {"role": "assistant", "tool_call": {"action": "tool_call", "tool": "get_weather", "arguments": {"location": "Moraira"}}}, 158 | {"role": "tool", "content": "Moraira: Sunny, 22°C"} 159 | ] 160 | ``` 161 | 162 | 6. **LLM responds with final answer:** Now having all the information, it returns `{"action": "final_answer", "answer": "It's sunny and 22°C in Moraira."}`, and the loop exits. 163 | 164 | For more sophistication, we could add reasoning capabilities, 165 | which can be handy for synthesizing results from multiple tool calls inside our agent loop from above. 166 | 167 | ```python 168 | def reasoning(user_input: str) -> str: 169 | """Generate an answer using step-by-step reasoning.""" 170 | prompt = user_input + "\nLet's think step by step:" 171 | reasoning_and_answer = inference(prompt) 172 | # The LLM naturally produces reasoning steps followed by the answer. 173 | # This works because LLMs are trained on Chain of Thought data. 174 | # For inspection/validation, use structured_inference with 175 | # reasoning_schema = {"reasoning_steps": "list[str]", "final_answer": "string"} 176 | return reasoning_and_answer # We could also return just the final_answer here. 177 | ``` 178 | 179 | And that's it. I hope agents are boring now. :) 180 | -------------------------------------------------------------------------------- /creating_a_replacement_for_the_switch_statement_in_cpp_that_also_works_for_strings.md: -------------------------------------------------------------------------------- 1 | # Creating a replacement for the switch statement in C++ that also works for strings 2 | 3 | The [switch statement](http://en.cppreference.com/w/cpp/language/switch) in C++ only works for `int`s, `enum`s or a values convertible to one of them. This probably will not change in the C++ standard soon, since some [low-level optimizations](https://en.wikipedia.org/wiki/Branch_table) of these statements depend on it. 4 | 5 | So the following chatbot, that likely passes every Turing test, sadly does not compile. 6 | 7 | ```c++ 8 | #include 9 | #include 10 | 11 | void say(const std::string& text) 12 | { 13 | std::cout << text << std::endl; 14 | } 15 | 16 | void quit(const std::string& msg, bool* running_flag) 17 | { 18 | if (running_flag) 19 | *running_flag = false; 20 | say(msg); 21 | } 22 | 23 | int main() 24 | { 25 | bool running = true; 26 | while (running) 27 | { 28 | std::string input; 29 | std::cin >> input; 30 | 31 | switch(input) { 32 | case "hi": say("hey"); break; 33 | case "whazzup": say("wazzuuuup"); break; 34 | case "bye": quit("see ya", &running); break; 35 | default: say("wat?"); break; 36 | } 37 | } 38 | } 39 | ``` 40 | 41 | Sure, we could [hash the string](https://stackoverflow.com/a/16388610/1866775) to an integral, and we probably will not run into [collisions](https://en.wikipedia.org/wiki/Hash_table#Collision_resolution) with that approach, but it restricts the case values to compile time constants, which would render it impossible to for example get them from an external configuration. Long `if - else if` chains also are not always nice to read. 42 | 43 | So let's - just out of curiosity, and as an exercise - create a replacement `switch`, that at least is suitable for our particular use case. Perhaps we learn something interesting on our way. 44 | 45 | OK, what does a `switch` actually do? It takes a mapping from possible values to statements, a default statement and of course the value to match. Since functions are first-class citizens in C++, we should be able to write our `switch2` without ugly [macros](http://en.cppreference.com/w/cpp/preprocessor/replace). 46 | 47 | A possible implementation could look as follows: 48 | 49 | ```c++ 50 | template 51 | void switch2(const Key_t& key, 52 | const std::unordered_map>& dict, 53 | const std::function& def) 54 | { 55 | const auto it = dict.find(key); 56 | if (it != dict.end()) 57 | it->second(); 58 | else 59 | def(); 60 | } 61 | ``` 62 | 63 | It takes the following three parameters 64 | * `key` is simply the value we want to switch on, `input` in our example from above. 65 | * `dict` is the mapping from possible values to functions that should be executed. Such a function can of course include multiple statements and execute an arbitrary number of side effects. 66 | * `def` is the function that should be executed if `key` is not present in `dict`. 67 | 68 | `switch2` then simply looks up `key` in `dict` and acts accordingly. 69 | 70 | Naively we could try to use it like that: 71 | 72 | ```c++ 73 | switch2(input, { 74 | {"hi", say("hey")}, 75 | {"whazzup", say("wazzuuuup")}, 76 | {"bye", quit("see ya", &running)}}, 77 | say("wat?")); 78 | ``` 79 | 80 | But even if it would compile (which it does not), it would immediately run all three calls do `say` and the call to `quit` before any switching happens. 81 | 82 | It could be done using lambdas: 83 | 84 | ```c++ 85 | switch2(input, { 86 | {"hi", [](){ say("hey"); }}, 87 | {"whazzup", [](){ say("wazzuuuup"); }}, 88 | {"bye", [&](){ quit("see ya", &running); }}}, 89 | [](){ say("wat?"); }); 90 | ``` 91 | 92 | But that adds quite some syntactical noise. So let's try to find a way to defer a function call in some other way. This is surprisingly easy in C++. We just need a function that takes another function `f` and a list of arguments and returns a nullary function, that runs f with the given arguments when called. Since we are doing some mental gymnastics exercise here anyway, we can write it on our own instead of using [`std::bind`](http://en.cppreference.com/w/cpp/utility/functional/bind). 93 | 94 | ```c++ 95 | template 96 | std::function defer(F f, Args ... args) 97 | { 98 | return [f, args...]() 99 | { 100 | f(args...); 101 | }; 102 | } 103 | ``` 104 | 105 | Now we are in the lucky situation to be able to do the following: 106 | 107 | ```c++ 108 | switch2(input, { 109 | {"hi", defer(say, "hey")}, 110 | {"whazzup", defer(say, "wazzuuuup")}, 111 | {"bye", defer(quit, "see ya", &running)}}, 112 | defer(say, "wat?")); 113 | ``` 114 | 115 | This reads not *that* bad. We need to write `defer` but we do not need `break` statements any more, which can accidentally be forgotten easily in usual `switch` blocks. But the most important thing is we can now talk to our awesome AI for hours. 116 | 117 | Is there an advantage over an `if - else if` chain like the following you ask? 118 | 119 | ```c++ 120 | if (input == "hi") say("hey"); else 121 | if (input == "whazzup") say("wazzuuuup"); else 122 | if (input == "bye") quit("see ya", &running); else 123 | say("wat?"); 124 | ``` 125 | 126 | And that is a very good question. Up to now there was none, but one fruit is hanging quite low. 127 | 128 | In longer chains it can happen that one case is covered more than once, which produces strongly unwanted run-time behavior. With our `switch2` we can prevent this from happening - not at compile time but at least at run time. 129 | 130 | ```c++ 131 | template 132 | void switch2(const Key_t& key, 133 | const std::vector>>& pairs, 134 | const std::function& def) 135 | { 136 | std::unordered_map> dict; 137 | for (const auto& entry : pairs) 138 | dict.insert(entry); 139 | assert(dict.size() == pairs.size()); 140 | const auto it = dict.find(key); 141 | if (it != dict.end()) 142 | it->second(); 143 | else 144 | def(); 145 | } 146 | ``` 147 | 148 | Now `switch2` takes a `vector` with key-function pairs. If the resulting dictionary does not have the same number of entries as the vector, at least one key was present more than once and our debugger will tell us immediately on the first call of switch with this invalid set of keys. 149 | 150 | Remember that some run-time overhead (hashing, lookup, no perfect forwarding in `defer`) is involved in using `switch2`, and that it possibly could show up in your profiler if used in a time-critical section of your code. 151 | 152 | If you are interested in learning more about functional programming using C++ you might enjoy [my video course on Udemy](https://www.udemy.com/functional-programming-using-cpp). I promise it contains code more useful in everyday usage than this article. ;) 153 | 154 | What do you think about our little switch replacement? I would be happy to read your comments in the [reddit discussion](https://www.reddit.com/r/programming/comments/6e0hzr/creating_a_replacement_for_the_switch_statement/). 155 | 156 | 157 | 158 | --- 159 | 160 | full code: 161 | 162 | ```c++ 163 | #include 164 | #include 165 | #include 166 | #include 167 | #include 168 | #include 169 | #include 170 | 171 | template 172 | void switch2(const Key_t& key, 173 | const std::vector>>& pairs, 174 | const std::function& def) 175 | { 176 | std::unordered_map> dict; 177 | for (const auto& entry : pairs) 178 | dict.insert(entry); 179 | assert(dict.size() == pairs.size()); 180 | const auto it = dict.find(key); 181 | if (it != dict.end()) 182 | it->second(); 183 | else 184 | def(); 185 | } 186 | 187 | template 188 | std::function defer(F f, Args ... args) 189 | { 190 | return [f, args...]() 191 | { 192 | f(args...); 193 | }; 194 | } 195 | 196 | void say(const std::string& text) 197 | { 198 | std::cout << text << std::endl; 199 | } 200 | 201 | void quit(const std::string& msg, bool* running_flag) 202 | { 203 | if (running_flag) 204 | *running_flag = false; 205 | say(msg); 206 | } 207 | 208 | int main() 209 | { 210 | bool running = true; 211 | while (running) 212 | { 213 | std::string input; 214 | std::cin >> input; 215 | 216 | switch2(input, { 217 | {"hi", defer(say, "hey")}, 218 | {"whazzup", defer(say, "wazzuuuup")}, 219 | {"bye", defer(quit, "see ya", &running)}}, 220 | defer(say, "wat?")); 221 | } 222 | } 223 | ``` 224 | -------------------------------------------------------------------------------- /functional_programming_in_cpp_with_the_functionalplus_library_today_hackerrank_challange_gemstones.md: -------------------------------------------------------------------------------- 1 | # Functional programming in C++ with the FunctionalPlus library; today: HackerRank challange "Gemstones" 2 | 3 | 4 | Functional programming becomes more prominent even in C++ these days, but sometimes the resulting code can feel awkward - especially with all the iterators one has to provide for the STL algorithms. **[FunctionalPlus](https://github.com/Dobiasd/FunctionalPlus)** (a small header-only library) can help overcome this issue and make your code concise and readable again. 5 | 6 | This article is targeted towards C++ programmers new to FunctionalPlus and with little experience in functional programming. It will show how the [HackerRank challange "Gemstones"](https://www.hackerrank.com/challenges/gem-stones) can be solved elegantly with the FunctionalPlus approach. 7 | 8 | 9 | ## The challange 10 | 11 | The (slightly modified) challange boils down to 12 | 13 | > Find the number of characters present in every line of an input text. 14 | 15 | As an example, the following input 16 | 17 | Lorem ipsum 18 | dolor sit amet, 19 | consectetur, 20 | adipisci velit 21 | 22 | should produce the following output 23 | 24 | 2 25 | 26 | since only `e` and `s` can be found in all four lines. The challange calls these characters "gems". 27 | 28 | 29 | ## Getting the data 30 | 31 | Assuming we already have the input in one single `std::string`, we first need to split it into lines. FunctionalPlus provides a convenient function (actually in form of a template) to do this: 32 | 33 | ```c++ 34 | // split_lines : (String, Bool) -> [String] 35 | vector fplus::split_lines(const string& str, bool allow_empty); 36 | ``` 37 | 38 | Using it looks like this: 39 | 40 | ```c++ 41 | const auto lines = fplus::split_lines(input, false); 42 | ``` 43 | 44 | 45 | ## Type annotations and API search 46 | 47 | The strange looking comment above the function declaration is a type annotation. 48 | 49 | ```haskell 50 | split_lines : (String, Bool) -> [String] 51 | ``` 52 | 53 | simply says 54 | 55 | > split_lines is a function taking a string and a bool and returns a container (i.e. `list`/`vector`/etc. ) of strings. 56 | 57 | Type annotations make it easier to find and understand function templates in `fplus` with some practice. If you are looking for a function implementing a certain small task, you can use the [API-search website](http://www.editgym.com/fplus-api-search/), where you can search by name, description or aforementioned annotation. Later we will see, why searching by type can come in very handy. 58 | 59 | 60 | ## The core algorithm 61 | 62 | OK, now that we have our data prepared, let's think about the core algorithm. If we regard every line as a set of characters, we need to calculate the [set intersection](https://en.wikipedia.org/wiki/Intersection_(set_theory)) of all given sets, since this would exactly contain the characters present in every line. 63 | 64 | So let's convert our `std::vector lines` into a set of characters. 65 | 66 | ```c++ 67 | typedef std::set character_set; 68 | ``` 69 | 70 | The function 71 | 72 | ```c++ 73 | // fplus::convert_container string -> character_set 74 | fplus::convert_container 75 | ``` 76 | 77 | can convert a `std::string` into a `character_set`. To apply it to all lines, we could write a for loop, but we have something better: 78 | 79 | ```haskell 80 | fplus::transform : ((a -> b), [a]) -> [b] 81 | ``` 82 | 83 | Let's disect its type annotation first. 84 | `fplus::transform` takes two parameters, the first one `(a -> b)` is a function taking a value of type `a` and returning a value of type `b`. These types could be anything from simple `int`s to complex classes. The second parameter is a container (e.g. `std::vector`) full of `a`s. `transform` applies the given function to all elements in the container and returns a container with the resulting elements. 85 | 86 | Using it to perform the conversion of all lines looks as follows: 87 | 88 | ```c++ 89 | const auto sets = fplus::transform( 90 | fplus::convert_container, 91 | lines); 92 | ``` 93 | 94 | So in our case the `(a -> b)` for `fplus::transform` is `(std::string -> character_set)`, which leads to: 95 | 96 | ```haskell 97 | fplus::transform : ((std::string -> character_set), [std::string]) -> [character_set] 98 | ``` 99 | 100 | Now that we have our sets, we need to intersect them. [`std::set_intersection`](http://en.cppreference.com/w/cpp/algorithm/set_intersection) can be used to calculate the intersection of two sets. FunctionalPlus provides a wrapper around this with `fplus::set_intersection`, which is easier to use, since one does not need to care about providing iterators. 101 | 102 | ```c++ 103 | // example 104 | fplus::set_intersection(set({1,2,3}), set({1,3,4})) == set({1,3}); 105 | ``` 106 | 107 | Now we need to intersect not only two but all sets that resulted from the lines of the input. We could write a loop to repeatedly call `set_intersection`, but the functional paradigm provides a more elegant solution. 108 | 109 | For easier visualisation let's write `fplus::set_intersection` as a binary infix operator called `^`. So `^` is a function that takes two sets and returns one set. In the type notation mentioned above, this would be: 110 | 111 | (^) : (Set, Set) -> Set 112 | 113 | (read: "`(^)` is a function taking two sets and returning a set.") 114 | 115 | So what we eventually want is: 116 | 117 | set1 ^ set2 ^ set3 ^ set4 ... 118 | 119 | This means we need something that combines a list of sets to one set by joining them via applications of our operator. If this would be a distinct function, it would take the operator and a list of sets and return a set. So its type would be: 120 | 121 | (Operator, [Set]) -> Set 122 | 123 | with 124 | 125 | Operator == (Set, Set) -> Set 126 | 127 | resulting in 128 | 129 | ((Set, Set) -> Set, [Set]) -> Set 130 | 131 | Wouldn't it be nice, if FunctionalPlus already provided such a thing? ;-) A quick request to the [API search](http://www.editgym.com/fplus-api-search/) gives the following result: 132 | 133 | ![api_search_fold_left_1_001](/functional_programming_in_cpp_with_the_functionalplus_library_today_hackerrank_challange_gemstones/api_search_fold_left_1_001.png) 134 | 135 | `fold_left_1` is just what we need! 136 | 137 | Actually, a [`fold` (sometimes known as `reduce`)](https://en.wikipedia.org/wiki/Fold_(higher-order_function)) is quite common in functional programming. With [`std::accumulate`](http://en.cppreference.com/w/cpp/algorithm/accumulate) the STL provides something similar, but like with `set_intersection`, the version from `fplus` is easier to use. 138 | 139 | And that is everything we need. With `fold_left_1` and `set_intersection` we can calculate the overall intersection set succinctly: 140 | 141 | ```c++ 142 | const auto gem_elements = fplus::fold_left_1( 143 | fplus::set_intersection, sets); 144 | ``` 145 | 146 | 147 | ## Input/Output 148 | 149 | The only missing piece is getting the text from standard input and writing the result to standard output. To avoid writing this boilderplate code, we can use `fplus::interact`, which already provides this functionality. 150 | 151 | ```haskell 152 | interact : (String -> String) -> Io () 153 | ``` 154 | 155 | `Io ()` simply says that we receive a function performing some side effects. 156 | 157 | `interact` - like our `fold` and `transform` - is a [higher-order function](https://en.wikipedia.org/wiki/Higher-order_function), because it takes a function as an argument. In our case we give it the instructions on how to process the input and it uses it to handle all the I/O stuff for us. 158 | 159 | 160 | ## Wrapping up 161 | 162 | Putting everything together, our solution looks like this: 163 | 164 | ```c++ 165 | #include 166 | 167 | std::string process(const std::string& input) 168 | { 169 | using namespace fplus; 170 | 171 | typedef std::set character_set; 172 | 173 | // Get lines from input. 174 | const auto lines = split_lines(input, false); 175 | 176 | // Convert string lines into sets of chars. 177 | const auto sets = transform( 178 | convert_container, 179 | lines); 180 | 181 | // Build the intersection of all character sets. 182 | const auto gem_elements = fold_left_1( 183 | set_intersection, sets); 184 | 185 | // Convert gem_elements.size() into a std::string. 186 | return show(size_of_cont(gem_elements)); 187 | } 188 | 189 | int main() 190 | { 191 | fplus::interact(process)(); 192 | } 193 | ``` 194 | 195 | What do you think? Is this code you personally would like to maintain compared to what an imperative version could look like? I would be happy to read your comments in the [reddit discussion](https://www.reddit.com/r/programming/comments/543rav/functional_programming_in_c_with_the/). 196 | 197 | By the way, if you are interested in learning more about functional programming using C++ you might enjoy [my video course on Udemy](https://www.udemy.com/functional-programming-using-cpp). -------------------------------------------------------------------------------- /simple_collaborative_filtering_in_pure_postgresql.md: -------------------------------------------------------------------------------- 1 | # Simple collaborative filtering in pure PostgreSQL 2 | 3 | Many apps/websites must decide which items to show their users to maximize engagement. Examples include: 4 | 5 | - streaming (e.g. Netflix) 6 | - e-commerce (e.g. Amazon) 7 | - jobs boards (e.g. Indeed) 8 | - dating (e.g. Tinder) (Let's temporarily ignore that labeling human beings as "items" is not poetic/romantic/nice/accurate.) 9 | 10 | The main approaches to obtain these recommendations are: 11 | 12 | - **content-based filtering**: Based on the features of each user and each item, we make predictions about which items the user might like, and recommend them. 13 | - **collaborative filtering**: 14 | - **user-user**: Find similar users and recommend items they liked. 15 | - **item-item**: Find items similar to the ones already liked by the user and recommend them. 16 | 17 | (For the pros and cons of each approach, just ask your favorite ~[bullshit machine](https://thebullshitmachines.com/)~ ~LLM~ AI.) 18 | 19 | This article demonstrates user-user collaborative filtering (with support for negative engagement signals) using only PostgreSQL. 20 | 21 | (An item-item approach can be implemented similarly.) 22 | 23 | ## Data model 24 | 25 | The sum of all user-item interactions gives a signal. Any interval could be chosen (e.g., 1 to 5 stars). In our case, we use: 26 | 27 | - `-1.0`: most negative (If you don't intend to provide negative signals, e.g. "dislikes", `0.0` works fine too.) 28 | - ` 1.0`: most positive 29 | 30 | ```sql 31 | CREATE TABLE signals ( 32 | user_id INTEGER NOT NULL, 33 | item_id INTEGER NOT NULL, 34 | signal REAL NULL CHECK (signal >= -1.0 AND signal <= 1.0), 35 | UNIQUE (user_id, item_id) 36 | ); 37 | CREATE INDEX signals_user_id_index ON signals (user_id) INCLUDE (item_id, signal); 38 | CREATE INDEX signals_item_id_index ON signals (item_id) INCLUDE (user_id, signal); 39 | CREATE INDEX signals_user_id_item_id_index ON signals (user_id, item_id) INCLUDE (signal); 40 | ``` 41 | 42 | The application would continuously insert rows into the table or [update existing signals](https://gist.github.com/Dobiasd/e12fbbb702e7da4c7677f65716fa2a27) on repeated interactions. 43 | 44 | ## Example data 45 | 46 | Let's insert some minimal test data for experimentation: 47 | 48 | ```sql 49 | INSERT INTO signals (user_id, item_id, signal) VALUES 50 | (10, 20, 0.9), 51 | (10, 21, -0.4), 52 | (10, 22, 0.7), 53 | (10, 23, -0.5), 54 | (15, 20, 0.8), 55 | (15, 21, -0.3) 56 | ; 57 | ``` 58 | 59 | So our user-item-interaction matrix looks like this: 60 | 61 | | user/item | 20 | 21 | 22 | 23 | 62 | | --------- | --- | ---- | --- | ---- | 63 | | 10 | 0.9 | -0.4 | 0.7 | -0.5 | 64 | | 15 | 0.8 | -0.3 | | 65 | 66 | Meaning: 67 | 68 | - User `10` likes items `20` and `22`, but dislikes items `21` and `23`. 69 | - User `11` also likes item `20` and dislikes item `21`. 70 | 71 | So, users `10` and `11` seem to have a similar taste, and we would thus expect user `11` to also like item `22`, but not `23`. 72 | 73 | ## Calculating user similarity 74 | 75 | To get to this prediction, we first need to calculate how similar these two users are. Since we will later have more users, let's calculate the similarity for all possible user pairs: 76 | 77 | ```sql 78 | CREATE VIEW user_similarities AS ( 79 | WITH pairs AS ( 80 | SELECT 81 | a.user_id user_id_a, 82 | b.user_id user_id_b, 83 | COUNT(1) overlap, 84 | AVG(1.0 - ABS(a.signal - b.signal)) agreement 85 | FROM signals a JOIN signals b ON a.item_id = b.item_id 86 | WHERE a.user_id != b.user_id 87 | GROUP BY a.user_id, b.user_id 88 | ) 89 | SELECT *, overlap * agreement user_similarity -- in PROD, only user_similarity is needed 90 | FROM pairs 91 | ); 92 | ``` 93 | 94 | In the above: 95 | 96 | - `overlap` counts the number of items, both users in a pair already have interacted with. 97 | - `agreement` is the Manhattan distance of the two interaction vectors, normalized to `[-1.0, 1.0]`. (Other similarity metrics would also be possible, e.g., euclidean distance or cosine similarity.) 98 | - `user_similarity` is `agreement` weighted by `overlap`. 99 | 100 | This gives a matrix of size `u × u` with `u` being the number of users. Example (not from the `INSERT` above): 101 | 102 | | user | 1 | 2 | 3 | 103 | | ---- | ---- | --- | ---- | 104 | | 1 | 1.0 | | -0.2 | 105 | | 2 | | 1.0 | 0.6 | 106 | | 3 | -0.2 | 0.6 | 1.0 | 107 | 108 | The diagonal of this matrix is not interesting, because it's just the fact, that a user is 100% similar to themselves. 109 | 110 | Also, the matrix is symmetric (because our user similarity metric is commutative), so only one half (top-right triangle) is needed. But later queries will become simpler by not getting rid of the second half, so we keep it. 111 | 112 | For bigger datasets, this matrix is likely very sparse, i.e., many user pairs have no overlap, and thus no similarity between them can be calculated. 113 | 114 | Negative user similarities can only occur when you use negative signals (`signals.signal < 0.0`). They mean, that this other user has opposite preferences. 115 | 116 | ## Generating recommendations 117 | 118 | Having this user-user-similarity matrix, we can derive our recommendations: 119 | 120 | ```sql 121 | CREATE VIEW user_recommendations AS ( 122 | WITH candidates AS ( 123 | SELECT DISTINCT a.user_id, b.item_id FROM signals a CROSS JOIN signals b 124 | ) 125 | SELECT 126 | candidates.user_id, 127 | candidates.item_id, 128 | AVG(user_similarity * s.signal) score 129 | FROM candidates 130 | JOIN user_similarities o ON o.user_id_a = candidates.user_id 131 | JOIN signals s ON s.user_id = o.user_id_b AND s.item_id = candidates.item_id 132 | LEFT JOIN signals known ON (candidates.user_id = known.user_id AND candidates.item_id = known.item_id) 133 | WHERE known.item_id IS NULL 134 | GROUP BY candidates.item_id, candidates.user_id 135 | ); 136 | ``` 137 | 138 | In the above: 139 | 140 | - `candidates` is the cartesian product, giving all potential user-item pairs. 141 | - We join with `user_similarities` to get all users with a known similarity. 142 | - We join with `signals` to collect the signals from these other users. 143 | - `signals` as `known` is only used to remove items the user already interacted with (`WHERE known.item_id IS NULL`). It's a "`LEFT ANTI JOIN`". 144 | 145 | ## Example recommendations 146 | 147 | Let's look into it: 148 | 149 | ```sql 150 | SELECT * FROM user_recommendations; 151 | ``` 152 | 153 | ``` 154 | user_id | item_id | score 155 | ---------+---------+--------------------- 156 | 15 | 22 | 1.2600000077486033 157 | 15 | 23 | -0.9000000208616257 158 | ``` 159 | 160 | Indeed, it gives the prediction we were hoping for. :D 161 | 162 | And that's already it! 163 | 164 | ## Load test 165 | 166 | Let's put some load on it: 167 | 168 | ```sql 169 | TRUNCATE signals; 170 | ``` 171 | 172 | ```sql 173 | INSERT INTO signals (user_id, item_id, signal) 174 | SELECT 175 | (random() * 100000)::integer, 176 | (random() * 1000)::integer, 177 | random() * 2.0 - 1.0 178 | FROM generate_series(1, 1100000) 179 | ON CONFLICT (user_id, item_id) DO NOTHING; -- to skip duplicates 180 | ``` 181 | 182 | We now have: 183 | 184 | - 100k users 185 | - 1k items 186 | - over 1M interactions 187 | 188 | So let's fetch the recommendations for a user: 189 | 190 | ```sql 191 | SELECT * FROM user_recommendations WHERE user_id = 42 ORDER BY score DESC LIMIT 10; 192 | ``` 193 | 194 | ``` 195 | user_id | item_id | score 196 | ---------+---------+--------------------- 197 | 42 | 438 | 0.09187011460708962 198 | 42 | 680 | 0.08926545207792841 199 | 42 | 897 | 0.08884137042722827 200 | 42 | 302 | 0.08332909557853353 201 | 42 | 906 | 0.08295293443873514 202 | 42 | 276 | 0.08270721418927421 203 | 42 | 340 | 0.08173571305819198 204 | 42 | 9 | 0.08054026719749253 205 | 42 | 299 | 0.08029276479669858 206 | 42 | 317 | 0.07946202537955073 207 | ``` 208 | 209 | On my machine (with a default Postgres running in Docker), queries like this take (after some warmup of the Postgres engine) ~200 ms, which is not bad because we applied almost zero optimizations so far. 210 | 211 | ## Potential optimizations 212 | 213 | In case it's too slow (or becomes too slow with more data), potential optimizations include: 214 | 215 | - Regularly materializing `user_similarities` (and maybe even `recommendations`) periodically 216 | - Limiting the number of pairs considered per user when generating `user_similarities` 217 | - Switch from the user-user approach to the item-item approach if this results in a smaller similarity table 218 | - Shard your data by certain properties of users/items, e.g., the user's country 219 | - Apply standard database tuning techniques 220 | - Use a fancier technology. PostgreSQL is great because of its simplicity and ubiquity, but at some scale, it might struggle. Other approaches could also enable techniques like decomposing the user-item matrix into the product of two smaller matrices to speed up downstream computations. 221 | 222 | ## Further enhancements 223 | 224 | In a mature real-life application, you might want to experiment with some additional ideas: 225 | 226 | - Include user demographics to calculate user similarities even for users without interactions to overcome their cold-start problem 227 | - Augmenting the results with the previously mentioned item-item approach (and include item features to overcome item-cold-start problems) 228 | - Calculate similarities transitively to a certain depth (currently it's only `1`) 229 | 230 | ## Closing words 231 | 232 | I hope you enjoyed reading this article as much as I did writing it. If you gather additional insights while using this approach, I'd be happy to hear from you. 233 | -------------------------------------------------------------------------------- /accurate_timing_of_strava_segments/accurate_timing_of_strava_segments.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | """ 4 | Accurate timing of Strava segments 5 | """ 6 | 7 | __author__ = "Tobias Hermann" 8 | __copyright__ = "Copyright 2023, Tobias Hermann" 9 | __license__ = "MIT" 10 | __email__ = "editgym@gmail.com" 11 | 12 | import argparse 13 | import datetime 14 | from typing import List, Tuple 15 | 16 | import geopy.distance 17 | import requests 18 | from sympy.geometry import Point, Line, Segment as GeoSegment 19 | from tcxreader.tcxreader import TCXReader, TCXTrackPoint 20 | 21 | 22 | def log_msg(msg: str) -> None: 23 | """Generic console logging with timestamp.""" 24 | print(f'{datetime.datetime.now()}: {msg}', flush=True) 25 | 26 | 27 | # Start of the actual algorithm 28 | 29 | 30 | def track_point_to_point(trackpoint: TCXTrackPoint) -> Point: 31 | """Point(longitude, latitude).""" 32 | return Point(trackpoint.longitude, trackpoint.latitude) 33 | 34 | 35 | def geo_projection(line: GeoSegment, point: Point) -> Point: 36 | """Orthogonal projection for geo coordinates.""" 37 | line_center: Point = (line.p1 + line.p2) / 2 38 | # If the line goes exactly through one of earth's poles, the numbers explode. 39 | x_scale = distance(line_center, Point(line_center.x + 0.1, line_center.y)) * 10 40 | y_scale = distance(line_center, Point(line_center.x, line_center.y + 0.1)) * 10 41 | euclidean_line = Line(Point(line.p1.x / x_scale, line.p1.y / y_scale), 42 | Point(line.p2.x / x_scale, line.p2.y / y_scale)) 43 | euclidean_point = Point(point.x / x_scale, point.y / y_scale) 44 | euclidean_projection = euclidean_line.projection(euclidean_point) 45 | projection = Point(euclidean_projection.x * x_scale, euclidean_projection.y * y_scale) 46 | return projection 47 | 48 | 49 | def closest_point_on_step(step_start: TCXTrackPoint, 50 | step_end: TCXTrackPoint, 51 | point: Point) -> TCXTrackPoint: 52 | """Find the closest point on a step (line segment) and interpolate the timestamp.""" 53 | step = GeoSegment(track_point_to_point(step_start), track_point_to_point(step_end)) 54 | 55 | step_length = distance(step.p1, step.p2) 56 | distance_to_step_start = distance(step.p1, point) 57 | distance_to_step_end = distance(step.p2, point) 58 | 59 | # Find the orthogonal projection 60 | projection = geo_projection(step, point) 61 | 62 | projection_is_outside_step = \ 63 | distance(step.p1, projection) > step_length or \ 64 | distance(step.p2, projection) > step_length 65 | if projection_is_outside_step: 66 | return step_start if distance_to_step_start < distance_to_step_end else step_end 67 | 68 | step_duration_s = (step_end.time - step_start.time).total_seconds() 69 | step_fraction = float(distance(step.p1, projection) / step_length) 70 | dt_s = step_fraction * step_duration_s 71 | exact_time = step_start.time + datetime.timedelta(seconds=dt_s) 72 | return TCXTrackPoint( 73 | longitude=float(projection.x), 74 | latitude=float(projection.y), 75 | time=exact_time) 76 | 77 | 78 | def closest_virtual_trackpoint(point: Point, trackpoints: List[TCXTrackPoint]) -> TCXTrackPoint: 79 | """Find the closest trackpoints (potentially interpolated) on a polygon chain of trackpoints.""" 80 | if len(trackpoints) == 1: 81 | return trackpoints[0] 82 | candidates = [closest_point_on_step(tp1, tp2, point) 83 | for tp1, tp2 in zip(trackpoints, trackpoints[1:])] 84 | return min(map(lambda tp: (distance(track_point_to_point(tp), point), tp), candidates))[1] 85 | 86 | 87 | def calc_effort_time(segment: GeoSegment, 88 | trackpoints: List[TCXTrackPoint], 89 | start_idx: int, 90 | end_idx: int) -> float: 91 | """Return the precise effort time.""" 92 | points_close_to_start = with_surrounding_trackpoints(trackpoints, start_idx) 93 | points_close_to_end = with_surrounding_trackpoints(trackpoints, end_idx) 94 | start = closest_virtual_trackpoint(segment.p1, points_close_to_start) 95 | end = closest_virtual_trackpoint(segment.p2, points_close_to_end) 96 | return float((end.time - start.time).total_seconds()) 97 | 98 | 99 | def with_surrounding_trackpoints( 100 | trackpoints: List[TCXTrackPoint], 101 | center_idx: int) -> List[TCXTrackPoint]: 102 | """Get trackpoint surrounding a center one.""" 103 | all_idxs = [center_idx - 2, center_idx - 1, center_idx, center_idx + 1, center_idx + 2] 104 | valid_idxs = sorted(list(set((filter(lambda idx: 0 <= idx < len(trackpoints), all_idxs))))) 105 | return [trackpoints[idx] for idx in valid_idxs] 106 | 107 | 108 | # Auxiliary things 109 | 110 | 111 | def is_trackpoint_close_to_point(trackpoint: TCXTrackPoint, point: Point) -> bool: 112 | """For performance, we simply compare latitude and longitude. 113 | An actual implementation would do probably something 114 | that also works on the earth's poles.""" 115 | return bool( 116 | float(point.y) - 0.0005 <= trackpoint.latitude <= float(point.y) + 0.0005 and 117 | float(point.x) - 0.0005 <= trackpoint.longitude <= float(point.x) + 0.0005) 118 | 119 | 120 | def distance(point1: Point, point2: Point) -> float: 121 | """Distance in meters between two geo coordinates""" 122 | return float(geopy.distance.geodesic((point1.y, point1.x), (point2.y, point2.x)).m) 123 | 124 | 125 | def find_indexes_of_trackpoints_closest_to_first_effort_start_and_end( 126 | segment: GeoSegment, trackpoints: List[TCXTrackPoint]) -> Tuple[int, int]: 127 | """ 128 | This could be the replaced by any other (probably already existing) way 129 | of finding the closest trackpoints. 130 | """ 131 | invalid_idx = -1 132 | invalid_distance = 99999999.9 133 | start_idx_dist: Tuple[int, float] = invalid_idx, invalid_distance 134 | end_idx_dist: Tuple[int, float] = invalid_idx, invalid_distance 135 | left_start_zone = None 136 | left_end_zone = None 137 | for point_idx, trackpoint in enumerate(trackpoints): 138 | 139 | # Find start of effort first. 140 | if is_trackpoint_close_to_point(trackpoints[point_idx], segment.p1): 141 | start_dist = distance(track_point_to_point(trackpoint), segment.p1) 142 | if start_idx_dist[0] == invalid_idx or \ 143 | start_dist < start_idx_dist[1] or \ 144 | left_start_zone: 145 | start_idx_dist = point_idx, start_dist 146 | left_start_zone = False # Skip previous passes though the start zone. 147 | else: 148 | left_start_zone = True 149 | 150 | # Only consider potential end points if they came after a start point. 151 | if start_idx_dist[0] != invalid_idx: 152 | if is_trackpoint_close_to_point(trackpoints[point_idx], segment.p2): 153 | end_dist = distance(track_point_to_point(trackpoint), segment.p2) 154 | if not end_idx_dist or end_dist < end_idx_dist[1]: 155 | end_idx_dist = point_idx, end_dist 156 | left_end_zone = False 157 | else: 158 | left_end_zone = True 159 | 160 | if left_start_zone and left_end_zone and \ 161 | start_idx_dist[0] != invalid_idx and end_idx_dist[0] != invalid_idx: 162 | break 163 | 164 | if start_idx_dist[0] == invalid_idx: 165 | raise RuntimeError("Did not find a suitable segment start point in the activity.") 166 | if end_idx_dist[0] == invalid_idx: 167 | raise RuntimeError("Did not find a suitable segment end point in the acticity.") 168 | return start_idx_dist[0], end_idx_dist[0] 169 | 170 | 171 | def calculate_effort_time(activity_tcx_path: str, segment: GeoSegment) -> None: 172 | """Calculate the effort time of an activity on a specific segment.""" 173 | tcx_reader = TCXReader() 174 | activity = tcx_reader.read(activity_tcx_path) 175 | log_msg(f'Analyzing activity: {activity_tcx_path}') 176 | 177 | trackpoints: List[TCXTrackPoint] = activity.trackpoints 178 | 179 | start_idx, end_idx = find_indexes_of_trackpoints_closest_to_first_effort_start_and_end( 180 | segment, trackpoints) 181 | 182 | coarse_segment_time = float( 183 | (trackpoints[end_idx].time - trackpoints[start_idx].time).total_seconds() 184 | ) 185 | 186 | # Refinement of the coarse start_idx-to-end_idx way. 187 | precise_segment_time = calc_effort_time(segment, trackpoints, start_idx, end_idx) 188 | 189 | log_msg(f'Coarse segment time: {coarse_segment_time=:0.1f}') 190 | log_msg(f'Precise segment time: {precise_segment_time=:0.1f}') 191 | 192 | 193 | def get_segment(access_token: str, segment_id: int) -> GeoSegment: 194 | """Download segment data using the Strava API""" 195 | 196 | # Hardcoded segment IDs, so one does not always need a valid access token. 197 | if segment_id == 4391619: # Marienfeld Climb 198 | return GeoSegment(Point(7.436902, 50.884516), Point(7.441928, 50.883243)) 199 | 200 | log_msg(f'Loading data for segment: {segment_id}') 201 | url = f'https://www.strava.com/api/v3/segments/{segment_id}' 202 | response = requests.get(url, 203 | headers={'Authorization': f'Bearer {access_token}'}, 204 | timeout=10 205 | ).json() 206 | log_msg(response) 207 | start_lat, start_lng = response['start_latlng'] 208 | end_lat, end_lng = response['end_latlng'] 209 | name = response['name'] 210 | log_msg(f'Loaded segment: {name}') 211 | return GeoSegment(Point(start_lng, start_lat), Point(end_lng, end_lat)) 212 | 213 | 214 | def main() -> None: 215 | """Parse command line and run calculation.""" 216 | parser = argparse.ArgumentParser('AccurateTimingOfStravaSegments') 217 | parser.add_argument('-a', '--activity_tcx_file', 218 | help='Use Sauce for Strava™ to export TCX files.') 219 | parser.add_argument('-s', '--segment_id', type=int, 220 | help='Can be copied from the URL in the browser.') 221 | parser.add_argument('-t', '--access_token', 222 | help='See: https://developers.strava.com/docs/authentication/') 223 | args = parser.parse_args() 224 | calculate_effort_time(args.activity_tcx_file, get_segment(args.access_token, args.segment_id)) 225 | 226 | 227 | if __name__ == '__main__': 228 | main() 229 | -------------------------------------------------------------------------------- /a_personal_generic_things_i_learned_as_a_software_developer_list.md: -------------------------------------------------------------------------------- 1 | # A personal, generic, things-I-learned-as-a-software-developer list 2 | 3 | With this article, I'd like to contribute my two subjective cents 4 | to the vast space of wisdom-offering mumblings. :) 5 | 6 | ## On the job 7 | 8 | ### Write code for humans 9 | 10 | Your code should clearly describe how it solves the domain problem 11 | in a way optimized for readability. 12 | The fact that a computer also has to be able to 13 | run it, is important of course, but secondary. 14 | Write and refactor for expressiveness first, 15 | and then, when your code is almost self-documenting, 16 | you might even only rarely need comments, and if so 17 | only to explain the "why" and not the "what" or "how". 18 | 19 | ### Consider the cost when deciding about optimizations 20 | 21 | We know "premature optimization is the root of all evil". 22 | But what if you have profiled your code and found a performance bottleneck? 23 | If it's a library and you don't know how it will be used, go crazy with fancy optimizations. 24 | But if the scope is known, consider the cost (initial development and maintenance) of 25 | implementing an optimization. 26 | Sometimes CPUs and RAM modules are cheaper than man-hours. 27 | 28 | ### Prototyping vs. production code 29 | 30 | When quickly hacking together a prototype 31 | it might make sense to write your code "bottom-up", 32 | i.e., start with some tricky implementation details to prove the feasibility of a solution's core. 33 | 34 | Also make it clear, that this is just a prototype, 35 | and later write new, clean and maintainable code for production. 36 | You might want to try to use the top-down approach here, 37 | so think from the outside in to obtain tidy interfaces and good abstraction layers. 38 | 39 | ### Don't blindly follow some methodology 40 | 41 | Some very project was developed using test-driven development? Great! Give it a try, 42 | but if it's not a good fit for your project, team or domain, work differently. 43 | 44 | > Research your own experience. Absorb what is useful, reject what is useless, add what is essentially your own. 45 | 46 | And while we are talking about testing: 47 | The usual testing pyramid, 48 | stating that one should have more unit tests than integration tests 49 | and more integration tests than system tests, 50 | might work perfectly in many situations: 51 | 52 | ```text 53 | /-------------- 54 | / system tests \ 55 | /------------------ 56 | / integration tests \ 57 | /---------------------- 58 | / unit tests \ 59 | --------------------------- 60 | ``` 61 | 62 | But maybe in a situation you don't need that many unit tests 63 | and better cover with integration tests if: 64 | 65 | - Your simple CRUD application does not have much logic to test. 66 | - You are using a programming language with a strong static type system, which already catches a whole class of bugs automatically. (Actually, if possible, try to express as much in types as possible, and constrain the possible states/values of your application as vigorously as possible by using types.) 67 | 68 | Other examples: 69 | 70 | - Some teams might be totally productive with weekly sprints and burn down charts, but that does not mean that this way of working is ideal for every team. 71 | - If you are working with multiple people on a project and you deploy regularly, using feature branches in your VCS might be mandatory. But in an earlier project stage with maybe just one or two devs, the simplicity of pushing to master directly might make sense. 72 | 73 | ### Learn about the domain 74 | 75 | Most software solves some real-world problems. 76 | Do not only focus on transforming some specifications into code. 77 | Try to understand the problem on a domain level. 78 | Speaking the same language as the experts not only helps in avoiding 79 | costly misunderstandings, but often by collaborating with 80 | the customer or product owner an even better solution might emerge. 81 | 82 | When I was developing image recognition software used in the processing 83 | of unsold newspapers, I learned a lot of helpful stuff 84 | by actually going on site and 85 | being confronted with the everyday problems arising in handling absurd amounts 86 | of sometimes fringed, wet or shrink-wrapped paper. 87 | 88 | Occasionally it even turns out the best solution does not even involve programming at all. 89 | And if you are not working for an agency, this might be great, 90 | because the code with the least complexity and defects is no code. 91 | Finding this such a maximally simple solution however is rarely easy and 92 | involves a "consultant" mindset. 93 | 94 | > It seems that perfection is attained not when there is nothing more to add, but when there is nothing more to remove. 95 | 96 | ### Weight technical dept economically 97 | 98 | When writing the code we usually want to have it nice and clean, 99 | so it can easily be maintained and extended. 100 | 101 | However technical debt is not something that is always bad. 102 | Sometimes you get a loan to buy a house, and you pay to it, including interest. 103 | It's the same with technical debt. 104 | You just have to be aware of the fact that you are creating it and why, 105 | so you can make conscious decisions about refactoring your code until it's perfect, 106 | or leaving a part of it in a messy state because: 107 | 108 | - the mess is isolated 109 | - the whole thing might be dropped soon anyways 110 | - there is something more valuable you can do with your time right now 111 | 112 | ### When stuck, create a minimal example 113 | 114 | Even if you might not post it as a question on stackoverflow.com 115 | creating a minimal example of your problem in a way that others 116 | could quickly understand it, often helps tremendously in finding 117 | the solution already. 118 | This kind of "Rubber duck debugging" helps to isolate the essence of the issue 119 | and can bring a deep understanding of the topic with it, 120 | instead of solving it by cargo culting. 121 | 122 | ## Personal advancement 123 | 124 | ### Basics > shiny things 125 | 126 | Having mastered just the very basics of computer science, different paradigms 127 | and engineering, in general, will help a lot in better using all these fancy new tools 128 | coming out every few months. 129 | If you understand the principles of 130 | concurrency, caching, design patterns, higher-order functions and SOLID 131 | you will see much more similarities between different languages/libraries/frameworks 132 | and thus have a much better learning curve. 133 | 134 | But this also involves low-level craftsmanship like being able to comfortably work 135 | with a Linux command line, git, a debugger, 136 | knowing your way around the shortcuts of a text editor, etc. 137 | 138 | ### Leave your comfort zone, regularly 139 | 140 | In order to grow, don't be afraid of something new 141 | that is guaranteed to make you feel uncomfortable 142 | because you will be a total newbie again. 143 | If you have worked with PHP and Symfony for years, 144 | spend some time to learn a bit of Kotlin, C or Haskell. 145 | Even if you not (yet?) use these things professionally, 146 | they will still make you better at your day-to-day tasks 147 | by expanding your perspective and bringing joy. 148 | 149 | So, don't be a "[programming-language-X] developer". 150 | Be a software engineer who just happens to currently use language X most, 151 | because it is the right tool for the job at hand. 152 | 153 | And if you don't know something, don't be scared to openly admit it. 154 | 155 | ### Be a jack of all trades 156 | 157 | Even if you specialize in something by building up deep knowledge about 158 | some technology, domain or abstraction level, 159 | try to also learn more broadly. 160 | This involves not only knowing about low-level implementations in Fortran and 161 | high-level code in Ruby, but also understanding systems architecture 162 | and the business in general. 163 | 164 | ### Immerse yourself in the culture, at least a bit 165 | 166 | You don't have to constantly go to conferences and read Hacker News all the time. 167 | 168 | But checking the top posts of the last month on /r/programming now and then might be a good idea. 169 | 170 | Not only will this help you know about the most important new tools and technologies, 171 | but you will also absorb some idioms and unconsciously develop a map in your head 172 | about what is possible. This, in turn, will help you make better decisions in your 173 | daily work. 174 | 175 | ### Consciously devote some time to learning 176 | 177 | Just a few hours of explicit learning a week can not only boost your 178 | excitement, but also your productivity in a profitable way. This can involve: 179 | 180 | - browsing interesting questions and answers on stackoverflow.com 181 | - reading some articles 182 | - watching a talk or listening to a podcast 183 | - playing with a toy project 184 | - reading one of the classics like 185 | - [Clean Code](https://www.amazon.com/Clean-Code-Handbook-Software-Craftsmanship/dp/0132350882) 186 | - [Design Patterns](https://www.amazon.com/Design-Patterns-Elements-Reusable-Object-Oriented/dp/0201633612) 187 | - [Domain driven design](https://www.amazon.com/Domain-Driven-Design-Tackling-Complexity-Software/dp/0321125215) 188 | - [The Mythical Man-Month](https://www.amazon.com/Mythical-Man-Month-Software-Engineering-Anniversary/dp/0201835959) 189 | - [The Pragmatic Programmer](https://www.amazon.com/Pragmatic-Programmer-Journeyman-Master/dp/020161622X) 190 | - [Structure and Interpretation of Computer Programs](https://www.amazon.com/Structure-Interpretation-Computer-Programs-Engineering/dp/0262510871) 191 | 192 | Maybe even maintain a "Things I don't know yet, but would like to learn" list. 193 | 194 | ### Don't treat social skills as something special 195 | 196 | They can, and should in order to be successful, be improved like 197 | technical skills too by theory and practice. Since many problems, 198 | one has to solve as a developer, are not purely technical 199 | (code reviews are only one of many examples) 200 | reading [Dale Carnegie](https://www.amazon.com/How-Win-Friends-Influence-People/dp/0671027034) 201 | or something similar should pay off quickly. 202 | 203 | ### Don't clutter your brain 204 | 205 | To think of cool and creative solutions, you need some space in your head. 206 | Try to use "external memory devices" for personal things too when possible, e.g.: 207 | 208 | - Kanban apps like Trello 209 | - Notes like Google Keep 210 | - Some calendar thingy 211 | - etc. 212 | 213 | The less stuff you have to keep in your head, the more freely you can think. 214 | (This btw. is also one the reasons why well-structured code is so valuable. 215 | It reduces cognitive load.) 216 | 217 | Also when you are totally stuck on a problem, leave it alone for some time 218 | and do something completely unrelated, away from the desk. 219 | Your brain might suddenly come up with a solution it would not have been 220 | able to generate while consciously focusing. 221 | -------------------------------------------------------------------------------- /internals_of_the_async_await_pattern_from_first_principles.md: -------------------------------------------------------------------------------- 1 | # Internals of the async/await pattern from first principles 2 | 3 | Many of us are using the async/await pattern, but understanding how it works under the hood is a different beast. This article sheds some (language-agnostic) light on how the needed machinery can be implemented. These insights are helpful when usage does not go according to plan, and I find it aesthetically pleasing too. 4 | 5 | A normal function (also known as a subroutine) returns once: 6 | 7 | ```java 8 | // Using an imaginary programming language to avoid eliciting language-specific associations 9 | // which could dilute the abstract concept with too many details. 10 | function foo() 11 | x = 21 12 | x = 2 * x 13 | return x 14 | 15 | assert foo() == 42 16 | ``` 17 | 18 | A Generator (also known as a semicoroutine, a special case of a coroutine) can return multiple times and preserve its state between invocations: 19 | 20 | ```java 21 | function foo() 22 | x = 21 23 | yield x 24 | x = 2 * x 25 | yield x 26 | 27 | g = foo() 28 | assert g.step() == 21 // Stepping the generator might look different in real languages. 29 | assert g.step() == 42 // But to be explicit, our imaginary language provides this method. 30 | ``` 31 | 32 | In many programming languages, the above generator is internally (by the interpreter or compiler) converted to a state machine as follows: 33 | 34 | ```java 35 | class foo_generator 36 | _state = 0 37 | running = true 38 | x = 21 39 | method step() 40 | if _state == 0 41 | _state = 1 42 | return x 43 | if _state == 1 44 | x = 2 * x 45 | _state = null 46 | running = false 47 | return Stopped x 48 | explode 49 | ``` 50 | 51 | And `g = foo()` simply obtains a new instance/object of that class. 52 | 53 | An additional (third) `g.step()` in the code above would return the special `Stopped` value. Some languages implement this as throwing an exception instead (`StopIteration` in Python for example). 54 | 55 | Even if your language does not actually transform the generator into a state machine, it probably does something with equivalent behavior, so it's a good mental model to hold on to. 56 | 57 | For our async/await pattern, we don't need to `yield` values from our generators, we just use it to suspend our function. 58 | 59 | ```java 60 | function foo() 61 | x = 21 62 | yield 63 | x = 2 * x 64 | return x 65 | ``` 66 | 67 | ```java 68 | class foo_generator 69 | _state = 0 70 | running = true 71 | x = 21 72 | method step() 73 | if _state == 0 74 | _state = 1 75 | return // ignoring the return type here for simplicity 76 | if _state == 1 77 | x = 2 * x 78 | _state = null 79 | running = false 80 | return Stopped x 81 | explode 82 | ``` 83 | 84 | So to get our answer from `foo`, we do something like this: 85 | 86 | ```java 87 | g = foo() 88 | g.step() 89 | answer = g.step() 90 | ``` 91 | 92 | From the outside, our function suspended itself during execution and continued where it left off to finally return the result. 93 | 94 | Composing such functions (including the suspend points) could look as follows: 95 | 96 | ```java 97 | function bar() 98 | x = 1 99 | yield 100 | x = 40 * x 101 | return x 102 | 103 | function foo() 104 | // Step through the whole thing, yielding (suspending) in between steps. 105 | await_gen = bar() 106 | await_result = await_gen.step() 107 | while await_gen.running 108 | yield 109 | await_result = await_gen.step() 110 | x = await_result 111 | x = x + 2 112 | return x 113 | ``` 114 | 115 | So `foo` would suspend itself once while awaiting `bar`. 116 | 117 | But this is syntactically rather clumsy. Luckily the designers of our imaginary language introduced the `await` keyword, which expands to something like the boilerplate above: 118 | 119 | ```java 120 | function foo() 121 | b = await bar() 122 | x = x + 2 123 | return x 124 | ``` 125 | 126 | (In reality, `await` might additionally do exception/error handling/propagation, but for compactness, we skip this here.) 127 | 128 | So the generator (calling it "coroutine" from here on) obtained from `foo()` now also operates step-wise, i.e., suspends itself before obtaining the result from `bar`. 129 | 130 | Btw., this way of suspending is called "suspend by return" (or sometimes "suspend up"), because `foo` and `bar` suspend themselves by returning to their caller. This is in contrast to normal function calls, in which a function "suspends" itself by calling some other function ("suspend by call" or "suspend down"). (When zooming in to the Assembly level, the return address might be passed to the callee via a pointer pushed to the stack, so when the callee is done, it knows where to jump to in the caller function.) The coroutines we're using here are called "stackless". The resume information is not stored on a call stack but in the (probably allocated on the heap) coroutine, each only knowing its own "stack frame". 131 | 132 | Let's get to the "co" (cooperative) in "coroutines". We now know how suspending/resuming works, so we'll look into how the implementation of an async library uses these building blocks. While we want things like IO to run in parallel in the background, our own functions only run concurrently, but not parallelly. I.e., we only use one thread, and the coroutines cooperate by purposely suspending to then be automatically resumed again later. To not manually resume them all the time, we need some executor to do this for us. 133 | 134 | ```java 135 | class Executor 136 | current = null 137 | _ready = empty list 138 | 139 | method submit(coroutine) 140 | _ready.push_right(coroutine) 141 | 142 | method run() 143 | while _ready.not_empty 144 | current = _ready.pop_left() 145 | current.step() 146 | if current.running 147 | submit(current) 148 | 149 | // Yes, it has to be global (or a Singleton) in this case. We'll see why in the next section. 150 | executor = Executor() 151 | ``` 152 | 153 | All this event loop does is advance (`step`) the coroutine (state machine) that is the most left in its `_ready` list, and in case it's not finished after that, put it back in the list. Once there is no more work left, it exits. 154 | 155 | We can already use this to run two coroutines concurrently! 156 | 157 | ```java 158 | function foo() 159 | yield 160 | print "a" 161 | yield 162 | print "b" 163 | yield 164 | print "c" 165 | 166 | function bar() 167 | yield 168 | print "x" 169 | yield 170 | print "y" 171 | yield 172 | print "z" 173 | 174 | executor.submit(foo()) 175 | executor.submit(bar()) 176 | executor.run() 177 | ``` 178 | 179 | Output (The two functions nicely take alternating turns.): 180 | 181 | ``` 182 | a 183 | x 184 | b 185 | y 186 | c 187 | z 188 | ``` 189 | 190 | But how about things that take a bit longer, like IO? To get there, we start with a simple `sleep`: 191 | 192 | ```java 193 | function foo() 194 | await async_sleep(1 second) 195 | print "moin" 196 | ``` 197 | 198 | We need to extend our executor: 199 | 200 | ```java 201 | class Executor 202 | current = null 203 | _ready = empty list 204 | _scheduled = empty priority queue 205 | 206 | method submit(coroutine) 207 | _ready.push_right(coroutine) 208 | 209 | method schedule(timestamp, coroutine) 210 | _scheduled.put(timestamp, coroutine) 211 | 212 | method run() 213 | while _ready.not_empty or _scheduled.not_empty 214 | if _ready.empty 215 | wakeup_time, coroutine = _scheduled.pop() 216 | sync_sleep(wakeup_time - now) 217 | submit(coroutine) 218 | 219 | current = _ready.pop_left() 220 | current.step() 221 | // We only re-submit the coroutine when it was not removed/scheduled by async_sleep. 222 | if current and current.running 223 | submit(current) 224 | 225 | executor = Executor() 226 | 227 | function async_sleep(duration) 228 | executor.schedule(now + duration, executor.current) 229 | executor.current = null 230 | ``` 231 | 232 | `run` has become more involved. If is nothing ready, it will sleep (actually block) until the next scheduled coroutine needs to be awakened. 233 | 234 | Since we're single-threaded, no urgent new tasks will be inserted "from the side", i.e., we will not oversleep. New tasks can only be added from inside a coroutine advanced from without the event loop. 235 | 236 | Finally, let's get to IO. 237 | 238 | When we submit a task waiting for IO, our `Executor` needs to do additional bookkeeping while handles are to be awaited. When no task is ready, it cancels its sleep when one of the handles becomes ready (`epoll` on Linux, `WaitForMultipleObjects` on Windows). 239 | 240 | 241 | ```java 242 | class Executor 243 | ... 244 | 245 | _io_pending = empty dictionary, mapping from handles to coroutines 246 | 247 | method run() 248 | while _ready.not_empty or _scheduled.not_empty or _io_pending.not_empty 249 | if _ready.empty 250 | timeout = null 251 | if _scheduled.not_empty 252 | wakeup_time, coroutine = _scheduled.first.timestamp 253 | timeout = wakeup_time - now 254 | 255 | // Sleep till the timeout expires or a handle becomes ready. 256 | io_ready = wait_for_handles(_io_pending.handles, timeout) 257 | for handle in io_ready 258 | submit(_io_pending.pop(handle)) 259 | 260 | current = _ready.pop_left() 261 | current.step() 262 | if current and current.running 263 | submit(current) 264 | 265 | // A coroutine sets itself as waiting for the socket. 266 | method recv(socket) -> bytes 267 | _io_pending[socket] = current 268 | current = null 269 | yield 270 | return sock.recv() 271 | ``` 272 | 273 | While the above only shows async input (for brevity), async output works analogously. 274 | 275 | [Here, you can find a (Python) implementation of all things discussed in this article](internals_of_the_async_await_pattern_from_first_principles/internals_of_the_async_await_pattern_from_first_principles.py), including an echo-ing TCP server (heavily inspired by [an amazing workshop](https://www.youtube.com/watch?v=Y4Gt3Xjd7G8) of David Beazley). 276 | 277 | Now you should have a rough idea of what happens with coroutines on the language level and the tooling the async libraries provide us with. I suggest to play around with (and purposely break) the linked example Python implementation to get a better feeling for everything. 278 | 279 | To put the treated concept into perspective, here is a spectrum of different multitasking approaches (from heavyweight to lightweight): 280 | - processes (OS, preemptive, stackful) 281 | - (OS) threads (OS, preemptive, stackful) 282 | - green threads (language VM, preemptive, stackful) 283 | - fiber (language VM, cooperative, stackful) 284 | - coroutines (async/await, cooperative, stackless, what this article is about) 285 | 286 | Some advantages of the coroutines approach described here are: 287 | - no synchronization mechanisms like locks/mutexes are needed to avoid data races 288 | - fast, suitable for some real-time use cases 289 | - light, it scales easily to thousands of concurrent tasks 290 | 291 | A downside is, that it sometimes can be hard to reason about. 292 | 293 | So choose wisely. ;) 294 | -------------------------------------------------------------------------------- /a_monad_is_just_a_monoid_in_the_category_of_endofunctors_explained.md: -------------------------------------------------------------------------------- 1 | # "A monad is just a monoid in the category of endofunctors." - explained 2 | 3 | ## Introduction 4 | 5 | The quote in the title sparked my curiosity after I had read it on [a blog](http://james-iry.blogspot.com/2009/05/brief-incomplete-and-mostly-wrong.html) using it jokingly. (Originally it's from a [math book](https://en.wikipedia.org/wiki/Categories_for_the_Working_Mathematician).) 6 | 7 | Despite good existing explanations on the web, achieving a level of at least somewhat satisfying understanding, however, turned out to be quite a ~~struggle~~ journey for me. This article is an attempt to ease this process for other software developers a bit, and to consolidate the things I learned for myself. ;-) 8 | 9 | (Disclaimer: The given explanations contain nasty shortcuts, some used consciously, some probably just based on my lack of deeper understanding.) 10 | 11 | ## Syntax 12 | 13 | First, let's agree on some syntax to use in the rest of the article. 14 | 15 | ```kotlin 16 | x : Int // x is a variable of type Int 17 | xs : List // xs is a list of integers 18 | f : Int -> String // f is a function from Int to String. 19 | ``` 20 | 21 | In functional programming, functions are first-class citizens. They can be passed around, and they can be returned from functions (higher-order functions). 22 | 23 | ```kotlin 24 | // isEven is a function taking an integer and returning a boolean. 25 | isEven : Int -> Boolean 26 | 27 | // The function keepIf takes two parameters. 28 | // The first one is a function taking an A and returning Boolean. 29 | // The second one is a list of elements of type A. 30 | // Finally, keepIf returns a sequence of elements of type A. 31 | keepIf : (A -> Boolean, List) -> List 32 | ``` 33 | 34 | `A` being a type variable, we could call `keepIf(isEven, [1, 2, 3, 4, 5])` and get `[2, 4]`. (Here, `A` is `Int`.) 35 | 36 | ### Currying 37 | 38 | While 39 | 40 | ```kotlin 41 | keepIf : (A -> Boolean, List) -> List 42 | ``` 43 | 44 | is a function that takes two parameters (a function and a list) and returns a list, we could also only apply the first parameter and get a function taking a `List` and returning a `List`. (partial function application) 45 | 46 | ```kotlin 47 | keepIf : (A -> Boolean) -> (List -> List) 48 | ``` 49 | 50 | This converts every function of arbitrary arity into a function of arity 1. 51 | 52 | We call this process currying. 53 | 54 | `->` is right-associative, so the above is equivalent to: 55 | 56 | ```kotlin 57 | keepIf : (A -> Boolean) -> List -> List 58 | ``` 59 | 60 | This way of thinking will help us in the rest of the article. 61 | 62 | ## What is a category? 63 | 64 | A category is a bunch of objects with a set of morphisms (arrows). 65 | 66 | ``` 67 | ----- f ----- 68 | | A | ----> | B | 69 | ----- ----- 70 | | 71 | | g 72 | | 73 | v 74 | ----- 75 | | C | 76 | ----- 77 | ``` 78 | 79 | Morphisms must be composable, so if there is a morphism `f` from `A` to `B` and a morphism `g` from `B` to `C`, then a morphism `h` from `A` to `C` must exist too. 80 | 81 | ```kotlin 82 | f : A -> B 83 | g : B -> C 84 | 85 | h : A -> C 86 | ``` 87 | 88 | with 89 | 90 | ```kotlin 91 | h(x) = g(f(x)) 92 | ``` 93 | 94 | ``` 95 | ----- f ----- 96 | | A | ----> | B | 97 | ----- ----- 98 | \ | 99 | \ | g 100 | \ h | 101 | \ v 102 | \ ----- 103 | >| C | 104 | ----- 105 | ``` 106 | 107 | Let's write that kind of (forward) function composition as `h = f >> g`. 108 | 109 | ```kotlin 110 | >> : (A -> B) -> (B -> C) -> (A -> C) 111 | ``` 112 | 113 | And finally, there is an identity morphism (endomorphism) for every object. 114 | 115 | ``` 116 | ------ 117 | / \ 118 | | | identity 119 | v | 120 | ----- / 121 | | A | --/ 122 | ----- 123 | ``` 124 | 125 | The category we work with in programming is the category of types, i.e., each object represents the set of possible values for the particular type. The morphisms are our functions. 126 | Since in functional programming functions are data too (they can be passed around, etc.) the function types themselves are also objects. 127 | 128 | Even though in this special category we can know something about the "content" objects (types) and what the concrete morphisms (functions) do to them, from a pure category-theoretical point of view, this is not needed. 129 | 130 | 131 | ## What is a functor? 132 | 133 | A functor is a structure-preserving mapping between categories. 134 | 135 | But for now, let's think about a functor in terms of a box. A typical example is the `Maybe` functor, which can either hold a value or nothing. (You might know it as `class Optional` or similar, depending on the programming languages you're familiar with.) 136 | 137 | For `Maybe` to become a functor, we must provide a function `lift` that converts a function of type `A -> B` to one of type `Maybe -> Maybe`: 138 | 139 | ```kotlin 140 | lift : (A -> B) -> Maybe -> Maybe 141 | ``` 142 | 143 | `lift` "lifts" a function `A -> B` from the world of normal types into the world of `Maybe`s. That's the structure-preserving property. The graph of morphisms in the original category also exists in the functor category. 144 | 145 | Since `Maybe` is also a type in our normal category of types, we call `Maybe` an endofunctor, i.e., it maps from our original category to a subcategory of it, not making us leave the source category. 146 | 147 | `lift` for `Maybe` returns a function that is implemented such that it applies the function `A -> B` to the value in `Maybe` if a value is present. In case there is nothing inside, it returns a `Maybe` with nothing inside. So the box can hold a value or be empty. 148 | 149 | Another generic type, which can be a functor, is `List`. For it, `lift` is usually implemented like `map`: 150 | 151 | ```kotlin 152 | map : (A -> B) -> List -> List 153 | ``` 154 | 155 | I.e., we can use it with a function `f: A -> B` and a `List`, and we get a `List`, with `f` applied to each original element. 156 | 157 | A functor must preserve identity morphisms: 158 | 159 | ```kotlin 160 | lift(identity) = identity 161 | ``` 162 | 163 | and composition of morphisms 164 | ```kotlin 165 | lift(f >> g) = lift(f) >> lift(g) 166 | ``` 167 | 168 | 169 | ## What is a monoid? 170 | 171 | A monoid is an algebraic structure consisting of: 172 | - a set, which in our case is a type, i.e., an object in the category of types 173 | - a binary operation (`<>`) 174 | - a neutral element for that operation (`neutral`) 175 | 176 | It must have the following properties: 177 | - closure: Combining two elements with the binary operation must return an element of the same type. (`<> : (A, A) -> A`) 178 | - associativity: `(x <> y) <> z = x <> (y <> z)` 179 | 180 | For the neutral element, the following must be true: 181 | - `x <> neutral = x` 182 | - `neutral <> x = x` 183 | 184 | A simple example of a monoid is the set of numbers under addition (`+`) with `0` as the neutral element: 185 | - `<> = +` 186 | - `neutral = 0` 187 | 188 | Take a minute to convince yourself that this fulfills the monoid requirements. 189 | 190 | Other examples of monoids in the category of types are: 191 | - numbers under multiplication (`*`) with `1` as the neutral element 192 | - sets under the union operation with the empty set as the neutral element 193 | 194 | These monoids so far were all commutative, i.e., `x <> y = y <> x` but that does not need to be true. 195 | 196 | Examples of non-commutative monoids are: 197 | - strings under concatenation with the empty string as the neutral element (similarly, lists under concatenation with the empty list as the neutral element) 198 | - endomorphism (`f : a -> a`) under composition (`>>`) with `identity` as the neutral element 199 | 200 | 201 | ## What is a monad? 202 | 203 | Let's say we want to read a text file containing a bunch of integers and calculate the median value of them. 204 | We have some functions that can fail, and thus they don't return a plain type, but a `Maybe`. 205 | 206 | ```kotlin 207 | // Takes a file path and returns the contents of the file, or nothing if the file can't be read. 208 | read_file : String -> Maybe 209 | 210 | // Takes a text and returns a list of the contained integers, or nothing if the parsing failed. 211 | parse_text : String -> Maybe> 212 | 213 | // Calculates the median value of a list of integers, which fails for empty lists. 214 | calc_median : List -> Maybe 215 | ``` 216 | 217 | We'd like to compose these functions, but normal function composition (`>>`) can't be used here, i.e., the types do not align. 218 | 219 | ```kotlin 220 | >> : (A -> B) -> (B -> C) -> (A -> C) 221 | ``` 222 | 223 | So we need some other kind of composition, a monadic one (called "(forward) Kleisli composition"): 224 | 225 | ```kotlin 226 | // The "fish" operator 227 | >=> : (A -> Functor) -> (B -> Functor) -> (A -> Functor) 228 | ``` 229 | 230 | If we provide (that is, implement) `>=>` for `Maybe`, our `Maybe` (endo)functor is now also a monad. So let's write it as: 231 | 232 | ```kotlin 233 | >=> : (A -> Monad) -> (B -> Monad) -> (A -> Monad) 234 | ``` 235 | 236 | In addition, we need to provide the "trivial" function `inject : A -> Monad` (also known as `pure` or `return`), which simply allows us to bring a value from the category of types to a value in the category of `Maybe` types. In that case, it's just the constructor of `Maybe`. 237 | 238 | In our original example, we can now write 239 | 240 | ```kotlin 241 | program : String -> Maybe = read_file >=> parse_text >=> calc_median 242 | ``` 243 | 244 | and are done. 245 | 246 | Composing functions this way is nice. In the case of `Maybe` we can think of it like so: 247 | - As long as we have an actual value, we apply the next function to it. 248 | - Once we divert to the `nothing` track, we stay there and just fall through to the end skipping the later functions. 249 | 250 | We spare all the boilerplate-like plumbing code by defining it only once (generically) in the implementation of `>=>`. 251 | 252 | Btw, instead of providing `>=>`, we could also provide `>>=` ("bind") (`>>= : Monad -> (A -> Monad) -> Monad`) or `join` (`join : Monad> -> Monad`). If `lift` and just one of the three (`>=>`/`>>=`/`join`) is given, the other two can always be derived from it, which is a fun exercise, but not needed here. 253 | 254 | Also, a monad must satisfy some laws: 255 | - `inject >=> f = f` (`inject(x) >>= f = f(x)`) 256 | - `m >=> inject = m` (`m >>= inject = m`) 257 | - `(f >=> g) >=> h = f >=> (g >=> h)` 258 | 259 | Staring at them long enough, they hopefully start to make sense. ;-) 260 | 261 | To summarize, a monad is a functor equipped with `>=>` (and `inject`). 262 | 263 | ## A monoid in the category of endofunctors 264 | 265 | The final piece in the puzzle is to understand, how this renders a monad a monoid in the category of endofunctors. 266 | 267 | The category of endofunctors is a category with the objects being the endofunctors and the morphisms being functions that map from one functor to another. (They are called natural transformations, and they, of course, must also satisfy the requirement of composability, for the whole thing to be a proper category. But their details are not important here.) 268 | 269 | ``` 270 | ------------ NT ----------- 271 | | Maybe | -------> | List | 272 | ------------ ----------- 273 | 274 | ... 275 | ``` 276 | 277 | What would a monoid in this category look like? 278 | 279 | For a monoid, we need an object, which is an endofunctor now, e.g., `Maybe`. 280 | 281 | (Usually, the mathematically clean way now involves a product category to create a monoidal category and functor composition, but we fiercely simplify here. ("Mathematicians hate this one (weird) trick.")) 282 | 283 | We can think of this object in terms of a function creating such a functorial value, `A -> Maybe`. So that's the first thing we use to form the monoid that we'd like to obtain. 284 | 285 | Second, we need a binary operation. Let's use our Kleisli composition, `>=> : (A -> Monad) -> (B -> Monad) -> (A -> Monad)`. 286 | 287 | Last, we need a neutral element for the operation. Let's use `inject : A -> Monad`. 288 | 289 | Does this algebraic structure fulfill our monoid requirements? 290 | - Closure (`<> : (A, A) -> A`)? Yes, because with `>=>` we don't suddenly land in a different monad, e.g., jump from `Maybe` to `List` or something like that. 291 | - Associativity (`(x <> y) <> z = x <> (y <> z)`)? Yes, because the monad laws already state that `>=>` must be associative (like normal function composition). 292 | - `x <> neutral = x` and `neutral <> x = x`? Yes, because `inject >=> f = f` (from the monad laws) and `f >=> inject = f` with `f : A -> Monad`. 293 | 294 | So there we have it! 295 | 296 | A monad in the category of types is (just) a monoid in the category of endofunctors. :tada: 297 | 298 | Data lives in a monad, monadic functions live in a monoid. 299 | 300 | ## Conclusion 301 | 302 | I hope the initial quote now sounds less alien. And maybe you also appreciate the beauty of it, because, in the end, compositionality is the essence of programming, i.e., building bigger things from simple things. And this provides a framework to formally think about this. 303 | 304 | Such insights can help to write leaner code, expressing the high-level logic clearly, without being blinded by implementation-specific (repeated) details/boilerplate. Implementers of compilers/libraries might even use such semantics (functors, monoids, monads) to perform certain optimizations based on the guarantees provided by the laws associated with these abstractions. --------------------------------------------------------------------------------