├── 1-regexp.md
├── 10-rag.md
├── 2-fts.md
├── 3-levenshtein.md
├── 4-lm.md
├── 5-classification.md
├── 6-ner.md
├── 7-classification-ner-llm.md
├── 8-neural.md
├── 9-qa.md
├── elastic
    ├── Dockerfile
    ├── config
    │   └── elasticsearch.yml
    ├── docker-compose.yaml
    └── query.sh
├── project
    ├── readme.md
    ├── spacex.zip
    └── text_description.txt
└── readme.md


/1-regexp.md:
--------------------------------------------------------------------------------
 1 | # Regular expressions (aka regex)
 2 | 
 3 | The task is concentrated on using regular expressions for extracting basic information from textual data. 
 4 | You will get more familiar with the regexp features that are particularly important for natural language processing applications, especially when you are preparing data for machine learning models.
 5 | 
 6 | ## Task
 7 | 
 8 | A FIQA-PL dataset containing Polish questions and answers is available at [Huggigface](https://huggingface.co/datasets/clarin-knext/fiqa-pl).
 9 | In this lab we only concentrate on the `corpus` split of the dataset.
10 | 
11 | Task objectives (8 points):
12 | 
13 | 1. Devise two regular expressions:
14 |    * extracting times, e.g. recognizing `20:30` as an instance of a time.
15 |    * extracting dates, e.g. recognizing `20 września` as an instance of a date.
16 | 2. Search for occurrences of times and dates in the dataset.
17 | 3. Plot results from point 2:
18 |    * for times create a bar plot for full hours (i.e. 17:35 -> is 17).
19 |    * for dates create a bar plot for months.
20 | 4. Compute the number of occurrences of `kwiecień` word in any inflectional form. Use a compact form for the query. The only forbidden form is a juxtaposition of the month names, i.e. `"kwiecień|kwietnia|kwietniu..."`.
21 | 5. As in 4, but preceded by a number and a space.
22 | 6. As in 4, but not preceded by a number and a space. Check if the results from 5 and 6 sum to 4.
23 | 7. Ask an LLM (e.g. [Bielik](https://chat.bielik.ai/)) to complete these tasks for you. Compare and criticize the code generated by the LLM.
24 | 
25 | 
26 | Answer the following questions (2 points):
27 | 1. Are regular expressions good at capturing times?
28 | 2. Are regular expressions good at capturing dates?
29 | 3. How one can be sure that the expression has matched all and only the correct expressions of a given type?
30 | 4. Is LLM able to generate regular expressions?
31 | 
32 | ## Hints
33 | 
34 | * Some programming languages allow to use Unicode classes in regular expressions, e.g.
35 |   * `\p{L}` - letters from any alphabet (e.g. a, ą, ć, ü, カ)
36 |   * `\p{Ll}` - small letters from any alphabet
37 |   * `\p{Lu}` - capital letters from any alphabet
38 | * Not all regular expressions engines support Unicode classes, e.g. `re` from Python does not.
39 |   Yet you can use `regex` library (`pip install regex`), which has much more features.
40 | * Regular expressions can include positive and negative lookahead and lookbehind constructions, e.g.
41 |   * *positive lookahead* - `(\w+)(?= has a cat)` will match in the string `Ann has a cat`, but it will match `Ann` only.
42 |   * *negative lookbehind* - `(?<!New )(York)`, will match `York` in `Yorkshire` but not in `New York`.
43 | * `\b` matches a word border. Regexp `fish` will match in `jellyfish`, but `\bfish\b` will only match `fish`.
44 |   In the case of Python you should use either `'\\bfish\\b'` or `r'\bfish\b'`.
45 | * `\b` is dependent on what is understood by "word". For instance in Ruby polish diacritics are not treated as parts of
46 |   a word, thus `\bpsu\b` will match both `psu` and `psuć`, since `ć` is a non-word character in Ruby.
47 | * Some languages, e.g. Ruby, support regexp match operator as well as regexp literals (`=~`, /fish/ respectively 
48 |   in the case of Ruby and Perl). Notably Python does not support either.
49 | * You should be very careful when copying regexps from Internet - different languages and even different versions of the
50 |   same language may interpret them differently, so make sure to always test them on a large set of diversified examples.
51 | * You can try regexes using https://regex101.com. It allows you to compare the matches between different programming languages.
52 | * You can use `datasets` Python library, to facilitate downloading and loading of the `clarin-knext/fiqa-pl` dataset.
53 | 


--------------------------------------------------------------------------------
/10-rag.md:
--------------------------------------------------------------------------------
 1 | # RAG-based Question Answering
 2 | 
 3 | The exercise introduces modern approaches to Question Answering using Retrieval Augmented Generation (RAG) with LLMs and vector databases.
 4 | 
 5 | ## Tasks
 6 | 
 7 | Objectives (8 points):
 8 | 
 9 | 1. Set up the QA environment:
10 |    * Install OLLAMA and select an appropriate LLM
11 |    * Configure [Qdrant](https://qdrant.tech/) vector database (or vector DB of your choosing)
12 |    * Install necessary Python packages for embedding generation
13 | 2. Find PDF file of your choosing. Example - some publication or CV file:
14 | 3. Write next procedures necessary for RAG pipeline. Use [LangChain](https://python.langchain.com/docs/introduction/) library:
15 |  
16 |    * Load PDF file using `PyPDFLoader`.  
17 |    * Split documents into appropriate chunks using `RecursiveCharacterTextSplitter`.
18 |    * Generate and store embeddings in Qdrant database
19 | 4. Design and implement the RAG pipeline with `LCEL`. As reference use this detailed guide created by LangChain community - [RAG](https://python.langchain.com/docs/tutorials/rag/). Next steps should involve:
20 |    * Create query embedding generation
21 |    * Implement semantic search in Qdrant
22 |    * Design prompt templates for context integration
23 |    * Build response generation with the LLM
24 | 
25 | Hint: You don't need to build it from scratch. A lot of this steps is already automated using LCEL pipeline definition.
26 | 
27 | 5. Implement basic retrieval strategies (semantic search).
28 | 6. Create basic QA prompt.
29 | 7. Determine 5 evaluation queries:
30 |     - Determine a few questions, which answers are confirmed by you.
31 | 8. Compare performance of RAG vs. pure LLM response.
32 | 
33 | Questions (2 points):
34 | 
35 | 1. How does RAG improve the quality and reliability of LLM responses compared to pure LLM generation?
36 | 2. What are the key factors affecting RAG performance (chunk size, embedding quality, prompt design)?
37 | 3. How does the choice of vector database and embedding model impact system performance?
38 | 4. What are the main challenges in implementing a production-ready RAG system?
39 | 5. How can the system be improved to handle complex queries requiring multiple document lookups?
40 | 
41 | ## Hints
42 | 
43 | 1. Careful chunk size selection is crucial for relevant context retrieval
44 | 2. To select LLM for answer generation you can consult [LLM leaderboard](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard#/) or [Polish LLM Leaderboard](https://huggingface.co/spaces/speakleash/open_pl_llm_leaderboard).
45 | 3. To select model for retrieval (embedding generation) you can consul [Embedding leaderboard](https://huggingface.co/spaces/mteb/leaderboard) or [Polish embedding leaderboard](https://huggingface.co/spaces/sdadas/pirb).
46 | 4. Consider implementing re-ranking of retrieved documents
47 | 5. Prompt engineering significantly impacts answer quality
48 | 6. Caching can greatly improve system performance during development
49 | 7. Consider using metadata filtering to improve retrieval precision
50 | 8. The choice of embedding model affects both accuracy and speed
51 | 


--------------------------------------------------------------------------------
/2-fts.md:
--------------------------------------------------------------------------------
 1 | # Lemmatization and full text search (FTS)
 2 | 
 3 | The task is concentrated on using full text search engine (ElasticSearch) to perform basic search
 4 | operations in a text corpus.
 5 | 
 6 | ## Tasks
 7 | 
 8 | Task objective (8 points):
 9 | 1. Install ElasticSearch (ES).
10 | 2. Install an ES plugin for Polish https://github.com/allegro/elasticsearch-analysis-morfologik 
11 | 3. Define an ES analyzer for Polish texts containing:
12 |    1. standard tokenizer
13 |    2. synonym filter with alternative forms for months, e.g. `kwiecień`, `kwi`, `IV`.
14 |    3. lowercase filter
15 |    4. Morfologik-based lemmatizer
16 |    5. lowercase filter (looks strange, but Morfologi produces capitalized base forms for proper names, so we have to lowercase them once more).
17 | 4. Define another analyzer for Polish, without the synonym filter.
18 | 5. Define an ES index for storing the contents of the corpus [FiQA-PL](https://huggingface.co/datasets/clarin-knext/fiqa-pl) using both analyzers.
19 |    Use different names for the fields analyzed with a different pipeline.
20 | 7. Load the data to the ES index.
21 | 8. Determine the number of documents and the number of matches containing the word `kwiecień` (in any form) including and excluding the synonyms.
22 | 9. Download the QA pairs for the [FiQA-PL dataset](https://huggingface.co/datasets/clarin-knext/fiqa-pl-qrels).
23 | 10. Compute NDCG@5 for the QA dataset (the test subset) for the following setusp:
24 |    * synonyms enabled and disabled,
25 |    * lemmatization in the query enabled and disabled.
26 | 11. (optional) Find three questions from the test subset with the following features:
27 |    * the relevant document is returned by ES at position 1,
28 |    * the relevant document is returned by ES at position 4 or 5.
29 |    * the relevant document is returned by ES is not found.
30 | 11. (optional) Analyze the possible reasons for these outcomes.
31 | 
32 | 
33 | Answer the following questions (2 points):
34 | 1. What are the strengths and weaknesses of regular expressions versus full text search regarding processing of text?
35 | 2. Can an LLM be applied in the context of searching for documents? Justify your answer, excluding the obvious observation that an LLM can be used to formulate the answer.
36 | 
37 | 
38 | ## Hints
39 | 
40 | 1. Full text search engines were developed for storing and searching textual data.
41 | 1. The most popular FTSes are SOLR and ElasticSearch (ES).
42 | 1. Some relational databases support full text search, but usually it is limited and not easy to adapt.
43 | 1. Both for SOLR and ES there are plugins supporting Polish.
44 | 1. FTSes use *inverted-index* to store the data. At loading time the text is split by *tokenizer* into 
45 |    *tokens* and individual tokens go through *filters*. The resulting tokens are placed as keys in a hash-like
46 |    structure. The values are so called *posting lists*, containing pointers to the documents where the tokens come from.
47 | 1. The minimal FTS configuration requires two elements: a tokenizer and a set of filters (the set might be empty in the extreme
48 |    case). **Changing the configuration of an index does not result in the new definitions being applied to the already
49 |    stored documents.** In such cases the index has to be *rebuilt*, meaning that the documents have to be loaded once
50 |    again.
51 | 1. FTSes contain a large number of tokenizers, e.g. they may know semantics of HTML documents and treat HTML tags as
52 |    tokens. Some popular tokenizers include:
53 |    1. *standard tokenizer* - based on the Unicode tokenization rules,
54 |    1. *whitespace tokenizer* - which splits the tokens by white spaces,
55 |    1. *url tokenizer* - which keeps the URLs as indivisible tokens.
56 | 1. Some tokens such as commas and full stops might be removed at the stage of filtering. Filtering of common tokens reduces the index size.
57 | 1. Some popular filters include:
58 |    1. *lowercase filter* - which downcases the letters,
59 |    1. *ASCII folding filter* - which removes Polish diacritics,
60 |    1. *stop token filter* - which removes the specified tokens (described above),
61 |    1. *lematizers* - which find the base form of a word,
62 |    1. etc. (present implementation of ES has more than 50 filters)
63 | 1. **Lemmatization** is a process when the inflected form of a word is replaced with its base form, e.g
64 |    the form *psu* is replaced with *pies*. You should notice that there are many ambiguous forms, e.g.
65 |    *goli* can have the following base forms: *golić*, *gol* and *goły*. To overcome the ambiguity, FTSes 
66 |    take very pragmatic approach - for a given inflected form all possible base forms are put in the index.
67 |    Even though it's not valid from the linguistics' point of view, it works well in practice.
68 | 1. [Term vector API](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-termvectors.html) allows to retrieve useful 
69 |    statistics of a given term in a particular document or in the whole document collection.
70 | 1. Polish retrieval models comparison is available at :https://huggingface.co/spaces/sdadas/pirb
71 | 2. In the `elastic` directory there's a basic configuration for runnig ElasticSearch with `docker compose`:
72 |    1. You can use the `docker-compose.yml` configuration - it will start ES with the morfologik plugin installed.
73 |    2. You can also modify the `Dockerfile` configuration and run it locally.
74 |    3. In the `query.sh` file there's a check for ES showing if it is possible to connect to the instance using `curl`. 
75 |    4. The correct output from curl is `[]` meaning there aren't any indices defined.
76 | 


--------------------------------------------------------------------------------
/3-levenshtein.md:
--------------------------------------------------------------------------------
 1 | # Levenshtein distance and spelling corrections
 2 | 
 3 | The task introduces the Levenshtein distance - a measure that is useful in tasks such as approximate string matching.
 4 | 
 5 | ## Tasks
 6 | 
 7 | Task objectives (8 points):
 8 | 1. Use FIQA-PL corpus, including documents, questions and relations between them.
 9 | 2. Use SpaCy [tokenizer API](https://spacy.io/api/tokenizer) to tokenize the text in the documents.
10 | 3. Compute **frequency list** for each of the processed files.
11 | 4. Aggregate the result to obtain one global frequency list. This frequency list gives you unigram statistics of the words
12 |    appearing in the corpus.
13 | 5. Apply a distortion function to the queries part of the corpus. In each query draw randomly one word and remove, add or change one letter in the word (2024/2025 was only change one letter, in 2025 should draw randomly the number of words and number of letters according to Normal distribution).
14 | 6. Compute nDCG@10 for the distorted queris, using the same approach as in lab 2. This result will be the baseline
15 |    for the other methods.
16 | 8. Install [Morfeusz](http://morfeusz.sgjp.pl/download/) (Binding dla Pythona) and use it to find all words from the queries that do not
17 |    appear in that dictionary. Only these words should be corected in the next step. This step is not obligatory - if there's a problem
18 |    installing Morfeusz, you can use one of the raw text files (Morfeusz "Słownik Polimorf" or "Słownik SGJP") and use the first column,
19 |    as a list of all valid forms in Polish.
20 | 10. Use Levenshtein distance and the frequency list, to determine the most probable correction of the words
21 |     in the queries that were identified as invalid. (**Note**: You don't have to apply the distance directly. Any method that is more
22 |     efficient than scanning the dictionary will be appreciated.)
23 | 11. Compute nDCG@10 for your implementation of the spelling correction method.
24 | 12. Use ElasticSearch's fuzzy match and compute nDCG@10 for this approach.
25 | 13. Compare the results of baseline with the 2 implemented methods. Take into account the nDCG score and the performance of the methods.
26 | 14. Use an LLM of your choice (you can use [Bielik](https://chat.bielik.ai/)) to fix 30 first queries from the distorted set and compare the results manually with the method based on the Levenshtein distance.
27 |    
28 | Draw conclusions regarding (2 points):
29 |   * the distribution of words in the corpus,
30 |   * the performance (speed) of your method compared to ElasticSearch,
31 |   * the results provided by your method compared to ElasticSearch,
32 |   * the validity of the obtained corrections,
33 |   * ability of an LLM to fix invalid queries.
34 | 
35 | ## Hints
36 | 
37 | 1. Levenshtein distance (Edit distance) is a measure defined for any pair of strings. It is defined as the minimal
38 |    number of single character edits (insertions, deletions or substitutions) needed to transform one string to the
39 |    other. The measure is symmetric.
40 | 1. The algorithm is usually implemented as a dynamic program,
41 |    see [Wikipedia article](https://en.wikipedia.org/wiki/Levenshtein_distance) for details.
42 | 1. The distance may be used to fix an invalid word by inspecting in the growing order of the distance, the words
43 |    that are *n* edits away from the invalid word. If there are no words *n* edits away, the words *n+1* edits away
44 |    are inspected.
45 | 1. The frequency list may be used to select the most popular word with given distance, if there are many candidate
46 |    corrections.
47 | 1. Usually the correction algorithm does not use the edit distance directly, since it would require to compare the
48 |    invalid word with all words in the dictionary. The algorithms work in the opposite way - they generate candidate
49 |    words that are 1 or 2 edits away from the invalid word (cf. P. Norvig's
50 |    [article](https://norvig.com/spell-correct.html) for the details). A different approach is to
51 |    use [Levenshtein automaton](https://norvig.com/spell-correct.html) for finding the corrections effectively.
52 | 1. ElasticSearch has
53 |    a [fuzziness](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-fuzzy-query.html)
54 |    parameter for finding approximate matches of a query.
55 | 


--------------------------------------------------------------------------------
/4-lm.md:
--------------------------------------------------------------------------------
 1 | # Language modelling
 2 | 
 3 | The exercise shows how a language model may be used to solve word-prediction tasks and to generate text.
 4 | 
 5 | 
 6 | ## Tasks
 7 | 
 8 | Objectives (8 points):
 9 | 
10 | 1. Read the documentation of [Language modelling in the Transformers](https://huggingface.co/transformers/task_summary.html#language-modeling) library.
11 | 2. Download three [Polish models](https://huggingface.co/models?filter=pl) from the Huggingface repository. These should be regular language models, which were not fine-tuned. E.g. `HerBERT` and `papuGaPT2` are good examples. You can also try using Bielik for that, but make sure you are using the model via Transformers API, not GUI.
12 | 3. Devise a method to test if the langage model understands Polish cases. E.g. testing for *nominal case* could be expressed as "Warszawa to największe `[MASK]`", and the masked word should be in nominative case. Create sentences for each case.
13 | 4. Devise a method to test long-range relationships such as gender. E.e. you can use two verbs with masculine and feminine gender, where one of the verbs is masked. Both verbs should have the same gender, assuming the subject is the same. Define at least 3 such sentences.
14 | 5. Check if the model captures real-world knolwedge. For instance a sentence "`[MASK]` wrze w temperaturze 100 stopni, a zamarza w temperaturze 0 stopni Celsjusza." checks if the model "knows" the description of water. Define at least 3 such sentences.
15 | 6. Check zero-shot learning capabilites of the models. Provide at least 5 sentences with different sentiment for the following scheme: "'Ten film to był kiler. Nie mogłem się oderwać od ekranu.' Wypowiedź ta ma jest zdecydowanie `[MASK]`" Try different prompts, to see if they make any difference.
16 | 7. Take into accout the fact, that causal language models such as PapuGaPT2 or plT5, will only generate continuations of the sentenes, so the examples have to be created according to that paradigm.
17 | 
18 | 
19 | Answer the following questions (2 points):
20 |    1. Which of the models produced the best results?
21 |    2. Was any of the models able to capture Polish grammar?
22 |    3. Was any of the models able to capture long-distant relationships between the words?
23 |    4. Was any of the models able to capture world knowledge?
24 |    5. Was any of the models good at doing zero-shot classification?
25 |    6. What are the most striking errors made by the models?
26 | 
27 | ## Hints
28 | 
29 | 1. Language modelling (LM) is a task concentrated on computing the probability distribution of words given a sequence of
30 |    preceding words.
31 | 2. In the past the most popular LM were based on n-gram counting. The distribution of probability of the next word was
32 |    approximated by the relative frequency of the last n-words, preceding this word. Usually n=3, since larger values
33 |    resulted in extremely large datasets.
34 | 3. Many algorithms were devised to improve that probability estimates for infrequent words. Among them Kneser-Ney was
35 |    the most popular.
36 | 4. SRI LM is the most popular toolkit for creating traditional language models.
37 | 5. At present recurrent neural networks, attention networks and transformers are the most popular neural-network
38 |    architectures for creating LMs.
39 | 6. The largest LM currently is GPT-3 described in (mind the number of authors!) *Language Models are Few-Shot Learners*
40 |    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav
41 |    Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon
42 |    Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler,
43 |    Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya
44 |    Sutskever, Dario Amodei
45 | 


--------------------------------------------------------------------------------
/5-classification.md:
--------------------------------------------------------------------------------
 1 | # Text classification
 2 | 
 3 | The task concentrates on classification of sentence pairs.
 4 | 
 5 | This type of classification is useful for problems such as determining the similarity of sentences or
 6 | checking if a text passage contains an answer to a question.
 7 | 
 8 | 
 9 | ## Tasks
10 | 
11 | Objectives (8 points):
12 | 
13 | 1. Use the FIQA-PL dataset that was used in lab 1 **and** lab lab 2 (so we need the passages, the questions and their
14 |    relations).
15 | 2. Create a dataset of positive and negative sentence pairs.
16 |    - In each pair the first element is a question and the second element is a passagei, i.e. "{question} {separator} {passage}",
17 |       where `separator` should be a separator taken from the model's tokenizer.
18 |    - Use the relations to mark the positive pairs (i.e. pairs where the question is answered
19 |       by the passage).
20 |    - Use your own strategy to mark negative pairs (i.e. you can draw the negative examples, but there are
21 |       better strategies to define the negative examples). The number of negative examples should be larger than the
22 |       number of positive examples.
23 | 3. The dataset from point 2 should be split into training, evaluation and testing subsets.
24 | 4. Train a text classifier using the Transformers library that distinguishes between the positive and the negative
25 |    pairs. To make the process manageable use models of size `base` and a runtime providing GPU/TPU acceleration.
26 |    Consult the discussions related to fine-tuning Transformer models to select sensible set of parameters.
27 |    You can also run several trainings with different hyper-parameters, if you have access to large computing resources.
28 | 5. Make sure you monitor the relevant metrics on the validation set during training. The last saved model might not be the
29 |    one with the best performance.
30 | 6. Report the results you have obtained for the model. Use appropriate measures, since the dataset is not balanced.
31 | 7. Use the classifier as a re-ranker for finding the answers to the questions. Since the re-ranker is slow, you
32 |    have to limit the subset of possible passages to top-n (10, 50 or 100 - depending on your GPU) texts returned by much faster model, e.g. FTS.
33 | 8. The scheme for re-ranking is as follows:
34 |    - Find passage candidates using FTS, where the query is the question.
35 |    - Take top-n results returned by FTS.
36 |    - Use the model to classify all pairs, where the first sentence is the question (query) and the second sentence is
37 |       the passage returned by the FTS.
38 |    - Use the score returned by the model (i.e. the probability of the **positive** outcome) to re-rank the passages.
39 | 9. Compute how much the result of searching the passages improved over the results from lab 2. Use NDCG to compare the
40 |    results.
41 | 
42 | Questions (2 points):
43 | 
44 | 1. Do you think simpler methods, like Bayesian bag-of-words model, would work for sentence-pair classification? Justify
45 |    your answer.
46 | 2. What hyper-parameters you have selected for the training? What resources (papers, tutorial) you have consulted to 
47 |    select these hyper-parameters?
48 | 3. Think about pros and cons of the neural-network models with respect to natural language processing. Provide at least
49 |    2 pros and 2 cons.
50 | 
51 | ## Hints
52 | 
53 | 1. You can use [this notebook](https://github.com/apohllo/sztuczna-inteligencja/tree/master/lab5) as a skeleton for the training procedure.
54 |    The relevant code starts in the "Klasyfikacja tekstu" section.
55 | 2. You can use [Google colab](https://colab.research.google.com/notebooks/intro.ipynb) to perform experiments which
56 |    require access to GPU or TPU.
57 | 3. [Huggingface Transformers](https://github.com/huggingface/transformers) library is a popular library for performing NLP tasks base on the transformer
58 |    architecture.
59 | 4. [Speech and Language Processing](https://web.stanford.edu/~jurafsky/slp3/) by Jurafsky and Martin 
60 |    has a [chapter](https://web.stanford.edu/~jurafsky/slp3/4.pdf) devoted to the problem of classification.
61 | 5. [Huggingface course](https://huggingface.co/course/chapter1/1) is a great resource related to all necessary knowledge related to the most up-to-date NLP model training.
62 | 


--------------------------------------------------------------------------------
/6-ner.md:
--------------------------------------------------------------------------------
 1 | # Named entity recognition
 2 | 
 3 | The exercise shows how we may extract elements such as names of companies, countries and similar objects from text.
 4 | 
 5 | ## Tasks
 6 | 
 7 | Objectives (8 points):
 8 | 
 9 | 1. Read the API of [NER](https://spacy.io/api/entityrecognizer) in Spacy
10 | 1. Take 1 thousand random passages from the FIQA-PL corpus.
11 | 1. Use the lemmatized and sentence split documents (using SpaCy API for [lemmatization](https://spacy.io/api/lemmatizer) and [sentence splitting](https://spacy.io/api/sentencizer)) to identify the expressions that consist of consecutive
12 |    words starting with a capital letter (you will have to look at the inflected form of the word to check its
13 |    capitalization) that do not occupy the first position in a sentence. E.g. the sentence:
14 |    ```
15 |    Wczoraj w Krakowie miało miejsce spotkanie prezydentów Polski i Stanów Zjednoczonych.
16 |    ```
17 |    should yield the following entries: `Kraków`, `Polska`, `Stan Zjednoczony`.
18 | 1. Compute the frequency of each identified expression and print 50 results with the largest number of occurrences.
19 | 1. Apply the NER from SpaCy to identify the named entities in the same set of documents.
20 | 1. Plot the frequency (histogram) of the identified classes.
21 | 1. Display 10 most frequent Named Entities for each identified type.
22 | 1. Display 50 most frequent Named Entities including their count and type.
23 | 2. Display 5 sentences containing at least 2 recognized named entities with different types. Highlight the recognized spans with color.
24 |    (For demo application [Streamlit](https://streamlit.io/) might be useful for displaying NER results).
25 | 
26 | 
27 | Answer the following questions (2 points):
28 | 
29 | 1. Which of the method (counting expressions with capital letters vs. NER) worked better for the task concerned with
30 |   identification of the proper names?
31 | 1. What are the drawbacks of the method based on capital letters?
32 | 1. What are the drawbacks of the method based on NER?
33 | 1. Which of the coarse-grained NER groups has the best and which has the worst results? Try to justify this
34 |   observation.
35 | 1. Do you think NER is sufficient for identifying different occurrences of the same entity (i.e. consider "USA" and
36 |   "Stany Zjednoczone" and "Stany Zjednoczone Ameryki Północnej")? If not, can you suggest an algorithm or a tool that
37 |   would be able to group such names together?
38 | 1. Can you think of a real world problem that would benefit the most from application of Named Entity Recognition
39 |   algorithm?
40 | 
41 | ## Hints
42 | 
43 | 1. Named entity recognition is a process aimed at the identification of entities mentioned in text by determining their
44 |    scope and classifying them to a predefined type. The larger the number of types, the more difficult the problem is.
45 | 1. Named entities are usually proper names and temporal expressions. They usually convey the most important information
46 |    in text.
47 | 1. IOB format is typically used to tag names entities. The name (IOB) comes from the types of tokens (_in_, _out_, _beginning_).
48 |    The following example shows how the format works:
49 |    ```
50 |    W            O
51 |    1776         B-TIME
52 |    niemiecki    O
53 |    zoolog       O
54 |    Peter        B-PER
55 |    Simon        I-PER
56 |    Pallas       I-PER
57 |    dokonał      O
58 |    formalnego   O
59 |    ...
60 |    ```
61 | 1. The set of classes used in NER is partially task dependent. Some general classes such as names of people or cities
62 |    are used universally, but categories such as references to law regulations is specific to legal information systems.
63 | 


--------------------------------------------------------------------------------
/7-classification-ner-llm.md:
--------------------------------------------------------------------------------
 1 | # NER and Classification with LLMs
 2 | 
 3 | The exercise demonstrates how to leverage Large Language Models for NER and text classification tasks, comparing their performance with traditional approaches.
 4 | 
 5 | ## Tasks
 6 | 
 7 | Objectives (8 points):
 8 | 
 9 | 1. Install and configure [OLLama](https://ollama.com/) with an appropriate LLM model (e.g. models from: Llama, Mistral, Bielik, Phi families). Rather not use models above 10B paramters.
10 | Sample LLM run command, when OLLama is installed: `ollama run phi3:3.8b`
11 | 2. Take 1 thousand random passages from the FIQA-PL corpus. INFO: You can play with new dataset, but it will be necessary to create baseline results (next excersise).
12 | 3. As baseline use traditional NER methods from lab 7 - SpaCy.
13 | 4. Design prompts for the LLM to:
14 |    * Identify named entities in text
15 |    * Classify them into predefined categories (person, organization, location, etc.)
16 | 5. Implement prompt variations to compare performance:
17 |    * Zero-shot prompting
18 |    * Few-shot prompting with 3-5 examples
19 | 6. Compare results between:
20 |    * Traditional NER (SpaCy)
21 |    * Pure LLM-based approach
22 | 7. Build a simple evaluation pipeline:
23 |    * Manually annotate 20 passages for ground truth (ideally, share those annotated passages in the group, so everyone have much more than 20)
24 |    * Compute precision, recall, and F1 score for each approach
25 |    * Analyze error patterns and classification mistakes
26 | 
27 | Questions (2 points):
28 | 
29 | 1. How does the performance of LLM-based NER compare to traditional approaches? What are the trade-offs in terms of accuracy, speed, and resource usage?
30 | 2. Which prompting strategy proved most effective for NER and classification tasks? Why?
31 | 3. What are the limitations and potential biases of using LLMs for NER and classification?
32 | 4. In what scenarios would you recommend using traditional NER vs. LLM-based approaches?
33 | 
34 | ## Hints
35 | 
36 | 1. Consider using prompt templates and systematic prompt engineering approaches
37 | 2. The quality of results heavily depends on the model size and prompt design
38 | 3. Consider implementing caching for LLM responses to speed up development
39 | 4. Pay attention to rate limits and resource usage when working with LLMs
40 | 


--------------------------------------------------------------------------------
/8-neural.md:
--------------------------------------------------------------------------------
 1 | # Neural search for question answering
 2 | 
 3 | The exercise introduces the problem of passage retrieval, an important step in factual question answering. 
 4 | This part concentrates on the methods for retrieving
 5 | the content of documents that might be useful for answering the question. We compare lexical text
 6 | representations (e.g. ElasticSearch default behaviour), with dense text representations (e.g. [multilingual E5](https://huggingface.co/intfloat/multilingual-e5-base) neural model).
 7 | 
 8 | ## Tasks
 9 | 
10 | Objectives (8 points)
11 | 
12 | 1. Read the documentation of the [document store](https://docs.haystack.deepset.ai/docs/document_store) and
13 |    the [retriever](https://docs.haystack.deepset.ai/docs/retriever) in the 
14 |    [Haystack framework](https://haystack.deepset.ai/).
15 | 2. Install Haystack framework (e.g. with `pip install 'farm-haystack[all]'`).
16 | 3. Configure a document store based on Faiss supported by multilingual E5 model:
17 |    1. For Faiss use [multilingual E5](https://huggingface.co/intfloat/multilingual-e5-base) or [silver retriever base](https://huggingface.co/ipipan/silver-retriever-base-v1) encoder.
18 |    3. **Warning:** If you use E5, make sure to [properly configure](https://github.com/deepset-ai/haystack/issues/5242) the store.
19 |    4. In the case you have problems using Faiss, you can use `InMemoryDocumentStore`, but this will require to re-index
20 |       all documents each time the script is run, which is time consuming.
21 | 4. Load the documents (passages) from the FiQA corpus.
22 | 5. Use the set of questions and the scorings defined in this corpus, to compute NDCG@5 for the dense retriever.
23 | 6. Compare the NDCG score from this exercise with the score from [lab 2](2-fts.md) and from [lab 6](6-classification.md).
24 | 7. **Bonus** (+2p) Combine dense retrieval with classification model from [lab 6](6-classification.md) to implement a two-step
25 |    retrieval. Compute NDCG@5 for this combined model.
26 | 8. **Bonus** (+2p) Use a different dense encoder, e.g. [E5 large](https://huggingface.co/intfloat/multilingual-e5-large) or [Polish Roberta Base](https://huggingface.co/sdadas/mmlw-retrieval-roberta-base) and compute NDCG@5.
27 | 
28 | Questions (2 points)
29 | 
30 | 1. Which of the methods: lexical match (e.g. ElasticSearch) or dense representation works better? 
31 | 2. Which of the methods is faster?
32 | 3. Try to determine the other pros and cons of using lexical search and dense document retrieval models.
33 |    
34 | 
35 | ## Hints
36 | 
37 | 1. Haystack is a framework for buliding question answering applications.
38 | 2. Lexical document retrieval is based on traditional NLP pipelines (e.g. lemmatization),
39 |    i.e. models based on bag of words. ElasticSearch typical usage is based on lexical search model.
40 | 3. Dense document retrieval is based on dense vector models provided by neural networks. These dense vectors might be 
41 |    generated directly, by e.g. avaraging the vectors of word embeddings belonging to a given text fragment. Yet such
42 |    models performance is inferior to sparse models.
43 | 4. More sophisticated models are trained directly on the document retrieval task. E.g. [DPR](https://arxiv.org/abs/2004.04906)
44 |    uses a bi-encoder architecture that has a separate neural network for encoding the question and for encoding the passage.
45 |    [E5](https://arxiv.org/abs/2212.03533) model has a shared encoder architecture.
46 |    These netowrks are trained to maximise the dot-product of the vectors produced by each of the networks.
47 | 5. Using dense vector representation requires computing the dense vectors for all passages in the dataset. 
48 |    These vectors might be stored in document stores such as [FAISS](https://github.com/facebookresearch/faiss) for faster retrieval, 
49 |    especially when the dataset is very large (does not fit into memory).
50 | 6. [Polish retrieval benchmark](https://huggingface.co/spaces/sdadas/pirb) lists and compares the models that implement dense retrieval for Polish.
51 | 


--------------------------------------------------------------------------------
/9-qa.md:
--------------------------------------------------------------------------------
 1 | # Contextual question answering
 2 | 
 3 | The aim of this exercise is building a neural model able to answer contextual questions in the legal domain.
 4 | 
 5 | ## Tasks
 6 | 
 7 | ## Objectives (8 points)
 8 | 
 9 | 1. Get acquainted with the [Simple legal questions dataset](https://github.com/apohllo/simple-legal-questions-pl) (you need to send your github login to gain access to the dataset).
10 | 2. **Bonus +5 points** Select one of the open issues in the dataset, provide the answers for the questions in the
11 |    package and open a pull request with the answers.
12 | 3. The legal questions dataset is your **test dataset**.
13 | 4. [PoQuAD](https://huggingface.co/datasets/clarin-pl/poquad) is your **train and validation dataset** (use the splits from the repo).
14 | 5. **Warning** PoQuAD has a python API compatible with the `datasets` library, but it only provides the **extractive answers**, even
15 |    though the abstractive answers are available in the JSON files. So you have to read the JSON files directly.
16 | 6. **Bonus +5 points** If you write a pull request with the changes to the API of the dataset that will expose the abstractive answers
17 |    and the impossible questions, and the PR will be accepted, you will gain additionl 5 points.
18 | 7. Train a neural model able to answer the legal questions. Make sure you are using a machine
19 |    with a GPU, since training the model on CPU will be very long. 
20 |    The training should include at least 3 epochs (depending on the size of the training set you are using). 
21 |    As the pre-trained models you can use (or any other model that is able to perform abstractive Question Answering):
22 |    * [plT5-base](https://huggingface.co/allegro/plt5-base)
23 |    * [plT5-large](https://huggingface.co/allegro/plt5-large)
24 | 8. If you have problems training the model, you can use [apohllo/plt5-base-poquad](https://huggingface.co/apohllo/plt5-base-poquad) which was trained on PoQuAD. **This will result in  subtraction of 2 points**. 
25 | 9. Report the obtained performance of the model (in the form of a table). The report should include *exact match* and *F1 score* 
26 |    for the tokens appearing both in the reference and the predicted answer.
27 | 10. Report the best results obtained on the validation dataset and the corresponding results on your test dataset. The results on the 
28 |    test set have to be obtained for the model that yield the best result on the validation dataset.
29 | 11. Generate, report and analyze the answers for at least 10 questions provided by the best model on you test dataset.
30 |    
31 | ## Questions (2 points)
32 | 
33 | 1. Does the performance on the validation dataset reflects the performance on your test set?
34 | 2. What are the outcomes of the model on your test questions? Are they satisfying? If not, what might be the reason
35 |    for that?
36 | 3. Why extractive question answering is not well suited for inflectional languages?
37 | 
38 | ## Hints
39 | 1. Contextual question answering can be resolved by at lest two approaches:
40 |    * extractive QA (EQA) - the model has to select a consecutive sequence of tokens from the context which form the question.
41 |    * abstractive QA (AQA) - the model has to generate a sequence of tokens, based on the question and the provided context.
42 | 2. Decoder-only models, like BERT, are not able to answer questions in the AQA paradigm, however they are very well suited for EQA.
43 | 3. To resolve AQA you need a generative model, such as (m)T%, BART or GPT. These model (generally) are called sequence-to-sequence
44 |    or text-to-text models, since they take text as the input and produce text as the output.
45 | 4. Text-to-text model generate the text autoregresively, i.e. they produce one token at a given step and then feed the generated token 
46 |    (and all tokens generated so far) as the input to the model when generating the next token.
47 |    As a result the generation process is pretty slow.
48 | 6. Many NLP tasks base on the neural networks can be solved with
49 |    [ready-made scripts](https://github.com/huggingface/transformers/tree/main/examples/pytorch) available in the Transformers library.
50 | 8. A model able to answer questions in the AQA paradigm may be trained with the
51 |    [run_seq2seq_qa.py](https://github.com/huggingface/transformers/tree/main/examples/pytorch/question-answering)
52 |    script available in Transfomers.
53 |    If using such a script make sure you are acquianted with the available training options - some of the are defined in the
54 |    [script itself](https://github.com/huggingface/transformers/blob/main/examples/pytorch/question-answering/run_seq2seq_qa.py#L56), 
55 |    but most of them are inherited from the general [trainer](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.TrainingArguments)
56 |    or [seq2seq trainer](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.Seq2SeqTrainingArguments).
57 | 
58 | 


--------------------------------------------------------------------------------
/elastic/Dockerfile:
--------------------------------------------------------------------------------
1 | ARG es_version
2 | FROM docker.elastic.co/elasticsearch/elasticsearch:$es_version
3 | # the ARG has to be doubled,
4 | # cf. https://docs.docker.com/engine/reference/builder/#understand-how-arg-and-from-interact
5 | ARG es_version
6 | RUN /usr/share/elasticsearch/bin/elasticsearch-plugin install pl.allegro.tech.elasticsearch.plugin:elasticsearch-analysis-morfologik:$es_version
7 | 


--------------------------------------------------------------------------------
/elastic/config/elasticsearch.yml:
--------------------------------------------------------------------------------
 1 | # Elasticsearch configuration for development environment
 2 | 
 3 | # Cluster settings
 4 | cluster:
 5 |   name: my-application-cluster
 6 | 
 7 | # Node settings
 8 | node:
 9 |   name: node-1
10 | 
11 | # Network settings
12 | network:
13 |   host: 0.0.0.0
14 | 
15 | # HTTP settings
16 | http:
17 |   port: 9200
18 | 
19 | # Discovery settings
20 | discovery:
21 |   type: single-node
22 | 
23 | # Memory settings
24 | bootstrap:
25 |   memory_lock: true
26 | 
27 | 
28 | # Path settings
29 | path:
30 |   data: /usr/share/elasticsearch/data
31 |   logs: /usr/share/elasticsearch/logs
32 | 
33 | # Security settings (disabled for development)
34 | xpack.security.enabled: false
35 | 
36 | # Allow CORS for development
37 | http.cors.enabled: true
38 | http.cors.allow-origin: "*"
39 | 
40 | # Increase the merge thread pool size
41 | thread_pool.write.queue_size: 1000
42 | 
43 | 


--------------------------------------------------------------------------------
/elastic/docker-compose.yaml:
--------------------------------------------------------------------------------
1 | version: '3.8'
2 | 
3 | services:
4 |   search:
5 |     image: apohllo/elasticsearch-morfologik:8.15.2
6 |     ports:
7 |       - "9200:9200"
8 |     volumes:
9 |       - ./config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml


--------------------------------------------------------------------------------
/elastic/query.sh:
--------------------------------------------------------------------------------
1 | #!/bin/bash
2 | 
3 | curl -k -X GET "http://localhost:9200/_cat/indices?v" -H "accept: application/json" -H "Content-Type: application/json"
4 | 


--------------------------------------------------------------------------------
/project/readme.md:
--------------------------------------------------------------------------------
 1 | # Project description
 2 | 
 3 | Your task is to design and implement 2 algorithms (or 4 algorithms if you work in a pair). These algorithms should either:
 4 | 1. group keyphrases,
 5 | 2. extract keyphrases from a document.
 6 | 
 7 | At least one of the designed algorithms needs to be a grouping algorithm, an extracting algorithm is not required.
 8 | Both algorithms should consider Polish as the target language.
 9 | 
10 | ## Keyphrase grouping
11 | 
12 | The grouping algorithm is given a number of lists of keyphrases, each list describes one document. 
13 | Additionaly it may be given those documents (it is up to you to decide). The task is to output partitions of keyphrases that
14 | should be merged together and the reference keyphrase they be merged into. E.g. `["nuclear energy", "wind energy",
15 | "energy"] -> "energy"` or `["wind energy", "hydropower", "solar energy"] -> "renewable energy sources"` (the second
16 | example is more difficult to achieve). The algorithm may use any publicly available data or tools, e.g. word embeddings,
17 | language models, onthologies or dictionaries.
18 | 
19 | 
20 | ## Keyphrase extraction
21 | 
22 | The extracting algorithm is given a set of documents and should output a set of lists of phrases (one list of phrases per
23 | document) that best describe (summarize) the document content. Those phrases should be present in the document.
24 | Lemmatization of the keyphrases is optional. Additionaly the algorith should rank those phrases, meaning that the phrase
25 | more important for the document (better describing its content) should be given higher score. E.g. in the article about
26 | Orlen-Lotos merger, the companies names would probably score higher than names of the CEOs, while the names of people
27 | quoted in the article could not be extracted as keywords at all. The algorithm may or may not process one document at a
28 | time. The algorithm may use any publicly available data or tools, e.g. word embeddings, language models, onthologies or
29 | dictionaries
30 | 
31 | 
32 | ## Resources
33 | 
34 | * [sample text corpus](https://github.com/the-ultimate-krol/spacex),
35 | * keyphrases assigned to each document by some algorithm. Please note, that those phrases may not be present in the
36 |   document, thus not fulfiling the requirement of being extractive keyphrases,
37 | * a reference algorithm extracting keypharses from documents in Polish: https://huggingface.co/Voicelab/vlt5-base-keywords
38 | 


--------------------------------------------------------------------------------
/project/spacex.zip:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/apohllo/nlp/034c26e00321fc45d1b3b8444a3eea5503b27f2d/project/spacex.zip


--------------------------------------------------------------------------------
/project/text_description.txt:
--------------------------------------------------------------------------------
 1 | Korpus spacex.zip zawiera 20 tekstów dotyczących różnych gałęzi działalności Elona Muska i jego projektów.
 2 | Każdy tekst zapisany jako .yaml zawiera metrykę:
 3 | 
 4 | subject: temat przewodni
 5 | title: "Tytuł artykułu zapisany w cudzysłowie"
 6 | author: autor lub autorzy
 7 | date: rok-miesiąc-dzień
 8 | publisher: nazwa portalu
 9 | references: media, na które powołują się autorzy tekstów
10 | source: agencje informacyjne, na które powołują się autorzy tekstów
11 | url: link do artykułu
12 | id: identyfikator artykułu
13 | tags:
14 |   - tagi przypisane przez redakcję
15 |   -
16 |   ...
17 | keys:
18 |   - tagi przypisane ręcznie
19 |   -
20 |   ...
21 | content: >- treść
22 | 


--------------------------------------------------------------------------------
/readme.md:
--------------------------------------------------------------------------------
 1 | # Natural language processing tasks
 2 | 
 3 | The tasks accompanying the Natural Language Processing classes at the Faculty of Computer Science, at AGH University of Science and Technology in Krakow.
 4 | 
 5 | 1. [Regular expressions](1-regexp.md)
 6 | 2. [Full text search](2-fts.md)
 7 | 3. [Levenshtein distance](3-levenshtein.md)
 8 | 4. [Language modelling](4-lm.md)
 9 | 5. [Classification](5-classification.md)
10 | 6. [Named entity recognition](6-ner.md)
11 | 7. [Classification and NER with LLMs](7-classification-ner-llm.md)
12 | 8. [Neural Search](8-neural.md)
13 | 9. [Question Answering](9-qa.md)
14 | 10. [RAG](10-rag.md)
15 | 
16 | # Staff
17 | 
18 | * Aleksander Smywiński-Pohl, PhD,
19 | * Magda Król,
20 | * Bartosz Minch, PhD.
21 | 


--------------------------------------------------------------------------------