├── bert.png ├── logo-banner.png ├── README.md ├── KeyBERT_gr.py └── 2023-12-03 18.41.12 Governing societies with Artificial Intelligence .txt /bert.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fabiomatricardi/MetadataIsAllYouNeed/main/bert.png -------------------------------------------------------------------------------- /logo-banner.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fabiomatricardi/MetadataIsAllYouNeed/main/logo-banner.png -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | # MetadataIsAllYouNeed 4 | Repo of the code from the Medium article Metadata Metamorphosis: from plain Data to Enhanced insights with Retrieval Augmented Generation 5 | 6 | ### Requirements 7 | ```python 8 | pip install pytorch 9 | pip install transformers 10 | pip install langchain 11 | pip install rich 12 | pip install gradio 13 | pip install keybert 14 | pip install tiktoken 15 | ``` 16 | 17 | In the repo 18 | - the Gradio GUI file for local run 19 | - the Google Colab Notebook to test it out yourself 20 | -------------------------------------------------------------------------------- /KeyBERT_gr.py: -------------------------------------------------------------------------------- 1 | import gradio as gr 2 | from keybert import KeyBERT 3 | #from ctransformers import AutoModelForCausalLM, AutoConfig, Config #import for GGML models 4 | import datetime 5 | 6 | doc = """ 7 | La filosofia della scienza è lo studio critico e riflessivo delle fondamenta, metodi e implicazioni di ogni branca della conoscenza scientifica. Si occupa di questioni come la natura della verità scientifica, le limitazioni del metodo scientifico, l'origine e il significato dell'interesse per la scoperta e il progresso scientifico, i rapporti tra scienza e tecnologia, la relazione tra scienza e filosofia stessa. 8 | La filosofia della scienza ha origine nelle prime riflessioni dei filosofi antichi sulla natura dell'esperimento scientifico e sull'uso delle teorie scientifiche per spiegare il mondo. Tra i primi pensatori che si occuparono di queste questioni ci furono Aristotele, Platone e Democrito. 9 | Nell'età moderna, la filosofia della scienza ha preso forma con le riflessioni dei grandi pensatori come Descartes, Bacon e Galilei sul metodo scientifico e sulla natura del progresso della conoscenza. In particolare, il metodo scientifico di Galilei è stato uno dei primi tentativi sistematici di definire le regole per la scoperta scientifica. 10 | Nel XX secolo, la filosofia della scienza ha avuto un ulteriore sviluppo con l'affermazione del positivismo logico e il dibattito tra realisti e antirealisti sul significato delle teorie scientifiche. I principali pensatori in questo campo sono stati Karl Popper, Thomas Kuhn e Imre Lakatos. 11 | In conclusione, la filosofia della scienza è un'area di studio critica che si occupa dei fondamenti epistemologici delle diverse branche della conoscenza scientifica. Ha avuto origine nelle riflessioni antiche sul metodo scientifico e ha continuato a svilupparsi nel XX secolo attraverso i dibattiti tra realisti, positivisti logici e antirealisti sulla natura del progresso scientifico. 12 | """ 13 | 14 | 15 | #MODEL SETTINGS also for DISPLAY 16 | modelfile = "multi-qa-MiniLM-L6-cos-v1" 17 | repetitionpenalty = 1.15 18 | contextlength=4096 19 | logfile = 'KeyBERT-promptsPlayground.txt' 20 | print("loading model...") 21 | stt = datetime.datetime.now() 22 | from keybert import KeyBERT 23 | kw_model = KeyBERT(model='multi-qa-MiniLM-L6-cos-v1') 24 | dt = datetime.datetime.now() - stt 25 | print(f"Model loaded in {dt}") 26 | 27 | def writehistory(text): 28 | with open(logfile, 'a', encoding='utf-8') as f: 29 | f.write(text) 30 | f.write('\n') 31 | f.close() 32 | 33 | """ 34 | gr.themes.Base() 35 | gr.themes.Default() 36 | gr.themes.Glass() 37 | gr.themes.Monochrome() 38 | gr.themes.Soft() 39 | """ 40 | def combine(text, ngram,dvsity): 41 | import datetime 42 | import random 43 | a = kw_model.extract_keywords(text, keyphrase_ngram_range=(1, ngram), stop_words='english', 44 | use_mmr=True, diversity=dvsity, highlight=True) 45 | output = ''' ''' 46 | colors = ['primary', 'secondary', 'success', 'danger','warning','info','light','dark'] 47 | for kw in a: 48 | s = random.randint(0,6) 49 | output = output + f'''{str(kw[0])} 50 | 51 | ''' 52 | STYLE = ''' 53 | 54 | 55 | 56 | ''' 57 | OUTPUT_OK = STYLE + '''
58 | ''' + output + "

" 59 | 60 | #print(OUTPUT_OK) 61 | 62 | return OUTPUT_OK 63 | 64 | #return generation, delta 65 | 66 | 67 | # MAIN GRADIO INTERFACE 68 | with gr.Blocks(theme=gr.themes.Soft()) as demo: #theme=gr.themes.Glass() 69 | #TITLE SECTION 70 | with gr.Row(variant='compact'): 71 | with gr.Column(scale=1): 72 | gr.HTML("
") 73 | with gr.Column(scale=3): 74 | gr.HTML("
" 75 | + "

Prompt Engineering Playground - Test your favourite LLM for advanced inferences

" 76 | + "

KeyBERT keyword extraction

") 77 | 78 | # PLAYGROUND INTERFACE SECTION 79 | with gr.Row(): 80 | with gr.Column(scale=1): 81 | gr.Markdown( 82 | f""" 83 | ### Tunning Parameters""") 84 | ngramrange = gr.Slider(label="nGram Range (1 to...)",minimum=1, maximum=4, step=1, value=2) 85 | diversity = gr.Slider(label="Text diversity", minimum=0.0, maximum=1.0, step=0.02, value=0.2) 86 | gr.Markdown( 87 | """ 88 | Change the NNGram range, the num of words in the keyword 89 | Change the diversity, lower number is little diversity 90 | And then click the Button below 91 | """) 92 | btn = gr.Button(value="Generate Keywords", size='lg', icon='./bert.png') 93 | with gr.Column(scale=4): 94 | text = gr.Textbox(label="Text to Analyze", lines=8) 95 | with gr.Group(): 96 | gr.Markdown(""" 97 | > Keywords""") 98 | labelled = gr.HTML() 99 | btn.click(combine, inputs=[text,ngramrange,diversity], outputs=[labelled]) 100 | 101 | if __name__ == "__main__": 102 | demo.launch(inbrowser=True) 103 | 104 | 105 | 106 | 107 | -------------------------------------------------------------------------------- /2023-12-03 18.41.12 Governing societies with Artificial Intelligence .txt: -------------------------------------------------------------------------------- 1 | Title: Governing societies with Artificial Intelligence  2 | published on Medium 3 | Author: Giles Crouch 4 | url: https://gilescrouch.medium.com/governing-society-and-artificial-intelligence-23882b9ce473 5 | 6 | Much of the hype and chatter around Artificial Intelligence today is how it will impact the workplace, the entertainment industry, human creativity and productivity. Increasingly, there are debates on ethics, human rights and human agency. These are important. But what about how we govern our societies with AI? 7 | Social governance is an element of culture which also includes economic systems, family systems, politics, militaries, art, architecture and literature. Sometimes, such as in a democracy, we have a say in these systems. In dictatorships, we do not. 8 | The mechanism by which we create and operate our social governance systems is bureaucracy. Which were well honed by corporations and go back many thousands of years. One might suggest that bureaucracy was humanity’s first multiplayer game. 9 | Humans have been fiddling about with all kinds of different governance systems for thousands of years, including hunter-gatherer and foraging societies. When we collectively decide we’re no longer thrilled with whatever system, we either run away and create something else or we have revolutions and whoever wins creates something new. The one constant? Bureaucracies. Rinse, wash, repeat. 10 | With the advent of Generative AI (GAI) through Large Language Models (LLMs) like ChatGPT, humanity finally had an opportunity to play with AI in a way it hadn’t before. Prior to GAI, our broader social understanding of AI was via Hollywood and Science-Fiction and it was very often rather dystopian. 11 | News articles today tend to either declare the end of the world is nigh or utopia will be in the earnings reports of Q4 of 2024. As the news media has often said, “if it bleeds, it leads” (a.k.a. clickbait today.) Unfortunately, just as so much else today is reduced to this or that arguments, the same has happened to AI. 12 | The reality is more likely to be somewhere in the middle when it comes to AI tools. Likely edging towards more benefits than harms. 13 | So we’ve not really been set up to receive AI in a nice way. It should be of little surprise in the launch and hype of GAI that we collectively panicked and ran about the backyard flailing our arms and screaming that the sky is falling. The prophets of AI in Silicon Valley didn’t much help with proclamations of the arrival of a new god in town and here’s your new religion; the new religious text is Large Language Models. 14 | Yet there is a very strong case for making AI work for us in governing our current and future societies. The increasing complexity of our systems of governance along with our sociocultural systems. Human brains, working collectively, are quite a powerful thing. But our collective cognitive abilities only go so far. There are risks such a Group Think, among others. 15 | We have co-evolved technologies because we need them to survive. As our societies and the technologies we use to run them have evolved, so has complexity. There’s a reason that frameworks and mental models such as design thinking and systems thinking have become ever more useful. They help us deal with complexity. So can Artificial Intelligence. 16 | The trick, as you’ve likely already considered, is how to do it right. That’s the tough part. It should not be left to the deities of Silicon Valley. While many are very smart indeed, the overwhelming evidence is that they are less than ideal stewards of society. They are technicians, they are not philosophers or stewards of culture. 17 | Musk, Bezos, Altman, Zuckerberg, Andreeson and so on with their litany of societal incompetencies and foibles must assure us this is the case. They see technology as the solution to everything. They see problems singularly and rarely apply any degree of critical thinking. And even more rarely, systems thinking, a method that’s increasingly important. Technology cannot solve societal issues. It never has. But humans can and do. 18 | We are best served by AI when we put it into context as to what AI actually is; a tool that augments humanity cognitively. 19 | If we are to find advantage in the use of AI to help us govern our societies, and I believe we should, we must then consider the implications. If the rise of the internet and social media has taught our species anything, it is that all technologies are a double-edged sword, that there are always unintended consequences and that no technology is neutral. These three principles should be etched in stone (mainly because stone is more durable as a storage device than a hard drive today.) 20 | If we accept these principles as inherently true, then we have a starting place from which to foster true societal innovations in how we govern ourselves. We can set regulations and laws into place that don’t stifle AI, but rather foster it. 21 | Alongside these tech leaders and engineers should sit philosophers, sociologists, anthropologists, psychologists, artists, architects, writers, lawyers and musicians. It is all these skills and professions that shape our societies today. There are some computer scientists working on AI that regularly engage with artists and musicians in developing their tools. They find it works. 22 | We already know that race and gender biases are common in AI tools, from LLMs to NLP and ML tools. We have starting points. 23 | In this way, we can take a human-centric approach towards how AI can be used in various aspects of helping us govern our society. 24 | For many of these techno-optimists, this will be seen as slowing things down. Stopping innovation and hurting more than helping. One might argue that the ideology of moving fast and breaking things has well, broken a lot of things and is why States are suing social media companies like Meta, looking to ban TikTok for espionage and well, a whole lot more. 25 | There are models and ways that enable AI to evolve. Yes, it may be a bit slower, but contrary to what the techno-optimists think, technology adoption move slower anyway. Real societal and business change, happens a lot slower than one might be lead to believe. 26 | Most of the time, tech industry leaders want to move faster but when asked why and what benefit that delivers, you’ll be met with a blank stare. They’ve just not thought that through beyond the narrow funnel of solving a problem, improving shareholder value, competition and damn the consequences. 27 | We have been experimenting with all kinds of different ways to govern our societies for thousands of years. Artificial Intelligence is not a holy grail that will serve up utopia at a drive-through window in a paper bag. It is simply another tool. Albeit a rather profound one. 28 | In the past, technologies have presented perceived risks to society such as the printing press, manufacturing machines, robots. Each was greatly feared. Each time the entrepreneurs cried that regulation would stifle innovation and society and markets would collapse. Yet here we are. 29 | However this goes, the only guarantee, the only solid prediction we can make is that it will be messy. At times, very messy. With two steps forward and one step backwards at other times. But move ahead we will. It’s the way we’ve always been. Perhaps, should we elevate our consciousness as some propose is inevitable, and we’ll figure something special out. Until then? Social enlightenment won’t come from AI delivered by Uber Eats. It will come from humans, with a little help from AI. 30 | 31 | --------------------------------------------------------------------------------