├── .demo ├── current-events.txt ├── imagebind-dog-audio.wav ├── llava-paper.txt ├── llava-view.jpg ├── test.mp3 ├── wiki-pink-flower.jpg └── wiki-yellow-flower.jpg ├── .gitignore ├── .vscode └── settings.json ├── Dockerfile ├── LICENSE ├── README.md ├── configs ├── zero2.json ├── zero3.json └── zero3_offload.json ├── multi_token ├── constants.py ├── data_tools.py ├── inference.py ├── language_models │ ├── __init__.py │ ├── base_model.py │ └── mistral.py ├── modalities │ ├── __init__.py │ ├── audio_clap.py │ ├── audio_whisper.py │ ├── base_modality.py │ ├── document_gte.py │ ├── imagebind.py │ ├── projectors.py │ ├── video_xclip.py │ └── vision_clip.py ├── model_utils.py ├── training.py └── training_data.py ├── requirements.txt ├── scripts ├── audio_setup.sh ├── clap_gpt_build_finetune_dataset.py ├── clap_gpt_build_pretrain_dataset.py ├── document_build_finetune_dataset.py ├── document_build_pretrain_dataset.py ├── document_setup.sh ├── gym_lunar_lander_build_dataset.py ├── gym_lunar_lander_client.py ├── imagebind_build_llava_finetune_dataset.py ├── imagebind_build_llava_pretrain_dataset.py ├── imagebind_setup.sh ├── llava_build_finetune_dataset.py ├── llava_build_pretrain_dataset.py ├── llava_download_finetune.sh ├── llava_download_pretrain.sh ├── llava_gpt_build_multi_image_finetune_dataset.py ├── merge_datasets.py ├── serve_model.py ├── train_model.py ├── upload_model.py ├── vastai_setup.sh ├── vastai_sync.sh ├── video_setup.sh ├── webui_download.py ├── whisper_build_pretrain_dataset.py ├── whisper_gpt_build_finetune_dataset.py ├── xclip_build_finetune_dataset.py └── xclip_build_pretrain_dataset.py └── setup.py /.demo/current-events.txt: -------------------------------------------------------------------------------- 1 | November 1, 2023 (Wednesday) 2 | edithistorywatch 3 | Armed conflicts and attacks 4 | 5 | 2023 Israel–Hamas war 6 | Evacuations during the 2023 Israel–Hamas war 7 | About 500 people, including injured Palestinians, foreigners and dual Palestinian–foreign nationals, are evacuated from the Gaza Strip through the Rafah Border Crossing to Egypt, marking the border crossing's first opening since the war began. (AFP via The Straits Times) 8 | Iran–Israel proxy conflict 9 | Israel deploys Sa'ar-class corvettes to the Red Sea in response to missile and drone attacks from Yemen. Israeli National Security Council official Tzachi Hanegbi says that the Houthi attacks on Israel are "intolerable". (Reuters) 10 | Iranian Foreign Affairs Minister Hossein Amir-Abdollahian warns of "harsh consequences" if Israel continues its invasion of the Gaza Strip. (Reuters) 11 | An Israeli soldier is killed in a mortar attack near the kibbutz of Be'eri in southern Israel. (The Times of Israel) 12 | Allied Democratic Forces insurgency 13 | The Ugandan military announces the capture of an Allied Democratic Forces commander who killed three people, including two tourists, in Queen Elizabeth National Park on October 27. (AP) 14 | Myanmar civil war 15 | The Three Brotherhood Alliance intensifies its attacks on junta outposts in northern Shan State as part of "Operation 1027". (Narinjara) 16 | Russian invasion of Ukraine 17 | Russian strikes against Ukrainian infrastructure 18 | Russia strikes 118 Ukrainian towns and villages with artillery, missiles, and drones over a 24-hour period, the most in a single day this year. More than 40 attacks are reported in Avdiivka. (BBC News) 19 | South Korea's National Intelligence Service reports that North Korea has supplied Russia with over one million rounds of artillery munitions in the ongoing invasion of Ukraine. (Al Jazeera) 20 | International relations 21 | 22 | International reactions to the 2023 Israel–Hamas war 23 | Israel–Jordan relations 24 | Jordan recalls its ambassador to Israel in condemnation of the ongoing war. (AFP via Zawya) 25 | Law and crime 26 | 27 | The European Data Protection Board bans American social media company Meta from acquiring its users' personal data for targeted advertising in the European Economic Area. (AFP via The Economic Times) 28 | Police in the United Kingdom, Germany, Belgium, and Italy arrest several people who threatened or planned terror attacks relating to the ongoing Israel–Hamas war. (The Jerusalem Post) 29 | Sports 30 | 31 | 2023 World Series 32 | In baseball, the Texas Rangers win the World Series against the Arizona Diamondbacks in 5 games, winning their first Major League Baseball championship in their 63-year existence. Corey Seager is named World Series MVP. (NBC News) (ESPN) 33 | October 31, 2023 (Tuesday) 34 | edithistorywatch 35 | Armed conflicts and attacks 36 | 37 | Arab–Israeli conflict 38 | 2023 Israel–Hamas war 39 | 2023 Israeli invasion of the Gaza Strip 40 | Eleven Israeli soldiers are killed and several others are wounded when hit by an anti-tank guided missile in northern Gaza. Seven of those killed were from the Givati Brigade. (The Guardian) (The Times of Israel) 41 | War crimes in the 2023 Israel–Hamas war 42 | Jabalia refugee camp airstrikes 43 | 31 October 2023 attack on Jabalia 44 | More than 50 people are killed and 150 others are injured by Israeli airstrikes at the Jabalia refugee camp in the Gaza Strip. Reports indicate that the camp has been completely destroyed. (Reuters) 45 | Amnesty International says that they have found that "the Israeli Army indiscriminately, and therefore unlawfully, used white phosphorous in an attack on Dhayra, in south Lebanon which "must be investigated as a war crime". (Amnesty International) 46 | Fifty-seven medical facilities have been damaged, with 32 currently inoperable due to ongoing airstrikes or lack of fuel, as Israel continue to target hospitals and medical centers. (Al Jazeera) 47 | The Office of the United Nations High Commissioner for Human Rights says that an immediate humanitarian ceasefire in Gaza is "absolutely imperative". (Anadolu Agency) 48 | The Patriarchate of Jerusalem blames Israel for the bombing of its cultural centre in Gaza City, saying that the attack "represents a stark embodiment of Israel's unwarranted determination to destroy the civil infrastructure and social service centres, as well as shelters for civilians". (Al-Ahram) 49 | The World Health Organization warns that Gaza is on the verge of a "public health catastrophe" due to overcrowding, widespread displacement, and severe damage to water and sanitation facilities, with only five percent of the normal water supplies accessible. (Al Jazeera) 50 | UNICEF says that at least 940 children have been reported missing in Gaza. They also state that infant dehydration is a growing threat, with Gaza becoming "a graveyard of children". (Times Now) 51 | UNICEF executive director Catherine Russell says that more than 420 children are being killed or injured in Gaza every day. (Al Jazeera) 52 | The Palestine Red Crescent Society reports continuous artillery and airstrikes near the al-Quds Hospital in northern Gaza. The hospital currently shelters hundreds of patients and more than 12,000 displaced civilians. (Al Jazeera) 53 | The Palestinian Health Ministry says that 8,525 Palestinians, including 3,542 children, have been killed in Israeli air raids. (Anadolu Agency) 54 | The UN agency for Palestinian refugees says that 64 of its employees have been killed during the war. (VOA) 55 | The Palestinian Prisoners Society says that Israeli forces arrested 60 Palestinians in the West Bank, adding that the "arrests are accompanied with torture and abusive attacks, as well as collective punishment and destruction of property". (Al Jazeera) 56 | The death toll in the West Bank increases to 130 as five more Palestinians are killed by Israeli soldiers. (Al Jazeera) 57 | Iran–Israel proxy conflict 58 | Israel intercepts three surface-to-surface missiles launched from Yemen, with at least one missile being shot down by its Arrow missile defence system. The Iran-backed Houthis claim responsibility for the launches. (Reuters via Ynet News) 59 | Mali War 60 | Tuareg rebels from Platform claim control over a camp in Kidal, northern Mali, following the evacuation of MINUSMA troops. (AFP via France 24) 61 | Russian invasion of Ukraine 62 | Crimea attacks 63 | Ukraine launches airstrikes on Crimea, destroying a Russian S-300 missile system near Yevpatoria. (The Kyiv Post) 64 | International relations 65 | 66 | International reactions to the 2023 Israel–Hamas war 67 | Foreign relations of Bolivia 68 | Bolivia suspends diplomatic relations with Israel. (Reuters) 69 | The Governments of Chile and Colombia recall their respective ambassadors to Israel as a response to Israel's military actions in the Gaza Strip. (ABC News) 70 | Canadian Foreign Affairs Minister Mélanie Joly calls for a "temporary pause in hostilities" in Gaza, saying that "the humanitarian situation facing the Palestinian people, facing Palestinian women and children, is dire". (National Post) 71 | Australian Foreign Affairs Minister Penny Wong expresses concern over growing attacks by residents of settlements against Palestinians in the occupied West Bank, saying that "such violence threatens to only inflame tensions and those responsible must be held accountable". (Al Jazeera) 72 | Jordanian Foreign Affairs Minister Ayman Safadi calls for an immediate stop to the "humanitarian catastrophe" in Gaza caused by Israeli bombardment and the implementation of international law, while stressing the necessity of humanitarian aid. (Al Jazeera) 73 | Law and crime 74 | 75 | 2019 Whakaari / White Island eruption 76 | The District Court of New Zealand in Auckland finds Whakaari's management guilty of insufficiently protecting visitors during the volcanic eruption that resulted in 22 deaths and 25 severe injuries. (AFP via VOA) 77 | 2022–2023 Bangladesh protests 78 | Two people are killed and dozens are injured in anti-government protests in Dhaka, Bangladesh. (BBC News) 79 | 2023 Israel–Hamas war protests 80 | Hundreds of demonstrators gather in front of the Israeli embassy in Manila, Philippines, to denounce the Israeli attacks on the Gaza Strip. (Al Jazeera) 81 | Anti-war protesters interrupt a U.S. Senate Senate Appropriations Committee hearing by U.S. Secretary of State Antony Blinken. Protesters are removed by police. (The Hill) 82 | A man in his 70s is arrested in Seoul, South Korea, after stabbing and injuring two police officers in front of the presidential residence. (Reuters) 83 | At least 380 people are arrested across Puerto Rico during a crackdown against 12 gangs dedicated to the sales of drugs and weapons. (AP) 84 | An explosion occurs at a military base near the agrotown of Kalodziščy in central Belarus. The Ministry of Defence claims that the explosion was deliberately orchestrated. (Euroradio) 85 | Sports 86 | 87 | Football Australia formally end their bid to host the 2034 FIFA World Cup, leaving Saudi Arabia as the only confirmed bid before FIFA's deadline for declarations of interest expires. (BBC Sport) 88 | October 30, 2023 (Monday) 89 | edithistorywatch 90 | Armed conflicts and attacks 91 | 92 | 2023 Israel–Hamas war 93 | 2023 Israeli invasion of the Gaza Strip 94 | Heavy clashes are reported as IDF tanks reach the outskirts of Gaza City. (Al Jazeera) 95 | Israeli troops block the Salah al-Din Road, the main highway in the Gaza Strip. (The Guardian) 96 | Israel announces that its forces rescued an IDF private who was taken hostage by Hamas during an overnight operation involving its intelligence agency Shin Bet. (The Times of Israel) 97 | Israeli Prime Minister Benjamin Netanyahu dismisses calls for a ceasefire in the war with Hamas, stating that Israel will continue its military operations. (AFP via ABS-CBN News) 98 | Yemeni civil war 99 | Houthi–Saudi Arabian conflict 100 | Saudi Arabia declares a "state of high alert" after four Saudi troops are killed in an attack by Houthi militants in Jazan Province near the border with Yemen. Saudi air defences also shoot down a ballistic missile over its territory that was travelling towards Israel. (Bloomberg) 101 | Three people are killed and five others injured during two shootouts between supporters of rival candidates in Mindanao, Philippines, amid elections for village leaders. (AFP via Al Jazeera) 102 | Business and economy 103 | 104 | 2023 United Auto Workers strike 105 | The United Auto Workers of the United States reaches a tentative deal with General Motors, bringing an official end to the six-week strike. (Reuters) 106 | International relations 107 | 108 | U.S. President Joe Biden announces that he will remove Gabon, Niger, Uganda and the Central African Republic from the African Growth and Opportunity Act trade program, because of "gross violations" of human rights and rampant political corruption, effective January 1, 2024. (Reuters) 109 | Law and crime 110 | 111 | 2023 Ybor City shooting 112 | Two people are killed and 16 others are injured in a mass shooting in the Ybor City neighborhood of Tampa, Florida, United States. A 22-year-old man is arrested. (CNN) 113 | Politics and elections 114 | 115 | 2023 Democratic Republic of the Congo general election 116 | The Constitutional Court of the Democratic Republic of the Congo dismisses a case aiming to disqualify former Katanga governor Moïse Katumbi from the presidential election, asserting that Katumbi solely possesses Congolese nationality despite claims of dual citizenship, which the country does not recognize. (AFP via EWN) 117 | 2023 Philippine barangay and Sangguniang Kabataan elections 118 | Filipinos elect members of the village and youth councils after a three-year delay. (AFP via Al Jazeera) 119 | David Adeang is elected as President of Nauru following a no-confidence vote in Russ Kun. (RNZ) 120 | German AfD politician Daniel Halemba is arrested for using the Nazi salute and possessing banned totalitarian symbols. (BBC News) (The Guardian) 121 | Science and technology 122 | 123 | In a study published in Nature Geoscience, researchers suggest that solid particulates (predominantly silicate) from the Chicxulub asteroid impact played a dominant role in the radiative forcing leading to the Cretaceous–Paleogene extinction event (impact winter), contrary to the widely accepted theory that sulfur aerosols led to the mass extinction. (AFP via RFI) (Nature) 124 | Japan suspends the pilot experiment of its first fully autonomous self-driving car following a minor collision. (AFP via Daily Tribune) 125 | Sports 126 | 127 | Rubiales affair 128 | FIFA bars former Royal Spanish Football Federation president Luis Rubiales from all football-related activities for three years due to Rubiales' inappropriate conduct during the 2023 FIFA Women's World Cup final in August. (AP) 129 | October 29, 2023 (Sunday) 130 | edithistorywatch 131 | Armed conflicts and attacks 132 | 133 | 2023 Israel–Hamas war 134 | War crimes in the 2023 Israel–Hamas war 135 | Impeding the delivery of aid to Gaza's residents may be considered a crime within the jurisdiction of the International Criminal Court, its top prosecutor has said. (Reuters) 136 | Reporters Without Borders says the killing of Reuters journalist Issam Abdallah in Lebanon earlier this month resulted from a deliberate strike from the direction of the Israeli border. (Reuters) 137 | Save the Children says more children have been killed in Gaza during the war, than the total killed in conflicts around the world since 2019. At least 3,324 children have been killed in Gaza, while 36 have died in the West Bank. (Al Jazeera) 138 | Two mortar shells hit a UN peacekeeper base in Hula, Nabatieh Governorate, Lebanon, injuring a peacekeeper. (WION) 139 | At least 8,005 Palestinians have been killed in Gaza since October 7. More than 1,400 people were killed in Israel. (Al Jazeera) 140 | Israel's ambassador to the United Nations, Gilad Erdan, calls on countries to cease funding the body after the UN General Assembly passed a non-binding resolution calling for an immediate ceasefire in Gaza. (The Business Standard) 141 | Israeli Prime Minister Benjamin Netanyahu says the war will be "long and difficult", with Israel intensifying its airstrikes and progressing to the "next phase" of operations. (NBC News) 142 | The Legal Center for Arab Minority Rights in Israel demands that Israel's government cancel emergency regulations extending the period which detainees suspected of "security offences" can be denied access to a lawyer. (Al Jazeera) 143 | 2023 Kerala bombing 144 | A bomb explodes during a Jehovah's Witnesses' convention at the Zamra International Convention Center in Kalamassery, Kerala, South India, killing two people and injuring around 40 others. A man claiming responsibility for the bombing surrenders to police. (Reuters) 145 | Russian invasion of Ukraine 146 | Crimea attacks 147 | Russia says that it has shot down 36 Ukrainian drones over the Black Sea and the Crimea. (DW) 148 | A person is killed and three more people are injured during a rocket attack by the Polisario Front in Smara, in the Morocco-occupied Western Sahara. (Middle East Online) 149 | Disasters and incidents 150 | 151 | 2023 Pacific hurricane season 152 | Hurricane Otis 153 | The Mexican government reports that at least 48 people were killed. Most of them in Acapulco, Guerrero. The government also reports widespread flooding, significant infrastructure damage, and aid has been slow to arrive. (AFP via The Journal) 154 | 2023 Rio Branco plane crash 155 | A plane crash in Rio Branco, Acre, Brazil, kills 12 people. (AP) 156 | 2023 Andhra Pradesh train collision 157 | Fourteen people are killed and 50 others are injured when two passenger trains collide in Andhra Pradesh, India. Human error is reported as the cause of the collision. (NDTV) 158 | The United Nations High Commissioner for Refugees reports that nine migrants have died in the Mediterranean Sea while heading to Italy from North Africa, with five of the bodies found on the coast of Sicily and four of the bodies recovered by the non-governmental organization Sea-Eye. (AFP via Daily Tribune) 159 | Law and crime 160 | 161 | 2023 Israel–Hamas war 162 | 2023 Israel–Hamas war protests 163 | 2023 wave of antisemitism in the North Caucasus 164 | A mob storms Uytash Airport in Makhachkala, Dagestan, Russia, as a Red Wings Airlines flight from Tel Aviv, Israel, arrives, shouting antisemitic slogans and attempting to board the plane. (AP) 165 | Protests are held in the United States including Florida, Michigan, and Colorado. (The Detroit News) (WPLG-TV) (The Denver Post) 166 | In Ottawa, Ontario, Canada, a pro-Palestinian protest is held calling for the Canadian government to help push for a ceasefire. (CTV News Ottawa) 167 | Antisemitism during the 2023 Israel–Hamas war 168 | Antisemitic threats are made toward Jewish students at Cornell University in Ithaca, New York, U.S. (WSYR-TV) 169 | Hijab and burka controversies in Europe 170 | The National Council of Switzerland votes 151–29 to approve a ban on full-face coverings, such as burqas worn by a small minority of Muslim women. Violators of the ban could face a fine of up to 1,000 Swiss francs. (AP) 171 | Iranian protests against compulsory hijab 172 | Human rights lawyer Nasrin Sotoudeh is arrested at the funeral of 17-year-old Armita Geravand in Tehran for not wearing a headscarf, according to local media. (AFP via Arab News) 173 | Bangladesh Nationalist Party secretary general Mirza Fakhrul Islam Alamgir and at least 164 party members are charged with the murder of a police officer during recent election-related demonstrations. (AFP via Daily Tribune) 174 | Politics and elections 175 | 176 | 2023 Omani general election 177 | Omanis elect the members of the Shura Council. (Times of Oman) 178 | October 28, 2023 (Saturday) 179 | edithistorywatch 180 | Armed conflicts and attacks 181 | 182 | 2023 Israel–Hamas war 183 | Taba and Nuweiba drone attacks 184 | An Egyptian Army spokesperson says that investigations into the two drones that fell in the towns of Nuweiba and Taba in the Sinai Peninsula yesterday showed that they were heading in a northerly direction across the Red Sea towards Israel. (Al-Ahram) 185 | War crimes in the 2023 Israel–Hamas war 186 | Thousands of Gazan workers employed in Israel go missing amid a campaign of mass arrests. Human rights groups and trade unions say that the workers have been illegally detained in military facilities in the West Bank, following the revocation of their work permits. (Al Jazeera) 187 | Israeli bombardments take place in the vicinities of the Al-Shifa Hospital and Indonesia Hospital while dozens of residents are at the facilities. (Al Jazeera) 188 | An Israeli shell hits the headquarters of the United Nations Interim Force in Lebanon, the second such incident since the war began. (WION) 189 | The UN Relief and Works Agency for Palestine Refugees says that 14 of its staff were killed in the past day, bringing the total to 53. (Al Jazeera) 190 | UN agencies say that they are unable to contact staff in Gaza amid the near-total communications blackout and also call for civilians to be protected. (UN) (AA) 191 | Human Rights Watch says that the near-total communication blackout "risks providing cover for mass atrocities and contributing to impunity for human rights violations" and will make it more difficult to "obtain critical information and evidence about human rights violations and war crimes being committed, and to hear directly from those experiencing the violations". (AFP via The Economic Times) 192 | The civil defense service in Gaza reports that Israeli airstrikes destroyed hundreds of buildings and damaged thousands of others in the Gaza Strip overnight. (CNN) 193 | Israeli warplanes bomb 150 underground targets in northern Gaza, including tunnels and underground infrastructure. (Al Jazeera) 194 | SpaceX CEO Elon Musk announces Starlink will be offered in Gaza to aid organizations after Israel disrupted most communication and internet services. Israel says it will prevent the use of the system. (Business Insider) (Anadolu) 195 | Business and economy 196 | 197 | 2023 United Auto Workers strike 198 | The United Auto Workers reaches an agreement with automaker Stellantis after signing a similar agreement with Ford in an effort to end the ongoing strike. (The Washington Post) (Politico) 199 | Disasters and accidents 200 | 201 | 2023 Darling Downs fires 202 | Evacuations are ordered for the towns of Tara, Wieambilla, and The Gums, Queensland, Australia, after more than 32 buildings are destroyed by ongoing bushfires. (7 News) 203 | Kostenko mine disaster 204 | At least 32 people are killed and 14 others are missing in a fire at a coal mine in Karaganda Region, Kazakhstan. (Reuters) 205 | At least 32 people are killed and 63 others are injured in a multiple-vehicle collision in Beheira Governorate, Egypt. (Reuters) 206 | At least 12 people die and four others are in critical condition after consuming bootleg alcohol at a wedding in Subang, West Java, Indonesia. Authorities arrest a suspect in relation to the incident. (AFP via Daily Tribune) 207 | Health and environment 208 | 209 | Sudan's health ministry reports over 100 deaths from cholera and dengue fever outbreaks since August, with war exacerbating healthcare challenges and leading to 70% of hospitals in war-affected areas becoming non-operational. (AFP via The Straits Times) 210 | International relations 211 | 212 | 2023 Israel–Hamas war 213 | 2023 Israel–Hamas war protests 214 | Hundreds of thousands of people demonstrate in cities across Europe, the Middle East, and Asia to show their support for Palestinians as Israel's military widens its air and ground offensive in the Gaza Strip. (Reuters) 215 | Thousands of Palestinians perform the Fajr prayer and stage demonstrations in Nablus, Tulkarem, Jenin and Tubas in the West Bank in a show of support for Gaza. (AA) 216 | International reactions to the 2023 Israel–Hamas war 217 | The UN General Assembly passes a resolution by a vote of 120–14, with 45 abstentions, calling for an immediate humanitarian truce in Gaza, despite US and Israeli opposition. (AP) 218 | The International Organization for Migration renews its call for a ceasefire, saying that "civilians must be protected. There are no winners in war. The most vulnerable pay the heaviest toll". (The Business Standard) 219 | Turkish President Recep Tayyip Erdoğan calls on Israel to "immediately come out of its state of madness and stop its attacks" on the Gaza Strip, saying that they have "targeted women, children and innocent civilians, deepening the humanitarian crisis". (The Business Standard) 220 | The Egyptian foreign ministry warns of the "humanitarian and security repercussions of the Israeli ground attack" on Gaza, saying that "we hold the Israeli government responsible for violating the United Nations General Assembly resolution for an immediate ceasefire and implementing a humanitarian truce". (Al Jazeera) 221 | Save the Children says that children will "bear the brunt" of Israel's intensified attacks on the Gaza Strip and that in the event of a full ground incursion, more than one million children's lives, nearly half of the 2.3 million population of Gaza, will be affected. (Scoop) 222 | Politics and elections 223 | 224 | 2024 United States presidential election 225 | Mike Pence 2024 presidential campaign 226 | Former U.S. Vice President Mike Pence suspends his campaign for the Republican Party nomination for the upcoming presidential election. (CNBC) 227 | Sports 228 | 229 | 2023 Rugby World Cup 230 | South Africa wins a record fourth Webb Ellis Cup after defeating New Zealand 12–11 in the final. (AFP via France 24) 231 | October 27, 2023 (Friday) 232 | edithistorywatch 233 | Armed conflicts and attacks 234 | 235 | 2023 Israel–Hamas war 236 | 2023 Israeli invasion of the Gaza Strip 237 | The Israel Defense Forces enter the Gaza Strip from multiple directions with infantry and tanks. Heavy clashes were reported. (New York Magazine) 238 | Israeli troops advance on the city of Beit Hanoun. (The Atlas News) 239 | War crimes in the 2023 Israel–Hamas war 240 | At least 33 Palestinians are killed and several are injured, following Israeli airstrikes on at least three residential neighbourhoods in Gaza City. (Al Jazeera) 241 | At least 10 Palestinians are killed, including journalist Yasser Abu Namous and his mother, in a series of airstrikes carried out by Israeli forces in southern Gaza. (Al Jazeera) 242 | Several people are killed or injured in an Israeli airstrike on the Al-Shati refugee camp which destroyed residential buildings and a nearby mosque. (The Business Standard) 243 | The Palestinian Red Crescent says that its medics were shot at by Israeli forces as they treated an injured person in Tubas in the West Bank. (The Business Standard) 244 | Six people are injured after a missile hits a building in Taba, Egypt. Israel says that the missile was launched from Yemen. (The Times of Israel) 245 | Four people are killed in a pre-dawn raid by Israeli forces in the West Bank. At least 19 people have been arrested. (Al Jazeera) 246 | Hamas reports that Israel has disrupted most communication and internet services in the Gaza Strip. (AFP via Onmanorama) 247 | Myanmar civil war 248 | Various rebel groups across Myanmar, including the Myanmar National Democratic Alliance Army, the Ta'ang National Liberation Army, the Arakan Army, and local defense groups, launch an offensive in northern Shan State and capture several towns and junta outposts. (The Irrawaddy) 249 | Russian invasion of Ukraine 250 | Crimea attacks 251 | Pro-Russian official Oleg Tsaryov is critically injured in an assassination attempt by unknown assailants in Crimea and is reported to be in intensive care. (Reuters) 252 | Syrian civil war 253 | American intervention in the Syrian civil war 254 | The U.S. military says that it has conducted airstrikes on Iran-backed targets in Syria linked to the Islamic Revolutionary Guard Corps. (AP) 255 | International relations 256 | 257 | The Financial Action Task Force adds Bulgaria to its "grey list" for increased monitoring due to strategic deficiencies in countering money laundering and terrorism financing. (AFP via Daily Tribune) 258 | Law and crime 259 | 260 | 2023 Israel–Hamas war protests 261 | Thousands of people gather in Chile's capital Santiago to protest against Israel's military actions in Gaza. (The Cairns Post) 262 | 2023 Lewiston shootings 263 | Authorities in Maine, United States, confirm the death of Robert Card, the suspected perpetrator of a mass shooting that killed 18 in Lewiston, with indications of a self-inflicted gunshot wound. (AFP via France 24) 264 | Aftermath of the 2023 Mozambican local elections 265 | Protests occur across Mozambique, particularly in the city of Nampula, against the results of local elections which locals claim were rigged. Ten people are injured in clashes with police. (Reuters) 266 | Aftermath of the assassination of Jovenel Moïse 267 | A retired Colombian army officer who participated in the 2021 assassination of Haitian president Jovenel Moïse is sentenced to life in prison by a court in Miami, United States. (AP) 268 | The Philippine National Police detain almost 600 individuals during a raid on a suspected human trafficking and Internet fraud hub in Manila, with nationals from multiple countries present. (AFP via CNA) 269 | Politics and elections 270 | 271 | Sultan Ibrahim Ismail of Johor is selected as Malaysia's 17th Yang di-Pertuan Agong by the nation's royalty, set to be inaugurated following the end of the current king's term on 31 January 2024. (AFP via Dawn) -------------------------------------------------------------------------------- /.demo/imagebind-dog-audio.wav: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sshh12/multi_token/6eb9813edf2e8ddbff951bca4b2f3d65b6b1206e/.demo/imagebind-dog-audio.wav -------------------------------------------------------------------------------- /.demo/llava-view.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sshh12/multi_token/6eb9813edf2e8ddbff951bca4b2f3d65b6b1206e/.demo/llava-view.jpg -------------------------------------------------------------------------------- /.demo/test.mp3: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sshh12/multi_token/6eb9813edf2e8ddbff951bca4b2f3d65b6b1206e/.demo/test.mp3 -------------------------------------------------------------------------------- /.demo/wiki-pink-flower.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sshh12/multi_token/6eb9813edf2e8ddbff951bca4b2f3d65b6b1206e/.demo/wiki-pink-flower.jpg -------------------------------------------------------------------------------- /.demo/wiki-yellow-flower.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sshh12/multi_token/6eb9813edf2e8ddbff951bca4b2f3d65b6b1206e/.demo/wiki-yellow-flower.jpg -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # Byte-compiled / optimized / DLL files 2 | __pycache__/ 3 | *.py[cod] 4 | *$py.class 5 | 6 | # C extensions 7 | *.so 8 | 9 | # Distribution / packaging 10 | .Python 11 | build/ 12 | develop-eggs/ 13 | dist/ 14 | downloads/ 15 | eggs/ 16 | .eggs/ 17 | lib/ 18 | lib64/ 19 | parts/ 20 | sdist/ 21 | var/ 22 | wheels/ 23 | share/python-wheels/ 24 | *.egg-info/ 25 | .installed.cfg 26 | *.egg 27 | MANIFEST 28 | 29 | # PyInstaller 30 | # Usually these files are written by a python script from a template 31 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 32 | *.manifest 33 | *.spec 34 | 35 | # Installer logs 36 | pip-log.txt 37 | pip-delete-this-directory.txt 38 | 39 | # Unit test / coverage reports 40 | htmlcov/ 41 | .tox/ 42 | .nox/ 43 | .coverage 44 | .coverage.* 45 | .cache 46 | nosetests.xml 47 | coverage.xml 48 | *.cover 49 | *.py,cover 50 | .hypothesis/ 51 | .pytest_cache/ 52 | cover/ 53 | 54 | # Translations 55 | *.mo 56 | *.pot 57 | 58 | # Django stuff: 59 | *.log 60 | local_settings.py 61 | db.sqlite3 62 | db.sqlite3-journal 63 | 64 | # Flask stuff: 65 | instance/ 66 | .webassets-cache 67 | 68 | # Scrapy stuff: 69 | .scrapy 70 | 71 | # Sphinx documentation 72 | docs/_build/ 73 | 74 | # PyBuilder 75 | .pybuilder/ 76 | target/ 77 | 78 | # Jupyter Notebook 79 | .ipynb_checkpoints 80 | 81 | # IPython 82 | profile_default/ 83 | ipython_config.py 84 | 85 | # pyenv 86 | # For a library or package, you might want to ignore these files since the code is 87 | # intended to run in multiple environments; otherwise, check them in: 88 | # .python-version 89 | 90 | # pipenv 91 | # According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control. 92 | # However, in case of collaboration, if having platform-specific dependencies or dependencies 93 | # having no cross-platform support, pipenv may install dependencies that don't work, or not 94 | # install all needed dependencies. 95 | #Pipfile.lock 96 | 97 | # poetry 98 | # Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control. 99 | # This is especially recommended for binary packages to ensure reproducibility, and is more 100 | # commonly ignored for libraries. 101 | # https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control 102 | #poetry.lock 103 | 104 | # pdm 105 | # Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control. 106 | #pdm.lock 107 | # pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it 108 | # in version control. 109 | # https://pdm.fming.dev/#use-with-ide 110 | .pdm.toml 111 | 112 | # PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm 113 | __pypackages__/ 114 | 115 | # Celery stuff 116 | celerybeat-schedule 117 | celerybeat.pid 118 | 119 | # SageMath parsed files 120 | *.sage.py 121 | 122 | # Environments 123 | .env 124 | .venv 125 | env/ 126 | venv/ 127 | ENV/ 128 | env.bak/ 129 | venv.bak/ 130 | 131 | # Spyder project settings 132 | .spyderproject 133 | .spyproject 134 | 135 | # Rope project settings 136 | .ropeproject 137 | 138 | # mkdocs documentation 139 | /site 140 | 141 | # mypy 142 | .mypy_cache/ 143 | .dmypy.json 144 | dmypy.json 145 | 146 | # Pyre type checker 147 | .pyre/ 148 | 149 | # pytype static type analyzer 150 | .pytype/ 151 | 152 | # Cython debug symbols 153 | cython_debug/ 154 | 155 | # PyCharm 156 | # JetBrains specific template is maintained in a separate JetBrains.gitignore that can 157 | # be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore 158 | # and can be added to the global gitignore or merged into this file. For a more nuclear 159 | # option (not recommended) you can uncomment the following to ignore the entire idea folder. 160 | #.idea/ 161 | wandb/ 162 | ssh*.json 163 | notebooks/ 164 | .checkpoints/ 165 | -------------------------------------------------------------------------------- /.vscode/settings.json: -------------------------------------------------------------------------------- 1 | { 2 | "[python]": { 3 | "editor.defaultFormatter": "ms-python.black-formatter" 4 | }, 5 | "python.formatting.provider": "none" 6 | } -------------------------------------------------------------------------------- /Dockerfile: -------------------------------------------------------------------------------- 1 | FROM pytorch/pytorch:2.0.1-cuda11.7-cudnn8-devel 2 | 3 | WORKDIR /app 4 | 5 | RUN apt-get update && apt-get install -y git curl nano wget unzip rsync jq 6 | 7 | RUN git clone https://github.com/sshh12/multi_token \ 8 | && cd multi_token \ 9 | && pip install -r requirements.txt \ 10 | && pip install -e . 11 | 12 | RUN pip install flash-attn --no-build-isolation 13 | 14 | CMD bash -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Apache License 2 | Version 2.0, January 2004 3 | http://www.apache.org/licenses/ 4 | 5 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 6 | 7 | 1. Definitions. 8 | 9 | "License" shall mean the terms and conditions for use, reproduction, 10 | and distribution as defined by Sections 1 through 9 of this document. 11 | 12 | "Licensor" shall mean the copyright owner or entity authorized by 13 | the copyright owner that is granting the License. 14 | 15 | "Legal Entity" shall mean the union of the acting entity and all 16 | other entities that control, are controlled by, or are under common 17 | control with that entity. For the purposes of this definition, 18 | "control" means (i) the power, direct or indirect, to cause the 19 | direction or management of such entity, whether by contract or 20 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 21 | outstanding shares, or (iii) beneficial ownership of such entity. 22 | 23 | "You" (or "Your") shall mean an individual or Legal Entity 24 | exercising permissions granted by this License. 25 | 26 | "Source" form shall mean the preferred form for making modifications, 27 | including but not limited to software source code, documentation 28 | source, and configuration files. 29 | 30 | "Object" form shall mean any form resulting from mechanical 31 | transformation or translation of a Source form, including but 32 | not limited to compiled object code, generated documentation, 33 | and conversions to other media types. 34 | 35 | "Work" shall mean the work of authorship, whether in Source or 36 | Object form, made available under the License, as indicated by a 37 | copyright notice that is included in or attached to the work 38 | (an example is provided in the Appendix below). 39 | 40 | "Derivative Works" shall mean any work, whether in Source or Object 41 | form, that is based on (or derived from) the Work and for which the 42 | editorial revisions, annotations, elaborations, or other modifications 43 | represent, as a whole, an original work of authorship. For the purposes 44 | of this License, Derivative Works shall not include works that remain 45 | separable from, or merely link (or bind by name) to the interfaces of, 46 | the Work and Derivative Works thereof. 47 | 48 | "Contribution" shall mean any work of authorship, including 49 | the original version of the Work and any modifications or additions 50 | to that Work or Derivative Works thereof, that is intentionally 51 | submitted to Licensor for inclusion in the Work by the copyright owner 52 | or by an individual or Legal Entity authorized to submit on behalf of 53 | the copyright owner. For the purposes of this definition, "submitted" 54 | means any form of electronic, verbal, or written communication sent 55 | to the Licensor or its representatives, including but not limited to 56 | communication on electronic mailing lists, source code control systems, 57 | and issue tracking systems that are managed by, or on behalf of, the 58 | Licensor for the purpose of discussing and improving the Work, but 59 | excluding communication that is conspicuously marked or otherwise 60 | designated in writing by the copyright owner as "Not a Contribution." 61 | 62 | "Contributor" shall mean Licensor and any individual or Legal Entity 63 | on behalf of whom a Contribution has been received by Licensor and 64 | subsequently incorporated within the Work. 65 | 66 | 2. Grant of Copyright License. Subject to the terms and conditions of 67 | this License, each Contributor hereby grants to You a perpetual, 68 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 69 | copyright license to reproduce, prepare Derivative Works of, 70 | publicly display, publicly perform, sublicense, and distribute the 71 | Work and such Derivative Works in Source or Object form. 72 | 73 | 3. Grant of Patent License. Subject to the terms and conditions of 74 | this License, each Contributor hereby grants to You a perpetual, 75 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 76 | (except as stated in this section) patent license to make, have made, 77 | use, offer to sell, sell, import, and otherwise transfer the Work, 78 | where such license applies only to those patent claims licensable 79 | by such Contributor that are necessarily infringed by their 80 | Contribution(s) alone or by combination of their Contribution(s) 81 | with the Work to which such Contribution(s) was submitted. If You 82 | institute patent litigation against any entity (including a 83 | cross-claim or counterclaim in a lawsuit) alleging that the Work 84 | or a Contribution incorporated within the Work constitutes direct 85 | or contributory patent infringement, then any patent licenses 86 | granted to You under this License for that Work shall terminate 87 | as of the date such litigation is filed. 88 | 89 | 4. Redistribution. You may reproduce and distribute copies of the 90 | Work or Derivative Works thereof in any medium, with or without 91 | modifications, and in Source or Object form, provided that You 92 | meet the following conditions: 93 | 94 | (a) You must give any other recipients of the Work or 95 | Derivative Works a copy of this License; and 96 | 97 | (b) You must cause any modified files to carry prominent notices 98 | stating that You changed the files; and 99 | 100 | (c) You must retain, in the Source form of any Derivative Works 101 | that You distribute, all copyright, patent, trademark, and 102 | attribution notices from the Source form of the Work, 103 | excluding those notices that do not pertain to any part of 104 | the Derivative Works; and 105 | 106 | (d) If the Work includes a "NOTICE" text file as part of its 107 | distribution, then any Derivative Works that You distribute must 108 | include a readable copy of the attribution notices contained 109 | within such NOTICE file, excluding those notices that do not 110 | pertain to any part of the Derivative Works, in at least one 111 | of the following places: within a NOTICE text file distributed 112 | as part of the Derivative Works; within the Source form or 113 | documentation, if provided along with the Derivative Works; or, 114 | within a display generated by the Derivative Works, if and 115 | wherever such third-party notices normally appear. The contents 116 | of the NOTICE file are for informational purposes only and 117 | do not modify the License. You may add Your own attribution 118 | notices within Derivative Works that You distribute, alongside 119 | or as an addendum to the NOTICE text from the Work, provided 120 | that such additional attribution notices cannot be construed 121 | as modifying the License. 122 | 123 | You may add Your own copyright statement to Your modifications and 124 | may provide additional or different license terms and conditions 125 | for use, reproduction, or distribution of Your modifications, or 126 | for any such Derivative Works as a whole, provided Your use, 127 | reproduction, and distribution of the Work otherwise complies with 128 | the conditions stated in this License. 129 | 130 | 5. Submission of Contributions. Unless You explicitly state otherwise, 131 | any Contribution intentionally submitted for inclusion in the Work 132 | by You to the Licensor shall be under the terms and conditions of 133 | this License, without any additional terms or conditions. 134 | Notwithstanding the above, nothing herein shall supersede or modify 135 | the terms of any separate license agreement you may have executed 136 | with Licensor regarding such Contributions. 137 | 138 | 6. Trademarks. This License does not grant permission to use the trade 139 | names, trademarks, service marks, or product names of the Licensor, 140 | except as required for reasonable and customary use in describing the 141 | origin of the Work and reproducing the content of the NOTICE file. 142 | 143 | 7. Disclaimer of Warranty. Unless required by applicable law or 144 | agreed to in writing, Licensor provides the Work (and each 145 | Contributor provides its Contributions) on an "AS IS" BASIS, 146 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 147 | implied, including, without limitation, any warranties or conditions 148 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 149 | PARTICULAR PURPOSE. You are solely responsible for determining the 150 | appropriateness of using or redistributing the Work and assume any 151 | risks associated with Your exercise of permissions under this License. 152 | 153 | 8. Limitation of Liability. In no event and under no legal theory, 154 | whether in tort (including negligence), contract, or otherwise, 155 | unless required by applicable law (such as deliberate and grossly 156 | negligent acts) or agreed to in writing, shall any Contributor be 157 | liable to You for damages, including any direct, indirect, special, 158 | incidental, or consequential damages of any character arising as a 159 | result of this License or out of the use or inability to use the 160 | Work (including but not limited to damages for loss of goodwill, 161 | work stoppage, computer failure or malfunction, or any and all 162 | other commercial damages or losses), even if such Contributor 163 | has been advised of the possibility of such damages. 164 | 165 | 9. Accepting Warranty or Additional Liability. While redistributing 166 | the Work or Derivative Works thereof, You may choose to offer, 167 | and charge a fee for, acceptance of support, warranty, indemnity, 168 | or other liability obligations and/or rights consistent with this 169 | License. However, in accepting such obligations, You may act only 170 | on Your own behalf and on Your sole responsibility, not on behalf 171 | of any other Contributor, and only if You agree to indemnify, 172 | defend, and hold each Contributor harmless for any liability 173 | incurred by, or claims asserted against, such Contributor by reason 174 | of your accepting any such warranty or additional liability. 175 | 176 | END OF TERMS AND CONDITIONS 177 | 178 | APPENDIX: How to apply the Apache License to your work. 179 | 180 | To apply the Apache License to your work, attach the following 181 | boilerplate notice, with the fields enclosed by brackets "[]" 182 | replaced with your own identifying information. (Don't include 183 | the brackets!) The text should be enclosed in the appropriate 184 | comment syntax for the file format. We also recommend that a 185 | file or class name and description of purpose be included on the 186 | same "printed page" as the copyright notice for easier 187 | identification within third-party archives. 188 | 189 | Copyright [yyyy] [name of copyright owner] 190 | 191 | Licensed under the Apache License, Version 2.0 (the "License"); 192 | you may not use this file except in compliance with the License. 193 | You may obtain a copy of the License at 194 | 195 | http://www.apache.org/licenses/LICENSE-2.0 196 | 197 | Unless required by applicable law or agreed to in writing, software 198 | distributed under the License is distributed on an "AS IS" BASIS, 199 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 200 | See the License for the specific language governing permissions and 201 | limitations under the License. 202 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # multi_token 2 | 3 | > Embed arbitrary modalities (images, audio, documents, etc) into large language models. 4 | 5 | This library is designed to be an extension of LLaVA for encoding ✨anything✨ (images, sounds, documents, videos, motion capture, screenshots, voice recordings, ...) into a format that can used in large language models. Its primary contribution is the ability to embed multiple instances and modalities into a single model and a framework for doing so fairly easily. 6 | 7 | Potentially with this you could ask Large Multimodal Models (LMMs): 8 | 9 | - > Read \ and give me a summary. 10 | 11 | - > Listen to \ and answer the spoke question. 12 | 13 | - > Compare and contrast \ and \ 14 | 15 | - > Given \ and \, what key should I press? 16 | 17 | Interested in how this works? See this [blog post](https://blog.sshh.io/p/large-multimodal-models-lmms). 18 | 19 | ## Usage 20 | 21 | ```bash 22 | git clone https://github.com/sshh12/multi_token \ 23 | && cd multi_token \ 24 | && pip install -r requirements.txt \ 25 | && pip install -e . 26 | 27 | pip install flash-attn --no-build-isolation 28 | ``` 29 | 30 | ### Model Zoo 31 | 32 | #### ⚠️ If you run into a missing `adapters.bin` see https://github.com/sshh12/multi_token/issues/12. ⚠️ 33 | 34 | | Base Model | Model | Modality | Notes | 35 | | - | - | - | - | 36 | | [mistralai/Mistral-7B-Instruct-v0.1](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1) | [sshh12/Mistral-7B-LoRA-DocumentGTE-16K-x8](https://huggingface.co/sshh12/Mistral-7B-LoRA-DocumentGTE-16K-x8) | **Long Document**

Encode a document as a series of `` and with `documents`. | ⚠️📚 A compression model pretrained on wikipedia and finetuned on LongAlpaca and Long-Data-Collections. Compresses chunks of 512 tokens into 64 using [gte-large](https://huggingface.co/thenlper/gte-large), as expected the results are fairly lossy. It performs similarly to the x128 version suggesting the bottleneck is the embedding model itself.

Compute: ~100 A6000 hours| 37 | | [mistralai/Mistral-7B-Instruct-v0.1](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1) | [sshh12/Mistral-7B-LoRA-DocumentGTE-260K-x128](https://huggingface.co/sshh12/Mistral-7B-LoRA-DocumentGTE-260K-x128) | **Long Document**

Encode a document as a series of `` and with `documents`. | ⚠️📚 A compression model pretrained on wikipedia and finetuned on LongAlpaca and Long-Data-Collections. Compresses chunks of 512 tokens into only 4 using [gte-large](https://huggingface.co/thenlper/gte-large), as expected the results are fairly lossy.

Compute: ~50 A6000 hours| 38 | | [mistralai/Mistral-7B-Instruct-v0.1](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1) | [sshh12/Mistral-7B-LoRA-ImageBind-LLAVA](https://huggingface.co/sshh12/Mistral-7B-LoRA-ImageBind-LLAVA) | **ImageBind (Vision/Audio/Text)**

Encode audio or image filenames as `` and with `imagebinds`. | ⚠️🖼️🔊📚 A model pretrained and finetuned on an augmented LLaVA dataset. Might hallucinate colors from audio and needs explicit mention of if the input is a sound/image/document.

Compute: ~180 4090 hours| 39 | | [mistralai/Mistral-7B-Instruct-v0.1](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1) | [sshh12/Mistral-7B-LoRA-VisionCLIP-LLAVA](https://huggingface.co/sshh12/Mistral-7B-LoRA-VisionCLIP-LLAVA) | **Vision**

Encode images as `` and with `images`. | ⭐🖼️ A model pretrained and finetuned on the LLaVA dataset. This should be comparable to [BakLLaVA](https://github.com/SkunkworksAI/BakLLaVA) and [LLaVA 1.5](https://llava-vl.github.io/).

Compute: ~160 3090 Ti hours| 40 | | [mistralai/Mistral-7B-Instruct-v0.1](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1) | [sshh12/Mistral-7B-LoRA-VisionCLIPPool-LLAVA](https://huggingface.co/sshh12/Mistral-7B-LoRA-VisionCLIPPool-LLAVA) | **Vision**

Encode images as `` and with `images`. | ⭐🖼️ A model pretrained and finetuned on the LLaVA dataset. This should be comparable to [BakLLaVA](https://github.com/SkunkworksAI/BakLLaVA) and [LLaVA 1.5](https://llava-vl.github.io/). Uses the last layer of CLIP encoded as 10-tokens (rather than the orignal 576).

Compute: ~100 A6000 hours| 41 | | [mistralai/Mistral-7B-Instruct-v0.1](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1) | [sshh12/Mistral-7B-LoRA-Multi-VisionCLIPPool-LLAVA](https://huggingface.co/sshh12/Mistral-7B-LoRA-Multi-VisionCLIPPool-LLAVA) | **Vision**

Encode images as `...` and with `images`. | ⭐🖼️🖼️ A model pretrained and finetuned on the LLaVA dataset and a synthetic multi-image dataset. Images encoded as 10-tokens each and this should support up to 6 images.

Compute: ~100 A6000 hours| 42 | | [mistralai/Mistral-7B-Instruct-v0.1](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1) | [sshh12/Mistral-7B-CLIP-LoRA-captions-only-demo](https://huggingface.co/sshh12/Mistral-7B-CLIP-LoRA-captions-only-demo) | **Vision**

Encode images as `` and with `images`. | ⚠️🖼️ This is a __very limited__ image model trained on only a few __caption-only__ examples for the sake of demonstrating a proof of concept.

Compute: ~10 3090 Ti hours | 43 | | [mistralai/Mistral-7B-Instruct-v0.1](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1) | [sshh12/Mistral-7B-LoRA-XCLIP](https://huggingface.co/sshh12/Mistral-7B-LoRA-XCLIP) | **Video**

Encode videos as `