├── LICENSE
├── README.md
├── migrate.py
└── requirements.txt


/LICENSE:
--------------------------------------------------------------------------------
 1 | MIT License
 2 | 
 3 | Copyright (c) 2025 Taylor Wilsdon
 4 | 
 5 | Permission is hereby granted, free of charge, to any person obtaining a copy
 6 | of this software and associated documentation files (the "Software"), to deal
 7 | in the Software without restriction, including without limitation the rights
 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
 9 | copies of the Software, and to permit persons to whom the Software is
10 | furnished to do so, subject to the following conditions:
11 | 
12 | The above copyright notice and this permission notice shall be included in all
13 | copies or substantial portions of the Software.
14 | 
15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21 | SOFTWARE.
22 | 


--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
  1 | # Open WebUI PostgreSQL Migration Tool 🚀
  2 | 
  3 | A robust, interactive tool for migrating Open WebUI databases from SQLite to PostgreSQL. Designed for reliability and ease of use.
  4 | 
  5 | ## Preview
  6 | <img width="600" alt="Screenshot 2025-02-20 at 5 25 31 PM" src="https://github.com/user-attachments/assets/d3e9cb13-3aff-455a-9860-8b1d530f5b9d" />
  7 | 
  8 | ## Migration Demo
  9 | https://github.com/user-attachments/assets/5ea8ed51-cc2d-49f0-9f1a-36e2f4e04f30
 10 | 
 11 | ## ✨ Features
 12 | 
 13 | - 🖥️ Interactive command-line interface with clear prompts
 14 | - 🔍 Comprehensive database integrity checking
 15 | - 📦 Configurable batch processing for optimal performance
 16 | - ⚡ Real-time progress visualization
 17 | - 🛡️ Robust error handling and recovery
 18 | - 🔄 Unicode and special character support
 19 | - 🎯 Automatic table structure conversion
 20 | 
 21 | ## 🚀 Quick Start
 22 | 
 23 | 1. **Clone the repository:**
 24 |    ```bash
 25 |    git clone https://github.com/taylorwilsdon/open-webui-postgres-migration.git
 26 |    cd open-webui-postgres-migration
 27 |    ```
 28 | 
 29 | 2. **Set up environment:**
 30 |    ```bash
 31 |    python -m venv venv
 32 |    source venv/bin/activate  # Windows: venv\Scripts\activate
 33 |    pip install -r requirements.txt
 34 |    ```
 35 | 
 36 | 3. **Run the migration:**
 37 |    ```bash
 38 |    python migrate.py
 39 |    ```
 40 | 
 41 | ## 📝 Best Practices
 42 | 
 43 | 1. **Before Migration:**
 44 |    - Backup your SQLite database
 45 |    - **Start Open-WebUI with `DATABASE_URL` configured pointing to the new PostgreSQL instance**
 46 |        - This is very important - you need to let Open-WebUI bootstrap the new DB before running the migration.
 47 |        - DATABASE_URL is formatted as `postgresql://user:password@host:port/dbname`
 48 |    - Verify PostgreSQL server access from host running script
 49 |    - Check available disk space
 50 | 
 51 | 2. **During Migration:**
 52 |    - Don't interrupt the process
 53 |    - Monitor system resources
 54 |    - Keep network connection stable
 55 | 
 56 | 3. **After Migration:**
 57 |    - Verify data integrity
 58 |    - Test application functionality
 59 |    - Keep SQLite backup until verified
 60 | 
 61 | 
 62 | ## 🔧 Configuration Options
 63 | 
 64 | During the migration, you'll be prompted to configure:
 65 | 
 66 | - **SQLite Database**
 67 |   - Path to your existing SQLite database
 68 |   - Automatic validation and integrity checking
 69 | 
 70 | - **PostgreSQL Connection**
 71 |   - Host and port
 72 |   - Database name
 73 |   - Username and password
 74 |   - Connection testing before proceeding
 75 | 
 76 | - **Performance Settings**
 77 |   - Batch size (100-5000 recommended)
 78 |   - Automatic memory usage warnings
 79 | 
 80 | ## ⚙️ System Requirements
 81 | 
 82 | - Python 3.8+
 83 | - PostgreSQL server (running and accessible)
 84 | - Sufficient disk space for both databases
 85 | - Network access to PostgreSQL server
 86 | 
 87 | ## 🛡️ Safety Features
 88 | 
 89 | - ✅ Pre-migration database integrity verification
 90 | - ✅ Transaction-based processing
 91 | - ✅ Automatic error recovery
 92 | - ✅ Failed row tracking and reporting
 93 | - ✅ Progress preservation on interruption
 94 | 
 95 | ## 🚨 Troubleshooting
 96 | 
 97 | Common issues and solutions:
 98 | 
 99 | | Issue | Solution |
100 | |-------|----------|
101 | | Connection Failed | Check PostgreSQL credentials and firewall settings |
102 | | Permission Denied | Verify PostgreSQL user privileges |
103 | | Memory Errors | Reduce batch size in configuration |
104 | | Encoding Issues | Ensure proper database character encoding |
105 | 
106 | 
107 | ## 🤝 Contributing
108 | 
109 | Contributions are welcome! Please feel free to submit a Pull Request.
110 | 
111 | ## 📄 License
112 | 
113 | This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
114 | 
115 | ## 💬 Support
116 | 
117 | If you encounter issues:
118 | 1. Check the troubleshooting section above
119 | 2. Search existing GitHub issues
120 | 3. Create a new issue with:
121 |    - Error messages
122 |    - Database versions
123 |    - System information
124 | 


--------------------------------------------------------------------------------
/migrate.py:
--------------------------------------------------------------------------------
  1 | import psycopg
  2 | import traceback
  3 | import sys
  4 | import sqlite3
  5 | from rich.console import Console
  6 | from rich.progress import Progress, TextColumn, BarColumn, SpinnerColumn
  7 | from rich.panel import Panel
  8 | from rich.table import Table
  9 | from rich.prompt import Prompt, IntPrompt, Confirm
 10 | from pathlib import Path
 11 | from typing import Dict, List, Tuple, Any, Optional
 12 | import asyncio
 13 | from contextlib import asynccontextmanager
 14 | 
 15 | console = Console()
 16 | 
 17 | # Configuration
 18 | MAX_RETRIES = 3
 19 | 
 20 | def get_sqlite_config() -> Path:
 21 |     """Interactive configuration for SQLite database path"""
 22 |     console.print(Panel("SQLite Database Configuration", style="cyan"))
 23 | 
 24 |     default_path = 'webui.db'
 25 |     while True:
 26 |         db_path = Path(Prompt.ask(
 27 |             "[cyan]SQLite database path[/]",
 28 |             default=default_path
 29 |         ))
 30 | 
 31 |         # Check if file exists
 32 |         if not db_path.exists():
 33 |             console.print(f"\n[red]Error: File '{db_path}' does not exist[/]")
 34 |             if not Confirm.ask("\n[yellow]Would you like to try a different path?[/]"):
 35 |                 console.print("[red]Migration cancelled by user[/]")
 36 |                 sys.exit(0)
 37 |             continue
 38 | 
 39 |         # Try to open the database to verify it's a valid SQLite file
 40 |         try:
 41 |             with sqlite3.connect(db_path) as conn:
 42 |                 cursor = conn.cursor()
 43 |                 cursor.execute("SELECT sqlite_version()")
 44 |                 version = cursor.fetchone()[0]
 45 |                 console.print(f"\n[green]✓ Valid SQLite database (version {version})[/]")
 46 |                 return db_path
 47 |         except sqlite3.Error as e:
 48 |             console.print(f"\n[red]Error: Not a valid SQLite database: {str(e)}[/]")
 49 |             if not Confirm.ask("\n[yellow]Would you like to try a different path?[/]"):
 50 |                 console.print("[red]Migration cancelled by user[/]")
 51 |                 sys.exit(0)
 52 | 
 53 | def test_pg_connection(config: Dict[str, Any]) -> Tuple[bool, Optional[str]]:
 54 |     """Test PostgreSQL connection and return (success, error_message)"""
 55 |     try:
 56 |         with psycopg.connect(**config, connect_timeout=5) as conn:
 57 |             with conn.cursor() as cur:
 58 |                 cur.execute("SELECT 1")
 59 |         return True, None
 60 |     except psycopg.OperationalError as e:
 61 |         error_msg = str(e).strip()
 62 |         if "role" in error_msg and "does not exist" in error_msg:
 63 |             return False, f"Authentication failed: The user '{config['user']}' does not exist in PostgreSQL"
 64 |         elif "password" in error_msg:
 65 |             return False, "Authentication failed: Invalid password"
 66 |         elif "connection failed" in error_msg:
 67 |             return False, f"Connection failed: Could not connect to PostgreSQL at {config['host']}:{config['port']}"
 68 |         else:
 69 |             return False, f"Database error: {error_msg}"
 70 |     except Exception as e:
 71 |         return False, f"Unexpected error: {str(e)}"
 72 | 
 73 | def get_pg_config() -> Dict[str, Any]:
 74 |     """Interactive configuration for PostgreSQL connection"""
 75 |     while True:
 76 |         console.print(Panel("PostgreSQL Connection Configuration", style="cyan"))
 77 | 
 78 |         config = {}
 79 | 
 80 |         # Default values
 81 |         defaults = {
 82 |             'host': 'localhost',
 83 |             'port': 5432,
 84 |             'dbname': 'postgres',
 85 |             'user': 'postgres',
 86 |         }
 87 | 
 88 |         config['host'] = Prompt.ask(
 89 |             "[cyan]PostgreSQL host[/]",
 90 |             default=defaults['host']
 91 |         )
 92 | 
 93 |         config['port'] = IntPrompt.ask(
 94 |             "[cyan]PostgreSQL port[/]",
 95 |             default=defaults['port']
 96 |         )
 97 | 
 98 |         config['dbname'] = Prompt.ask(
 99 |             "[cyan]Database name[/]",
100 |             default=defaults['dbname']
101 |         )
102 | 
103 |         config['user'] = Prompt.ask(
104 |             "[cyan]Username[/]",
105 |             default=defaults['user']
106 |         )
107 | 
108 |         config['password'] = Prompt.ask(
109 |             "[cyan]Password[/]",
110 |             password=True
111 |         )
112 | 
113 |         # Show summary
114 |         summary = Table(show_header=False, box=None)
115 |         for key, value in config.items():
116 |             if key != 'password':
117 |                 summary.add_row(f"[cyan]{key}:[/]", str(value))
118 |         summary.add_row("[cyan]password:[/]", "********")
119 | 
120 |         console.print("\nConnection Details:")
121 |         console.print(summary)
122 | 
123 |         # Test connection
124 |         with console.status("[cyan]Testing database connection...[/]"):
125 |             success, error_msg = test_pg_connection(config)
126 | 
127 |         if not success:
128 |             console.print(f"\n[red]Connection Error: {error_msg}[/]")
129 | 
130 |             if not Confirm.ask("\n[yellow]Would you like to try again?[/]"):
131 |                 console.print("[red]Migration cancelled by user[/]")
132 |                 sys.exit(0)
133 | 
134 |             console.print("\n")  # Add spacing before retry
135 |             continue
136 | 
137 |         console.print("\n[green]✓ Database connection successful![/]")
138 | 
139 |         if not Confirm.ask("\n[yellow]Proceed with these settings?[/]"):
140 |             if not Confirm.ask("[yellow]Would you like to try different settings?[/]"):
141 |                 console.print("[red]Migration cancelled by user[/]")
142 |                 sys.exit(0)
143 |             console.print("\n")  # Add spacing before retry
144 |             continue
145 | 
146 |         return config
147 | 
148 | def get_batch_config() -> int:
149 |     """Interactive configuration for batch size"""
150 |     console.print(Panel("Batch Size Configuration", style="cyan"))
151 | 
152 |     console.print("[cyan]The batch size determines how many records are processed at once.[/]")
153 |     console.print("[cyan]A larger batch size may be faster but uses more memory.[/]")
154 |     console.print("[cyan]Recommended range: 100-5000[/]\n")
155 | 
156 |     while True:
157 |         batch_size = IntPrompt.ask(
158 |             "[cyan]Batch size[/]",
159 |             default=500
160 |         )
161 | 
162 |         if batch_size < 1:
163 |             console.print("[red]Batch size must be at least 1[/]")
164 |             continue
165 | 
166 |         if batch_size > 10000:
167 |             if not Confirm.ask("[yellow]Large batch sizes may cause memory issues. Continue anyway?[/]"):
168 |                 continue
169 | 
170 |         return batch_size
171 | 
172 | def check_sqlite_integrity(db_path: Path) -> bool:
173 |     """Run integrity check on SQLite database"""
174 |     console.print(Panel("Running SQLite Database Integrity Check", style="cyan"))
175 | 
176 |     try:
177 |         with sqlite3.connect(db_path) as conn:
178 |             cursor = conn.cursor()
179 | 
180 |             checks = [
181 |                 ("Integrity Check", "PRAGMA integrity_check"),
182 |                 ("Quick Check", "PRAGMA quick_check"),
183 |                 ("Foreign Key Check", "PRAGMA foreign_key_check")
184 |             ]
185 | 
186 |             table = Table(show_header=True)
187 |             table.add_column("Check Type", style="cyan")
188 |             table.add_column("Status", style="green")
189 | 
190 |             for check_name, query in checks:
191 |                 cursor.execute(query)
192 |                 result = cursor.fetchall()
193 |                 status = "✅ Passed" if (result == [('ok',)] or not result) else "❌ Failed"
194 |                 table.add_row(check_name, status)
195 | 
196 |                 if status == "❌ Failed":
197 |                     console.print(f"[red]Failed {check_name}:[/] {result}")
198 |                     return False
199 | 
200 |             try:
201 |                 cursor.execute("SELECT COUNT(*) FROM sqlite_master;")
202 |                 cursor.fetchone()
203 |             except sqlite3.DatabaseError as e:
204 |                 console.print(f"[bold red]Database appears to be corrupted:[/] {e}")
205 |                 return False
206 | 
207 |             console.print(table)
208 |             return True
209 | 
210 |     except Exception as e:
211 |         console.print(f"[bold red]Error during integrity check:[/] {str(e)}")
212 |         return False
213 | 
214 | def sqlite_to_pg_type(sqlite_type: str, column_name: str) -> str:
215 |     # Special handling for known JSON columns in the group table
216 |     json_columns = {'data', 'meta', 'permissions', 'user_ids'}
217 |     if column_name in json_columns:
218 |         return 'JSONB'
219 | 
220 |     types = {
221 |         'INTEGER': 'INTEGER',
222 |         'REAL': 'DOUBLE PRECISION',
223 |         'TEXT': 'TEXT',
224 |         'BLOB': 'BYTEA'
225 |     }
226 |     return types.get(sqlite_type.upper(), 'TEXT')
227 | 
228 | def get_sqlite_safe_identifier(identifier: str) -> str:
229 |     """Quotes identifiers for SQLite queries"""
230 |     return f'"{identifier}"'
231 | 
232 | def get_pg_safe_identifier(identifier: str) -> str:
233 |     """Quotes identifiers for PostgreSQL if they're reserved words"""
234 |     reserved_keywords = {'user', 'group', 'order', 'table', 'select', 'where', 'from', 'index', 'constraint'}
235 |     return f'"{identifier}"' if identifier.lower() in reserved_keywords else identifier
236 | 
237 | @asynccontextmanager
238 | async def async_db_connections(sqlite_path: Path, pg_config: Dict[str, Any]):
239 |     sqlite_conn = None
240 |     pg_conn = None
241 | 
242 |     try:
243 |         # Try SQLite connection first
244 |         try:
245 |             sqlite_conn = sqlite3.connect(sqlite_path, timeout=60)
246 |             sqlite_conn.execute('PRAGMA journal_mode=WAL')
247 |             sqlite_conn.execute('PRAGMA synchronous=NORMAL')
248 |         except sqlite3.Error as e:
249 |             console.print(f"[bold red]Failed to connect to SQLite database:[/] {str(e)}")
250 |             raise
251 | 
252 |         # Try PostgreSQL connection
253 |         try:
254 |             pg_conn = psycopg.connect(**pg_config)
255 |         except psycopg.OperationalError as e:
256 |             console.print(f"[bold red]Failed to connect to PostgreSQL database:[/] {str(e)}")
257 |             if sqlite_conn:
258 |                 sqlite_conn.close()
259 |             raise
260 | 
261 |         yield sqlite_conn, pg_conn
262 | 
263 |     finally:
264 |         if sqlite_conn:
265 |             try:
266 |                 sqlite_conn.close()
267 |             except sqlite3.Error:
268 |                 pass
269 | 
270 |         if pg_conn:
271 |             try:
272 |                 pg_conn.close()
273 |             except psycopg.Error:
274 |                 pass
275 | 
276 | async def process_table(
277 |     table_name: str,
278 |     sqlite_cursor: sqlite3.Cursor,
279 |     pg_cursor: psycopg.Cursor,
280 |     progress: Progress,
281 |     batch_size: int
282 | ) -> None:
283 |     # Special handling for group table
284 |     is_group_table = table_name.lower() == 'group'
285 |     if is_group_table:
286 |         console.print("[cyan]Processing group table - enabling detailed logging[/]")
287 | 
288 |     pg_safe_table_name = get_pg_safe_identifier(table_name)
289 |     sqlite_safe_table_name = get_sqlite_safe_identifier(table_name)
290 | 
291 |     task_id = progress.add_task(
292 |         f"Migrating {table_name}...",
293 |         total=100,
294 |         visible=True
295 |     )
296 | 
297 |     try:
298 |         # Truncate existing table
299 |         try:
300 |             pg_cursor.execute(f"TRUNCATE TABLE {pg_safe_table_name} CASCADE")
301 |             pg_cursor.connection.commit()
302 |         except psycopg.Error as e:
303 |             console.print(f"[yellow]Note: Table {table_name} does not exist yet or could not be truncated: {e}[/]")
304 |             pg_cursor.connection.rollback()
305 | 
306 |         # Get PostgreSQL column types
307 |         try:
308 |             pg_cursor.execute("""
309 |                 SELECT column_name, data_type
310 |                 FROM information_schema.columns
311 |                 WHERE table_name = %s
312 |             """, (table_name,))
313 |             pg_column_types = dict(pg_cursor.fetchall())
314 |             pg_cursor.connection.commit()
315 |         except psycopg.Error:
316 |             pg_cursor.connection.rollback()
317 |             pg_column_types = {}
318 | 
319 |         # Get SQLite schema
320 |         retry_count = 0
321 |         while retry_count < MAX_RETRIES:
322 |             try:
323 |                 sqlite_cursor.execute(f'PRAGMA table_info({sqlite_safe_table_name})')
324 |                 schema = sqlite_cursor.fetchall()
325 |                 break
326 |             except sqlite3.DatabaseError as e:
327 |                 retry_count += 1
328 |                 console.print(f"[yellow]Retry {retry_count}/{MAX_RETRIES} getting schema for {table_name}: {e}[/]")
329 |                 if retry_count == MAX_RETRIES:
330 |                     raise
331 | 
332 |         # Create table if it doesn't exist
333 |         if not pg_column_types:
334 |             try:
335 |                 columns = [f"{get_pg_safe_identifier(col[1])} {sqlite_to_pg_type(col[2], col[1])}"
336 |                           for col in schema]
337 |                 create_query = f"CREATE TABLE IF NOT EXISTS {pg_safe_table_name} ({', '.join(columns)})"
338 |                 console.print(f"[cyan]Creating table with query:[/] {create_query}")
339 |                 pg_cursor.execute(create_query)
340 |                 pg_cursor.connection.commit()
341 |             except psycopg.Error as e:
342 |                 console.print(f"[red]Error creating table {table_name}: {e}[/]")
343 |                 pg_cursor.connection.rollback()
344 |                 raise
345 | 
346 |         # Process rows
347 |         sqlite_cursor.execute(f"SELECT COUNT(*) FROM {sqlite_safe_table_name}")
348 |         total_rows = sqlite_cursor.fetchone()[0]
349 |         processed_rows = 0
350 |         failed_rows = []
351 | 
352 |         while processed_rows < total_rows:
353 |             try:
354 |                 sqlite_cursor.execute(
355 |                     f"SELECT * FROM {sqlite_safe_table_name} LIMIT {batch_size} OFFSET {processed_rows}"
356 |                 )
357 |                 raw_rows = sqlite_cursor.fetchall()
358 | 
359 |                 if not raw_rows:
360 |                     break
361 | 
362 |                 rows = []
363 |                 for raw_row in raw_rows:
364 |                     cleaned_row = []
365 |                     for item in raw_row:
366 |                         if isinstance(item, bytes):
367 |                             try:
368 |                                 cleaned_row.append(item.decode('utf-8', errors='replace'))
369 |                             except:
370 |                                 cleaned_row.append(item.decode('latin1', errors='replace'))
371 |                         elif isinstance(item, str):
372 |                             try:
373 |                                 cleaned_row.append(item.encode('utf-8', errors='replace').decode('utf-8'))
374 |                             except:
375 |                                 cleaned_row.append(item.encode('latin1', errors='replace').decode('latin1'))
376 |                         else:
377 |                             cleaned_row.append(item)
378 |                     rows.append(tuple(cleaned_row))
379 | 
380 |                 for row_index, row in enumerate(rows):
381 |                     try:
382 |                         if is_group_table:
383 |                             console.print(f"[cyan]Processing group row {processed_rows + row_index}[/]")
384 |                         col_names = [get_pg_safe_identifier(col[1]) for col in schema]
385 |                         values = []
386 |                         for i, value in enumerate(row):
387 |                             col_name = schema[i][1]
388 |                             col_type = pg_column_types.get(col_name)
389 | 
390 |                             if value is None:
391 |                                 values.append('NULL')
392 |                             elif col_type == 'boolean':
393 |                                 values.append('true' if value == 1 else 'false')
394 |                             elif isinstance(value, str):
395 |                                 # Check if this is a JSON column
396 |                                 if col_type == 'jsonb':
397 |                                     try:
398 |                                         # Try to parse as JSON to validate
399 |                                         import json
400 |                                         json.loads(value)
401 |                                         values.append(f"'{value}'::jsonb")
402 |                                     except json.JSONDecodeError as e:
403 |                                         console.print(f"[yellow]Warning: Invalid JSON in {col_name}: {e}[/]")
404 |                                         values.append("'{}'::jsonb")
405 |                                 else:
406 |                                     escaped_value = value.replace(chr(39), chr(39)*2)
407 |                                     escaped_value = escaped_value.replace('\x00', '')
408 |                                     values.append(f"'{escaped_value}'")
409 |                             else:
410 |                                 values.append(str(value))
411 | 
412 |                         insert_query = f"""
413 |                             INSERT INTO {pg_safe_table_name}
414 |                             ({', '.join(col_names)})
415 |                             VALUES ({', '.join(values)})
416 |                         """
417 |                         if is_group_table:
418 |                             console.print(f"[cyan]Executing query:[/]\n{insert_query}")
419 |                         pg_cursor.execute(insert_query)
420 |                     except Exception as e:
421 |                         if is_group_table:
422 |                             console.print(f"[red]Error processing group row {processed_rows + row_index}:[/]")
423 |                             console.print(f"[red]Row data:[/] {row}")
424 |                             console.print(f"[red]Error details:[/] {str(e)}")
425 |                         else:
426 |                             console.print(f"[red]Error processing row in {table_name}: {e}[/]")
427 |                         failed_rows.append((table_name, processed_rows + len(failed_rows), str(e)))
428 |                         continue
429 | 
430 |                 processed_rows += len(rows)
431 |                 pg_cursor.connection.commit()
432 |                 progress.update(task_id, completed=(processed_rows / total_rows) * 100)
433 | 
434 |             except sqlite3.DatabaseError as e:
435 |                 console.print(f"[red]SQLite error during batch processing: {e}[/]")
436 |                 console.print("[yellow]Attempting to continue with next batch...[/]")
437 |                 processed_rows += batch_size
438 |                 continue
439 | 
440 |         if failed_rows:
441 |             console.print(f"\n[yellow]Failed rows for {table_name}:[/]")
442 |             for table, row_num, error in failed_rows:
443 |                 console.print(f"Row {row_num}: {error}")
444 | 
445 |         progress.update(task_id, completed=100)
446 |         console.print(f"[green]Completed migrating {processed_rows} rows from {table_name}[/]")
447 |         if failed_rows:
448 |             console.print(f"[yellow]Failed to migrate {len(failed_rows)} rows from {table_name}[/]")
449 | 
450 |     except Exception as e:
451 |         pg_cursor.connection.rollback()
452 |         console.print(f"[bold red]Error processing table {table_name}:[/] {str(e)}")
453 |         raise
454 | 
455 | async def migrate() -> None:
456 |     # Get SQLite database path
457 |     sqlite_path = get_sqlite_config()
458 | 
459 |     if not check_sqlite_integrity(sqlite_path):
460 |         console.print("[bold red]Aborting migration due to database integrity issues[/]")
461 |         sys.exit(1)
462 | 
463 |     # Get PostgreSQL configuration
464 |     pg_config = get_pg_config()
465 | 
466 |     # Get batch size configuration
467 |     batch_size = get_batch_config()
468 | 
469 |     console.print(Panel("Starting Migration Process", style="cyan"))
470 | 
471 |     async with async_db_connections(sqlite_path, pg_config) as (sqlite_conn, pg_conn):
472 |         sqlite_cursor = sqlite_conn.cursor()
473 |         pg_cursor = pg_conn.cursor()
474 | 
475 |         sqlite_cursor.execute("SELECT name FROM sqlite_master WHERE type='table'")
476 |         tables = sqlite_cursor.fetchall()
477 | 
478 |         with Progress(
479 |             SpinnerColumn(),
480 |             TextColumn("[progress.description]{task.description}"),
481 |             BarColumn(),
482 |             TextColumn("[progress.percentage]{task.percentage:>3.0f}%"),
483 |         ) as progress:
484 |             try:
485 |                 for (table_name,) in tables:
486 |                     if table_name in ("migratehistory", "alembic_version"):
487 |                         continue
488 | 
489 |                     await process_table(
490 |                         table_name,
491 |                         sqlite_cursor,
492 |                         pg_cursor,
493 |                         progress,
494 |                         batch_size
495 |                     )
496 | 
497 |                 console.print(Panel("Migration Complete!", style="green"))
498 | 
499 |             except Exception as e:
500 |                 console.print(f"[bold red]Critical error during migration:[/] {e}")
501 |                 console.print("[red]Stack trace:[/]")
502 |                 console.print(traceback.format_exc())
503 |                 pg_conn.rollback()
504 |                 sys.exit(1)
505 | 
506 | if __name__ == "__main__":
507 |     asyncio.run(migrate())
508 | 


--------------------------------------------------------------------------------
/requirements.txt:
--------------------------------------------------------------------------------
1 | markdown-it-py==3.0.0
2 | mdurl==0.1.2
3 | psycopg[binary]==3.2.3
4 | Pygments==2.19.1
5 | rich==13.9.4
6 | typing_extensions==4.12.2
7 | 


--------------------------------------------------------------------------------