, ) if they better represent the original layout. Utilize `colspan` and `rowspan` attributes as necessary to accurately represent merged cells.
14 | - Preserve all formatting elements such as bold, italic, underline, strikethrough text, font sizes, and colors using appropriate HTML tags and inline styles if needed.
15 | - Maintain the hierarchy (h1-h6) and styling of headings and subheadings using appropriate HTML tags or Markdown.
16 | - Visual Elements:
17 | * Images: If there is text within the image, try to recreate the structure within the image. If there is no text, describe the image content and position, and use placeholder `![]() ` tags to represent their location in the document. Capture the image meaning in the alt attribute. Don't specify src if not known.
18 | * Emojis: Use Unicode characters instead of images.
19 | * Charts/Diagrams: For content that cannot be accurately represented in text format, provide a detailed textual description within an HTML element that visually represents its position in the document.
20 | * Complex visuals: Mark with [?] and make a note for ambiguities or uncertain interpretations in the document. Use HTML comments for conversion notes. Only output notes with comment tags.
21 | - Special Characters:
22 | * Letters with ascenders are usually: b, d, f, h, k, l, t
23 | * Letters with descenders are usually: g, j, p, q, y. Lowercase f and z also have descenders in many typefaces.
24 | * Pay special attention to these commonly confused character pairs,
25 | Letter 'l' vs number '1' vs exclamation mark '!'
26 | Number '2' vs letter 'Z'
27 | Number '5' vs letter 'S'
28 | Number '51' vs number '±1'
29 | Number '6' vs letter 'G' vs letter 'b'
30 | Number '0' vs letter 'O'
31 | Number '8' vs letter 'B'
32 | Letter 'f' vs letter 't'
33 | * Contextual clues to differentiate:
34 | - If in a numeric column, interpret 'O' as '0'
35 | - If preceded/followed by numbers, interpret 'l' as '1'
36 | - Consider font characteristics, e.g.
37 | '1' typically has no serif
38 | '2' has a curved bottom vs 'Z's straight line
39 | '5' has more rounded features than 'S'
40 | '6' has a closed loop vs 'G's open curve
41 | '0' is typically more oval than 'O'
42 | '8' has a more angular top than 'B'
43 | {custom_instructions}
44 | - Return only the correct markdown without additional text or explanations.
45 | - DO NOT use code blocks such as "```html" or "```markdown" in the output unless there is a code block in the content.
46 | - Think before generating the output in tags.
47 |
48 | Remember, your primary objective is to create an output that, when rendered, structurally replicates the original document's content as closely as possible without losing any textual details.
49 | Prioritize replicating structure above all else.
50 | Use tables without borders to represent column-like structures.
51 | Keep the font color black (#000000) and the background white (#ffffff).
52 |
53 | OUTPUT FORMAT:
54 | Enclose the response within XML tags as follows:
55 |
56 | [Step-by-step analysis and generation strategy]
57 |
58 |
61 |
62 | Quality Checks:
63 | 1. Verify structural and layout accuracy
64 | 2. Verify content completeness
65 | 3. Visual element handling
66 | 4. Hierarchy preservation
67 | 5. Confirm table alignment and cell merging accuracy
68 | 6. Spacing fidelity
69 | 7. Verify that numbers fall within expected ranges for their column
70 | 8. Flag any suspicious characters that could be OCR errors
71 | 9. Validate markdown syntax
72 | """
73 |
74 | OPENAI_USER_PROMPT = """\
75 | Convert the following document to markdown.
76 | Ensure accurate representation of all content, including tables and visual elements, per your instructions.
77 | """
78 |
79 | INSTRUCTIONS_ADD_PG_BREAK = "Insert a `` tag between the content of each page to maintain the original page structure."
80 |
81 | LLAMA_PARSER_PROMPT = """\
82 | You are a document conversion assistant. Your task is to accurately reproduce the content of an image in Markdown and HTML format, maintaining the visual structure and layout of the original document as closely as possible.
83 |
84 | Instructions:
85 | 1. Use a combination of Markdown and HTML to replicate the document's layout and formatting.
86 | 2. Reproduce all text content exactly as it appears, including preserving capitalization, punctuation, and any apparent errors or inconsistencies in the original.
87 | 3. Use appropriate Markdown syntax for headings, emphasis (bold, italic), and lists where applicable.
88 | 4. Always use HTML (``, ``, ``) to represent tabular data. Include `colspan` and `rowspan` attributes if needed.
89 | 5. For figures, graphs, or diagrams, represent them using `![]() ` tags and use appropriate `alt` text.
90 | 6. For handwritten documents, reproduce the content as typed text, maintaining the original structure and layout.
91 | 7. Do not include any descriptions of the document's appearance, paper type, or writing implements used.
92 | 8. Do not add any explanatory notes, comments, or additional information outside of the converted content.
93 | 9. Ensure all special characters, symbols, and equations are accurately represented.
94 | 10. Provide the output only once, without any duplication.
95 | 11. Enclose the entire output within tags.
96 |
97 | Output the converted content directly in Markdown and HTML without any additional explanations, descriptions, or notes.
98 | """
99 |
--------------------------------------------------------------------------------
/lexoid/core/utils.py:
--------------------------------------------------------------------------------
1 | import asyncio
2 | import io
3 | import mimetypes
4 | import os
5 | import re
6 | import sys
7 | from difflib import SequenceMatcher
8 | from hashlib import md5
9 | from typing import Dict, List, Optional
10 | from urllib.parse import urlparse
11 |
12 | import nest_asyncio
13 | import pikepdf
14 | import pypdfium2
15 | import requests
16 | from bs4 import BeautifulSoup
17 | from docx2pdf import convert
18 | from loguru import logger
19 | from markdown import markdown
20 | from markdownify import markdownify as md
21 | from PIL import Image
22 | from PyQt5.QtCore import QMarginsF, QUrl
23 | from PyQt5.QtGui import QPageLayout, QPageSize
24 | from PyQt5.QtPrintSupport import QPrinter
25 | from PyQt5.QtWebEngineWidgets import QWebEngineView
26 | from PyQt5.QtWidgets import QApplication
27 |
28 | # Source: https://stackoverflow.com/a/12982689
29 | HTML_TAG_PATTERN = re.compile("<.*?>|&([a-z0-9]+|#[0-9]{1,6}|#x[0-9a-f]{1,6});")
30 |
31 |
32 | def split_pdf(input_path: str, output_dir: str, pages_per_split: int):
33 | paths = []
34 | with pikepdf.open(input_path) as pdf:
35 | total_pages = len(pdf.pages)
36 | for start in range(0, total_pages, pages_per_split):
37 | end = min(start + pages_per_split, total_pages)
38 | output_path = os.path.join(
39 | output_dir, f"split_{str(start + 1).zfill(4)}_{end}.pdf"
40 | )
41 | with pikepdf.new() as new_pdf:
42 | new_pdf.pages.extend(pdf.pages[start:end])
43 | new_pdf.save(output_path)
44 | paths.append(output_path)
45 | return paths
46 |
47 |
48 | def create_sub_pdf(
49 | input_path: str, output_path: str, page_nums: Optional[tuple[int, ...] | int] = None
50 | ) -> str:
51 | if isinstance(page_nums, int):
52 | page_nums = (page_nums,)
53 | page_nums = tuple(sorted(set(page_nums)))
54 | with pikepdf.open(input_path) as pdf:
55 | indices = page_nums if page_nums else range(len(pdf.pages))
56 | with pikepdf.new() as new_pdf:
57 | new_pdf.pages.extend([pdf.pages[i - 1] for i in indices])
58 | new_pdf.save(output_path)
59 | return output_path
60 |
61 |
62 | def convert_image_to_pdf(image_path: str) -> bytes:
63 | with Image.open(image_path) as img:
64 | img_rgb = img.convert("RGB")
65 | pdf_buffer = io.BytesIO()
66 | img_rgb.save(pdf_buffer, format="PDF")
67 | return pdf_buffer.getvalue()
68 |
69 |
70 | def remove_html_tags(text: str):
71 | html = markdown(text, extensions=["tables"])
72 | return re.sub(HTML_TAG_PATTERN, "", html)
73 |
74 |
75 | def calculate_similarity(text1: str, text2: str, ignore_html=True) -> float:
76 | """Calculate similarity ratio between two texts using SequenceMatcher."""
77 | if ignore_html:
78 | text1 = remove_html_tags(text1)
79 | text2 = remove_html_tags(text2)
80 | return SequenceMatcher(None, text1, text2).ratio()
81 |
82 |
83 | def convert_pdf_page_to_image(
84 | pdf_document: pypdfium2.PdfDocument, page_number: int
85 | ) -> bytes:
86 | """Convert a PDF page to an image."""
87 | page = pdf_document[page_number]
88 | # Render with 4x scaling for better quality
89 | pil_image = page.render(scale=4).to_pil()
90 |
91 | # Convert to bytes
92 | img_byte_arr = io.BytesIO()
93 | pil_image.save(img_byte_arr, format="PNG")
94 | img_byte_arr.seek(0)
95 | return img_byte_arr.getvalue()
96 |
97 |
98 | def get_file_type(path: str) -> str:
99 | """Get the file type of a file based on its extension."""
100 | return mimetypes.guess_type(path)[0]
101 |
102 |
103 | def is_supported_file_type(path: str) -> bool:
104 | """Check if the file type is supported for parsing."""
105 | file_type = get_file_type(path)
106 | if (
107 | file_type == "application/pdf"
108 | or "wordprocessing" in file_type
109 | or "spreadsheet" in file_type
110 | or "presentation" in file_type
111 | or file_type.startswith("image/")
112 | or file_type.startswith("text")
113 | ):
114 | return True
115 | return False
116 |
117 |
118 | def is_supported_url_file_type(url: str) -> bool:
119 | """
120 | Check if the file type from the URL is supported.
121 |
122 | Args:
123 | url (str): The URL of the file.
124 |
125 | Returns:
126 | bool: True if the file type is supported, False otherwise.
127 | """
128 | supported_extensions = [".pdf", ".png", ".jpg", ".jpeg", ".tiff", ".bmp", ".gif"]
129 | parsed_url = urlparse(url)
130 | ext = os.path.splitext(parsed_url.path)[1].lower()
131 |
132 | if ext in supported_extensions:
133 | return True
134 |
135 | # If no extension in URL, try to get content type from headers
136 | try:
137 | response = requests.head(url)
138 | except requests.exceptions.ConnectionError:
139 | return False
140 | content_type = response.headers.get("Content-Type", "")
141 | ext = mimetypes.guess_extension(content_type)
142 |
143 | return ext in supported_extensions
144 |
145 |
146 | def download_file(url: str, temp_dir: str) -> str:
147 | """
148 | Downloads a file from the given URL and saves it to a temporary directory.
149 |
150 | Args:
151 | url (str): The URL of the file to download.
152 | temp_dir (str): The temporary directory to save the file.
153 |
154 | Returns:
155 | str: The path to the downloaded file.
156 | """
157 | response = requests.get(url)
158 | file_name = os.path.basename(urlparse(url).path)
159 | if not file_name:
160 | content_type = response.headers.get("Content-Type", "")
161 | ext = mimetypes.guess_extension(content_type)
162 | file_name = f"downloaded_file{ext}" if ext else "downloaded_file"
163 |
164 | file_path = os.path.join(temp_dir, file_name)
165 | with open(file_path, "wb") as f:
166 | f.write(response.content)
167 | return file_path
168 |
169 |
170 | def find_dominant_heading_level(markdown_content: str) -> str:
171 | """
172 | Finds the most common heading level that occurs more than once.
173 | Also checks for underline style headings (---).
174 |
175 | Args:
176 | markdown_content (str): The markdown content to analyze
177 |
178 | Returns:
179 | str: The dominant heading pattern (e.g., '##' or 'underline')
180 | """
181 | # Check for underline style headings first
182 | underline_pattern = r"^[^\n]+\n-+$"
183 | underline_matches = re.findall(underline_pattern, markdown_content, re.MULTILINE)
184 | if len(underline_matches) > 1:
185 | return "underline"
186 |
187 | # Find all hash-style headings in the markdown content
188 | heading_patterns = ["#####", "####", "###", "##", "#"]
189 | heading_counts = {}
190 |
191 | for pattern in heading_patterns:
192 | # Look for headings at the start of a line
193 | regex = f"^{pattern} .*$"
194 | matches = re.findall(regex, markdown_content, re.MULTILINE)
195 | if len(matches) > 1: # Only consider headings that appear more than once
196 | heading_counts[pattern] = len(matches)
197 |
198 | if not heading_counts:
199 | return "#" # Default to h1 if no repeated headings found
200 |
201 | return min(heading_counts.keys(), key=len)
202 |
203 |
204 | def split_md_by_headings(markdown_content: str, heading_pattern: str) -> List[Dict]:
205 | """
206 | Splits markdown content by the specified heading pattern and structures it.
207 |
208 | Args:
209 | markdown_content (str): The markdown content to split
210 | heading_pattern (str): The heading pattern to split on (e.g., '##' or 'underline')
211 |
212 | Returns:
213 | List[Dict]: List of dictionaries containing metadata and content
214 | """
215 | structured_content = []
216 |
217 | if heading_pattern == "underline":
218 | # Split by underline headings
219 | pattern = r"^([^\n]+)\n-+$"
220 | sections = re.split(pattern, markdown_content, flags=re.MULTILINE)
221 | # Remove empty sections and strip whitespace
222 | sections = [section.strip() for section in sections]
223 |
224 | # Handle content before first heading if it exists
225 | if sections and not re.match(r"^[^\n]+\n-+$", sections[0], re.MULTILINE):
226 | structured_content.append(
227 | {
228 | "metadata": {"page": "Introduction"},
229 | "content": sections.pop(0),
230 | }
231 | )
232 |
233 | # Process sections pairwise (heading, content)
234 | for i in range(0, len(sections), 2):
235 | if i + 1 < len(sections):
236 | structured_content.append(
237 | {
238 | "metadata": {"page": sections[i]},
239 | "content": sections[i + 1],
240 | }
241 | )
242 | else:
243 | # Split by hash headings
244 | regex = f"^{heading_pattern} .*$"
245 | sections = re.split(regex, markdown_content, flags=re.MULTILINE)
246 | headings = re.findall(regex, markdown_content, flags=re.MULTILINE)
247 |
248 | # Remove empty sections and strip whitespace
249 | sections = [section.strip() for section in sections]
250 |
251 | # Handle content before first heading if it exists
252 | if len(sections) > len(headings):
253 | structured_content.append(
254 | {
255 | "metadata": {"page": "Introduction"},
256 | "content": sections.pop(0),
257 | }
258 | )
259 |
260 | # Process remaining sections
261 | for heading, content in zip(headings, sections):
262 | clean_heading = heading.replace(heading_pattern, "").strip()
263 | structured_content.append(
264 | {
265 | "metadata": {"page": clean_heading},
266 | "content": content,
267 | }
268 | )
269 |
270 | return structured_content
271 |
272 |
273 | def html_to_markdown(html: str, title: str, url: str) -> str:
274 | """
275 | Converts HTML content to markdown.
276 |
277 | Args:
278 | html (str): The HTML content to convert.
279 | title (str): The title of the HTML page
280 | url (str): The URL of the HTML page
281 |
282 | Returns:
283 | Dict: Dictionary containing parsed document data
284 | """
285 | markdown_content = md(html)
286 |
287 | # Find the dominant heading level
288 | heading_pattern = find_dominant_heading_level(markdown_content)
289 |
290 | # Split content by headings and structure it
291 | split_md = split_md_by_headings(markdown_content, heading_pattern)
292 |
293 | content = {
294 | "raw": markdown_content,
295 | "segments": split_md,
296 | "title": title,
297 | "url": url,
298 | "parent_title": "",
299 | "recursive_docs": [],
300 | }
301 |
302 | return content
303 |
304 |
305 | def get_webpage_soup(url: str) -> BeautifulSoup:
306 | try:
307 | from playwright.async_api import async_playwright
308 |
309 | nest_asyncio.apply()
310 |
311 | async def fetch_page():
312 | async with async_playwright() as p:
313 | browser = await p.chromium.launch(
314 | headless=True,
315 | args=[
316 | "--disable-blink-features=AutomationControlled",
317 | "--no-sandbox",
318 | "--window-size=1920,1080",
319 | ],
320 | )
321 | context = await browser.new_context(
322 | viewport={"width": 1920, "height": 1080},
323 | user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
324 | bypass_csp=True,
325 | )
326 | page = await context.new_page()
327 |
328 | # Add headers to appear more like a real browser
329 | await page.set_extra_http_headers(
330 | {
331 | "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
332 | "Accept-Language": "en-US,en;q=0.5",
333 | "Sec-Fetch-Dest": "document",
334 | "Sec-Fetch-Mode": "navigate",
335 | "Sec-Fetch-Site": "none",
336 | "Sec-Fetch-User": "?1",
337 | }
338 | )
339 |
340 | await page.goto(url)
341 |
342 | # Wait for Cloudflare check to complete
343 | await page.wait_for_load_state("networkidle")
344 |
345 | # Additional wait for any dynamic content
346 | try:
347 | await page.wait_for_selector("body", timeout=30000)
348 | except Exception:
349 | pass
350 |
351 | html = await page.content()
352 | await browser.close()
353 | return html
354 |
355 | loop = asyncio.get_event_loop()
356 | html = loop.run_until_complete(fetch_page())
357 | soup = BeautifulSoup(html, "html.parser")
358 | except Exception as e:
359 | logger.debug(
360 | f"Error reading HTML content from URL, attempting with default https request: {str(e)}"
361 | )
362 | response = requests.get(url)
363 | soup = BeautifulSoup(
364 | response.content, "html.parser", from_encoding="iso-8859-1"
365 | )
366 | return soup
367 |
368 |
369 | def read_html_content(url: str) -> Dict:
370 | """
371 | Reads the content of an HTML page from the given URL and converts it to markdown or structured content.
372 |
373 | Args:
374 | url (str): The URL of the HTML page.
375 |
376 | Returns:
377 | Dict: Dictionary containing parsed document data
378 | """
379 |
380 | soup = get_webpage_soup(url)
381 | title = soup.title.string.strip() if soup.title else "No title"
382 | url_hash = md5(url.encode("utf-8")).hexdigest()[:8]
383 | full_title = f"{title} - {url_hash}"
384 | return html_to_markdown(str(soup), title=full_title, url=url)
385 |
386 |
387 | def extract_urls_from_markdown(content: str) -> List[str]:
388 | """
389 | Extracts URLs from markdown content using regex.
390 | Matches both [text](url) and bare http(s):// URLs.
391 |
392 | Args:
393 | content (str): Markdown content to search for URLs
394 |
395 | Returns:
396 | List[str]: List of unique URLs found
397 | """
398 | # Match markdown links [text](url) and bare URLs
399 | markdown_pattern = r"\[([^\]]+)\]\((https?://[^\s\)]+)\)"
400 | bare_url_pattern = r"(? Dict:
412 | """
413 | Recursively reads HTML content from URLs up to specified depth.
414 |
415 | Args:
416 | url (str): The URL to parse
417 | depth (int): How many levels deep to recursively parse
418 | visited_urls (set): Set of already visited URLs to prevent cycles
419 |
420 | Returns:
421 | Dict: Dictionary containing parsed document data
422 | """
423 | if visited_urls is None:
424 | visited_urls = set()
425 |
426 | if url in visited_urls:
427 | return {
428 | "raw": "",
429 | "segments": [],
430 | "title": "",
431 | "url": url,
432 | "parent_title": "",
433 | "recursive_docs": [],
434 | }
435 |
436 | visited_urls.add(url)
437 |
438 | try:
439 | content = read_html_content(url)
440 | except Exception as e:
441 | print(f"Error processing URL {url}: {str(e)}")
442 | return {
443 | "raw": "",
444 | "segments": [],
445 | "title": "",
446 | "url": url,
447 | "parent_title": "",
448 | "recursive_docs": [],
449 | }
450 |
451 | if depth <= 1:
452 | return content
453 |
454 | # Extract URLs from all content sections
455 | urls = extract_urls_from_markdown(content["raw"])
456 |
457 | # Recursively process each URL
458 | recursive_docs = []
459 | for sub_url in urls:
460 | if sub_url not in visited_urls:
461 | sub_content = recursive_read_html(sub_url, depth - 1, visited_urls)
462 | recursive_docs.append(sub_content)
463 |
464 | content["recursive_docs"] = recursive_docs
465 | return content
466 |
467 |
468 | def save_webpage_as_pdf(url: str, output_path: str) -> str:
469 | """
470 | Saves a webpage as a PDF file using PyQt5.
471 |
472 | Args:
473 | url (str): The URL of the webpage.
474 | output_path (str): The path to save the PDF file.
475 |
476 | Returns:
477 | str: The path to the saved PDF file.
478 | """
479 | if not QApplication.instance():
480 | app = QApplication(sys.argv)
481 | else:
482 | app = QApplication.instance()
483 | web = QWebEngineView()
484 | web.load(QUrl(url))
485 |
486 | def handle_print_finished(filename, status):
487 | print(f"PDF saved to: {filename}")
488 | app.quit()
489 |
490 | def handle_load_finished(status):
491 | if status:
492 | printer = QPrinter(QPrinter.HighResolution)
493 | printer.setOutputFormat(QPrinter.PdfFormat)
494 | printer.setOutputFileName(output_path)
495 |
496 | page_layout = QPageLayout(
497 | QPageSize(QPageSize.A4), QPageLayout.Portrait, QMarginsF(15, 15, 15, 15)
498 | )
499 | printer.setPageLayout(page_layout)
500 |
501 | web.page().printToPdf(output_path)
502 | web.page().pdfPrintingFinished.connect(handle_print_finished)
503 |
504 | web.loadFinished.connect(handle_load_finished)
505 | app.exec_()
506 |
507 | return output_path
508 |
509 |
510 | def convert_to_pdf(input_path: str, output_path: str) -> str:
511 | """
512 | Converts a file or webpage to PDF.
513 |
514 | Args:
515 | input_path (str): The path to the input file or URL.
516 | output_path (str): The path to save the output PDF file.
517 |
518 | Returns:
519 | str: The path to the saved PDF file.
520 | """
521 | if input_path.startswith(("http://", "https://")):
522 | return save_webpage_as_pdf(input_path, output_path)
523 | file_type = get_file_type(input_path)
524 | if file_type.startswith("image/"):
525 | img_data = convert_image_to_pdf(input_path)
526 | with open(output_path, "wb") as f:
527 | f.write(img_data)
528 | elif "word" in file_type:
529 | return convert_doc_to_pdf(input_path, os.path.dirname(output_path))
530 | else:
531 | # Assume it's already a PDF, just copy it
532 | with open(input_path, "rb") as src, open(output_path, "wb") as dst:
533 | dst.write(src.read())
534 |
535 | return output_path
536 |
537 |
538 | def has_image_in_pdf(path: str):
539 | with open(path, "rb") as fp:
540 | content = fp.read()
541 | return "Image".lower() in list(
542 | map(lambda x: x.strip(), (str(content).lower().split("/")))
543 | )
544 |
545 |
546 | def has_hyperlink_in_pdf(path: str):
547 | with open(path, "rb") as fp:
548 | content = fp.read()
549 | # URI tag is used if Links are hidden.
550 | return "URI".lower() in list(
551 | map(lambda x: x.strip(), (str(content).lower().split("/")))
552 | )
553 |
554 |
555 | def router(path: str, priority: str = "speed") -> str:
556 | """
557 | Routes the file path to the appropriate parser based on the file type.
558 |
559 | Args:
560 | path (str): The file path to route.
561 | priority (str): The priority for routing: "accuracy" (preference to LLM_PARSE) or "speed" (preference to STATIC_PARSE).
562 | """
563 | file_type = get_file_type(path)
564 | if (
565 | file_type.startswith("text/")
566 | or "spreadsheet" in file_type
567 | or "presentation" in file_type
568 | ):
569 | return "STATIC_PARSE"
570 |
571 | if priority == "accuracy":
572 | # If the file is a PDF without images but has hyperlinks, use STATIC_PARSE
573 | # Otherwise, use LLM_PARSE
574 | has_image = has_image_in_pdf(path)
575 | has_hyperlink = has_hyperlink_in_pdf(path)
576 | if file_type == "application/pdf" and not has_image and has_hyperlink:
577 | logger.debug("Using STATIC_PARSE for PDF with hyperlinks and no images.")
578 | return "STATIC_PARSE"
579 | logger.debug(
580 | f"Using LLM_PARSE because PDF has image ({has_image}) or has no hyperlink ({has_hyperlink})."
581 | )
582 | return "LLM_PARSE"
583 | else:
584 | # If the file is a PDF without images, use STATIC_PARSE
585 | # Otherwise, use LLM_PARSE
586 | if file_type == "application/pdf" and not has_image_in_pdf(path):
587 | logger.debug("Using STATIC_PARSE for PDF without images.")
588 | return "STATIC_PARSE"
589 | logger.debug("Using LLM_PARSE because PDF has images")
590 | return "LLM_PARSE"
591 |
592 |
593 | def convert_doc_to_pdf(input_path: str, temp_dir: str) -> str:
594 | temp_path = os.path.join(
595 | temp_dir, os.path.splitext(os.path.basename(input_path))[0] + ".pdf"
596 | )
597 |
598 | # Convert the document to PDF
599 | # docx2pdf is not supported in linux. Use LibreOffice in linux instead.
600 | # May need to install LibreOffice if not already installed.
601 | if "linux" in sys.platform.lower():
602 | os.system(
603 | f'lowriter --headless --convert-to pdf --outdir {temp_dir} "{input_path}"'
604 | )
605 | else:
606 | convert(input_path, temp_path)
607 |
608 | # Return the path of the converted PDF
609 | return temp_path
610 |
611 |
612 | def get_uri_rect(path):
613 | with open(path, "rb") as fp:
614 | byte_str = str(fp.read())
615 | pattern = r"\((https?://[^\s)]+)\)"
616 | uris = re.findall(pattern, byte_str)
617 | rect_splits = byte_str.split("/Rect [")[1:]
618 | rects = [
619 | list(map(float, rect_split.split("]")[0].split())) for rect_split in rect_splits
620 | ]
621 | return {uri: rect for uri, rect in zip(uris, rects)}
622 |
--------------------------------------------------------------------------------
/pyproject.toml:
--------------------------------------------------------------------------------
1 | [tool.poetry]
2 | name = "lexoid"
3 | version = "0.1.14"
4 | description = ""
5 | authors = []
6 | readme = "README.md"
7 |
8 | [tool.poetry.dependencies]
9 | python = "^3.10"
10 | google-generativeai = "^0.8.1"
11 | openai = "^1.47.0"
12 | pikepdf = "^9.3.0"
13 | pdfplumber = "^0.11.4"
14 | pandas = "^2.2.3"
15 | tabulate = "^0.9.0"
16 | bs4 = "^0.0.2"
17 | markdownify = "^0.13.1"
18 | opencv-python = "^4.10.0.84"
19 | pypdfium2 = "^4.30.0"
20 | markdown = "^3.7"
21 | python-dotenv = "^1.0.0"
22 | loguru = "^0.7.2"
23 | playwright = "^1.49.0"
24 | docx2pdf = "^0.1.8"
25 | python-docx = "^1.1.2"
26 | nest-asyncio ="^1.6.0"
27 | pyqt5 = {version = "^5.15.11", markers = "platform_system != 'debian'"}
28 | pyqtwebengine = {version = "^5.15.7", markers = "platform_system != 'debian'"}
29 | huggingface-hub = "^0.27.0"
30 | together = "^1.4.0"
31 | openpyxl = "^3.1.5"
32 | pptx2md = "^2.0.6"
33 |
34 | [tool.poetry.group.dev.dependencies]
35 | ipykernel = "^6.29.5"
36 | pytest-asyncio = "^0.23.8"
37 | pytest = "^8.3.2"
38 |
39 |
40 | [tool.poetry.group.docs.dependencies]
41 | sphinx = "^8.1.3"
42 | pydata-sphinx-theme = "^0.16.1"
43 | docutils = "^0.21.2"
44 |
45 | [build-system]
46 | requires = ["poetry-core", "wheel"]
47 | build-backend = "poetry.core.masonry.api"
48 |
--------------------------------------------------------------------------------
/tests/api_cost_mapping.json:
--------------------------------------------------------------------------------
1 | {
2 | "gemini-2.5-flash-preview-04-17": {
3 | "input": 0.15,
4 | "output": 0.6
5 | },
6 | "gemini-2.5-pro-preview-03-25": {
7 | "input": 1.25,
8 | "output": 10
9 | },
10 | "gemini-2.0-flash": {
11 | "input": 0.1,
12 | "output": 0.4
13 | },
14 | "gemini-2.0-pro-exp": {},
15 | "gemini-2.0-flash-thinking-exp": {},
16 | "gemini-2.0-flash-001": {
17 | "input": 0.1,
18 | "output": 0.4
19 | },
20 | "gemini-1.5-flash-8b": {
21 | "input": 0.0375,
22 | "output": 0.15
23 | },
24 | "gemini-1.5-flash": {
25 | "input": 0.075,
26 | "output": 0.3
27 | },
28 | "gemini-1.5-pro": {
29 | "input": 1.25,
30 | "output": 5
31 | },
32 | "gpt-4o": {
33 | "input": 2.5,
34 | "output": 10
35 | },
36 | "gpt-4o-mini": {
37 | "input": 0.15,
38 | "output": 0.6
39 | },
40 | "meta-llama/Llama-3.2-11B-Vision-Instruct": {
41 | "input": 0,
42 | "output": 0
43 | },
44 | "meta-llama/Llama-3.2-11B-Vision-Instruct-Turbo": {
45 | "input": 0.18,
46 | "output": 0.18
47 | },
48 | "meta-llama/Llama-3.2-90B-Vision-Instruct-Turbo": {
49 | "input": 1.2,
50 | "output": 1.2
51 | },
52 | "meta-llama/Llama-Vision-Free": {
53 | "input": 0,
54 | "output": 0
55 | },
56 | "google/gemma-3-27b-it": {
57 | "input": 0.1,
58 | "input-image": 0.0000256,
59 | "output": 0.2
60 | },
61 | "qwen/qwen-2.5-vl-7b-instruct": {
62 | "input": 0.2,
63 | "input-image": 0.0001445,
64 | "output": 0.2
65 | },
66 | "microsoft/phi-4-multimodal-instruct": {
67 | "input": 0.05,
68 | "input-image": 0.0001769,
69 | "output": 0.1
70 | },
71 | "accounts/fireworks/models/llama4-maverick-instruct-basic": {
72 | "input": 0.22,
73 | "output": 0.88
74 | },
75 | "accounts/fireworks/models/llama4-scout-instruct-basic": {
76 | "input": 0.15,
77 | "output": 0.6
78 | }
79 | }
--------------------------------------------------------------------------------
/tests/benchmark.py:
--------------------------------------------------------------------------------
1 | import os
2 | import time
3 | from dataclasses import dataclass
4 | from glob import glob
5 | from pathlib import Path
6 | from statistics import mean, stdev
7 | from typing import Dict, List, Optional, Tuple
8 |
9 | import pandas as pd
10 | from dotenv import load_dotenv
11 |
12 | from lexoid.api import parse
13 | from lexoid.core.utils import calculate_similarity
14 |
15 | load_dotenv()
16 |
17 |
18 | @dataclass
19 | class BenchmarkResult:
20 | config: Dict
21 | similarity: List[float] # Store all similarity scores for iterations
22 | execution_time: List[float] # Store all execution times for iterations
23 | cost: Optional[List[float]] = None
24 | error: Optional[str] = None
25 |
26 |
27 | def get_input_output_pairs(input_path: str, output_dir: str) -> List[Tuple[str, str]]:
28 | """Get matching input and ground truth file pairs."""
29 | if os.path.isfile(input_path):
30 | # Single file mode
31 | base_name = Path(input_path).stem
32 | ground_truth_path = os.path.join(output_dir, f"{base_name}.md")
33 | if os.path.exists(ground_truth_path):
34 | return [(input_path, ground_truth_path)]
35 | return []
36 |
37 | # Directory mode
38 | input_files = sorted(glob(os.path.join(input_path, "*")))
39 | pairs = []
40 |
41 | for input_file in input_files:
42 | base_name = Path(input_file).stem
43 | ground_truth_path = os.path.join(output_dir, f"{base_name}.md")
44 |
45 | if os.path.exists(ground_truth_path):
46 | pairs.append((input_file, ground_truth_path))
47 |
48 | return pairs
49 |
50 |
51 | def run_benchmark_config(
52 | input_path: str,
53 | ground_truth: str,
54 | config: Dict,
55 | output_save_dir: str = None,
56 | iterations: int = 1,
57 | ) -> BenchmarkResult:
58 | """Run a single benchmark configuration for a specified number of iterations."""
59 | similarities = []
60 | execution_times = []
61 | costs = []
62 | error = None
63 |
64 | for _ in range(iterations):
65 | try:
66 | start_time = time.time()
67 | config["parser_type"] = config.get(
68 | "parser_type",
69 | (
70 | "LLM_PARSE"
71 | if "model" in config
72 | else ("STATIC_PARSE" if "framework" in config else "AUTO")
73 | ),
74 | )
75 | result = parse(
76 | input_path,
77 | pages_per_split=1,
78 | api_cost_mapping="tests/api_cost_mapping.json",
79 | **config,
80 | )
81 | execution_time = time.time() - start_time
82 |
83 | if output_save_dir:
84 | filename = (
85 | f"{Path(input_path).stem}_"
86 | + ", ".join(
87 | [
88 | f"{key}={str(value).replace('/', '_')}"
89 | for key, value in config.items()
90 | ]
91 | )
92 | + f"{int(start_time)}.md"
93 | )
94 | with open(os.path.join(output_save_dir, filename), "w") as fp:
95 | fp.write(result["raw"])
96 |
97 | similarity = calculate_similarity(result["raw"], ground_truth)
98 | similarities.append(similarity)
99 | execution_times.append(execution_time)
100 | costs.append(
101 | result["token_cost"]["output"] if "token_cost" in result else 0.0
102 | )
103 | except Exception as e:
104 | print(f"Error running benchmark for config: {config}\n{e}")
105 | error = str(e)
106 | break # Stop further iterations if an error occurs
107 |
108 | return BenchmarkResult(
109 | config=config,
110 | similarity=similarities,
111 | execution_time=execution_times,
112 | cost=costs,
113 | error=error,
114 | )
115 |
116 |
117 | def aggregate_results(results: List[BenchmarkResult]) -> BenchmarkResult:
118 | """Aggregate multiple benchmark results into a single result."""
119 | if not results:
120 | return None
121 |
122 | valid_results = [r for r in results if r.error is None]
123 | if valid_results:
124 | all_similarities = [s for r in valid_results for s in r.similarity]
125 | all_execution_times = [t for r in valid_results for t in r.execution_time]
126 | all_costs = [c for r in valid_results for c in r.cost]
127 | avg_similarity = mean(all_similarities)
128 | std_similarity = stdev(all_similarities) if len(all_similarities) > 1 else 0.0
129 | avg_execution_time = mean(all_execution_times)
130 | avg_cost = mean(all_costs)
131 | error = (
132 | None
133 | if len(valid_results) == len(results)
134 | else f"Failed: {len(results) - len(valid_results)}/{len(results)}"
135 | )
136 | else:
137 | avg_similarity = 0.0
138 | std_similarity = 0.0
139 | avg_execution_time = 0.0
140 | avg_cost = 0.0
141 | error = f"Failed: {len(results)}/{len(results)}"
142 |
143 | return BenchmarkResult(
144 | config=results[0].config,
145 | similarity=[avg_similarity, std_similarity], # Store mean and std dev
146 | execution_time=[avg_execution_time],
147 | cost=[avg_cost],
148 | error=error,
149 | )
150 |
151 |
152 | def generate_test_configs(input_path: str, test_attributes: List[str]) -> List[Dict]:
153 | """
154 | Generate different configuration combinations to test based on specified attributes.
155 | """
156 | config_options = {
157 | "parser_type": ["LLM_PARSE", "STATIC_PARSE", "AUTO"],
158 | "model": [
159 | # # Google models
160 | "gemini-2.5-flash-preview-04-17",
161 | # "gemini-2.5-pro-preview-03-25",
162 | # "gemini-2.0-pro-exp",
163 | "gemini-2.0-flash",
164 | # "gemini-2.0-flash-thinking-exp",
165 | # "gemini-2.0-flash-001",
166 | # "gemini-1.5-flash-8b",
167 | # "gemini-1.5-flash",
168 | # "gemini-1.5-pro",
169 | # # OpenAI models
170 | "gpt-4o",
171 | "gpt-4o-mini",
172 | # # Meta-LLAMA models through HF Hub
173 | # "meta-llama/Llama-3.2-11B-Vision-Instruct",
174 | # # Meta-LLAMA models through Together AI
175 | # "meta-llama/Llama-3.2-11B-Vision-Instruct-Turbo",
176 | "meta-llama/Llama-3.2-90B-Vision-Instruct-Turbo",
177 | # "meta-llama/Llama-Vision-Free",
178 | # # Model through OpenRouter
179 | "google/gemma-3-27b-it",
180 | "qwen/qwen-2.5-vl-7b-instruct",
181 | # "microsoft/phi-4-multimodal-instruct",
182 | # # Model through fireworks
183 | "accounts/fireworks/models/llama4-maverick-instruct-basic",
184 | # "accounts/fireworks/models/llama4-scout-instruct-basic",
185 | ],
186 | "framework": ["pdfminer", "pdfplumber"],
187 | "pages_per_split": [1, 2, 4, 8],
188 | "max_threads": [1, 2, 4, 8],
189 | "as_pdf": [True, False],
190 | "temperature": [0.2, 0.7],
191 | }
192 |
193 | # Only test as_pdf if input is not a PDF
194 | is_pdf = input_path.lower().endswith(".pdf")
195 | if is_pdf and "as_pdf" in test_attributes:
196 | test_attributes.remove("as_pdf")
197 |
198 | configs = [{}]
199 |
200 | for attr in test_attributes:
201 | new_configs = []
202 | for config in configs:
203 | if attr == "parser_type" or attr == "temperature":
204 | for value in config_options[attr]:
205 | new_config = config.copy()
206 | new_config[attr] = value
207 | new_configs.append(new_config)
208 | elif attr == "model" and (
209 | "parser_type" not in config or config.get("parser_type") == "LLM_PARSE"
210 | ):
211 | for value in config_options[attr]:
212 | new_config = config.copy()
213 | new_config[attr] = value
214 | new_configs.append(new_config)
215 | elif attr == "framework" and (
216 | "parser_type" not in config
217 | or config.get("parser_type") == "STATIC_PARSE"
218 | ):
219 | for value in config_options[attr]:
220 | new_config = config.copy()
221 | new_config[attr] = value
222 | new_configs.append(new_config)
223 | elif attr in ("pages_per_split", "max_threads"):
224 | for value in config_options[attr]:
225 | new_config = config.copy()
226 | new_config[attr] = value
227 | new_configs.append(new_config)
228 | elif attr == "as_pdf" and not is_pdf:
229 | for value in config_options[attr]:
230 | new_config = config.copy()
231 | new_config[attr] = value
232 | new_configs.append(new_config)
233 | else:
234 | new_configs.append(config)
235 | configs = new_configs
236 |
237 | return configs
238 |
239 |
240 | def format_results(results: List[BenchmarkResult], test_attributes: List[str]) -> str:
241 | """Format benchmark results as a markdown table, including only tested attributes."""
242 | sorted_results = sorted(results, key=lambda x: x.similarity[0], reverse=True)
243 |
244 | # Dynamically generate table headers based on test_attributes
245 | headers = ["Rank"]
246 | for attr in test_attributes:
247 | headers.append(attr.replace("_", " ").title())
248 | headers.extend(["Mean Similarity", "Std. Dev.", "Time (s)", "Cost ($)", "Error"])
249 |
250 | md_lines = [
251 | "# Parser Benchmark Results\n",
252 | "| " + " | ".join(headers) + " |",
253 | "|" + "|".join(["---"] * len(headers)) + "|",
254 | ]
255 |
256 | for i, result in enumerate(sorted_results, 1):
257 | config = result.config
258 | error_msg = result.error if result.error else "-"
259 |
260 | row = [str(i)]
261 | for attr in test_attributes:
262 | row.append(str(config.get(attr, "-")))
263 | row.extend(
264 | [
265 | f"{result.similarity[0]:.3f}",
266 | f"{result.similarity[1]:.3f}",
267 | f"{result.execution_time[0]:.2f}",
268 | f"{result.cost[0]}",
269 | error_msg,
270 | ]
271 | )
272 | md_lines.append("| " + " | ".join(row) + " |")
273 |
274 | return "\n".join(md_lines)
275 |
276 |
277 | def run_benchmarks(
278 | input_path: str,
279 | output_dir: str,
280 | test_attributes: List[str],
281 | benchmark_output_dir: str,
282 | iterations: int = 3,
283 | ) -> List[BenchmarkResult]:
284 | """Run all benchmarks for given input(s) and return results."""
285 | # Get input/output file pairs
286 | file_pairs = get_input_output_pairs(input_path, output_dir)
287 | if not file_pairs:
288 | print("No matching input/output file pairs found!")
289 | return []
290 |
291 | # Generate test configurations based on first input file
292 | configs = generate_test_configs(file_pairs[0][0], test_attributes)
293 |
294 | # Run benchmarks
295 | results = []
296 | total_configs = len(configs)
297 | total_files = len(file_pairs)
298 |
299 | print(
300 | f"Running {total_configs} configurations across {total_files} file(s) for {iterations} iterations..."
301 | )
302 |
303 | all_results = []
304 | for i, config in enumerate(configs, 1):
305 | print(f"Progress: {i}/{total_configs} - Testing config: {config}")
306 |
307 | # Run benchmark for each file
308 | file_results = []
309 | for input_file, ground_truth_path in file_pairs:
310 | print(f"Running benchmark for file: {input_file}")
311 | with open(ground_truth_path, "r", encoding="utf-8") as f:
312 | ground_truth = f.read()
313 | result = run_benchmark_config(
314 | input_file, ground_truth, config, benchmark_output_dir, iterations
315 | )
316 | file_results.append(result)
317 | all_results.append((input_file, result))
318 |
319 | result = aggregate_results(file_results)
320 |
321 | results.append(result)
322 |
323 | # Format and save results
324 | save_format = "csv"
325 | if save_format == "markdown":
326 | markdown_report = format_results(results, test_attributes)
327 | result_path = os.path.join(benchmark_output_dir, "results.md")
328 | with open(result_path, "w", encoding="utf-8") as f:
329 | f.write(markdown_report)
330 | elif save_format == "csv":
331 | df = pd.DataFrame(
332 | [
333 | {
334 | "Model": result.config.get("model", "-"),
335 | "Mean Similarity": result.similarity[0],
336 | "Std. Dev.": result.similarity[1],
337 | "Time (s)": result.execution_time[0],
338 | "Cost($)": result.cost[0],
339 | }
340 | for result in results
341 | ]
342 | )
343 | result_path = os.path.join(benchmark_output_dir, "results.csv")
344 | df.to_csv(result_path, index=False)
345 |
346 | print(f"\nBenchmark complete! Results saved to {result_path}")
347 |
348 | # Save document-wise results to CSV
349 | doc_results = []
350 | for input_file, result in all_results:
351 | doc_result = {
352 | "Input File": os.path.basename(input_file),
353 | "Mean Similarity": result.similarity[0],
354 | "Time (s)": result.execution_time[0],
355 | "Cost($)": result.cost[0],
356 | }
357 | for key, value in result.config.items():
358 | doc_result[key] = value
359 | doc_results.append(doc_result)
360 | doc_df = pd.DataFrame(doc_results)
361 | doc_result_path = os.path.join(benchmark_output_dir, "document_results.csv")
362 | doc_df.to_csv(doc_result_path, index=False)
363 | print(f"Document-wise results saved to {doc_result_path}")
364 |
365 | return results
366 |
367 |
368 | def main():
369 | # Can be either a single file or directory
370 | input_path = "examples/inputs"
371 | output_dir = "examples/outputs"
372 |
373 | benchmark_output_dir = f"tests/outputs/benchmark_{int(time.time())}/"
374 | os.makedirs(benchmark_output_dir, exist_ok=True)
375 |
376 | # Specify which attributes to test
377 | test_attributes = [
378 | # "parser_type",
379 | "model",
380 | # "framework",
381 | # "pages_per_split",
382 | # "max_threads",
383 | # "as_pdf",
384 | # "temperature",
385 | ]
386 |
387 | # Number of iterations for each benchmark
388 | iterations = 5
389 |
390 | results = run_benchmarks(
391 | input_path, output_dir, test_attributes, benchmark_output_dir, iterations
392 | )
393 |
394 | # Print top 3 configurations
395 | top_results = sorted(results, key=lambda x: x.similarity[0], reverse=True)[:3]
396 | print("\nTop 3 Configurations:")
397 | for i, result in enumerate(top_results, 1):
398 | print(
399 | f"{i}. Similarity: {result.similarity[0]:.3f} (±{result.similarity[1]:.3f}), Time: {result.execution_time[0]:.2f}s"
400 | )
401 | print(f" Config: {result.config}")
402 |
403 |
404 | if __name__ == "__main__":
405 | main()
406 |
--------------------------------------------------------------------------------
/tests/env_template:
--------------------------------------------------------------------------------
1 | GOOGLE_API_KEY = ""
2 | OPENAI_API_KEY = ""
3 | HUGGINGFACEHUB_API_TOKEN = ""
4 | TOGETHER_API_KEY = ""
--------------------------------------------------------------------------------
/tests/test_parser.py:
--------------------------------------------------------------------------------
1 | # python3 -m pytest tests/test_parser.py -v
2 | # With logs: python3 -m pytest tests/test_parser.py -v -s
3 |
4 | import os
5 |
6 | import pytest
7 | from dotenv import load_dotenv
8 | from loguru import logger
9 |
10 | from lexoid.api import parse
11 | from lexoid.core.utils import calculate_similarity
12 |
13 | load_dotenv()
14 | output_dir = "tests/outputs"
15 | os.makedirs(output_dir, exist_ok=True)
16 | models = [
17 | # Google models
18 | "gemini-2.0-pro-exp",
19 | "gemini-2.0-flash",
20 | "gemini-1.5-flash",
21 | "gemini-1.5-flash-8b",
22 | "gemini-1.5-pro",
23 | # OpenAI models
24 | "gpt-4o",
25 | "gpt-4o-mini",
26 | # Meta-LLAMA models through HF Hub
27 | "meta-llama/Llama-3.2-11B-Vision-Instruct",
28 | # Meta-LLAMA models through Together AI
29 | "meta-llama/Llama-3.2-11B-Vision-Instruct-Turbo",
30 | "meta-llama/Llama-3.2-90B-Vision-Instruct-Turbo",
31 | "meta-llama/Llama-Vision-Free",
32 | ]
33 |
34 |
35 | @pytest.mark.asyncio
36 | @pytest.mark.parametrize("model", models)
37 | async def test_llm_parse(model):
38 | input_data = "examples/inputs/test_1.pdf"
39 | expected_ouput_path = "examples/outputs/test_1.md"
40 | config = {"parser_type": "LLM_PARSE", "model": model, "verbose": True}
41 | result = parse(input_data, **config)["raw"]
42 | assert isinstance(result, str)
43 |
44 | # Compare the result with the expected output
45 | expected_ouput = open(expected_ouput_path, "r").read()
46 | # save the result to a file
47 | with open(f"{output_dir}/input_table_{model.replace('/', '_')}.md", "w") as f:
48 | f.write(result)
49 | score = calculate_similarity(result, expected_ouput)
50 | assert round(score, 3) > 0.75
51 |
52 |
53 | @pytest.mark.asyncio
54 | @pytest.mark.parametrize("model", models)
55 | async def test_jpg_parse(model):
56 | input_data = "examples/inputs/test_4.jpg"
57 | expected_ouput_path = "examples/outputs/test_4.md"
58 | config = {"parser_type": "LLM_PARSE", "model": model}
59 | result = parse(input_data, **config)["raw"]
60 | assert isinstance(result, str)
61 |
62 | # Compare the result with the expected output
63 | expected_ouput = open(expected_ouput_path, "r").read()
64 | # save the result to a file
65 | m_name = model.replace("/", "_")
66 | with open(f"{output_dir}/input_image_{m_name}.md", "w") as f:
67 | f.write(result)
68 | score = calculate_similarity(result, expected_ouput)
69 | assert round(score, 3) > 0.8
70 |
71 |
72 | @pytest.mark.asyncio
73 | @pytest.mark.parametrize(
74 | "sample",
75 | [
76 | "examples/inputs/test_explicit_hyperlink_n_img.pdf",
77 | "examples/inputs/test_hidden_link_with_image.pdf", # currently fails
78 | "examples/inputs/test_with_hidden_links_no_img.pdf",
79 | ],
80 | )
81 | async def test_url_detection_auto_routing(sample):
82 | patterns = ["http", "https", "www"]
83 | model_type = "gemini-1.5-pro"
84 | config = {"parser_type": "AUTO", "model": model_type, "verbose": True}
85 | result = parse(sample, **config)["raw"]
86 | assert isinstance(result, str)
87 | found = [True if p in result else False for p in patterns]
88 | assert any(found)
89 |
90 |
91 | @pytest.mark.asyncio
92 | @pytest.mark.parametrize(
93 | "sample",
94 | [
95 | "examples/inputs/test_explicit_hyperlink_n_img.pdf",
96 | "examples/inputs/test_hidden_link_with_image.pdf",
97 | "examples/inputs/test_with_hidden_links_no_img.pdf",
98 | ],
99 | )
100 | async def test_url_detection_pdfplumber(sample):
101 | patterns = ["http", "https", "www"]
102 | framework = "pdfplumber"
103 | config = {"parser_type": "STATIC_PARSE", "framework": framework}
104 | result = parse(sample, **config)["raw"]
105 | assert isinstance(result, str)
106 | found = [True if p in result else False for p in patterns]
107 | assert any(found)
108 |
109 |
110 | @pytest.mark.parametrize("model", models)
111 | @pytest.mark.asyncio
112 | async def test_url_detection_multi_page_auto_routing(model):
113 | sample = "examples/inputs/sample_test_doc.pdf"
114 | patterns = ["http", "https", "www"]
115 | config = {"parser_type": "AUTO", "model": model, "verbose": True}
116 | results = parse(sample, pages_per_split=1, **config)["segments"]
117 |
118 | assert len(results) == 6
119 | for res in results:
120 | content = res["content"]
121 | if res["metadata"]["page"] == 1:
122 | # Page 1: Fails to detect the URL
123 | found = [p in content for p in patterns]
124 | assert not any(found)
125 | elif res["metadata"]["page"] == 2:
126 | # Page 2: Detects the URL
127 | found = [p in content for p in patterns]
128 | assert any(found)
129 | elif res["metadata"]["page"] == 3:
130 | # Page 3: Does not contain any URL
131 | found = [p in content for p in patterns]
132 | assert not any(found)
133 | elif res["metadata"]["page"] == 4:
134 | # Page 4: Detects the URL
135 | found = [p in content for p in patterns]
136 | assert any(found)
137 | elif res["metadata"]["page"] == 5:
138 | # Page 5: Detects all the URLs
139 | found = [p in content for p in patterns]
140 | assert all(found)
141 | elif res["metadata"]["page"] == 6:
142 | # Page 6: Detects the URL
143 | found = "https://github" in content
144 | assert found
145 |
146 |
147 | @pytest.mark.asyncio
148 | @pytest.mark.parametrize("depth", [1, 2])
149 | async def test_recursive_url_parsing(depth):
150 | results = parse("https://example.com/", depth=depth)["segments"]
151 |
152 | # Not necessarily always the case. Just the case for "example.com".
153 | assert len(results) == depth
154 |
155 |
156 | @pytest.mark.asyncio
157 | async def test_recursive_url_parsing_in_pdf():
158 | sample = "examples/inputs/sample_test_doc.pdf"
159 | parser_type = "AUTO"
160 | results = parse(sample, parser_type, pages_per_split=1, depth=2)
161 | assert len(results["recursive_docs"]) >= 7, results
162 |
163 |
164 | @pytest.mark.asyncio
165 | async def test_parsing_txt_type():
166 | sample = "examples/inputs/sample_test.txt"
167 | parser_type = "AUTO"
168 | results = parse(sample, parser_type)["segments"]
169 | assert len(results) == 1
170 | assert results[0]["content"] is not None
171 |
172 |
173 | @pytest.mark.asyncio
174 | async def test_parsing_docx_type():
175 | sample = "examples/inputs/sample.docx"
176 | parser_type = "STATIC_PARSE"
177 | results = parse(sample, parser_type)["segments"]
178 | assert len(results) >= 1
179 | assert results[0]["content"] is not None
180 |
181 | parser_type = "LLM_PARSE"
182 | results = parse(sample, parser_type)["segments"]
183 | assert len(results) > 1
184 | assert results[0]["content"] is not None
185 |
186 |
187 | @pytest.mark.asyncio
188 | async def test_parsing_xlsx_type():
189 | sample = "examples/inputs/sample.xlsx"
190 | parser_type = "STATIC_PARSE"
191 | results = parse(sample, parser_type)["segments"]
192 | assert len(results) >= 1
193 | assert results[0]["content"] is not None
194 |
195 |
196 | @pytest.mark.asyncio
197 | async def test_parsing_pptx_type():
198 | sample = "examples/inputs/sample.pptx"
199 | parser_type = "STATIC_PARSE"
200 | results = parse(sample, parser_type)["segments"]
201 | assert len(results) >= 1
202 | assert results[0]["content"] is not None
203 |
204 |
205 | @pytest.mark.asyncio
206 | async def test_dynamic_js_parsing():
207 | test_url = "https://go.contentsquare.com/ab-testing-playbook"
208 | results = parse(test_url, parser_type="AUTO")["raw"]
209 | # Check if the content contains the expected information
210 | should_contain_info = "6 Types of experimentation"
211 | assert should_contain_info.lower() in results.strip().lower()
212 |
213 |
214 | @pytest.mark.asyncio
215 | async def test_pdfplumber_table_parsing():
216 | sample = "examples/inputs/test_1.pdf"
217 | parser_type = "STATIC_PARSE"
218 | results = parse(sample, parser_type, framework="pdfplumber")["raw"]
219 | assert [token in results for token in ["|", "Results", "Accuracy"]]
220 |
221 |
222 | @pytest.mark.asyncio
223 | @pytest.mark.parametrize(
224 | "sample",
225 | [
226 | ("examples/inputs/stress_test/large_doc_1.pdf", 527),
227 | ("examples/inputs/stress_test/large_doc_2.pdf", 117),
228 | ],
229 | )
230 | async def test_large_pdf_parsing(sample):
231 | parser_type = "AUTO"
232 | file_name = sample[0]
233 | n_pages = sample[1]
234 | results = parse(file_name, parser_type, pages_per_split=1)["segments"]
235 | assert len(results) == n_pages
236 | assert results[0]["content"] is not None
237 |
238 |
239 | token_usage_models = [
240 | # Google models
241 | "gemini-2.0-flash-001",
242 | # OpenAI models
243 | "gpt-4o",
244 | # Meta-LLAMA models through HF Hub
245 | "meta-llama/Llama-3.2-11B-Vision-Instruct",
246 | # Meta-LLAMA models through Together AI
247 | "meta-llama/Llama-3.2-11B-Vision-Instruct-Turbo",
248 | ]
249 |
250 |
251 | @pytest.mark.parametrize("model", token_usage_models)
252 | @pytest.mark.asyncio
253 | async def test_token_usage_api(model):
254 | sample = "examples/inputs/test_1.pdf"
255 | parser_type = "LLM_PARSE"
256 | config = {"parser_type": parser_type, "model": model}
257 | token_usage = parse(sample, **config)["token_usage"]
258 | logger.info(f"Token usage: {token_usage}")
259 | assert token_usage["input"] > 0
260 | assert token_usage["output"] > 0
261 | assert token_usage["total"] > 0
262 |
263 |
264 | @pytest.mark.asyncio
265 | async def test_pdf_save_path():
266 | sample = "https://example.com/"
267 | parser_type = "LLM_PARSE"
268 | result = parse(
269 | sample,
270 | parser_type,
271 | as_pdf=True,
272 | save_dir="tests/outputs/temp",
273 | save_filename="test_output.pdf",
274 | )
275 | assert "pdf_path" in result
276 | assert result["pdf_path"].endswith(".pdf")
277 | assert os.path.exists(result["pdf_path"])
278 |
279 | # Clean up
280 | os.remove(result["pdf_path"])
281 | os.rmdir("tests/outputs/temp")
282 |
283 |
284 | @pytest.mark.asyncio
285 | async def test_page_nums():
286 | sample = "examples/inputs/sample_test_doc.pdf"
287 | result = parse(sample, "LLM_PARSE", page_nums=(3, 4), pages_per_split=1)
288 | assert len(result["segments"]) == 2
289 | assert all(keyword in result["raw"] for keyword in ["Table 24", "apple"])
290 | assert all(keyword not in result["raw"] for keyword in ["Aenean", "Lexoid"])
291 |
292 | result = parse(sample, "LLM_PARSE", page_nums=(3, 3), pages_per_split=1)
293 | assert len(result["segments"]) == 1
294 | assert "Table 24" in result["raw"]
295 |
296 | sample = "https://www.dca.ca.gov/acp/pdf_files/lemonlaw_qa.pdf"
297 | result = parse(sample, "STATIC_PARSE", page_nums=2, pages_per_split=1)
298 | assert len(result["segments"]) == 1
299 | assert "ATTEMPTS" in result["raw"]
300 | assert "acp@dca.ca.gov" not in result["raw"]
301 |
302 |
303 | @pytest.mark.parametrize(
304 | "model",
305 | [
306 | "gemini-2.0-flash",
307 | "gpt-4o",
308 | "meta-llama/Llama-3.2-11B-Vision-Instruct-Turbo",
309 | ],
310 | )
311 | @pytest.mark.asyncio
312 | async def test_token_cost(model):
313 | sample = "examples/inputs/test_1.pdf"
314 | parser_type = "LLM_PARSE"
315 | api_cost_path = os.path.join(os.path.dirname(__file__), "api_cost_mapping.json")
316 | config = {
317 | "parser_type": parser_type,
318 | "model": model,
319 | "api_cost_mapping": api_cost_path,
320 | }
321 | result = parse(sample, **config)
322 | assert "token_cost" in result
323 | assert result["token_cost"]["input"] > 0
324 | assert result["token_cost"]["output"] > 0
325 | assert result["token_cost"]["total"] > 0
326 |
327 |
328 | @pytest.mark.asyncio
329 | async def test_blockquote():
330 | sample = "examples/inputs/bench_md.pdf"
331 | parser_type = "STATIC_PARSE"
332 | results = parse(sample, parser_type, framework="pdfplumber")["raw"]
333 | # Assert that there is at least one fenced code block
334 | assert " " * 3 in results
335 |
336 |
337 | @pytest.mark.asyncio
338 | async def test_monospace_code_block():
339 | sample = "examples/inputs/bench_md.pdf"
340 | parser_type = "STATIC_PARSE"
341 | results = parse(sample, parser_type, framework="pdfplumber")["raw"]
342 | # Assert that there is at least one fenced code block
343 | assert "```" in results
344 |
345 |
346 | @pytest.mark.asyncio
347 | async def test_pdf_headings():
348 | sample_path = "examples/inputs/bench_md.pdf"
349 | parser_type = "STATIC_PARSE"
350 | results = parse(sample_path, parser_type, framework="pdfplumber")["raw"]
351 |
352 | # Test for h1 (should have # in markdown)
353 | assert "#" in results
354 | assert "##" in results
355 |
356 |
357 | @pytest.mark.asyncio
358 | async def test_email_address():
359 | sample = "examples/inputs/bench_md.pdf"
360 | parser_type = "STATIC_PARSE"
361 | results = parse(sample, parser_type, framework="pdfplumber")["raw"]
362 | assert "" in results
363 |
364 |
365 | @pytest.mark.asyncio
366 | async def test_horizontal_lines():
367 | sample = "examples/inputs/bench_md.pdf"
368 | parser_type = "STATIC_PARSE"
369 | results = parse(sample, parser_type, framework="pdfplumber")["raw"]
370 | assert "\n---\n" in results, "Markdown horizontal rule not found"
371 |
372 |
373 | @pytest.mark.asyncio
374 | async def test_strikethrough_words():
375 | sample = "examples/inputs/bench_md.pdf"
376 | parser_type = "STATIC_PARSE"
377 | results = parse(sample, parser_type, framework="pdfplumber")["raw"]
378 | assert "~~" in results, "Markdown strikethrough text not found"
379 |
--------------------------------------------------------------------------------
| |