The clever search engine from next door
Prototype of a web search engine with a crawler network, full-text search and AI-assisted image search – built in 2018/2019 as a bachelor's thesis at Hochschule Merseburg.
The idea
Build a search engine yourself – from the crawler to the search form
ZyperSearch was an attempt to build a complete information retrieval system from scratch: a network of crawlers collects web pages, documents and images, a server stores everything in a database, and a web front end answers queries. Classic full-text search for text, a neural network for images.
Full-text search
Boolean search over title, meta keywords and page content, stored in MySQL. Results are ordered by hit frequency – a later iteration added a ranking of search terms.
Image search with a neural network
Every image found is classified with TensorFlow (Inception model) and scanned for text with Tesseract OCR. So you search for what is in the picture – not just its file name.
Documents included
PDF, Word, Excel and PowerPoint files are read with Apache PDFBox and Apache POI and indexed like web pages. The search form filters by web pages, PDF and Office.
Architecture
Three separate modules, connected by sockets and a database
The prototype ran inside the Hochschule Merseburg intranet: crawlers on lab PCs and two retired servers, the database and web server in the DMZ. Each crawler asks the server for the oldest unprocessed link (FIFO), analyses it and reports content, MIME type and newly discovered links.
Crawler network
Network A · university intranet
- 6 lab PCs + 2 old servers, Ubuntu 16.04
- Java, jsoup for HTML, PDFBox/POI for documents
- TensorFlow Inception + Tesseract OCR for images
- Client-side cache against duplicate links
Server
Network B · university DMZ
- Dell PowerEdge T300, Ubuntu 18.04
- Java socket server, port 2156
- MySQL: tables Hosts, Links, Emails
- FIFO hand-out of the oldest link, timestamps
Web front end
Apache Tomcat 9 · port 8080
- JSP + servlet, Ajax requests with XML responses
- Standard search, image search, statistics
- Results: title, link, 140-character snippet
- Footer shows the number of known links (cached)
Opened ports: SSH 22022 and web 8080 from the internet, crawler interface 2156 from the intranet – every port had to be requested from the network administrator.
Facts & figures
55 M
Known links
as of March 2019, according to the search form footer
20 M
Records in the test run
after which an SSD RAID 0 was added for MySQL
8
Crawler machines
6 lab PCs + 2 former university servers
Dell PowerEdge T300
Server
Xeon X3363, 4 cores, 20 GB RAM
16
Repositories
crawler variants, server, image recognition, web, API
102 pages
Length of the thesis
6 chapters, 26 figures, 88 references
Tech stack
As of 2019/2020 – deliberately not modernised
Core
Analysis
Later experiments
The prototype in action
Screenshots from the thesis (figures 23–25)
The cats stayed: years later, the image search turned into the idea for CatCatch, our app for stray cats. cat-catch.de
History
From the first search form to this website
Idea and first crawlers
Project start alongside the degree: crawler clients in Java, socket server and MySQL schema. First test runs on the university network with retired server hardware.
First search form
The web front end is born on Tomcat: main.jsp with logo, input and a “Suchen!” button, plus the image search.
Search options and Boolean search
Filters for web pages, PDF and Office, search on Enter, Boolean combination of search terms.
Thesis submitted
102 pages on information retrieval, neural networks and the prototype. On 28 March 2019 the university library publishes the thesis under CC BY 4.0.
zypersearch.de
This project site goes live. The cats from the image search live on in CatCatch.
Honest conclusion
What the prototype showed – and what it did not
“As the development of the prototype has shown, a good web search engine should not follow just one of the two approaches, but combine both.”
- 1
For phrase search, classic text analysis is largely sufficient. Neural networks only add value there once usage data is available to personalise the ranking.
- 2
For image search, neural networks are indispensable: people search for what is in a picture, and only the network can see that. Titles and OCR text still mattered because the Inception model was not trained on proper names.
- 3
Scaling requires clustering and strictly encapsulated modules. New links belong in a temporary database first, which is optimised before it replaces the search database.
- 4
The market leaders are successful for a reason. A new search engine could only hold its own in Europe by playing by different rules: fully transparent, open source, ad-free, donation-funded and hosted in Europe.
- 5
Realistically, no new search engine will dethrone a market leader. The prototype did show, however, that a complete retrieval system is feasible with university resources.
Curious?
Read or download the full thesis, or try the old search form yourself.