Research IT

People in bright white building milling around looking at paintings hung on black boards

Semantic Search in Action: Building a Multi-Modal Cognitive Engine

Two of our Research Software Engineers, in conjunction with the Department of Computer Science at the University, partnered with mindcubes to prototype a multi-modal semantic search system that links images to structured knowledge, delivering a scalable AI pipeline for smarter, concept-aware retrieval.

Over a focused six-month sprint (Jul 2024 – Jan 2025), the team set out to prototype a multi-modal semantic-search system working with mindcubes, a start-up based in Manchester who are building the next generation of semantic AI technologies. Their search system can take a picture, understand what is in the picture and then match it to the most relevant ideas in their database, a database which organises knowledge in the form of interconnected maps called metasets.

The Challenge

mindcubes has a range of customers including museums, art galleries, trade fairs and exhibitions. All of these customers wanted more than basic image recognition; they needed a system that could understand meaning, not just pixels.

For example, if a visitor at a museum says, “I’m interested in Japanese culture,” the system should surface relevant artefacts from the past. If the visitor then says, “I’m interested in poetry,” it should highlight items connected to poetry, such as a vase created by a poet to celebrate life.

In other words, the system needs to capture the semantic meaning behind objects as well as the objects themselves. Achieving this meant bridging the gap between raw visual input and structured semantic knowledge, which required:

  • Building robust joint embeddings – this is a method that represents different data sources, such as images and ontology terms, in a unified way. This allows for their meaningful (semantic) comparison and integration within a shared space.
  • Designing a reproducible and scalable development workflow.

The RSE Contribution

Under the guidance of Dr Jiaoyan Chen (Senior Lecturer in Data Intensive Systems, Department of Computer Science), two Research Software Engineers, Erdem Atbas and Lasse Schmieding, led the development efforts.

We explored image data and, through weekly Agile check-ins with mindcubes, co-designed the retrieval pipeline to meet evolving needs. To keep development efficient and collaborative, we adopted a GitHub-first workflow and containerised all services using Docker Compose. As part of this effort, we built Aurora, a FastAPI-based microservice that handles embedding, tagging, and semantic search. To maintain reliability, we reinforced the codebase with integration tests using Pytest and Testcontainers, ensuring quality at every stage.

Summary of the Key Tools & Technologies

  • Python, FastAPI & Docker for a modular, API-first architecture.
  • OpenSearch for vector-based retrieval and search engine with MongoDB for storage.
  • Cohere & CLIP for multi-modal embedings; LangChain for retrieval-augmented generation (RAG) experiments.
  • Thorough testing and handover materials, including Pytest suites, and detailed README.

Outcomes

  • Aurora prototype: a one-click UI for auto-tagging & semantic search.
  • Cognitive Engine API that connects image input to knowledge graph concepts.
  • A clear roadmap for scaling: ingesting PDF/HTML and applying the engine across multi-domain content

Here is an exclusive first look! The figure below offers a sneak peek at the interactive REST API endpoints. It’s just a taste of what’s to come.

mindcubes

Reflections & Lessons Learned

Success came from tight feedback loops driven by regular input from mindcubes: regular check-ins kept scope realistic and progress on track. The team learned to start simple, focusing on strong baseline models first, before iterating with context-aware and fine-tuned systems later.

Looking Ahead

Future work will focus on expanding the system to handle richer input formats, such as PDF and HTML content, while refining OpenSearch ranking algorithms to deliver more precise results. Further gains could be achieved by testing the pipeline in new domains, with museums next on the roadmap. Building on the collaboration’s foundation, the team is now well-positioned to deepen its understanding of AI as modular components and to integrate them into a highly functional system capable of delivering transformative impacts for users and, ultimately, society.

Does Your Research Require an AI Software Solution?

If you’re interested in working with the RSE team on an AI-related project (large or small), get in touch with us via Connect and we’ll organise a meeting to discuss how we can work together. Find out how the University of Manchester is driving growth across Greater Manchester via AI collaboration AI Innovation: The Turing Innovation Catalyst's Success