DualNeighbors: Multilingual Curation

• Multilingual curation algorithm is a method that combines lexical (TF-IDF) and semantic (embedding) analyses to select and connect documents across languages.
• It uses dual similarity via word neighbors and embedding neighbors to offer precise recommendations alongside expansive cross-lingual links.
• The algorithm, exemplified by DualNeighbors, enhances cultural and thematic connectivity in diverse datasets as validated by quantitative network metrics.

A multilingual curation algorithm is an automated method designed to select, filter, link, or optimize resources—such as documents, datasets, or media—across multiple languages, with the aim of improving connectivity, relevance, and semantic or functional quality in large, heterogeneous collections. Unlike monolingual or purely keyword-based approaches, these algorithms combine language-agnostic representations (e.g., embeddings), cross-lingual alignment techniques, and statistical or structural heuristics to enable discovery, recommendation, and organization of content that is thematically or functionally parallel but expressed in diverse linguistic forms. The DualNeighbors algorithm exemplifies such an approach, facilitating exploration and curation of textual corpora by traversing both lexical and semantic similarities within and across languages (Arnold et al., 2018).

1. Algorithmic Foundations and Workflow

The DualNeighbors algorithm operates on the principle of dual similarity, integrating both lexical (TF-IDF word-based) and semantic (embedding-based) proximity into a unified curation workflow. The process is formalized as follows:

• Lexicon L and a p-dimensional word embedding function are established.
• For each word in L, a neighborhood function $f$ retrieves its M closest neighbors from the multilingual embedding space.
• Each document $c_i$ is represented as a set of words $\{ w_{i,k} \}$ in L.
• A binary term frequency matrix Y is computed, and the TF-IDF matrix X is constructed as

$$X_{i,j} = Y_{i,j} \times \log \left( \frac{n}{\sum_i Y_{i,j}} \right)$$

• The “embedded corpus” replaces each word with its M nearest embedding neighbors, recomputing TF-IDF on this virtual expansion.
• Similarity matrices for direct TF-IDF (S) and embedding-based (S^emb) modes are calculated via cosine similarity:

$$S_{i,i'} = \frac{X_{i'} \cdot X_i^\top}{\sqrt{X_i^\top X_i}};\quad S^{(\text{emb})}_{i,i'} = \frac{X^{(\text{emb})}_{i'} \cdot X_i^\top}{\sqrt{X_i^\top X_i}}$$

(with diagonals zeroed out).

• For a query index $\tilde{\imath}$, TopN rankings extract the most similar documents by both metrics, outputting their union as recommendations.

This pipeline enables the algorithm to traverse not only direct word matches, which tend to be language- and discourse-specific, but also embedding-based conceptual links, which may cross linguistic and cultural boundaries.

2. Semantic and Linguistic Bridging via Multilingual Embeddings

The key to multilingual curation in DualNeighbors is the embedding neighbor step, which replaces terms with their nearest neighbors from aligned multilingual word embedding spaces (e.g., fastText covering 157 languages). Rotational alignment techniques, often via bilingual dictionaries, ensure that semantically similar words from different languages (e.g., “school” and “école”) are mapped closely. This alignment enables the embedding replacement strategy to surface documents that, while lexically distinct, share thematic or semantic content, producing recommendations that bridge both linguistic and discursive divides.

This dual similarity approach (word neighbors for high lexical precision, embedding neighbors for semantic generality) delivers a balanced exploration strategy: it maintains relevance while revealing broader, sometimes unexpected, cultural or linguistic connections—addressing a key limitation of traditional monolingual or keyword-based exploration.

3. Evaluation: Qualitative Connectivity and Quantitative Metrics

DualNeighbors has been evaluated qualitatively and quantitatively on datasets with substantial linguistic and cultural diversity:

• In the FSA-OWI photography captions archive, embedding-based neighbors linked regionally disparate descriptions of related events (e.g., agricultural tasks), while word-based neighbors found lexically proximate captions.
• In parallel news feeds (e.g., The Guardian and Le Figaro), cross-lingual connections emerged linking direct translations and cultural event coverage (e.g., “red carpet” ↔ “tapis rouge”).

Quantitative analysis leveraged network connectivity metrics: algebraic connectivity, average distances, in-degree distributions, and third-degree ego-scores. Results show that even a small number of embedding neighbors, when mixed with word neighbors, substantially increase global connectivity. Manual validation over thousands of links across datasets confirms that embedding neighbor recommendations have a high rate of semantically valid matches, both within and across languages.

4. Open-Source Implementation and Computational Considerations

The algorithm is provided as an open-source R package (“cdexplo”), enabling researchers to process raw multilingual text and generate interactive websites for document exploration. The package performs automatic language detection, appropriate lexical filtering, and supports the inclusion of metadata and images. Example usage includes annotation, dual neighbor computation, and local website generation:

library(cdexplo) data <- read.csv("input.csv") anno <- cde_annotate(data) link <- cde_dual_neigh(anno, nw = 10, ne = 2) cde_make_page(link, "output_location")

Resource considerations include the computational cost of embedding neighbor expansion and similarity matrix computation, manageable for typical humanities/social science corpora. The design is modular, allowing adaptation to alternative embedding sources or additional preprocessing steps without altering the core workflow.

5. Applications Across Disciplines

The DualNeighbors algorithm’s dual similarity curation paradigm has broad applicability:

• Digital archives and cultural heritage research: Discovery of thematic or cultural links not evident in monolingual search.
• Comparative media and cross-national studies: Mapping event or topic coverage across media in different languages.
• Recommendation systems in digital humanities and libraries: Combining narrow (word neighbor) and broad (embedding neighbor) exploration prevents user "filter bubbles" and supports serendipitous discovery.
• Cross-disciplinary, cross-cultural research: Enables scholars to traverse disciplinary or linguistic silos, surfacing previously hidden connections among documents from disparate sources.

6. Implications and Future Directions

The integration of TF-IDF and multilingual embedding-based similarity in document curation represents a substantive advance for researchers in the humanities and social sciences tasked with exploring large, heterogeneous, and linguistically diverse corpora. By facilitating recommendations and explorations that are at once precise and semantically rich, the algorithm supports a form of cross-cultural and cross-discursive connectivity unattainable with traditional curation methods.

Further directions might entail integrating contextualized embeddings, improving scalability to even larger corpora, and expanding to domains with more complex or hierarchical document structures. Extensions could also target more granular alignment of embeddings, incorporation of user feedback, or adaptation to tasks such as automated dataset construction for supervised machine learning or digital humanities analysis.

---

Summary Table: Key Characteristics of DualNeighbors Multilingual Curation Algorithm

Feature	Lexical Mode (TF-IDF)	Embedding Mode (Multilingual space)
Similarity metric	Cosine over TF-IDF vectors	Cosine over embedding-augmented TF-IDF
Scope of connection	Intra-language, precise word overlaps	Cross-language, thematic/semantic links
Cross-cultural capability	Limited	High
Data requirements	Lexicon, lemmatizer, POS-tagger	Multilingual word embeddings
Example Application	Captions archive, media feeds	Comparative news/events, cultural links

The dual-pronged curation approach—leveraging both precise word-based and cross-lingual semantic neighbor recommendations—positions the DualNeighbors algorithm as a foundational method for advancing multilingual corpus curation and cultural analytics (Arnold et al., 2018).