Machine Learning for SEO Are you ready for this?🤯 I just generated top ten internal link recommendations in a dataset of 100,000 pages that don't link to each other but should. Rather than using traditional means of matching pages such as TF-IDF, LDA, n-grams and keyword frequency I use machine learning. Embeddings Scraped, pre-processed and tokenised data is embedded to a 768-dimensional vector space using Google's language-agnostic BERT sentence embedding model. Page > Page Combined and averaged sentence vectors for each URL are used to form a page similarity matrix in numpy which is then used to build a list of semantically related pages through cosine similarity. That was the easy part. Now onto the tricky business of anchor text recommendation by selecting a portion of existing text on the proposed linking page that is also relevant to the target page. Sentence > Page Iterating over all sentence vectors nested within each identified link candidate page, once again, looking for the one with the shortest cosine distance in respect to the page-level embedding of the target URL as a whole. Once the most suitable sentence embedding has been identified, it's then possible to, on the fly, reconstitute that sentence by mapping the sentence ID and URL which are stored as raw text key column at the end of the vector values. Word > Page Regenerated raw text content is used to produce word IDs + word-level embeddings which are cycled through one final similarity lookup. Again, looking for the shortest cosine distance between the selected word vectors of the source page and page-level vectors of the target page. It's now possible to re-build the keyword text (otherwise lost in vector data) by mapping word ID keys kept in the program memory during this temporary cycle same as we did earlier with sentences. Word = Anchor Text All identified keywords are then recorded in the "anchor_text" column of the link recommendation csv (source,target,anchor_text) in descending order of relevance. Note: The reason word-level vector embeddings are generated on the fly and not during sentence-level embeddings is due to it being computationally demanding and requiring hundreds of gigabytes of storage for a dataframe of 100,000 scraped pages. Tech Stack For word embeddings I use SentenceTransformers [1], LaBSE [2], PyTorch [3] and Nvidia's CUDA [4] on a Tesla T4 16GB GPU. LaBSE is a good choice for websites with many herflang silos due to it being completely language agnostic. You can of course use vanilla BERT if you like, but I would recommend BERT-large-cased. Pro Tips: 1. My tests show significantly better outcomes when using L2 normalised [5] LaBSE embeddings. 2. Don't strip full stops during tokenisation process otherwise you'll lose sentence boundary data. Useful Links: [1] sbert.net/ [2] ai.googleblog.com/2020/08/la… [3] pytorch.org/ [4] developer.nvidia.com/cuda-to… [5] numpy.org/doc/stable/referen…