Scraping Xing with code and without code (one is 100x faster)

Hallo 🇩🇪 !

Scraping is probably my favorite discipline in the Growth field and that’s why I’ll explain today how to do a little Xing scraping in different ways.

First I’ll explain how to scrape Xing with a somewhat advanced stack because Xing’s technology doesn’t allow for the usual scraping stack like Python’s Scrapy or BeautifulSoup.

Secondly, I’ll explain how you can do it yourself with XingZap and no other tools or skills, saving you at worst hours of development time and debugging.

If you don't want to know how to do the technical dive, simply scroll down or just get your XingZap account ! (you get 1000 credits free to get started.

Let’s get into it.

Scraping Xing with Python, Selenium and Chrome
‍

Here’s what you need to get started smoothly:

Experience in Python
Experience with Selenium’s library
Experience with a Python HTML parser (I use BeautifulSoup)
Experience in HTML/CSS and CSS selectors
A Xing account

Before we do anything we’ll need to login programmatically to your account but I’ll start with a disclaimer first:‍

Please be aware that there can be disruptions if you login too often from different IPs/locations.
Xing (like Linkedin) has their own policy on account banning, and their word is final.
XingZap and I cannot be held responsible for the actions you decided to undertake with your own account

With that out of the way:

The first thing to do is to wait for the cookie consent window and accept all the cookies
Then we want to type our email and passwords in the appropriate fields and press enter.
I work with CSS selectors to pinpoint every element that needs to be filled or clicked

Here’s what the final logging function could look like raw:

‍

This was only the beginning !

We still need to find the selectors for the data we want (name, position, company...) and output it.For the sake of simplicity, I’ll just show how I would get the first page of results of a given Xing search:

I load a search, here I searched for “fullstack developers”
I wait for the search results to load with Puppeteer and store the source HTML so I can parse it with BeautifulSoup a very powerful Python HTML parser
After inspecting with the developer console, the results are stored in the div with the attribute data-qa with a value of results-list
Let’s say i want to scrape only the names and the URLs:

I pick all the spans that have the x85e3j-0 class name and get the the hrefs from the a tags which have the“search-card-style-containerLink-70ccdd46 class name:

unformatted result from selenium scraping

With a bit of Python list comprehension I get the the actual formatted clean data

python scraping result with beautifulsoup

Putting it all together, it would look like this after amending the first script:

‍

This script is functional as of December 2021, but I should note that nothing guarantees that it will still hold up in the future as you may already know.

With start, you’ll have to test and learn the rest on your own to see the limitations and tailor the Xing scraper script to your needs.

There are obviously more steps for this script to be ready for production : we still need to implement the paging, improve the input and the outputs, add human behavior and waiting times and finally make sure our errors are handled with screenshots in case Xing changes their layout.

You can take all this into account or simply read the next section to get started in seconds.

‍