(in just a few sentences; without going into details)

A transformer is a neural network architecture. Transformers are based on the so-called attention mechanism.

Transformers were proposed in a paper called "Attention Is All You Need" that was published in 2017 by a team of AI researchers from Google.

In the years following the publication, transformers started to dominate over and replace previous architectures, which had been recurrent neural networks (RNNs), including long short-term memory (LSTM) and GRU architectures.

Transformers became widely used in many application domains, like computer vision and natural language processing (NLP).

basics conceptual

XGBoost stands for "Extreme Gradient Boosting". XGBoost is a scalable, distributed gradient-boosted decision tree (GBDT) machine learning library.

It provides parallel tree boosting and is the leading machine learning library for regression, classification, and ranking problems.

XGBoost is not based on neural networks but decision trees. XGBoost dominated Kaggle competitions in recent years.

To understand how XGBoost, you need to understand the following concepts:

The term "boosting" in "gradient boosting" refers to the concept improving a single weak model by combining it with a number of other weak models in order to generate a collectively strong model.

Gradient boosting is an extension of boosting where the process of additively generating weak models is formalized as a gradient descent algorithm over an objective function.

Gradient boosting sets targeted outcomes for the next model in an effort to minimize errors. Targeted outcomes for each case are based on the gradient of the error (hence the name gradient boosting) with respect to the prediction.

GBDTs iteratively train an ensemble of shallow decision trees, with each iteration using the error residuals of the previous model to fit the next model. The final prediction is a weighted sum of all of the tree predictions. Random forest “bagging” minimizes the variance and overfitting, while GBDT “boosting” minimizes the bias and underfitting.

(Source: https://www.nvidia.com/en-us/glossary/data-science/xgboost/)