net/mlx5e: CHECKSUM_COMPLETE offload for VLAN/QinQ packets

net/mlx5e: CHECKSUM_COMPLETE offload for VLAN/QinQ packets

When the VLAN tag is present in the packet buffer (i.e VLAN stripping disabled, QinQ)
the driver will currently report CHECKSUM_UNNECESSARY.
Instead of using CHECKSUM_COMPLETE offload for packets with first
ethertype of IPv4/6, use it for packets with last ethertype of IPv4/6 to
cover the former cases as well.

The checksum field present in the CQE is calculated from the IP header
until the end of the packet. When the first ethertype is different than
IPv4/6 (for ex. 802.1Q VLAN) a checksum of the VLAN header/s should be
added. The small header/s checksum calculation will allow us to use
CHECKSUM_COMPLETE instead of CHECKSUM_UNNECESSARY.

Testing bandwidth of one and 8 TCP streams to a single RQ,
LRO and VLAN stripping offloads disabled:
CPU: Intel(R) Xeon(R) CPU E5-2660 v2 @ 2.20GHz
NIC: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]

Before:
+--------------+--------------------+---------------------+----------------------+
| Traffic type | 1 Stream BW [Mbps] | 8 Streams BW [Mbps] |   Checksum offload   |
+--------------+--------------------+---------------------+----------------------+
| Untagged     |          28,247.35 |           24,716.88 | CHECKSUM_COMPLETE    |
| VLAN         |          27,516.69 |           23,752.26 | CHECKSUM_UNNECESSARY |
| QinQ         |           6,961.30 |           20,667.04 | CHECKSUM_UNNECESSARY |
+--------------+--------------------+---------------------+----------------------+

Now:
+--------------+--------------------+---------------------+-------------------+
| Traffic type | 1 Stream BW [Mbps] | 8 Streams BW [Mbps] | Checksum offload  |
+--------------+--------------------+---------------------+-------------------+
| Untagged     |          28,521.28 |           24,926.32 | CHECKSUM_COMPLETE |
| VLAN         |          27,389.37 |           23,715.34 | CHECKSUM_COMPLETE |
| QinQ         |           6,901.77 |           20,845.73 | CHECKSUM_COMPLETE |
+--------------+--------------------+---------------------+-------------------+

No performance degradation observed.

Signed-off-by: Gal Pressman <galp@mellanox.com>
Reviewed-by: Tariq Toukan <tariqt@mellanox.com>
Signed-off-by: Saeed Mahameed <saeedm@mellanox.com>